In the light of increased external pressure and new societal challenges, a new Swedish innovation and business policy is emerging whose purpose is to drive an overall transition to a sustainable society.

The requisite changes are of an unprecedented scale and nature, in that they span several policy areas, institutions, groups of stakeholders and professions. A core component of the new transition policy is therefore transboundary cooperation. This also establishes new needs for assessments and continuous learning.

This report analyses the specific challenges and opportunities presented in the evaluation of policies to promote sustainability transition. The discussion is based on three central issues:

• What are the basic assumptions and unique characteristics of the policy?
• What purposes can the evaluation fulfil in transformative processes?
• What specific challenges do transformative policies raise for the design, methodology and practical implementation of evaluations?

The ambition is to generate a first broader overview of the role of evaluation in relation to the new transition policy.

In conclusion, we note that:

The new transition policy reflects a development in which three policy areas – environmental policy, policy for economic and regional growth, and innovation policy – have gradually grown together in a meta-policy with the sustainability transition as the overarching objective and transboundary interaction as the primary administrative method. (Question 1)

With this, the policy also takes on a number of new dimensions in the form of: 1) a pronounced direction towards sustainability (directionality); 2) an emphasis on experiments and trials as the primary mechanism for change; and 3) the ambition to involve the end-beneficiaries of the policy to a greater extent. Implicitly, the innovation policy is expected to be the engine facilitating the achievement of the sustainability transition. (Question 1)

With the goal of successfully implementing this transition, the policy has gradually developed a number of instruments for the purpose of encouraging and supporting various pilot activities. The latter will thus be empowered to replace the established economic, social and technological structures constructed around established technologies, such as fossil-fuel powered automobiles. (Question 1)

Therefore, in reality, the transition requires a comprehensive coordination of measures and stakeholders in a wide range of policy areas. With this, transboundary cooperation also emerges as the policy’s primary principle of governance. A core aspect of these collaborative efforts is continuous learning. (Question 1)

This, in turn, underlines evaluation as an integral part of the policy’s governance doctrine. Evaluation is ultimately about learning. However, learning can take place at various levels and for different purposes. In terms of the sustainability transition, the evaluation work primarily serves two purposes: 1) to exercise control in a management structure based on delegation; and 2) to contribute to the further development of policies. Both of these functions are needed and therefore must be exercised in parallel. Meanwhile, each individual issue raises different needs. One important message is therefore that the practical evaluation work – for its design, methodology and purpose – must always be based on the policies being pursued. This, in turn, underlines the need for methodological pluralism and collaborative efforts among evaluators as well. (Question 2)

The report then identifies five themes that require special attention when evaluating sustainability transitions: 1) the directionality of the policy; 2) transboundary cooperation; 3) the implementation of pilot activities; 4) the end-beneficiaries of the policy; and 5) the more general issue of causality. (Question 3)

These challenges, which are connected to all parts of the policy cycle (inflows, implementation, and outcomes) and the policy’s primary governance principle (collaboration), are then matched up with a number of design and methodological approaches from the evaluation literature. The purpose of this discussion is to: 1) provide an initial overview of the various challenges and opportunities that arise in particular from the evaluation of the sustainability transition; 2) provide examples of how to address some of these challenges; and 3) at the same time illustrate the potential of continued identification of challenges and the various tools of evaluation discipline. (Question 3)

The latter discussion will lead to a number of more comprehensive observations.

One first observation concerns the need to recreate, as a first step, the policy’s original intervention theory. This is particularly important when evaluating major transition processes that, via their complexity and long-term horizon, often leave the evaluator without a “map and compass.” The intervention theory establishes a starting point, which the evaluation then aims to improve. Similarly, intervention theory forms the basis for hypothesis testing and continued methodology development.

A second observation concerns the three specific governance elements that we believe together establish the transition itself and which may therefore be subject to combined evaluations. In brief, they deal with: 1) the coordination between measures/initiatives and various different policy areas taking place at an aggregated level (meta-policy); 2) the conscious governance towards side effects and the concept of pilots projects and trials as an instrument to create added value beyond one’s own area of activity; and (3) the process by which the pilot/exploratory activities are brought back to the line in a series of implementation stages.

A third observation concerns the need to monitor the process. Not only is the transition policy complex, it is also constantly evolving. This means that the recreated intervention theory must be regularly revised. One way to continuously monitor this development is to work in parallel with process tracing.

This means, as a fourth observation, that the evaluation must also be close and intertwined with the policy. Therefore in practice, the scrutiny and developing evaluation should be kept separate. Both are centrally important to continued further learning.

There is a clear risk however that they will contaminate each other if they are not carried out in completely different forms.

Finally, the sustainability transition accentuates the need for continuous formative and summative learning for the continued development of policy. With this, potentially, the evaluation work will have an increased strategic purpose in the formulation of future policies.