The purpose of this report is to (1) make an analysis of the so-called learning plans that the counties have been commissioned by the government to draw up; (2) study the results of the ongoing evaluation method used in the national structural funds programme and the regional structural funds programme; and (3) analyse how learning within the regional growth policy can be developed.
In an earlier study (Tillväxtanalys, 2011b), Growth Analysis reviewed the so-called learning plans that the counties have been commissioned by the government to draw up for the current year on the basis of Growth Analysis’ guidelines. The aim is to contribute to the process that is now being initiated by the regional self-government bodies, municipal cooperation bodies and county administrative boards to strengthen the regional growth policy.
The learning plans can be viewed as an attempt to build a system for learning within the regional growth policy but the difficulties are many and must not be underestimated. There are many players and levels involved, which can lead to delays and difficulties in establishing new lines of thought and interventions. Growth Analysis sees a need for reflection on a kind of learning that concerns both form and content. This also means that many of the interventions that are important in getting a leaning system to work better at regional level encompasses both administrative aspects and questions of content.
In collaboration with Kontigo, Growth Analysis has analysed the learning plans and made proposals for how learning primarily at regional level can be strengthened with the learning plans submitted by the counties as the starting point. The purpose of the commission was thus to deepen this knowledge concerning the application of the regions’ learning plans in practice and on the basis of this knowledge draw up proposals for how regional learning can be developed.
In the continuing work of following up and evaluating the learning plans, Growth Analysis recommends (Tillväxtanalys (2011b) that the regions consider the following: a future-oriented view; evaluation of the previous learning process to identify relevant obstacles to learning; clarity as regards the level of ambition and delimitations; planning of various types of intervention to strengthen all three phases (planning, implementation and sign-off); and more reflection around the project’s content.
Within the framework of the structural fund programmes, the focus on evaluation and results is further strengthened over the 2007-2013 period and relatively substantial resources are dedicated to ongoing evaluation. Here Growth Analysis presents the concept of ongoing evaluation and its results within the national structural fund programme and the regional structural fund programmes and suggests how learning at the regional level can be strengthened.
Growth Analysis has established that ongoing evaluation has had a very high level of ambition, which has to a significant degree led to a reinforcement of learning within the structural fund projects and programmes initiated during the current programme period. But Growth Analysis’ overall conclusion is that based on the programmes and project evaluations that we have reviewed. The evaluators have not managed to resolve the difficulties in producing credible evaluation results. They have, however, contributed valuable results that have to do with the actual implementation of programmes and projects.
In summary, we draw the following conclusions as regards the results of ongoing evaluation based on the review we have made:
Growth Analysis therefore proposes that the evaluators’ request concerning qualitative indicators that are used during the current programme period be further reviewed and that the changes necessary to be able to better control and monitor result development over the course of the next programme period be made.
It must, however, be pointed out that reporting credible results within these structural fund programmes is no easy task due to the way the programmes and projects are structured and implemented. The Swedish Agency for Economic and Regional Growth has earlier stated that this kind of additional or innovative programme cannot be evaluated in any simple way using so-called robust assessment methods and monitoring indicators. The agency’s opinion is that the indicators are often not reliable and that it is difficult for robust assessment methods to capture the level of ambition by influencing structures, institutions and attitudes. At EU level, both impact assessments and theory-based evaluations have been held up as legitimate ways of trying to get at the results of the structural fund programmes and projects. In this case, the Swedish Agency for Economic and Regional Growth has prescribed theory-based evaluation as a way to try to get at the problems that exist as regards reporting effects and results.
Growth Analysis is of the opinion that we shall prescribe neither the one nor the other method but instead try to combine different methods as best we can in order to as far as possible find valid indications that these interventions really deliver results based on the resources applied. All methods have their own weaknesses and the theory-based method of assessment that ongoing evaluation has embraced relies strongly on the belief that on the basis of scientific theories it is possible to capture and interpret results in programmes that have largely been formulated in a political context. This does not make the interpretations any easier but lead to other kinds of difficulties in interpreting the results. Based on the review we have made, there are examples of such difficulties where the evaluators encounter problems when they try to put the reported results in a scientific theoretical framework. Theory-based approaches to evaluation are good in that they are broad approaches and can be used to analyse causal mechanisms in different parts of the organisation in greater depth, but at the same time require that information of different kinds be collected, processed and analysed. Attempts have been through theory-based impact evaluation to find evidence and knowledge of long-term effects, and in the international arena a mix of methods has been held up that largely consist of combining impact assessment with theory-based evaluations. These, however, are both resource-intensive and time-consuming activities, but give an opportunity to determine what works and does not work when the focus is on long-term effects.
Finally, it is impossible to take all steps at once, or it may be necessary to limit the levels of ambition as regards individual parts of the growth policy system described above or different types of learning. It would therefore appear to be a good idea to integrate a plan for how learning can be developed in stages in documents such as the regional development strategy.
Serial number: Report 2013:02
Reference number: 2011/056