Effective industrial and innovation policy requires learning and evaluation of public interventions. Evaluation in turn requires documentation in the form of, for example, intervention plans and descriptions, decision texts and call tenders. Without such documentation, the expertise and logic behind interventions is lost over time and evaluations risk asking the wrong questions.

In this report, we have analyzed the documentation and associated routines at Vinnova, the Energy Agency and the Agency for Economic and Regional Growth. The analysis focuses on the scope, contents and handling of documentation. Our conclusion is that the state of documentation at the studied authorities is likely to complicate the evaluation of both individual interventions and overall policy.

The authorities' reflections on the logic behind interventions are lost if and when documentation is not established, does not contain important aspects or is lost. In the long term, the deficiencies could have negative consequences for the effectiveness of policy and governance. Identified gaps and deficiencies also risk complicating aspects of the authorities' internal work such as handovers and collaboration. Documented intervention logic for previous interventions would save time and resources in the design of new interventions as well as lead to better interventions.

Strengths and weaknesses in the authorities' documentation of interventions

Vinnova

Vinnova's handling of established documentation is usually excellent. The authority's work with open data has made the processes for handling of documentation uniform. It has also simplified access to documentation and increased the transparency as to which documentation is available and not.

However, the scope of the documentation, i.e. the types of interventions that are covered by documentation, is not comprehensive. The non-public calls for proposals that make up a very important part of Vinnova's operations are not documented to a sufficient extent. The strategic innovation programs and interventions focused on spreading information would also benefit from more documentation, to facilitate evaluation and learning. Since the main form of documentation of individual interventions, i.e. call texts, is aimed at an external audience, contents that is important for learning and evaluation is often missing.

The Energy Agency

The contents of documentation of interventions at the Swedish Energy Agency is generally detailed, homogenous and well structured, enabling evaluation and learning. The documentation of individual applications, however, is missing the authority's own purposes and goals, which complicates evaluating and learning about these interventions.

The authority's handling of documentation seems to have failed during transitions between IT systems. There are now gaps in central databases and archive systems, which risk creating problems for future evaluations.

The Agency for Economic and Regional Growth

The documentation of interventions at the Agency for Economic and Regional Growth is relatively decentralized. Individual departments seem to have their own routines for what is to be documented, what the documentation should contain and how it should be handled. The non-homogeneous handling has led to so significant deficiencies in handling, such that it has been difficult to make an informed assessment on the scope and content of program documentation at the authority. These deficiencies risk making it more difficult to evaluate many of the agency’s efforts in a fair and accurate manner.

A clear area for improvement is the documentation of interventions responding to government assignments. The agency receives many of these, often at short notice, and is expected to initiate interventions quickly. In these cases, the agency seems not to have the time to document their own thoughts and reflections on the interventions, which complicates evaluation and learning, as well as the design of interventions.

Our analytical process

Our analysis is based on the documentation that we have collected within our method development project Prerequisites and methods for mapping growth policy initiatives. Our focus has been the state of documentation for 2019. In addition, we have analyzed the authorities’ documentation as far back as digital records is available, something that has varied between the three authorities.

The data collection has been carried out in collaboration with the authorities to ensure that the data is representative. To ensure that we base our analysis on a correct picture of the state of documentation, we have repeatedly requested additional documentation, pointed out gaps and requested clarifications. The collection was only considered completed when the respective agencies announced that we had received all relevant documentation. To enable best possible coverage, we have allowed the collection process to take the time it needs, over a year.

In the concluding part of the report, we address recommendations to the three authorities, and the government, on how to strengthen and further support the agencies’ work with documentation. Our recommendations are a contribution to the discussion on the need for more impact evaluations in industrial and innovation policy.