A testbed offers private firms the opportunity to test and demonstrate products and processes in pre-commercial stages. The establishment and development of a testbed is a complicated process influenced by technological, organizational, and institutional factors. The process requires not just technological expertise, but also tight collaboration and dialogue among key players: testbed operators, their potential customers, and other stakeholders.

The following summarizes the report Factors affecting the establishment of a testbed: An analysis of the“Environmental Technology Testbeds” program, which is an analysis of a government funded program aimed at stimulating the establishment of testbeds.

Background and purpose

Vinnova is a Swedish government agency that aims to stimulate innovation. A key underlying assumption to the agency’s work with testbeds is that test and demonstration is an essential step in the innovation process and that markets alone tend to undersupply these opportunities for firms seeking to innovate. The studied program is different from previous public investments in test and demonstration in that it “casts the net wide” in terms of eligible applicants (i.e., aspiring testbed operators), fosters a broad variety of testing environ­ments and has an explicit focus on cooperation with stakeholders. Evaluation of this type of program is limited, as is previous academic research. Sweden’s Growth Analysis, an agency under the direction of the Ministry of Enterprise and Innovation charged with evaluating and analysing Swedish growth policy, has undertaken this study to better understand factors that affect the establishment of testbeds. This qualitative study is based on interviews with managers of 21 independent testbed projects that have received funding from Vinnova’s “Environmental Technology Testbed” program since 2012. The studied projects were managed by research institutes, universities, public utilities, founda­tions, and private firms, and included a range of testbed types and diverse stakeholders.

Key findings

The studied program was one of Vinnova’s first applications of the increasingly popular concept of a ‘testbed’ – which proved to be a somewhat ambiguous concept for project managers. The study’s findings suggest that the funding and implementation of the testbed projects were, to a large extent, influenced by the project managers’ own interpretation and understanding of the concept and how it applied to their organizations’ operations and plans. Further, discrepancies between Vinnova’s objective for the program and the project managers’ interpretation led to a number of implementation challenges.

The heterogeneous nature of the funded projects led to varied implementation as well as varying levels of complexity. When a project was characterized by a specific task and few stakeholders, the implementation was relatively easy to manage as both the testbed operator and the stakeholders tended to better understand their role in the process. However, a number of time-consuming challenges related to coordination and administra­tion arose when a project’s focus was more diffuse and involved large numbers of stake­holders.

The supply of commercial testbed services depended on the readiness of the technology behind the testbed. For example, some technologies were still in the research stage while others were mature enough to enable test and demonstration of commercial products and processes. While supply and demand for testbed services balanced in about half of the studied testbed projects, there were nonetheless several cases of mismatch problems. One cause for mismatch (weak demand) was conservatism and reluctance among potential testbed customers. Another cause for mismatch (high demand) was immature technology or underdeveloped testing infrastructure.

Although marketing and promotion of a testbed can, in theory, increase matching possibilities, such activities were found to be inadequate, i.e., project managers tended to underestimate the need for marketing and/or fail to allocate a sufficient budget. This problem was magnified in the case of small and medium-sized enterprises, which is noteworthy given that Vinnova has a specific desire to attract such firms, which are assumed to have greater growth potential. From the perspective of the testbed operators, these small and medium-sized firms were simply unaware of testing opportunities, were reluctant to involve testbeds in their innovation processes, and/or lacked the funds to purchase testing services.

The institutional context of each project, which varied extensively, influenced the implementation process. The majority of the projects were managed by established organizations (i.e. research institutes, public utilities, universities, etc.), while others were managed by private sector firms. Being embedded in a larger organizational context had both pros and cons. Benefits included, in some of the cases, access to valuable expertise (e.g. marketing, management, business development etc.) that were not represented in the project organization itself. Disadvantages included, in some cases, higher costs associated with expensive equipment, where the increased price affected demand, especially among small firms. Being embedded in a larger organization also imposed a number of restrictions related to culture, politics, and the mission statement of the larger organization. This was a particular problem with public utilities, which had little to no experience in offering testbed services. Further, managers within a larger organizational context reported time-consuming and non-productive activity related to project approval and internal justification.

The majority of projects encountered obstacles associated with local, regional and inter­national compliance; either specific to the individual project (e.g. rules for handling chemicals) or to the institutional context within which the project was implemented (e.g. managing priorities of the needs of citizens and customers).

Conclusions and policy recommendations

A key conclusion of this analysis is the need for a shared understanding of the ‘testbed’ concept in order to achieve program goals. Further, the establishment and successful development of a testbed requires a balance between supply and demand. The establish­ment process requires a supportive context and expertise in non-technical fields. The development of a commercially sound testbed also requires a comprehensive budget that considers all relevant costs related to its establishment.

Growth Analysis recommends that Vinnova abstain from introducing new concepts or modifying the definition of existing concepts that have gained traction in the target population. Vinnova should establish higher standards for applicants regarding competence, budget, and coordination of stakeholders. Another key recommendation is the importance of outreach to inform potential customers of available testbed opportunities.