Assessment tool of the month

This month we are profiling the R&D Outlook(TM).

The R&D Outlook(TM) is a review of the evidence necessary to substantiate proposed R&D based on the anticipated scientific, technological, market and social outcomes.  The evidence, analyses, conclusions presented in the R&D Outlook support the development of R&D proposals that positions the value of the proposed research on the basis of scientific, technological, economic and social impacts of the anticipated results.

COMMUNITY OF PRACTICE OF THE MONTH

Learn how Researchers/Scientists can engage VISTA's art, science and passion to develop more strategic and competitive R&D projects with improved innovation outcomes.

REPORT OF THE MONTH

Download a copy of Developing a College-Based Applied R&D Impact Assessment Framework completed in December 2007.

EXECUTIVE SUMMARY

Traditional research performance indicators, such as publications, are not good measures of the contribution of College R&D programs. Assessing the socio-economic effects of R&D could provide a more comprehensive measure of research performance at Colleges. In this respect, the logic model appears to be the strongest overall framework for segmenting the effects of R&D programs and linking these effects to R&D activities.

There are several conceptual frameworks for the effects of R&D activities. These frameworks can be grouped based on their focus: research (Knott and Wildvasky’s Six Stages of Impact; Kautz and Larsen’s Five Stages of Innovation), socio-economic (Salter and Martin’s Six Benefits; B.P.I.R. System; Buxton and Hanny’s Research Payback; Allen Group’s Typology of Benefits; Godin and Dore’s Eleven Dimensions of the Impact of Science), and knowledge (Howard’s Four Types of Knowledge; Molas-Gallart’s Knowledge Exchange Networks). However, many frameworks lacked the concrete metrics necessary to measure the proposed conceptual dimensions.

Measurement techniques can be focused on the R&D organization itself, and the client organization. Case studies were the most prominent type of study conducted, and common measurement techniques included documentary analysis, questionnaires, in-depth interviews and focus groups. The use of multiple measurement techniques has been recommended in the literature. However, the most important aspect of selecting evaluation techniques for R&D activities is consulting with the stakeholders for whom the evaluations are being conducted. It is critical that stakeholders be involved in the design of the evaluation framework because the actual validity of an evaluation is only as good as the validity of the measurement in the mind of stakeholders.

Three R&D-specific frameworks are recommended for Colleges, with Godin and Dore’s Eleven Dimensions of the Impact of Science being the strongest of these conceptual frameworks. In terms of measurement techniques, questionnaires appear to be the most effective and versatile measure, but all techniques should be considered. Considering that no concrete metrics were available, issues pertinent to variable design must be considered. Three key variable types were proposed: retrospective-change (e.g., individuals using retrospective thought to gauge how much change occurred), real-change (e.g., comparing status measures at the start and end of the project), and R&D contribution (e.g., the perceived degree to which the R&D organization contributed to a change).

A template for data collection for the evaluation of new R&D projects is proposed. The framework emphasizes the use of a multi-technique approach to gauge real-change. Following the completion of the R&D project, a “follow the client” and “follow the people” approach to data collection is recommended. This monitoring could be continued for several years. However, given that the development and testing of new metrics will be required, it is recommended that completed projects be evaluated first. By using historical analysis and retrospective-change questions, problems and issues with data collection could be easily identified, with little risk to ongoing or future projects.