Bridging from Project Case Study to Portfolio Analysis in a Public R&D Program

Rosalie Ruegg
This paper presents a framework, rooted in the case study method, for evaluating both individual projects and a portfolio of projects. It introduces a prototype evaluation tool that offers new capabilities for the evaluation of a public research and development program in the intermediate period before long-run effects can be measured. An assembly of steps produces linked layers of information in a framework that results in a synergistic evaluative capability that enables bridging from project case study to portfolio analysis. This interlinked framework for evaluation can be used both by program administrators, taking a top-down approach, and by project managers, taking a bottom-up approach. It also produces results of interest to policy makers and other program stakeholders. The prototype evaluation tool, the Composite Performance Rating System (CPRS), is designed specifically for the Advanced Technology Program (ATP), but is potentially adaptable to other programs. CPRS uses uniformly collected output and outcome data (indicator metrics) to compute an overall performance rating for each of ATP’s completed projects, using a four-star system. The distribution of ratings provides an easy-to-grasp measure of the overall portfolio’s performance. This report, organized in two parts, presents in Part I the framework for evaluating individual projects and the entire portfolio, and in Part II a detailed account of CPRS developed specifically for ATP. Part I is expected to be of general interest to the broader evaluation community; Part II, of primary interest to ATP and to other multi-goal, public research and development program administrators interested in new ways of using indicator data for portfolio evaluation.
Access through the Cases and Simulations Portal from Rutgers SPAA
Copyright © 2018, Rutgers, The State University of New Jersey. All rights reserved.