Project to Research Portfolio : Meeting Scale and Complexity

Citation: Linton J., Vonortas N. (2015) From Research Project to Research Portfolio: Meeting Scale and Complexity. Foresight-Russia, vol. 9, no 2, pp. 38–43. DOI: 10.17323/1995-459x.2015.2.38.43 DOI: 10.17323/1995-459X.2015.2.38.43 Nicholas Vonortas — Professor, George Washington University, United States; São Paulo Excellence Chair, Department of Science and Technology Policy, State University of Campinas, Brazil. Address: 1957 E Street, NW, Washington, DC 20052, United States. E-mail: vonortas@gwu.edu

T he mere scale of the scientific enterprise and its core role in the growth and well being of nations urgently requires a better understanding of the achievements and challenges to these investments [Lane, 2009]. The expanding volume and complexity of the enterprise -due to both the convergence of traditionally separate fields and the grand challenges science is asked to address (health, environment, energy, education) -fuel arguments for a reconfiguration of the existing systems of science and innovation to better serve society [Kintisch, 2006].
Scientists need to proactively engage in the discussion over the need to improve the efficiency and effectiveness of societal investments to ensure that the next generation of the management and decision-making process for our science, technology and innovation system is rooted in sound principles rather than shaped largely by political interests and budgetary concerns. We propose supplementing peer review with research portfolio evaluation approaches and decision-making tools that can better assess research uncertainties and other special features of the transformation of the resulting knowledge into improved social well-being. A coupling of research quality review by peers with more systematic portfolio meta-analysis of recommended projects is both possible and essential.
The need for change has been evident for some time. Both industry and public sector have gradually changed the way they account for research. In the public sector, GPRA was put into place by the American Congress twenty years ago and the Program Assessment Rating Tool (PART) by the Office of Management and Budget only a few years later (now retracted). Proposed changes can be dangerous. Recently the proposed High Quality Research Act [Mervis, 2013] created much unhappiness and tension in the United States. Nobody is against quality, but the demand to certify how each piece of the science-funding puzzle is needed independently for the future benefit of a country is a tall order. Yet, who can stop elected policy officials from making such "worthy" demands? Scientists can! By counter-proposing a principled research portfolio evaluation system allowing vast improvements in investment decisions. Rather than a radical proposition, this falls clearly within the Science of Science and Innovation Policy remit.

Challenges of the Peer Review System
While the peer review system has served science well, it has not been free of debate [Feller, 2013]. Chief concerns include: lock-in to existing trajectories of scientific inquiry and the difficulty of linking the best single research projects to the pragmatic challenges that society faces. None of this is insurmountable, but the opaqueness of scientific inquiry for the majority of the public (the ultimate user) combined with the perception that scientists are a 'club' -supported generously by society so that members of the 'club' can decide how to allocate funds to each other -has weakened public support. This perception can be reduced by the systematic use of research portfolios governed by appropriate objectives and appraised in responsible ways.
The use of objectives for research is a problem only if misapplied. Hence, the scientific community must state how they are applied. The existing concerns of outside stakeholders can be taken into account without violating the fundamental principles of the peer review system. We need to enhance the process of dividing research funds into smaller disciplinary or functional budgets, by further clarifying the objectives of each fund and by making a clear statement regarding subject coverage for each fund/portfolio -including insights into breadth, completeness of coverage, and even the research overlap and/or duplication that may be helpful. The overall risk and risk distribution must be specified. Finally, whether a portfolio is directed/undirected and focuses on a specific field, discipline or is multidisciplinary should be stated.
Having set objectives for each research fund, some enhancements of the peer-review process are required. Currently, the review system's consider-ation of portfolio fit ceases once resources are assigned to a project fund. Successful implementation of a portfolio-based approach requires assessing both the inherent quality of the proposed research project and its similarities/ differences in terms of subject content and risk with other projects under consideration. This modification still ensures that the best projects based on peer-review are selected. However, if lesser projects are ill-fitted with the objectives of the overall portfolio, they will be replaced with the next best peerreviewed project(s) that fit the portfolio better. Portfolio-based peer-review committees will have to assess the risk and fit between projects, thus insuring that detractors cannot offer a rational argument against either individual or overall outcomes. Reviewers will undertake this task 'armed' with a toolbox that has been significantly enriched during the past couple of decades 1 .

Moving to Portfolios
While the existing portfolio management knowledge infrastructure is advanced already, further developments are required given that scientific knowledge differs significantly from other investment subjects in which such methodologies are applied. Science projects and portfolios differ from their non-science counterparts (physical capital and financial investments) because the likelihood of sudden and extreme successes and failures is higher. They differ because of the presence of technological as well as market uncertainty. They differ because of intellectual property and the complications of imperfect appropriability. And they differ because science is cumulative -it typically builds progressively on previous results. The current state of the art for project and portfolio management of science, technology, and innovation confronts us with short, medium, and long-term needs.
In the short-term it is critical to abandon tools that while popular and appropriate for engineering economics, capital budgeting and financial analysis, are inherently biased against the desirable uncertainty associated with research. Tools such as NPV (Net Present Value) are counter-productive, appropriate only for projects with predictable outcomes and cash flows (such as a toll bridge) 2 . The lack of a priori predictability is a characteristic of science that must be accepted and embraced. The interrelation of projects -both synergies and duplication -must be considered as they lead to portfolio modification, thereby providing better overall value. Sufficient replication is needed to provide the necessary breadth and depth of research [Nelson, 1990]; this should be distinguished from careless duplication -such as duplicating infrastructure that has sufficient capacity to be utilized for multiple research projects. The challenges of cumulativeness of science and of the incomplete appropriability of results must be taken into consideration.
Several approaches for assessing research investment portfolios have been identified [Linquiti, 2012]: Qualitative methods • focusing on multiple objectives. Projects are sorted into ad hoc categories based on a taxonomy of characteristics. Categories are then reconciled with overall organizational strategies and resources allocated; Quantitative methods • focusing on multiple objectives. A weighted scoring system for each important attribute of research projects is used to provide an overall score for each project. Linear programming selects projects to maximize the aggregate score within the portfolio, subject to constraints.
1 Several agencies have such committees already, including NSF, DOE, NIH, etc. What we are really proposing is a significant extension of their role and their systematic use of formal analytical tools and models.
, where N is the total number of periods, t is the period under consideration, R is the return for period t, and i is the interest rate.

Наука
Alternatively, Data Envelopment Analysis determines the set of attribute weights that maximize the efficiency of each project.
Quantitative methods focusing on a single objective • . Non-linear programming methods consider the correlation between projects and, for a target portfolio return, identify the projects that offer the lowest variance in portfolio return. For complex portfolios that exhibit multiple non-linear relationships, non-standard probability distributions and complex decision rules, stochastic like Monte Carlo simulation are used.

Dynamic methods •
. Interactions among research projects, economies of scale and scope, and complementarity/substitutability offer different risks and returns to alternative combinations and sequences. Real option and decision tree methods can be applied to determine alternative research portfolios [Vonortas, Desai, 2007]. Still largely untested in public sector research environments, these methods are quite promising and deserve a close look.
In the medium term, we need to obtain the most appropriate scope and balance of risk and return through the selection of a combination of high-risk high-return and lower-risk lower-return projects. Review procedures that better determine the term 'impact' could, for instance, be based on real options for methodologies to capture both potential upside benefits and downside costs (and failures) while utilizing critical peer review input for appraising technical risk [Vonortas, 2008]. Viewing a research portfolio as a set of options, this approach can vastly improve the handling of market and technical risk that affects research. It is dependent on committees of scientific experts providing core research features.
Of course, as scientific discovery has an underlying distribution that is non-Gaussian, we cannot adopt the tools of financial portfolio management unchecked [Casault, Groen, 2012]. Balancing research portfolios requires models able to deal with the inherent differences in the underlying distributions. Further development and testing on real research project portfolios is necessary. Research in real options [Triantis, 2003;Brosch, 2008] must address questions like: How does the performance of a research portfolio depend on the relationship among the likely performances of the research projects? What types of relationships drive the risk and the return of the whole portfolio? How can the information about these relationships be utilized to improve the process of research project portfolio formation?
In the long-term the need is to institutionalize research portfolio analysis, thereby, also abandoning the current common practice of accepting low risk projects utilizing traditional techniques, while approving high-risk projects on the basis of vague arguments of societal interest. Versatile analytical approaches capable of handling different levels of risk will replace the inherently negative term 'risk' with more neutral terminology reflecting both the positive upside and negative downside variations. Broad coverage of basic science must continue to avoid deep losses from ignoring unpredicted areas of future importance and their associated socio-economic benefits.
This goal is closer than it appears. A series of objectives associated with economic returns, societal benefits (e.g., health, environment, national security), risk tolerance, the coverage of different fields and disciplines, and degree of concentration on different parts of the discovery path from pure science to product development and application must be determined. Once these objectives are identified, the most suitable portfolio can be identified and eventually adjusted on a more dynamic basis -reflecting the acceleration of scientific progress and the convergence of traditionally separate fields. Interesting methods already under development [Van Bekkum, Pennings, 2009;Zapata, Reklaitis, 2010;Bhattacharyya, 2011] can become first-rate decision-making tools that prevent non-expert policy makers and managers from making simplistic statements justifying or rejecting research on the basis of emotional (rather than rational) appeal. ФОРСАЙТ Т. 9. № 2 2015 Наука Implications Differences between implications from applying conventional and portfolio approaches to research funding are represented in Figure 1. Each research project is like a piece of a puzzle. The classic peer review process (image on the left) tends to provide unintended overlap and allows for an ill fit between some of the pieces and unwanted gaps to occur. Areas of high risk and high return can be missed due to their controversial nature and split decisions typically resulting in negative funding decisions. Very high-risk projects (bright colors) are included in the system if they are deemed to be of national interest, thus circumventing split decisions. The portfolio approach (image on the right) includes more colors as it selects projects based on optimizing portfolio objectives, therefore not eliminating projects with split decisions which are the best available to satisfy a stated objective. In the classic peer review process, bright colors associated with high risk and a high frequency of split decisions have tended to be replaced with gray color associated with lower risk. The portfolio of projects provides a complete picture with good coverage of both field and risk level and no unintended overlap between projects. Still, the distortion of the target bands indicates that the portfolio approach is an improvement, not a panacea.
The implications of widespread use of research portfolio management are: Better allocation of resources: • -decision-making based on the merit of both projects and entire portfolios of projects; -availability of early warning systems of 'gaps' in scientific inquiry; -ability to see the 'highways' between different types of research; -ability to holistically address the 'grand challenges' of research.
Better justification of public resources for research: while risky research • projects can -and should -fail, portfolios of projects are constructed so that the overall risk of technical and market failure is minimal.
A leap forward in terms of greater value from our research base and clear-• er identification of extant gaps and opportunities, while preserving the critical role of the peer review system.
For enduring results, far-reaching policy actions are required: Institutionalize portfolio methods of analysis and decision-making in the 1.
public sector and boost fledgling efforts in federal research agencies such as the National Institutes of Health (NIH), the National Science Foundation (NSF) and the Department of Energy (DOE) in the United States Source: compiled by the authors. to move in that direction. In order to achieve this, one must consider the interrelation of projects as well as the risk/return trade-off for each project and between research projects that may or may not be part of the set of research projects selected. This depends on the availability of richer data both before and after program implementation that enables richer insights than is presently the case. It also requires program managers who understand how to use the information to assess the risk/return trade-off, the impact of correlation, synergy, and mutual exclusivity.
A new generation of research managers must be trained in the intrica-2.
cies of science and technology policy and the complexities of modern decision-making support tools to ensure that portfolios are considered in relation to socio-economic objectives. Translating complex policy objectives into effective research portfolios remains partly an art -despite the emergence of numerous rigorous techniques -depending as it does on non-numeric decision-making processes.
Better define the 'impact' criterion in the evaluation of research propos-3.
als. Whereas individual projects are evaluated on their merit as sound scientific endeavors, their impact must be defined firmly in terms of association with and contribution to the broader objectives of the respective portfolio(s).
In conclusion, the scale and complexity of the contemporary technical enterprise as well as the convergence of traditionally distinct science fields has overpowered our current techniques for the management of science, technology and innovation. We echo the calls for a Science of Science Policy [OSTP, 1998]. More specifically we underline the urgent need for further developments in the support system of research management. The consideration of R&D as part of portfolio management offers a great leap ahead for obtaining greater value from our current research base and for clearly identifying the gaps and opportunities that will move us forward.