Roadmapping in the Era of Uncertainty: How to Integrate Data-Driven Methods with Expert Insights

Roadmapping has long been regarded as a practical tool for supporting decision-making for science and technology innovation and it has received recent attention for its potential use in responses to uncertainty. Indeed, roadmapping enables forward-looking strategy making and thus helps to reduce uncertainty. Accordingly, numerous studies have been conducted to propose new approaches to roadmapping for a wide range of contexts, including the data-driven and expert-based approaches. Although these two main approaches have distinct advantages and disadvantages, few previous studies have focused on how to integrate them into roadmapping to better support decision-making related to science and technology innovation. To address this research gap, this study investigated how to integrate data-driven approaches with expert insights during roadmapping. For this purpose, a workshop-based roadmapping method was combined with data-driven methods to test this approach in the context of technology planning for the automobile industry. An ethnographic approach was used to collect data on when, where, and how data analysis must be conducted to support experts’ discussions. The research findings open a discussion regarding how to integrate data-driven methods with expert insights during roadmapping based on the trade-offs between the two types of data, that is, hard data for data-driven methods and soft data from expert insights and suggest possible opportunities for future roadmapping developments. Professor, Department of Industrial Engineering, sungjoo@ajou.ac.kr Sungjoo Lee

FORESIGHT AND STI GOVERNANCE Vol. 15 No 2 2021 1 Ethnographic approach suggests non formalized, contextually adaptive gathering and analysis of empirical data. T echnology and strategic roadmaps have long been regarded as flexible tools that can support strategic and long-range planning by matching longterm goals with short-term actions and specific technology solutions . Since its first introduction by Motorola in the 1980s, roadmapping has been applied to various contexts, including technology forecasting [Gerdsri, 2007], new product development [Petrick, Echols, 2004;Lee et al., 2008], service planning [Cho, Lee, 2014], and R&D project planning [Cho et al., 2016]. On the one hand, regardless of its context, the forward-looking feature of roadmapping helps organizations manage the fuzzy front-end of innovation and survive in turbulent environments, enabling them to reduce uncertainty by collecting the information required to plan for the future. On the other hand, it must be combined with other methods to be suitable for the context. Among the approaches proposed for roadmapping, the two main streams are the data-driven and expert-based approaches. The data-driven approach uses hard data such as patents to investigate past trends to predict the future [Geum et al., 2015], whereas the expert-based approach relies heavily on expert insights -that is, soft data produced during roadmapping workshops . The representative research group of the former approach is the Seoul School, whereas that of the latter is the Cambridge School [Park et al., 2020]. These two approaches have distinct advantages and disadvantages. The data-driven approach, although it takes a retrospective perspective, utilizes the insights derived from reliable sources to implement systematic analysis techniques. Given recent advances in data analytics, such as natural language processing, deep learning, and artificial intelligence, as well as expanded data sources for investigating innovation activities, the potential for data to support human decision-making is considerable. However, the expert-based approach facilitates the use of tacit knowledge that is not available in the public domain. Furthermore, based on this knowledge, such an approach enables the setting of goals to achieve a desirable future outcome, supporting normative forecasting and exploratory forecasting. Accordingly, the two approaches can be complementary, and if they are implemented well together in the roadmapping process, they can support better decision-making related to science and technology innovation. Nevertheless, few previous attempts have been made to address this issue; the few exceptions include the work [Kostoff, Schaller, 2001], which mentions a hybrid method that combines a computer-based approach (considered a data-driven approach in this paper, emphasizing the importance of data) and an expert-based approach, and [Lee et al., 2007], which summarizes the data analysis techniques that can be used at each roadmapping stage. Indeed, the role of data in successful roadmapping has been highlighted in previous studies [Lee et al., 2011;Schimpf, Abele, 2019]. The quantity and quality of information provided to support roadmapping can affect its results. To address this gap in the research, therefore, this study investigates the integration of data-driven approaches with expert insights during roadmapping. For this purpose, a workshop-based roadmapping method was combined with data-driven methods to test this approach in the context of technology planning for the automobile industry, Hyundai Motor Company. A single case study using an ethnographic approach 1 was adopted to collect data on when, where, and how data analysis must be conducted to support experts' discussions. Thus, data needs during the roadmapping process were presented along with the techniques to visualize the data analysis results. The research findings open a discussion on how to integrate data-driven methods with expert insights during roadmapping based on the trade-offs between the two data types and suggest possible opportunities for the future development of roadmapping.

Literature Review
Roadmapping is defined as "A process that mobilizes structured systems thinking visual methods (e.g., roadmap 'canvas' and participative approaches to address organizational challenges and opportunities), supporting communication and alignment for strategic planning and innovation management within and between organizations at the firm and sector levels" [Park et al., 2020, p. 2]. It helps organizations better prepare for technological change and offers a tool for corporate foresight [Linton, Walsh, 2004]. Indeed, organizations that use corporate foresight more often are more likely to be engaged in roadmapping and produce more innovation [Yoon et al., 2019]. Due to these advantages, roadmapping has gained significant attention recently and relevant research has increased notably in number [Carvalho et al., 2013;Park et al., 2020]. The number of roadmapping studies exceeded those on other popular planning tools such as Delphi, scenario planning and modeling/simulation [Park et al., 2020]. They also noted that the growth and prosperity of roadmapping studies have led to several research streams, with two distinguished ones including the focus on the design of roadmapping processes, outputs, and on the development of supportive tools for roadmapping [Park et al., 2020]. These research streams are related to roadmapping approaches that are classified into three categories: expert-based, computer-based, and hybrid approaches [Kostoff, Schaller, 2001]. The expert-based approach relies on insights in developing roadmaps and involves holding a series of workshops to identify roadmap elements and their relationships [Wells et al., 2004;Phaal, Muller, 2009;Phaal et al., 2007]. Moreover, cross-functional roadmapping teams can be organized to provide suf-ficient knowledge for successful roadmapping Gerdsri et al., 2010;Phaal et al., 2004]. The relevant studies mainly focus on the development of roadmapping processes or roadmap canvases and on identifying key success factors for procedure. As a result, numerous roadmapping methods have been proposed for choosing technology alternatives [Garcia, Bray, 1997], introducing scenario planning [Groenveld, 1997], supporting fast-starting roadmapping , and managing emerging technologies [Gerdsri, 2007]. In addition to the roadmapping process, previous studies also proposed roadmap canvases, both as roadmapping outputs and as roadmapping guidelines. The most representative format is a timebased multi-layered chart, with the top layer mapping business trends and drivers, the middle layer mapping products/services/functions, and the bottom layer mapping technologies. However, these formats can change according to the purposes behind the roadmapping effort. Geum et al. [Geum et al., 2013] proposed a roadmap canvas specifically for open innovation, titled the dual-technology roadmap. Likewise, the relevant studies have investigated the design and customization of the roadmapping process, the structure of the roadmap canvas, and the key success factors of roadmapping. Although expert-based roadmapping based on expert insights is appropriate for corporative foresight, its success may depend highly upon individuals' capabilities-that is, their ability to innovate, willingness to share information, and prior experiences used to justify their decisions. Therefore, it may not be effective in some cultures in which discussions are not encouraged or in areas of convergence in which expert insights are not sufficient to provide all necessary knowledge for roadmapping. On the other hand, applying data analysis techniques that require a computer to create roadmaps has increased rapidly, possibly with advances in big-data analytics. Producing roadmaps solely via computerbased analysis, where roadmap elements and their relationships are identified without expert intervention, is called a computer-based approach. On the contrary, roadmapping that involves expert and computer-based analyses constitutes a hybrid approach. Previous studies adopting these approaches have generally proposed novel roadmapping methods (i.e., analysis methods) and tested them in practice. Accordingly, these approaches align with the 'application-and-proposition' research stream. First, computer-based roadmapping commonly employs patent data, which is regarded as one of the most rich and reliable sources of innovation. These roadmaps, sometimes called patent roadmaps, were developed to investigate technology trends , monitor competitors [Lee et al., 2012;Yu, Zhang, 2019], or establish R&D strategies [Suh, Park, 2009]. Using patent data for roadmapping is advantageous because such data improves the credibility of roadmapping outputs. Nevertheless, patent roadmaps have shortcomings when used for corporate foresight due to its inherent retrospective nature and its limited consideration of corporate-level strategies in developing roadmaps. Second, a hybrid approach aims to overcome the limitations of computer-based approaches and those of expert-based approaches to increase the results' objectivity while maintaining interactions among experts during roadmapping. Existing studies on hybrid roadmapping concentrated on tools to support decisions before, during, and/or after the process. Indeed, expert-based roadmapping often introduced strategy-making methods such as scenario planning and evaluation such as technology valuation to adopt information obtained from them. Many studies combined roadmapping with other decision-supporting tools emphasizing responses to uncertainty. For example, multiple scenarios and their impact on roadmaps are considered [Geum et al., 2014;Lee, Geum, 2017], the robustness of roadmaps is analyzed [Lee et al., 2016], and the impact of changes from external and internal factors on roadmaps is evaluated to determine whether the roadmap needs revision [Gerdsri et al., 2019]. Other studies proposed a set of tools for various other purposes, such as a patent and portfolio analysis for prioritization [Lee et al., 2007] and a design-structure matrix for analyzing relationships among roadmap elements [Son et al., 2018]. These decision-supporting tools should greatly improve roadmapping performance, but few attempts have been made to understand how they are used in a real organizational setting. Our knowledge regarding the data source of hybrid roadmapping's needs, how it embeds into the overall organizational process, and how it evolved amongst rapid change is relatively limited. This study aligns with existing literature on hybrid roadmapping and tries to fill the research gap. Consequently, it highlights issues regarding a methodological aspect of hybrid roadmapping and its application at a large corporate organization [Park et al., 2020;Amati et al., 2020;Simonse et al., 2015]. Furthermore, we focus on the information sources necessary to ensure roadmaps' quality.

Proposed Hybrid Approach
The proposed approach has three stages, as presented in Figure 1. This process is based on Cambridge's S-Plan [Phaal et al., 2007], in that the first two stages correspond to S-Plan's landscaping and the last stage is related to the S-Plan's topic mapping. The first stage focuses on the ideation of innovation opportunities, which may come from short-term market and business needs or long-term changes in the technological and business environment. Thus, data for internal and external environment analyses are necessary. The ideas proposed in the first stage are then evaluated in the second stage. After the ideas are grouped into several topics, these topics are prioritized. Finally, for the topics selected in the second stage, detailed plans for pursuing each of them are established. Given that detailed planning requires a comprehensive review of relevant technologies, data analysis results that represent technology trends and available technology solutions are needed at this stage. Using the workshop-based approach, we listed data sources that can support decision-making at each stage of roadmapping based on practical needs and we aimed to integrate these datadriven methods with expert insights.

Stage 1. Ideation
The first stage of roadmapping aims to identify innovation opportunities. The first roadmapping workshop is organized for this purpose. Workshop participants are encouraged to propose such opportunities by answering certain questions -why, what, and how. Those opportunities may arise from the technology-push and market-pull approaches. Technology push derives from new technologies seeking a useful application, whereas market-pull considers innovation opportunities based on market and business demands. Taking a longer perspective, it is necessary to address emerging changes in technological and business environments to facilitate insights that lead to disruptive innovation. During this discussion, although experts propose new business and technology opportunities based on their insights and data analysis, the provision of further information regarding internal and external environment analyses can promote their discussions. Internal sources of information, including customer complaints and survey results, help experts understand current market needs, as well as system, product, or service failures, to examine the limitations of current business offerings and past projects to identify actions taken to overcome the limitations of strategies at a higher level. However, external sources of information can be used to describe trends in terms of patents, publications, and media based on which emerging technologies and competitor's activities are investigated to identify opportunities. Particularly in cases of high uncertainty, future scenarios can be developed to derive various ideas for each scenario; innovation opportunities are captured for the scenario. Figure 2 describes a workshop template for this stage. If j scenarios are built, j opportunity maps need to be created: some of innovation opportunities may be common across multiple scenarios, while others may be specific to a particular scenario. In the figure, topic (i,j) indicates the i th topic from the j th scenario. In order to help understand scenarios and seize innovation opportunities easily, the use of a value proposition canvas is recommended. The value proposition canvas is a graphical expression of what customers do, need, and suffer from in a specific context and further help design product and service offerings to satisfy the customers [Osterwalder et al., 2014].

Stage 2. Selection
Whereas the first stage is aimed at identifying various ideas, the second stage targets idea selection. As the ideas developed in the first stage may involve similar concepts, they must be grouped into several topics. Likewise, if common ideas are submitted from different scenarios, those ideas must be merged into a single topic. At this stage, many discussions among the workshop participants are held to define the scope and concept of each topic. Once the concept of each topic becomes clear, those topics are prioritized to select the most important ones. Generally, as promising topics that an organization is capable of implementing are considered valuable as business opportunities, two criteria -attractiveness and feasibility -are used for prioritization. Then, a strategic technology roadmap is established on a selected topic, for which experts' panels are used. The experts use their insight to set a vision for the topic, discuss the current state of the topic, and establish the milestones to achieve the vision based on the current state. During this process, panels often face situations in which all information on the relevant technologies for the topic are not available, particularly in the era of uncertainty and convergence. Nevertheless, a compre- External environment analysis (scenarios, patents, publications, media etc.) • Identifying ideas to respond to market / business needs (short-term) • Identifying ideas to prepare for changes in environment (long-term) Prioritisation analysis (patents, publications, media etc.) Portfolio analysis (patents, publications, media etc.) • Grouping the ideas (product/service/technology) into topics Planning hensive understanding of the corresponding technology trends and potential competitors is required for robust planning. Here, patent analyses discover available technology solutions within the target area as well as other areas. In particular, the recent advances in data analytics and visualization allow one to extract useful technical information from a large number of patent documents effectively. Emerging technological trends can be identified in the form of keywords or key concepts from the collection and analysis of topic-related patent documents. On the other hand, patent documents published in other areas can also be analyzed to summarize their trends to be referenced or converged, which supports the discussion during roadmapping.
Here, the level of specificity for a strategic technology roadmap may vary by resource constraints and the roadmap's purpose. Figure 3 represents workshop templates for Stage 2. The map on the left is used for detailed planning for each topic, while the map on the right is for an aggregated level planning for all topics of concern.

Stage 3. Planning
The final stage is aimed at developing a detailed plan to pursue high-priority topics, particularly those focusing on technology planning. The technologies relevant to the topics are evaluated by four selection criteria: importance, urgency, development risk, and technological capabilities. Importance evaluates the criticality level of the organization's acquiring the technology. If the technology is likely to have a strong, positive impact on the organization and aligns with the organizational strategy, it will have a higher value. Urgency measures how immediately the technology is needed at the organization. Risk evaluates the degree of risk associated with the technology; if the technology requires complex technologies and high costs, its development risk will be high. Capability indicates the level of technology-related knowledge or expertise within the organization. These criteria can be adjusted to roadmapping contexts. Sub-criteria can be designed, where decision-supporting techniques such as scoring models, the analytic hierarchical processes, or the analytical network processes are used to synthesize experts' evaluation results. Furthermore, data analysis can also support experts' evaluation at this stage. For example, patent analysis can serve as a reference for technological capabilities and impact analysis can serve as a reference for importance. Based on the analysis results, two portfolios are proposed: one to prioritize the items and the other to establish an action plan (see Figure 4).

Background
The three-stage workshop-based roadmapping process was performed in collaboration with Hyundai Motor Company, a South Korean automobile company. Recently, the automobile industry has encountered considerable challenges related to a dramatically changing business landscape caused by the emergence of the sharing economy, pressures caused by environmentally friendly automobiles, and the opportunities available for various forms of personal mobility. These unpredictable factors make the industry risky for incumbents but easy to enter for newcomers. Organizations in the industry are introducing roadmapping aggressively to search for new business and technology opportunities and they are investing in their R&D to cope with the expected changes in their industry Accordingly, the automobile industry was suitable for a case study, which was conducted over the course of two months (February and March 2018). The research team played the role of a roadmapping team that designed a process, recruited participants, facilitated the process, provided relevant information, and summarized the roadmapping results. During the process, we observed when and where information needs occurred, what kind of information was required, and how the data needed to be analyzed to support expert's decisions effectively during roadmapping.

Roadmapping Process
The three-stage roadmapping was implemented using two workshops along with tasks imposed between the workshops. The first and second stages were covered in the first workshop, while the third stage was conducted in the second workshop. The participating team was in charge of technology development related to the noise, vibration, and harshness (NVH) of automobiles and aimed to establish a long-term R&D plan for about 10 years given the new role of NVH technology in future mobility services: in general, a ten-year time horizon is considered to be appropriate for many organizations [Phaal, Muller, 2009]. All team members were involved in the roadmapping as a taskforce team. The roadmapping process started from process design, followed by the development of scenarios, identification of opportunities, and the development of strategies along with action plans. Here, short-term opportunities were searched for in the trend analysis of patents and publications, while long-term opportunities were identified from expert discussions. From the requirements of the participating team, the main target for data analysis was set to patents and publications within and outside of the sector. As a result, a standardized process proposed in this study was customized as shown in Table 1.
The first workshop was carried out on February 9, 2018 and was aimed at identifying new business and technology opportunities (Stage 1). Before the workshop, the team requested that future mobility scenarios be prepared because all the team members were engineers and, although NVH technologies can be expected to be influenced significantly by external factors, they needed sufficient time to think about those factors. Accordingly, five scenarios were proposed using three criteria -vehicle control, vehicle ownership, and new vehicles (see Figure 5). Then, to facilitate discussion, customer profiles that described the activities that customers were involved in while using the vehicles were developed for each scenario as well as a value proposition map that investigated the needs and wants expected during the activities. Finally, the participants generated new business and technology opportunities and explained them in terms of why the opportunity is needed, what the opportunities are, and how the op-   Second-top priority portunity can be pursued (that is, the technologies required to do so). These opportunities were discussed among participants and similar opportunities were merged, leading to 18 topics (Stage 2). A further evaluation of the attractiveness and feasibility of the 18 topics led to the selection of 11 topics along with 46 corresponding technologies for detailed planning. Of these, five topics were associated with multiple scenarios and were called G-topics (general-topics), whereas six were scenario-specific and were called C-topics (context-topics). Given the limited number of topics, internal discussions were carried out to choose the topics and technologies for further investigation.
On the other hand, trend analysis results based on patents and publications were provided to the participants for the identification of available technologies within and outside of the sector (see Figure 6). For this purpose, 27,411 publications were collected from the Scopus database. In total, 5,988 patents from the USPTO, 1,181 patents from the EPO, and 329 patents from the KPO were collected on automobile NVH technologies; these were published between January 2016 and March 2018. To summarize the contents of the patents and publications effectively, topic modeling based on LDA 2 was performed. This resulted in nine topics (29 subtopics) from patents and publications. The relationships between the topics proposed during the workshop and the topics obtained from data analysis were investigated by the research team and proposed to the roadmapping participants for reference. In addition, for each topic, the relevant keywords, the number of relevant documents, the major organizations, and key documents were summarized, as shown in Table 2. We also highlighted hot topics that are gaining more attention and cold topics that are losing attractiveness, which we defined based on the increasing rate of relevant documents, along with outlier patents defined as unique patents in terms of their technological content on the premise that those patents could have the potential to be disruptive technologies. This data analysis process helped the participants understand what opportunities are in line with the main technology trends and, more specifically, obtain a list of patents and publications worth reviewing while investigating each topic. Furthermore, the participants suggested that recent psychoacoustic technologies could be applied to improve automobile NVH system, particularly for G-Topic 1. However, the topics were outside their areas of expertise, so another round of data analysis was necessary. Focusing on the psychoacoustic technologies, we collected 1,534 publications published from January 2016 to March 2018 from the Scopus database. Again, LDA-based topic modeling was used, resulting in ten topics (40 subtopics). For this study, we focused only on the publication data because the company was seeking collaboration partners in academia, which was the main source of the publications. Again, the research team matched relationships between the 13 technologies for G-topic 1 and ten topics identified during data analysis to help the roadmapping participants introduce emerging psychoacoustic technologies in the NVH system. Furthermore, these patent and publication analyses enabled the participating team to identify potential collaborators for pursuing the topics.
The second workshop, which targeted more focused discussions on the selected topics and relevant technologies, was held on March 29, 2018 (Stage 3). Between the first and second workshops, we asked five key participants to evaluate the 46 technologies associated with the 11 topics using four criteria -urgency, risk, importance, and capability -as well as precedent relationships between the technologies via a technology cross-impact matrix. This task encouraged the participants to think deeply about the topics and technologies and let them search for those technologies individually. This was expected to support discussions in the second workshop. Given the broad scope of the topics, we allowed the participants to leave some questions partly unanswered if they lacked expertise on the technology or failed to collect relevant information. On acquiring the results, the basic statistics of the technology -mean and standard deviation -and technological relationships were used to develop a preliminary technology roadmap for the second workshop, as shown in Figure 8. In this workshop, participants presented and discussed various viewpoints, particularly in terms of the technologies with high standard deviations in their evaluation results. An in-depth discussion of the technologies with diverse viewpoints led the participants to share their ideas and reach a consensus. In addition, a first draft roadmap developed based on the evaluation results was presented to be modified according to the discussion results. This preliminary roadmap could have been developed for each topic or for all topics at an aggregated level. As the roadmapping was conducted at the team level, not the organizational level, we put all the topics and relevant technologies onto a single roadmap, as shown in Figure 9. Here, it should be noted that five C-topics were merged into a single topic due to the small number of relevant technologies for each topic. Accordingly, the roadmap included seven topic layers with 46 technologies. FFirst, the technologies were positioned according to their urgency values

Figure 6. Topic Analysis Results for within the Sector
Source: authors.     Adding or deleting more technologies was allowed at this stage, although it did not occur in this case study. Finally, the technology evaluation results were aggregated by topic (see Table 3), based upon which portfolio map types were developed -one for prioritization and another for action plans as shown in Figure 10. With the average index value to separate high and low space in the map, five out of 11 topics (G-Topic 1, G-Topic 3, G-Topic 4, C-Topic 2, and C-Topic 3) were positioned in the fourth quadrant of the prioritization map, signifying top priorities of development. Among these five topics, four (G-Topic 1, G-Topic 3, G-Topic 4, and C-Topic 3) were located in the fourth quadrant, while only C-Topic 2 was in the first quadrant of the action plan map. Thus, an R&D collaboration strategy was recommended for the first four topics, producing an industry-academic collaboration lab funded by Hyundai Motor Company. Internal R&D was designed for C-Topic 2. When investigating the relationships between shortterm emerging topics identified from the patent and publication data analysis and long-term promising topics identified from expert insights, we found that improving the internal capabilities in E, F, and I, which are associated with the five top-priority topics (G-Topic 1, G-Topic 3, G-Topic 4, C-Topic 2, and C-Topic 3), would be greatly helpful in preparing for the future and gaining competitive advantages. Furthermore, we also found that G-Topic 1 could benefit significantly from recent developments in psychoacoustic technologies, since the topic is characterized as a technology convergence between NVH and psychoacoustic technologies. Therefore, a link between technologies for G-Topic 1 and the topic analysis results for psychoacoustic technologies were considered to identify potential collaboration partners.

Recommended situations for applying the method
With the emergence of disruptive technologies and innovative business models, companies are challenged by ongoing technological revolutions and rapidly changing business conditions, leading to an increasing level of uncertainty. The proposed approach that integrates data-driven methods with expert insights is useful particularly for technology planning under such uncertainty. Expert insights are key to long-term planning by setting a vision and strategy for the organization, while data-driven methods enable users to understand and prioritize innovation opportunities made available by recent disruptive technologies. In addition, the data-driven methods help one to collect and synthesize expert insights systematically. Under uncertainty, experts may have different subjective opinions on the same candidate opportunities, possibly due to their different experiences and knowledge of the target technological domain. Thus, it is essential to integrate their knowledge for reliable decision-making, where datadriven methods provide objective information around candidate opportunities.

Bias
Despite the value of combining expert insights with data-driven methods, using several data-based methods is not always advantageous because data collection and analysis usually requires a huge amount of time and effort, thus delaying the roadmapping process. On the other hand, agile roadmapping is valuable under uncertainty, capturing value in short-term initiatives. Accordingly, understanding the needs for data analysis is indispensable for designing the roadmapping process along with accessing the right data and having the most qualified people as roadmapping participants. Notably, said participants may depend too much upon data analysis results, thus limiting their creativity in developing new ideas. Hence, an optimal use of data analysis results for effective roadmapping must be determined. Furthermore, additional data sources necessitate a process for selecting reliable ones, constructing a balanced portfolio, and developing effective analysis methods -issues worth addressing.

Implications for theory
While most existing studies on roadmapping focused either on workshop-based or data-driven approaches [Park et al., 2020], this study emphasized the integrated use of expert insights and data-driven methods. On the one hand, opportunity capture by eliciting knowledge from experts is important since it involves tacit knowledge sharing. On the other hand, opportunity capture by analyzing technological and market data is also significant given that it enables one to generate, identify, and evaluate more ideas. How to best combine the two types of knowledge -one from expert insights and the other from data analysis -can be an important topic for roadmapping research.

Implications for practice
The research findings offered several implications for practice in the field of roadmapping. First, data analysis needs largely occurred in three areas: understanding research trends within the sector, identifying available technologies in another relevant sector, and collecting and summarizing expert opinions. With today's increased technological complexity and emerging breakthrough and/or converging technologies, data analysis is essential to improving roadmap quality. Furthermore, offering data analysis results may enhance communication among experts. By providing objective information, a data analysis prevents a single person, usually a senior manager, from dominating the discussion. Future research is needed to examine how the data analysis results can be used to enhance roadmapping quality.

Application notes
Before initiating a roadmapping process, it is necessary to clearly define the scope and purpose of such an endeavor including a plan of how the outputs will be used. Designing a roadmapping process within a limited budget is also critical for successful roadmapping, particularly for a hybrid method where both experts and data-driven approaches are used. Regarding the experts, the most motivated and qualified people need to be identified to be engaged in roadmapping. Using appropriate templates can help to elicit and share expert knowledge. As to the data, the data analysis results should be able to fill in the knowledge gaps of the experts. Therefore, the data analysis needs, procedures, and results are recommended to be carefully planned. Otherwise, the analysis results cannot be integrated into workshop-based roadmapping. The cost and benefit of analyzing each type of data should be considered as well to optimize the use of data-driven methods.

Lessons from the case study
First, expert insights produced creative ideas with information about relevant projects (past and ongoing) and about learning from them, whereas data analysis results mostly offered feasible ideas regarding competitor trends and available technologies in other sectors. These two approaches are complementary, but analysis results must be provided in a correct form at the correct level of detail for data analysis results to be useful for experts. Recent data analysis and visualization techniques can be introduced to effectively use data. Second, most computer-based roadmapping literature mainly considered two data sources as targets for analysis: patents and publications, which are useful in analyzing past trends. However, more data sources that present competitors' plans (e.g., news and You-Tube) and/or expert insights from outside of organizations (e.g., LinkedIn and podcasts) will be available, going beyond traditional analyses of patents and publications. These data sources can be linked to generate more valuable implications.

Areas for future research
The workshop observation raised several research gaps to address, suggesting future research directions. First, in the case study, the data analysis needs stem from roadmapping at a team level, and different needs could be addressed via roadmapping at the organizational level. The needs may also vary by roadmapping context, such as long-term versus short-term planning, product versus service development, defining versus solving problems, and internal R&D versus collaboration strategies. According to [Schimpf, Abele, 2019], data sources useful for roadmapping include external roadmaps, associations, market analyses, suppliers, customers, users, research organizations and universities, competitors, consulting companies, legislation, journals, and media. Useful public and private data sources for context-specific roadmapping must be studied further. Second, continuous research is needed to develop hybrid roadmapping processes and methodologies. 3 Combined with newly available data sources and methods, the roadmapping process can become more efficient. Moreover, a hybrid process can also benefit significantly from introducing roadmapping systems, which enables systematic data collection, sharing, and elaboration across organizational units [Amati et al., 2020]. It enables the integrated application of those methods and easy updates of roadmaps Lee, Park, 2005]. Thus, future research is necessary to create a framework for developing, evaluating, and improving a hybrid roadmapping process. Finally, future research must address challenges by the timely updating of roadmaps to keep them alive. Changing business environments and emerging promising technologies can be monitored continuously via data collection and analysis. How to evaluate a roadmap's status and how to choose when to revise it need further investigation.

Conclusions
This study discussed how to combine data-driven methods with expert insights during roadmapping. Today's business environment is characterized as being full of uncertainties. On the one hand, the rapid emergence of disruptive technologies 4 has changed the way organizations do business. On the other hand, social pressures on technologies or other unpredictable social changes such as the present pandemic also make it hard for an organization to plan the future. Accordingly, a workshop-based approach can help an organization to quickly respond to the changing environment, enabling agile roadmapping, and allowing organizations to also establish long-term planning based on expert insights. Here, an appropriate use of data is expected to supplement limited knowledge during the expert-driven roadmapping.
In this study, a workshop-based roadmap aiming for 10-year technology planning to deal with uncertainties was implemented in collaboration with Hyundai Motor Company. We focused on the data analysis needs of the workshop participants and the way the analysis results were used in participants' decisionmaking. The research findings indicated that discussions among participants were facilitated by the use of data analysis results: they were useful in having a forward-looking perspective for the ideation stage and collecting diverse opinions to develop a first-cut roadmap for the selection stage. For the planning stage, data were useful for understanding emerging trends within the sector and identifying available technologies and collaboration partners outside of the sector. Consequently, this study contributes to the advances in roadmapping methodologies by suggesting the necessity of a hybrid approach that combines data analysis results and expert insights. Furthermore, this is one of the few studies on corporate roadmapping in a real setting, and hence, it can have practical contributions as well. Despite its meaningful contributions, this study has several limitations. First, only a single case study was conducted in each of the contexts of 1) team-level roadmapping; 2) the automobile industry; and 3) the Asian context. The level of hard data required for robust roadmapping may change due to various factors such as the purpose of a given roadmapping, industry condition and organizational and possibly national culture. More cases are needed to improve the external validity of the research findings. Second, most of the findings were derived from observations on the roadmapping process itself. Although an ethnographical approach was adopted in this study, interviews with participants or surveys on data analysis needs during roadmapping across organizations could produce more meaningful insights. Accordingly, further analysis on the beforeand after-roadmapping stages can be conducted to understand data needs and satisfaction. Finally, this study discussed how to integrate data-driven methods with expert insights but failed to propose a new roadmapping framework to integrate a data-driven approach with expert insights. Recent advances in data analysis techniques along with the emergence of numerous data sources are expected to provide various implications to support expert ideation and decision-making during roadmapping. Future research will continue to address those issues.