Carayannis Goletsis Grigoroudis 2018 Composite innovation metrics

Prévia do material em texto

Technological Forecasting & Social Change 131 (2018) 4–17
Contents lists available at ScienceDirect
Technological Forecasting & Social Change
Composite innovation metrics: MCDA and the Quadruple Innovation
Helix framework
Elias G. Carayannis a,d,⁎, Yorgos Goletsis b, Evangelos Grigoroudis c
a GWU School of Business, Washington, DC, USA
b University of Ioannina, Department of Economics, GR45110 Ioannina, Greece
c Technical University of Crete, School of Production Engineering and Management, GR73100 Chania, Greece
d National Research University Higher School of Economics, Moscow, Russian Federation
⁎ Corresponding author.
E-mail addresses: caraya@gwu.edu (E.G. Carayannis),
vangelis@ergasya.tuc.gr (E. Grigoroudis).
http://dx.doi.org/10.1016/j.techfore.2017.03.008
0040-1625/© 2017 Elsevier Inc. All rights reserved.
a b s t r a c t
a r t i c l e i n f o
Article history:
Received 21 February 2017
Accepted 5 March 2017
Available online 20 March 2017
Innovation is a complex, dynamic, socio-technical, socio-economic and socio-political phenomenonwhich needs
to be approached in a holisticmanner to be properlymeasured and assessed. In this paper, we revisit the national
and regional Innovation Scoreboards using a multiple criteria decision analysis (MCDA) approach in the context
of the Quadruple Innovation Helix (QIH) framework. We deploy anMCDA approach combining AHP and TOPSIS
methodswhichmerges data fromGovernment, University, Industry, and Civil Society sectors (the fourQIH actors
or helices) and overcomes limitations of the existing Innovation Scoreboard approach by incorporating the differ-
ent preference systems of the QIH Helix actors. The findings illustrate the power and promise of our approach as
an alternative composite innovationmetric. Estimating the different preferences of innovation stakeholders gives
the ability to develop policies and practices oriented towards specific QIH actors. Estimating the importance that
eachQIH actor assigns to different innovation aspects is critical policy-wise and practice-wise as it provides a per-
spective on relative efficacies and potential ways andmeans to calculate differential efficacies for alternative con-
figurations of resource allocations. These results underlie specific policies, practices, and priorities therein based
on the relative re-distribution of weights.
© 2017 Elsevier Inc. All rights reserved.
Keywords:
Innovation
Innovation Scoreboard
Innovation systems
MCDA
AHP
TOPSIS
Quadruple Innovation Helix Framework
Composite innovation metrics
1. Introduction
Innovation is considered as a key driver to economic growth and
competitiveness. Traditional industrial economies are now transformed
to knowledge economies where innovation is considered to one of the
mains drivers for sustained economic growth, if not the single one
(Grupp and Mogee, 2004). Policy makers need efficient and effective
tools to measure and monitor the innovation related performance so
that they develop new measures, policies, and evaluate current
approaches.
In order to understand the innovation concept and model the inno-
vation process, initially a linear approach has been considered for sever-
al years, including a sequence of research (basic and applied) and
commercialization (market test and diffusion). This linear approach
was later changed with the introduction of a dynamic/systemic behav-
iour in the chain-link model of innovation (Kline and Rosenberg,
1986). In this direction different actors are considered to be interacting
into non-linear path characterized by feedback mechanisms. In this
framework a systems approach is applied for describing the knowledge
goletsis@cc.uoi.gr (Y. Goletsis),
creation (‘Mode 1’ to ‘Mode 2’ (Gibbons et al., 1994) or even ‘Mode 3’
(Carayannis and Campbell, 2006, 2009) (Carayannis et al., 2016a). In
particular, ‘Mode 1’ knowledge production is associated with the linear
model of innovation (i.e., invention-innovation-diffusion), where there
is a sequential ‘first-then’ relationship between basic research (knowl-
edge production) and innovation (knowledge application). This ap-
proach has been challenged by the concept of ‘Mode 2’ of knowledge
production (Gibbons et al., 1994), which is related to a context-driven re-
search (i.e., knowledge application and knowledge-based problem-solv-
ing in a context of application), while ‘Mode 3’ knowledge production
(Carayannis et al., 2016a) focuses on and leverages higher order learning
processes and dynamics that allow for both top-down government, uni-
versity and industry policies and practices as well as bottom-up civil soci-
ety and grassroots movements initiatives and priorities to interact and
engage with each other toward a more intelligent, effective, and efficient
synthesis (see also (Carayannis and Campbell, 2006, 2009, 2012)).
An innovation systems approach at a national level was introduced
(Lundvall, 1992; Nelson, 1993) defining a complex set of relationships
at a country level. Specifically, a National Innovation System (NIS) is de-
fined as consisting of a network of institutions whose activities initiate,
import, modify and diffuse new technologies and which provide the
framework within which governments form and implement policies
to influence the innovation process (Freeman, 1987; Metcalfe, 1995).
http://crossmark.crossref.org/dialog/?doi=10.1016/j.techfore.2017.03.008&domain=pdf
http://dx.doi.org/10.1016/j.techfore.2017.03.008
mailto:vangelis@ergasya.tuc.gr
http://dx.doi.org/10.1016/j.techfore.2017.03.008
http://www.sciencedirect.com/science/journal/00401625
1
2
3
de
5E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
TheNIS concept has since then been used as a tool for analyzing country
specificities in the innovation process in a globalized economy, as well
as a guide for policy formulation (OECD, 1999). This systematic ap-
proach has been extended to regional level and the concept of a Region-
al Innovation System has been introduced in the research and policy
agendas (Cooke, 2001; Cooke et al., 1997). Regional concepts are con-
sidered as tools to generate an effective national innovation system
(Chung, 2002) as in the globalized knowledge-based economies the re-
gion state is becoming the focal point of the economic activity instead of
the nation state. Regions are more dynamic and responsive than nation
states, regions can better exploit knowledge advantages and stocks, and
they can focus on region specific capabilities, while interaction and co-
operation (and clustering) are feasible at the regional level. As (Hajek
et al., 2014) note, the benefits of spatial proximity are realized at region-
al level, and the benefits are reflected in the effective formal and infor-
mal cooperation among regional actors (investors, entrepreneurs,
researchers, enterprises, public institutions, and consumers). In turn,
policy formulation should focus on regional specificities and capabilities
so as to optimize the results (see also the smart specialization concept).
In this National/Regional innovation systems framework, four main
interacting actors are involved, this leading to the concept of the Qua-
druple Innovation Helix (QIH) framework. Within the QIH framework,
academia and industry interact and collaborate while government co-
ordinates and facilitates applying top-down policy instruments accord-
ing to visions and perspectives for the future, while civil society forms
the fourth helix interacting with all the above in a bottom-up fashion.
Given that innovation is a complex dynamic socio-technical, socio-eco-
nomic and socio-political phenomenon, civil society plays a central role
in driving user-centric innovation that serves both the society and the
economy (social innovation inter alia).
Within this framework several approaches have been applied in
order to measure innovation-related performance. An indicator ap-
proach (single, multiple or composite) has been mostly followed; the
need for international comparisons have led tothe use and the develop-
ment of internationally comparable indicators. In general, the single in-
dicator approach (e.g., R&D expenditures, number of patents) has been
found to offer only a limited view of such a broad and complex concept
such as innovation (Tidd and Bessant, 2013); therefore the role of com-
posite indicators has been significantly enhanced in recent years
(Carayannis and Grigoroudis, 2016; Carayannis and Provance, 2008;
Paas and Poltimäe, 2010). Sub-indicators used in these approaches
and related surveys have been following the evolution of the innovation
concept with the introduction of the idea of incremental innovations,
the introduction of non-technological innovation, the focus on co-oper-
ation co-opetion and recently on open innovation(Chesbrough, 2006),
as well as targeted open innovation (Carayannis and Meissner, 2016;
Carayannis et al., 2016c; Meissner et al., 2016). Innovation scoreboards,
based onmultiple indicators have been developed aiming at facilitating
evaluation, benchmarking and policy formulation. Among the most
widely used instruments are the European Innovation Scoreboard1
(EIS) and the Regional Innovation Scoreboard2 (RIS).3 IUS and RIS are
considered to be the central authoritative sources for the European
Commission and other EU as well as national policy making bodies
(Adam, 2014). EIS is composed of 25 indicators covering 8 dimensions
structured under 3main blocks/pillars. In a similar approach, RIS is com-
posed of 12 indicators (existing also at the EIU).
Although these innovation scoreboards are major attempts to grasp
the multiple facets of innovation, they have received significant criti-
cism on how the composite index is calculated (Archibugi et al., 2009;
Grupp and Mogee, 2004; Grupp and Schubert, 2010), as well as due to
the equal weighting scheme (Adam, 2014; Grupp and Mogee, 2004;
https://ec.europa.eu/growth/industry/innovation/facts-figures/scoreboards_en
https://ec.europa.eu/growth/industry/innovation/facts-figures/regional_en
Other approaches include the Global Innovation Scoreboard, the Global Innovation In
x, etc.
-
Schubert, 2006) applied in indicator aggregation. It should be noted that
equal weighting does not imply the absence of weighting, because the
former implies an explicit judgment on the weights being equal. Its ef-
fect also depends on how component indicators are divided into catego-
ries or groups:weighting equally categories (composed of different sub-
indicators) could disguise different weights applied to each single sub-
indicator (Goletsis and Chletsos, 2011; OECD, 2008).
In order to address the above criticism in our work we apply a mul-
tiple criteria approach for estimating the different innovation indicators'
weights and constructing country and regional rankings. Althoughmul-
tiple criteria decision analysis (MCDA) manages to effectively capture
the multiple dimensions of the evaluation problem, as well as the exis-
tence of multiple stakeholders, there are limited applications focusing
on innovation; most of them apply MCDA for innovation planning
(e.g., (Meesapawong et al., 2014)), technology roadmapping (see e.g.
(Cho and Lee, 2013)). Recently, MCDA has been applied to assess the
competitiveness of nations (Perez-Moreno et al., 2016) A multiple
criteria method (namely TODIM) has been applied by Paredes-
Frigolett et al. (2014) to rank innovation systems of Latin America and
Iberian Peninsula countries; in their approach they suppose that prefer-
ence weights are already known to the decision maker.
Our approach is based on combining AHP (Saaty, 1990) and
TOPSIS methods (Hwang and Yoon, 1981). Specifically, in our
approach the weighting issue is addressed through the application
of the Analytical Hierarchy Process (AHP) method. In the first step
the AHP method applies pairwise comparisons to hierarchical struc-
tures. Although it is usually applied in ranking problems, in the
current application, the AHP method provides a structured frame-
work for setting priorities on each level of the hierarchy. Given the
hierarchy implied in the scoreboard indicators, we apply AHP for
weight elicitation. The AHP method has been effectively applied in
numerous MCDA applications with intangible criteria, while Saaty
(Saaty, 2005) notes that using pairwise comparison measurements,
the AHP method is able to provide relative measurements. In the sec-
ond step, the TOPSIS (Technique for Order Preference by Similarity to
Ideal Solution) is applied in order to evaluate the innovation performance
of countries and regions. TOPSIS' basic principle is that the chosen alterna-
tives should have the shortest distance from the positive ideal solution
and the farthest distance from the negative ideal solution. As Tavana
and Hatami-Marbini (2011) note, TOPSIS has been shown to be one of
the bestMCDAmethods in addressing the rank reversal issue. This consis-
tency feature is largely appreciated in practical applications. Another rel-
ative advantage of TOPSIS is its ability to identify the best alternative
quickly (Parkan and Wu, 1997).
The proposed MCDA approach overcomes the need for conversion
into a single metric (Cherchye et al., 2004) which can raise significant
criticism especially in the case of aggregation of monetary with non-
monetary outcomes (e.g., patents with employment and high tech ex-
ports). Through the TOPSIS applicationwe also overcome the constraint
for mutual independence of preferences that could occur if we applied a
weighting average approach into the scoreboards' sub-indicators (see
(Keeney and Raiffa, 1976) for further discussion).
The evaluation of criteria (or attributes) weights is one of the major
issues discussed in theMCDA literature, particularly in aggregation pro-
cedures using an additive model. A detailed overview on weights elici-
tation methods for additive models can be found in (Eisenführ et al.,
2010)), while it should be noted that this problem can be examined
by different perspectives. For example, (Keeney and Raiffa, 1976) pres-
ent and discuss the trade-off elicitation procedure, while experimental
analysis has been also used in several studies (see for example
(Borcherding et al., 1991;Weber and Borcherding, 1993). Also, rank or-
dering criteriamethodshave beenwidely used for evaluatingweights in
an MCDA context (see (Ahn and Choi, 2012). Additional literature in
weight elicitation includes the works of (Grupp and Mogee, 2004),
who use a heuristic approach, while (Cherchye et al., 2008; Saisana
and Tarantola, 2002) use models based on linear programming.
https://ec.europa.eu/growth/industry/innovation/facts-figures/scoreboards_en
https://ec.europa.eu/growth/industry/innovation/facts-figures/regional_en
Fig. 1. Structure and indicators of EIS.
Direction of time
First Helix:
Academia/
Universities
Second Helix:
Academia/
Universities
Third Helix:
State/
Government
Fourth Helix:
Media-based and
culture-based public,
and civil society
Fig. 2. Conceptualization of the QIH framework (Carayannis and Campbell, 2009).
6 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
Freudenberg (2003) and Jacobs et al. (2004) adopt statistical methods
based on Monte Carlo simulations.
Our approach, apart from addressing technical criticisms on score-
board construction, introduces a QIH framework perspective applied
to the abovementioned approach, as we get criteria (sub-indicator)
weights from a pool covering (i) multiple stakeholders and (ii) stake-
holders from each of the four main actor or helices of the QIH frame-
work. Each of them is characterized by a different preference system
corresponding to different priorities and rankings. Incorporating the
multiple actor view in our approach, the analysis can support the use
of innovation-related data for assessing innovation related policies (at
different levels/nation-region) along the precepts of the QIH
framework.
The main contribution of this work is that the proposed approach
can be a valuable toolfor innovation policy makers as well as practi-
tioners. Having estimated the different preferences of innovation
stakeholders, it is possible to develop policies and practices oriented
towards specific QIH actors or helices or potentially involve to a
different extent the QIH actors in existing policies and practices. It
should be emphasized that the QIH weights are both policy-wise
and practice-wise critical, as they provide a perspective on
relative efficacies and potential ways and means to calculate differ-
ential efficacies for alternative configurations of resource allocations.
These results underlie specific policies, and practices as well as prior-
ities therein based on the relative re-distribution of weights. In
addition, combining the results of the AHP and the TOPSIS
methods it is possible to implement a performance/importance anal-
ysis in order to identify strengths and weaknesses of national or re-
gional innovation systems. Finally, contrary to the existing
approaches, the presented work considers the different aspects of
several decisionmakers in a national/regional innovation ecosystem,
thus the existence and involvement of decision makers, which is ab-
solutely necessary in MCDA problems, is clearly recognized and
represented.
In the following sections, we first briefly present the national and
regional innovation scoreboards to form the basis of our approach. We
then present the multiple criteria methodology proposed. Results in-
cluding different weights and rankings are presented and commented
in the next section which is followed by concluding remarks.
2. Innovation Scoreboards and the Quadruple Innovation Helix
Framework
- The European and the Regional Innovation Scoreboards
The European Innovation Scoreboard (EIS) is a widely used compos-
ite innovation index. It is used tomeasure at an annual basis the innova-
tion performance of its member countries. EIS, previously named
Innovation Union Scoreboard, is used as a major tool to assess relative
strengths and weaknesses of national innovation systems and helps
countries identify areas they need to address. Innovation performance
is measured using a composite indicator – the Summary Innovation
Fig. 3. AHP based weight calculation
7E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
Index – which summarizes the performance of a series of indicators,
structured in a hierarchical way. The Summary Innovation Index is cal-
culated as the unweighted average of the normalized innovation
indicators.
Three types of indicators are distinguished: Enablers, which refer to
innovation drivers external to the firm, Firm activities, referring to the
innovation efforts at the level of the firm andOutputs referring to the ef-
fects of firms' innovation activities. The current version of EIS considers
in total 25 indicators, structured in 8 innovation dimensions and cover-
ing the following 3 main blocks (Hollanders et al., 2016a):
a. Enablers: This block of indicators captures the main drivers of inno-
vation performance external to thefirm, including the innovation di-
mensions of human resources (availability of a high-skilled and
educated workforce), open, excellent and attractive research sys-
tems (international competitiveness of the science base) andfinance
and support (availability of finance and support for research/innova-
tion projects).
b. Firm activities: It refers to the innovation efforts at the firm level and
consists of dimensions related to firm investments (R&D and non-
R&D innovation investments), linkages and entrepreneurship di-
mension (entrepreneurial and collaboration efforts with the public
sector and among innovating enterprises), and intellectual assets
(different forms of Intellectual Property Rights (IPR) as an output
of the innovation process).
c. Outputs: This final block is related to the effects of enterprises' inno-
vation activities and includes the dimensions related to innovators
(enterprises that have introduced innovations in different forms),
and economic effects (economic success of innovation in terms of
employment, exports, and sales).
Fig. 1 presents an overview of the EIS, including the detailed innova-
tion indicators. It should be noted that EIS concept has been evolving
over the year mainly in terms of the included dimensions/indicators
(e.g. including new forms of innovation). These revisions show the on-
going debate on defining innovation and justify the lack of a universally
accepted constant framework for measuring it (Carayannis and
Grigoroudis, 2016; Carayannis and Provance, 2008).
The Regional Innovation Scoreboard (RIS) is a regional extension of
the EIS, assessing the innovation performance of European regions on
a limited number of indicators. The RIS uses as many indicators as pos-
sible from the EIS including regional data from the Community Innova-
tion Survey (CIS). The RIS is limited to using regional data for 12 of the
25 indicators used in the EIS (see also Fig. 1). For some indicators, slight-
ly different definitions have been used, as regional data would not be
available if the definitions were the same as in the EIS.
Similar to the EIS, a Regional Innovation Index is calculated as the
unweighted average of the normalized innovation indicators (see
(Hollanders et al., 2016b) for further details). Based on this composite
index, regions are classified to innovation leaders, strong innovators,
moderate innovators and modest Innovators, using EU28 average per-
formance as a threshold. In general, as regional and national innovation
is linked, variation is limited for most countries. Still, RIS manages to
identify regional performance groups and identify in some cases stron-
ger regional variation in some countries; this highlighting regional spec-
ificities and the existence of regional ‘pockets of excellence’.
- Quadruple Innovation Helix Framework
As noted in the previous section, the ‘Mode3’ knowledge production
system consists of innovation networks and knowledge clusters for
knowledge creation, diffusion and use (Carayannis and Campbell,
2006; Prainsack, 2012): “It is a multi-layered, multi-modal, multi-nodal
and multi-lateral system, encompassing mutually complementary and re-
inforcing innovation networks and knowledge clusters consisting of
human and intellectual capital, shaped by social capital and underpinned
by financial capital” (Carayannis and Campbell, 2009). The Quadruple
Innovation Helix (QIH) concept is associated with the ‘Mode 3’ knowl-
edge production system (see (Carayannis et al., 2012) for a discussion
of the evolution of knowledge creation models).
The QIH framework is a transformation and enhancement of the Tri-
ple Helix model developed by Etzkowitz and Leydesdorff (2000). The
Triple Helix model considers only three helices that intertwine and by
this generate a national innovation system limited to a top-down ap-
proach: Universities, Industry, and Government, while in the QIH
framework, Carayannis and Campbell (2009) propose to add a fourth
helix, identified as “media-based and culture-based public” and “civil
society”, thus introducing and incorporating the bottom-up approach.
As shown in Fig. 2, this helix is related with “media”, “creative indus-
tries”, “culture”, “values”, “lifestyles”, “art”, and perhaps also the notion
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Human Resources
Research Systems
Finance &
Support
Firm Investments
Linkages and
Entrepreneurship
Intellectual
Assets
Innovators
Economic Effects
Universities
Government
Industry
Civil Society
Fig. 4. Innovation Scoreboard weights per QIH actor.
8 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
of the “creative class” (see also (Carayannis et al., 2012; Carayannis and
Campbell, 2012; Carayannis and Grigoroudis, 2016).
It is important to note the dynamically intertwined processes of co-
opetition, co-evolution and co-specializationwithin and across regional
and sectoral innovationecosystems emphasized by the QIH concept
(Carayannis and Rakhmatullin, 2014). Thus, the QIH actors may have
different roles (e.g. enabler, utilizer, developer or user) that should be
identified and analyzed.
3. Methodology
In the proposed approach we apply a three step approach: (i) First,
we identify the relevant indicators from the innovation scoreboards
and establish the decision matrix. (ii) We apply the AHPmethod to ob-
tain indicator (criteria) weights. In order to capture the preferences of
the four different actors we use specially built questionnaires for this.
Aggregate weights are then calculated. (iii) We apply the TOPSIS meth-
od to get national or regional rankings according to the respective deci-
sionmatrices (i.e., indicator scores) and theweights of each actor. Apart
from the global ranking, individual actor rankings are obtained in order
to reveal the diversified preference systems.
In the following paragraphs the two MCDA methods applied are
presented:
- AHP – analytic hierarchy process
AHP is a multiple criteria method that supports decisions by
organising perceptions and judgements into a (multi-level) hierarchical
structure that exhibit the forces that influence a decision (Saaty, 1994).
AHP provides the framework for setting priorities on each level of the
hierarchy using pairwise comparisons. This approach makes it ideal
for estimation of weights in the scoreboard case (see also (Clinton et
al., 2002; Reisinger et al., 2003) on the appropriateness of AHP for indi-
cator prioritization).
In the proposed approach we only apply the AHP approach for the
elicitation of the indicators (criteria) weights, as TOPSIS is expected to
better handle potential rank reversal issues. Moreover its dual
comparison against positive and negative ideal solutions help avoiding
the predicament that the units of same value cannot be appropriately
ranked (see related discussion in (Hsieh et al., 2006; Joshi et al., 2011)).
In AHP, the criteria weights W = (w1, w2, ….wm) are determined
through pairwise comparisons using an m x m comparison matrix:
A ¼
a11 ⋯ a1m
⋮ ⋱ ⋮
am1 ⋯ amm
0
@
1
A
where aij denotes the relative importance of criterion i over criterion j
(in a scale 1–9) (aii = 1 and aij = 1/aji, i, j = 1.m). The weight vector
can be then determined by solving the characteristic equation AW =
λmaxW, where λmax is the maximum eigenvalue of A. Saaty (1990) also
proposes the use of a Consistency Ratio (CR) to examine the consistency
of A.
In our approach we first apply AHP based pairwise comparisons for
the calculation of the weights of the 8 EIS innovation dimensions. In a
similar way the individual indicator weights are calculated for the indi-
cators of each axis. The weights for each of the k actors are then the
products of the twoweights. In order to capture the preferences of mul-
tiple actors, mean weights are then calculated (see Fig. 3).
- TOPSIS - Technique for Order Preference by Similarity to Ideal
Solution
TOPSIS is an outrankingMCDAmethod initially proposed by Hwang
and Yoon (1981) that can be used for ranking or selection problems. Its
main idea is that the best alternative is the one with shortest distance
from the ideal (positive ideal) solution and the farthest distance from
the negative ideal solution, among the whole criteria set. In our case,
TOPSIS application is composed of the following steps:
i. Establish the decision matrix for the ranking based on sub indicator
scores,
X ¼ xij; i ¼ 1::n; j ¼ 1::m
� �
Table 1
Innovation dimension weights per QIH actor (%).
Innovation dimension Universities Government Industry
Civil
Society
1. Human resources 6.86 5.26 10.72 13.91
2. Research systems 4.06 5.90 8.13 12.05
3. Finance & support 12.36 6.52 14.59 15.75
4. Firm investments 17.26 15.40 18.61 13.23
5. Linkages and 20.73 4.82 15.85 9.65
9E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
where n denotes the number of alternatives (i.e., countries/regions) and
m the number of scoreboard indicators (criteria).
ii. Calculate the normalized decision matrix, R = [rij] where
rij xð Þ ¼
xijffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
∑nj¼1 x2ij
q ; i ¼ 1;…;n; j ¼ 1;…;m:
iii. Calculate the weight normalized decision matrix, V= [vij] by multi-
plying with respective criteria weights:
vij xð Þ ¼ wjrij xð Þ; i ¼ 1;…;n; j ¼ 1;…;m:
In the proposed approach the weights are calculated though the ap-
plication of the AHP method.
iv. Determine the positive ideal solution (A+) and the negative ideal so-
lution (A−)
Aþ ¼ vþ1 xð Þ; vþ2 xð Þ;…; vþj xð Þ;…; vþm xð Þ
n o
¼ maxvij xð Þj j∈ J1Þ
� �
; minvij xð Þj j∈ J2
� �ji ¼ 1;…;n� �
A− ¼ v−1 xð Þ; v−2 xð Þ;…; v−j xð Þ;…; v−m xð Þ
n o
¼ minvij xð Þj j∈ J1Þ
� �
; maxvij xð Þj j∈ J2
� �ji ¼ 1;…;n� �
where J1 is associated with the positive indicators and J2 with the nega-
tive indicators (if any).
v. Calculate the distances of the alternatives to the positive ideal solu-
tion Di+ and to the negative ideal solution Di−:
Dþi
� �
≔
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
∑
m
j¼1
vij xð Þ−vþi xð Þ
� �2s
; i ¼ 1;…;n:
D−i
� �
:¼
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
∑
m
j¼1
vij xð Þ−v−i xð Þ
� �2s
; i ¼ 1;…;n:
vi. Calculate the relative closeness to the ideal solution and rank the al-
ternatives according to this distance (in descending order)
Ci
∗ = Di−/(Di+ + Di−), i = 1,...,n
4. Application and results
The proposed methodology has been applied to the two Innovation
Scoreboards, namely the EIS and the RIS. For the EIS case, 25 indicators
for 34 countries have been selected.4 As per the EIS hierarchy, these in-
dicators have been grouped under 8 dimensions, namely Human Re-
sources, Research Systems, Finance & Support, Firm Investments,
Linkages and Entrepreneurship, Intellectual Assets, Innovators, Eco-
nomic Effects. In a similar way, 11 indicator data for 190 European re-
gions have been collected from RIS.5 Few missing values were
replaced by average values in the country case. In the region case, the re-
spective country value was used in case of a missing value. The
4 2015 scoreboard data have been used. In 2015, EIS was called IUS-Innovation Union
Scoreboard.
5 RIS 2014 data have been used.
dimension and indicator weights were extracted through a specially
built questionnaire facilitating AHP like pairwise comparisons. A 1–9
scale was used, with 1 indicating equal importance between indicators
and 9 indicating extreme importance.
Aiming at capturing the different views of the QIH actors the ques-
tionnaire was sent to four distinct groups:
• Group Universities: This included academics active in innovation re-
lated subjects as well as technology transfer offices, entrepreneurship
offices, science and technology parks and related structures as well as
research institutes.
• Group Government: This included government officials (national and
regional levels) dealing with entrepreneurship and innovation sup-
port including Regional Innovation Councils.
• Group Industry: This included Chambers of Commerce and Industry,
listed companies in Athens Stock Exchange, accelerators and incuba-
tors.
• Group Civil Society: This included NGOs and students.
The collected responses showed significant variation in the percep-
tion of the importance of each innovation dimension (Fig. 4). As
shown in Table 1, for University actors Linkages and Entrepreneurships,
Firm Investments and Innovators (including SMEs with innovation and
employment in fast growing firms) are themost important factors. Sur-
prisingly, the (open, excellent and attractive) Research Systems dimen-
sion is considered the least important for University respondents. The
composition of the indicator could be an explanation to this: The related
indicators include the international co-publications, the publications
among top 10% cited and the numberof non-EU doctoral students. Al-
though this is to be investigated, it seems that a series of other enablers
belonging to research systems may be missing here. Government offi-
cials have a totally different view giving strong emphasis on the Eco-
nomic Effects of the innovation process, on Innovators and on Firm
investment; all of which reside on the firm level. Contrary to the acade-
mia, they consider Linkages and Entrepreneurship (including collabora-
tions and in-house innovators) as the least important factor. The view of
Industry people is relatively close to the Universities, with greater im-
portance given to research systems. Intellectual Assets (patents, trade-
marks, etc.) seem to be the least important factor to them. In the Civil
Society case weights are more or less equally dispersed, with Finance
and Support being considered as themost important factor and linkages
being considered the least important one. In addition, it seems that the
Research Systems and the Intellectual Assets have a very low impor-
tance to all actors. On the other side, directly firm-related dimensions
(i.e., Firm investments and Economic effects) have the highest impor-
tance to all four actors.
Combing the aforementioned estimated weights with the criteria
performance it is possible to develop a series of performance/impor-
tance diagrams, which are able to identify innovation strengths and
weakness, as perceived by the different QIH actors. These diagrams
are similar to SWOT maps or actions charts, since they can identify po-
tential improvement actions, as well threats, opportunities or leverage
entrepreneurship
6. Intellectual assets 6.19 10.27 3.34 10.14
7. Innovators 16.92 19.16 12.28 12.06
8. Economic effects 15.62 32.67 16.48 13.20
Fig. 5. Relative importance/performance diagrams per QIH actor.
Table 2
Weight correlations among actors.
Universities Government Industry
Civil
society
Universities Pearson
correlation
1 0.372 0.805⁎ −0.111
Sig. (2-tailed) 0.364 0.016 0.794
N 8 8 8 8
Government Pearson
correlation
0.372 1 0.356 0.102
Sig. (2-tailed) 0.364 0.387 0.809
N 8 8 8 8
Industry Pearson
correlation
0.805a 0.356 1 0.369
Sig. (2-tailed) 0.016 0.387 0.369
N 8 8 8 8
Civil society Pearson
correlation
−0.111 0.102 0.369 1
Sig. (2-tailed) 0.794 0.809 0.369
N 8 8 8 8
a Correlation is significant at the 0.05 level (2-tailed).
10 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
opportunities (see (Grigoroudis and Siskos, 2010) for a detailed discus-
sion). The performance/importancediagrams for the different actors are
presented in Fig. 5, where the importance of the innovation indicators,
as estimated by the AHP method, corresponds to the horizontal axis,
while the vertical axis refers to the average performance of these indica-
tors across all countries, as given by the EIS (raw data). All values have
been standardized using the average and the variance of data (so that
the intercept of axes corresponds to the centroid of all points in the
graph) in order to have fully comparable results. It is important to
note that this standardization process produces relative diagrams,
thus, the categorization of innovation criteria is relative, i.e., the impor-
tance (or performance) value is considered high (or low) with compar-
ison to the performance (or importance) values of the other criteria.
Although the weights estimated by the AHPmethod reveal different
preference systems for each QIH actor, the relative performance/impor-
tance diagrams show several similarities across all actors. As shown in
Fig. 5, public R&D expenditure is considered as a strength,while venture
capital, business R&D expenditure, non-R&D innovation expenditure,
innovative SMEs collaboratingwith others, and SMEs introducing prod-
uct or process innovations are considered asweaknesses fromalmost all
actors. It is important to note that the majority of the aforementioned
weaknesses refer to firm activities, and particularly firm investments.
Thus, these specific indicators should be a top improvement priority in
any potential innovation policy.
Different strengths and weaknesses may also be found for the QIH
actors. For example new doctorate graduates is a strength for the Indus-
try and the Civil Society, while employment in fast-growing firms of in-
novative sectors is considered as a strength from the Government and
the Civil Society. On the other hand, SMEs innovating in-house is a crit-
ical indicator for the Academia and the Industry. These results may
adopt or orient potential innovation policies to specific actors, in order
to maximize the effectiveness of improvement actions or programs.
Based on the above, the QIH is characterized by actors with quite di-
verging views (see Table 2 for weight correlations among actors). This
can raise major questions on how the co-evolution of the helix can be
realized and how common goals (if they exist) can be achieved. Univer-
sities and Government officials seem to focus on indicators external to
them (Universities consider Research Systems and Intellectual Assets
of lesser importance; Governments do not consider the Finance and
Table 3
Country rankings.
Country Universities Industry Government Civil society QIH EIS (IUS)
AT 11 11 16 9 9 13
BE 4 6 8 6 6 11
BG 34 34 34 34 34 33
CH 6 1 2 1 1 1
CY 19 21 20 24 22 18
CZ 16 17 15 18 18 17
DE 5 2 1 4 3 5
DK 1 4 4 3 4 3
EE 15 13 9 12 12 15
EL 23 26 21 31 28 25
ES 29 31 32 30 32 22
FI 28 23 22 21 23 4
FR 13 12 13 10 10 12
HR 9 9 12 7 8 27
HU 14 15 11 11 13 24
IE 12 14 7 14 14 9
IS 7 7 5 8 7 10
IT 27 25 30 26 26 19
LT 31 29 29 28 29 29
LU 30 28 31 25 27 7
LV 21 24 27 23 24 30
MK 26 30 23 32 30 32
MT 25 27 24 27 25 21
NL 22 22 19 22 21 6
NO 17 18 26 16 17 16
PL 8 8 14 13 11 28
PT 32 32 28 29 31 20
RO 18 20 25 19 20 34
RS 20 16 10 17 16 23
SE 3 5 6 5 5 2
SI 33 33 33 33 33 14
SK 10 10 17 15 15 26
TR 24 19 18 20 19 31
UK 2 3 3 2 2 8
11E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
Support or theHuman Capital as important factors). Similarly, but not to
the same extent, Civil society gives less importance to Collaborations. Fi-
nally, one should note the very low importance Industry gives to Intel-
lectual Assets; this could be indicating major deficiencies in
knowledge transfer mechanisms.
Applying the TOPSIS method, we obtained the country rankings as
reported in Table 3. Average weights for four actor types have been
used in calculations. Individual actor ranking are also provided. Table
3 reveals differences compared to EIS rankings with a moderate
Spearman's correlation coefficient between our ranking and EIS
(Spearman's rho= 0.523). Although Switzerland gets the top positions
in the both rankings and there is a small difference in rankings of
Table 4
Spearman's correlation co-efficient (national level).
Universities Govern
Universi-ties Correlation coefficient 1.000 0.899a
Sig. (2-tailed) 0.000
N 34 34
Govern-ment Correlation coefficient 0.899a 1.000
Sig. (2-tailed) 0.000
N 34 34
Industry Correlation coefficient 0.971a 0.924a
Sig. (2-tailed) 0.000 0.000
N 34 34
Civil Society Correlation coefficient 0.941a 0.899a
Sig. (2-tailed) 0.000 0.000
N 34 34
QIH Correlation coefficient 0.959a 0.928a
Sig. (2-tailed) 0.000 0.000
N 34 34
EIS Correlation coefficient 0.441a 0.515a
Sig. (2-tailed) 0.009 0.002
N 34 34
a Correlation is significant at the 0.01 level (2-tailed).
countries such as Bulgaria, Latvia, Czech Republic, Norway, FYROM
and France, there is a significant deviation in countries such as Croatia,
Poland, Netherlands, Finland, Luxembourg and Slovenia. We can also
note some patterns. Denmark and UK get top positions according the
Universities' ranking. Both, Industry and Government provide similar
top rankings to Switzerland and Denmark while according to the Civil
Society Switzerland and UK get the top positions. Finland is also an in-
teresting case, which although in the EIS ranking gets a Leader position
(fourth) in our case gets an inferiorposition (number 23); this being at-
tributed to the different weighting system. Finland's strongest relative
strengths are in International scientific co-publications, License and pat-
ent revenues from abroad, PCT patent applications, and Public-private
co-publications. However, in our approach considering actual prefer-
ence systems of the innovation actors, as already noted Research Sys-
tems and the Intellectual Assets have a very low importance to all
actors. This example can reveal the shortcomings of the equalweighting
approach followed by EIS and the merits of our MCDA based approach.
In the following table (Table 4), the Spearman's correlation coeffi-
cient among the different rankings is calculated. Despite the significant
variations in the preference systemof each actor (i.e., weights) rankings
are quite similar.
In a similar way regional weights are presented in Table A2 and the
respective regional rankings in Table A3 at the Appendix A. As shown in
Table 5 the correlations among individual regional rankings are high.
Direct comparisonwith RIS is not feasible as RIS does not provide in-
novation scores, rather it follows a classification schema to Innovation
Leaders, Followers, Moderate and Modest Innovators. Following the
same classification schema, it is clear that our approach provides quite
different results (Table 6). The same applies to the national approach
as well. This is due not only to the different weighting system our ap-
proach but also to the relative comparison approach (to the ideal) of
the TOPSIS method. It could also become an indication for the re-exam-
ination of the classification thresholds used. (See Table 6.)
In 79 cases (i.e., regions) there is a coincidence between the national
and the regional classification, while in the rest 111 case there are vari-
ations, some of them being quite major. As shown in Fig. 6, there are in-
novative regions significantly overperforming national average (see e.g.
in Czech Republic, Spain, the Netherlands, Portugal, Slovenia, Finland
and Norway). Moreover, and this should be especially considered by
the policy makers, there are significant variations in innovation within
the majority of countries; these variations being more prominent in
Spain, Poland and Croatia. Although each region has its own specificities
and smart specializations it is this analysis can provide major support
for benchmarking, decisionmaking and policy formulation on the direc-
tion of reduction of inequalities among European nations and regions.
ment Industry Civil Society QIH EIS
0.971a 0.941a 0.959a 0.441a
0.000 0.000 0.000 0.009
34 34 34 34
0.924a 0.899a 0.928a 0.515a
0.000 0.000 0.000 0.002
34 34 34 34
1.000 0.976a 0.990a 0.491a
0.000 0.000 0.003
34 34 34 34
0.976a 1.000 0.991a 0.540a
0.000 0.000 0.001
34 34 34 34
0.990a 0.991a 1.000 0.523a
0.000 0.000 0.001
34 34 34 34
0.491a 0.540a 0.523a 1.000
0.003 0.001 0.001
34 34 34 34
Table 5
Spearman's correlation co-efficient (regional level).
Universities Government Industry Civil Society QIH
Universities Correlation coefficient 1.000 0.978a 0.945a 0.938a 0.985a
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 190 190 190 190 190
Govern-ment Correlation coefficient 0.978a 1.000 0.932a 0.979a 0.993a
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 190 190 190 190 190
Industry Correlation coefficient 0.945a 0.932a 1.000 0.901a 0.961a
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 190 190 190 190 190
Civil Society Correlation coefficient 0.938a 0.979a 0.901a 1.000 0.975a
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 190 190 190 190 190
QIH Correlation coefficient 0.985a 0.993a 0.961a 0.975a 1.000
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 190 190 190 190 190
a Correlation is significant at the 0.01 level (2-tailed).
Table 6
Innovation performance classifications.
Classification
National level Regional level
Our approach NIS Our approach RIS
Innovation leaders 7 4 57 34
Innovation followers 9 7 56 57
Moderate innovators 10 13 59 68
Modest innovators 2 3 18 31
12 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
Caseswhere regional performance significantly outweighs national per-
formance (see e.g. Bulgaria, Slovenia, Finland) should be mainly attrib-
uted to the additional indicators included to EIS compared to RIS.
5. Concluding remarks and future research
We have proposed a more holistic but also targeted structured ap-
proachwhere we use AHP and TOPSIS to check and balance acrossmul-
tiple and varying factors related to capturing and quantifying the
presence and effects of regional and sectoral innovation dimensions.
We have adopted the Quadruple Innovation Helix (QIH) framework
developed by one of the authors of this paper (Carayannis and
Campbell, 2009) and which has become an increasingly popular policy
making instrument among policy makers with the European Commis-
sion and other such organizations at the national and supra-national
levels to serve as an integrative device and enabler of an AHP-based bal-
anced scorecard approach to help identify and formulate insights for
Fig. 6. Regional innovation perform
theory, policy and practice concerning the design, deployment and de-
velopment of innovation and entrepreneurship ecosystems.
Some key findings and insights from our empirical research for this
paper are:
• Observed variation in indicator importance (i.e., QIH actor weights)
may suggest the need for different policies and practices tailored to
the preference system and contingencies to QIH actors.
• Our methodology and empirical findings seem to overcome earlier
criticismof the balanced scoreboard approach concerning aggregation
and equal weighting of indicators.
• Based on our methodology and empirical findings the power of the
Regional Innovation System concept is demonstrated, however the
limitations of the RIS approach due to limited data availability is also
highlighted.
• Based on our methodology and empirical findings we document sig-
nificant variation in innovation performance within countries which
constitute a strong indicator of regional level inequalities and varia-
tion in innovation performance.
• As a result, we empirically confirm the existence of regional innova-
tion hubs (pockets of excellence).
The authors have past and ongoing related research on the model-
ling and analytics of innovation and entrepreneurship regional and sec-
toral ecosystems (Carayannis et al., 2015, 2016b,d; Carayannis and
Grigoroudis, 2016; Carayannis and Rakhmatullin, 2014) and this work
will help advance the state of the art regarding this stream of research
ance range across countries.
13E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
that is increasingly becoming amajor research thrust among academics
and policy makers.
Future research will thus aim to collect more breadth and depth of
data to help validate our working hypotheses and thus help advance
our research agenda dealing with remaining challenges of validity, reli-
ability and generalizability as well as policy relevance and impact and
advance the envelop of effective efficiency using the QIH approach.
On a meta-level, this research has the potential of making a major
contribution to research protocol design, in the form of a qualitatively
transformational methodological innovation and contribution using
multi-level approaches and a synthetic, holistic multi-method approach
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
P
R
R
N
SM
In
E
P
M
E
B
B
B
B
B
C
C
with MCDA to enable more effectively efficient tools for ecosystem de-
sign, as well as the creation and support of more effectively efficient in-
novation and entrepreneurship ecosystems.
Acknowledgement
The article was prepared partly within the framework of the Basic
Research Program at the National Research University Higher School of
Economics (HSE) and partially supported within the framework of the
subsidy granted to the HSE by the Government of the Russian Federation
for the implementationof the Global Competitiveness Program.
Appendix A
Table A1
Indicator weights per actor type_NIS
Universi-ties
 Industry
 Govern ment
 Civil Society
 QIH
.1.1 New doctorate graduates
 0.033
 0.066
 0.029
 0.059
 0.055
.1.2 Population completed tertiary education
 0.031
 0.033
 0.017
 0.042
 0.037
.1.3 Youth with upper secondary level education
 0.005
 0.008
 0.037
 0.038
 0.030
.2.1 International scientific co-publications
 0.014
 0.025
 0.015
 0.038
 0.032
.2.2 Scientific publications among top 10% most cited
 0.021
 0.038
 0.038
 0.059
 0.050
.2.3 Non-EU doctorate students
 0.006
 0.018
 0.004
 0.024
 0.020
.3.1 Public R&D expenditure
 0.049
 0.071
 0.066
 0.083
 0.076
.3.2 Venture capital
 0.075
 0.075
 0.048
 0.087
 0.080
.1.1 Business R&D expenditure
 0.122
 0.103
 0.062
 0.059
 0.072
.1.2 Non-R&D innovation expenditure
 0.050
 0.083
 0.099
 0.069
 0.072
.2.1 SMEs innovating in-house
 0.058
 0.064
 0.012
 0.023
 0.032
.2.2 Innovative SMEs collaborating with others
 0.099
 0.064
 0.040
 0.052
 0.057
.2.3 Public-private co-publications
 0.050
 0.031
 0.014
 0.019
 0.023
.3.1 PCT patent applications
 0.017
 0.009
 0.035
 0.017
 0.017
.3.2 PCT patent applications in societal challenges
 0.013
 0.006
 0.022
 0.034
 0.027
.3.3 Community trademarks
 0.022
 0.006
 0.016
 0.014
 0.014
.3.4 Community designs
 0.010
 0.013
 0.013
 0.033
 0.026
.1.1 SMEs introducing product or process innovations
 0.055
 0.054
 0.090
 0.045
 0.051
.1.2 SMEs introducing marketing/organisational innovations
 0.092
 0.037
 0.028
 0.031
 0.038
.1.3 Employment fast-growing firms of innovative sectors
 0.021
 0.032
 0.066
 0.041
 0.040
.2.1 Employment in knowledge-intensive activities
 0.008
 0.030
 0.012
 0.020
 0.020
.2.2 Contribution of MHT product exports to trade balance
 0.032
 0.035
 0.075
 0.031
 0.036
.2.3 Knowledge-intensive services exports
 0.032
 0.034
 0.089
 0.035
 0.040
.2.4 Sales of new to market and new to firm innovations
 0.027
 0.042
 0.034
 0.025
 0.029
.2.5 Licence and patent revenues from abroad
 0.057
 0.024
 0.039
 0.021
 0.026
3
Table A2
Indicator weights per actor type-RIS
Universi-ties
 Industry
 Govern ment
 Civil Society
 QIH
opulation having completed tertiary education
 0.072
 0.117
 0.088
 0.158
 0.136
&D expenditure in the public sector
 0.129
 0.159
 0.121
 0.193
 0.174
&D expenditure in the business sector
 0.105
 0.130
 0.066
 0.071
 0.082
on-R&D innovation expenditure
 0.075
 0.073
 0.104
 0.075
 0.079
Es innovating in-house
 0.091
 0.077
 0.017
 0.040
 0.047
novative SMEs collaborating with others
 0.125
 0.096
 0.053
 0.068
 0.078
PO patents
 0.065
 0.036
 0.090
 0.112
 0.093
roduct or process innovators
 0.060
 0.074
 0.150
 0.077
 0.081
arketing or organisational innovators
 0.117
 0.060
 0.046
 0.056
 0.063
mployment in medium-high/high-tech manufacturing and knowledge-intensive services
 0.039
 0.060
 0.070
 0.066
 0.066
les of new-to-market and new-to-firm innovations
 0.124
 0.120
 0.195
 0.085
 0.102
Sa
Table A3
Regional rankings per actor type.
Region (NUTSII)
 University
 Government
 Industry
 Civil Society
 All 4 helices
E1
 55
 35
 45
 60
 47
E2
 41
 44
 41
 49
 44
E3
 54
 55
 56
 69
 59
G3
 186
 177
 184
 188
 186
G4
 167
 174
 163
 163
 165
Z01
 75
 87
 49
 39
 62
Z02
 87
 102
 83
 122
 99
(continued on next page)
T
14 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
able A3 (continued)
Region (NUTSII)
C
C
C
C
C
C
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
IE
IE
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
FR
FR
FR
FR
FR
FR
FR
FR
FR
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
IT
University
 Government
 Industry
 Civil Society
 All 4 helices
Z03
 138
 144
 134
 143
 139
Z04
 116
 99
 135
 158
 129
Z05
 90
 63
 99
 124
 95
Z06
 93
 100
 90
 96
 92
Z07
 104
 76
 112
 130
 108
Z08
 128
 142
 139
 160
 143
K01
 9
 20
 8
 12
 14
K02
 23
 49
 26
 44
 34
K03
 72
 69
 76
 58
 66
K04
 61
 54
 69
 41
 55
K05
 35
 53
 29
 43
 42
E1
 1
 8
 5
 9
 3
E2
 8
 10
 17
 25
 17
E3
 3
 9
 2
 2
 1
E4
 42
 32
 60
 46
 45
E5
 29
 25
 23
 15
 21
E6
 31
 22
 38
 32
 31
E7
 15
 15
 34
 45
 27
E8
 45
 40
 53
 51
 46
E9
 17
 16
 35
 34
 25
EA
 30
 18
 43
 47
 37
EB
 32
 19
 46
 54
 41
EC
 43
 30
 65
 65
 48
ED
 7
 11
 10
 10
 7
EE
 53
 48
 70
 74
 61
EF
 49
 37
 72
 67
 56
EG
 13
 12
 28
 26
 20
01
 64
 52
 64
 73
 63
02
 50
 47
 50
 63
 49
L1
 68
 39
 79
 91
 72
L2
 77
 93
 97
 104
 89
L3
 82
 50
 85
 83
 77
L4
 70
 56
 86
 87
 75
S11
 135
 133
 125
 118
 127
S12
 119
 79
 100
 89
 100
S13
 126
 114
 104
 86
 106
S21
 92
 73
 67
 71
 76
S22
 84
 57
 63
 70
 69
S23
 120
 97
 109
 115
 110
S24
 106
 72
 96
 94
 93
S3
 108
 113
 78
 68
 88
S41
 123
 85
 106
 110
 109
S42
 147
 128
 142
 150
 144
S43
 153
 152
 140
 128
 150
S51
 111
 108
 95
 82
 101
S52
 132
 130
 123
 114
 124
S53
 187
 187
 182
 173
 184
S61
 136
 126
 120
 116
 123
S62
 150
 141
 143
 142
 147
S63
 181
 180
 190
 169
 183
S64
 180
 179
 188
 168
 182
S7
 160
 156
 160
 152
 159
1
 40
 58
 31
 23
 38
2
 121
 138
 126
 127
 128
3
 107
 116
 114
 112
 115
4
 88
 84
 87
 75
 85
5
 100
 112
 102
 95
 103
6
 67
 74
 48
 57
 60
7
 63
 70
 52
 37
 53
8
 80
 91
 62
 48
 68
9
 130
 146
 127
 101
 130
C1
 83
 61
 84
 99
 82
C2
 131
 136
 150
 159
 146
C3
 118
 109
 111
 108
 112
C4
 96
 75
 103
 109
 97
H1
 114
 106
 137
 138
 126
H2
 105
 101
 105
 90
 102
H3
 109
 80
 124
 120
 111
H4
 69
 51
 80
 78
 71
H5
 94
 65
 98
 92
 86
I1
 110
 90
 108
 98
 105
I2
 127
 118
 128
 117
 122
I3
 133
 125
 146
 146
 136
I4
 97
 86
 77
 64
 81
F1
 115
 92
 119
 121
 118
F2
 152
 143
 155
 157
 153
T
15E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
able A3 (continued)
Region (NUTSII)
IT
IT
IT
IT
IT
IT
H
H
H
H
H
H
H
N
N
N
N
N
N
N
N
N
N
N
N
A
A
A
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
R
R
R
R
R
R
R
R
SI
SI
SK
SK
SK
SK
FI
FI
FI
FI
FI
SE
SE
SE
SE
SE
University
 Government
 Industry
 Civil Society
 All 4 helices
F3
 113
 105
 113
 113
 114
F4
 145
 137
 145
 140
 141
F5
 144
 139
 144
 144
 142
F6
 137
 129
 147
 153
 140
G1
 129
 131
 136
 137
 133
G2
 139
 134
 141
 135
 134
U1
 140
 149
 118
 107
 131
U21
 168
 169
 167
 176
 171
U22
 164
 163
 165
 171
 167
U23
 178
 183
 177
 178
 179
U31
 179
 184
 176
 179
 181
U32
 162
 178
 161
 166
 166
U33
 155
 167
 154
 141
 156
L11
 37
 23
 32
 14
 24
L12
 99
 94
 115
 134
 116
L13
 95
 78
 117
 131
 107
L21
 58
 41
 58
 62
 52
L22
 36
 26
 37
 24
 30
L23
 62
 43
 59
 61
 54
L31
 34
 13
 24
 7
 19
L32
 48
 33
 40
 29
 36
L33
 44
 31
 39
 28
 33
L34
 98
 95
 121
 133
 117
L41
 39
 21
 44
 38
 39
L42
 57
 42
 57
 59
 50
T1
 10
 29
 20
 31
 22
T2
 28
 46
 30
 42
 35
T3
 46
 60
 66
 76
 64
L11
 165
 168
 158
 139
 160
L12
 141
 145
 110
 79
 121
L21
 151
 150
 131
 103
 135
L22
 161
 158
 159
 147
 158
L31
 163
 173
 157
 132
 157
L32
 154
 157
 151
 145
 154
L33
 174
 170
 166
 161
 168
L34
 188
 188
 174
 165
 180
L41
 166
 166
 162
 149
 161
L42
 172
 165
 170
 164
 169
L43
 189
 182
 183
 180
 185
L51
 156
 153
 156
 151
 155
L52
 176
 161
 175
 177
 174
L61
 184
 181
 179
 174
 178
L62
 173
 171
 171
 167
 170
L63
 171
 176
 164
 155
 163
T11
 101
 82
 93
 93
 94
T15
 142
 140
 153
 156
 151
T16
 78
 71
 82
 88
 79
T17
 33
 38
 36
 53
 40
T18
 122
 127
 138
 154
 132
T2
 124
 147
 149
 148
 145
T3
 157
 160
 169
 172
 162
O11
 177
 164
 181
 182
 177
O12
 175
 155
 178
 187
 176
O21
 158
 162
 173
 181
 173
O22
 169
 159
 180
 186
 175
O31
 159
 151
 168
 183
 164
O32
 149
 154
 130
 111
 138
O41
 185
 185
 187
 190
 188
O42
 190
 186
 186
 184
 187
01
 102
 111
 107
 126
 113
02
 65
 81
 42
 40
 57
01
 91
 59
 74
 72
 74
02
 148
 135
 152
 170
 152
03
 143
 132
 148
 162
 148
04
 170
 172
 172
 175
 172
13
 51
 66
 47
 52
 51
18
 11
 27
 9
 8
 15
19
 27
 28
 13
 27
 23
1A
 6
 34
 4
 16
 16
2
 112
 121
 129
 136
 125
11
 5
 36
 7
 5
 9
12
 2
 14
 1
 1
 2
21
 79
 110
 92100
 90
22
 4
 24
 6
 4
 5
23
 24
 62
 15
 20
 29
(continued on next page)
T
16 E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
able A3 (continued)
Region (NUTSII)
S
S
S
U
U
U
U
U
U
U
U
U
U
U
U
C
C
C
C
C
C
C
N
N
N
N
N
N
N
H
H
University
 Government
 Industry
 Civil Society
 All 4 helices
E31
 89
 122
 101
 102
 104
E32
 76
 123
 94
 105
 98
E33
 38
 64
 25
 6
 26
KC
 66
 104
 73
 84
 80
KD
 52
 103
 55
 77
 73
KE
 81
 119
 88
 85
 87
KF
 60
 107
 71
 80
 78
KG
 74
 117
 89
 106
 91
KH
 12
 67
 12
 33
 32
KI
 73
 83
 61
 56
 67
KJ
 26
 68
 33
 35
 43
KK
 47
 89
 54
 66
 65
KL
 71
 115
 75
 81
 84
KM
 56
 88
 51
 36
 58
KN
 85
 124
 91
 97
 96
H01
 18
 3
 14
 17
 8
H02
 21
 5
 19
 21
 12
H03
 19
 2
 16
 18
 10
H04
 16
 1
 11
 13
 6
H05
 25
 7
 27
 30
 18
H06
 22
 4
 21
 22
 13
H07
 20
 6
 18
 19
 11
OO1
 59
 45
 22
 11
 28
OO2
 117
 98
 122
 125
 119
OO3
 125
 120
 116
 119
 120
OO4
 146
 148
 133
 123
 137
OO5
 86
 77
 68
 50
 70
OO6
 14
 17
 3
 3
 4
OO7
 103
 96
 81
 55
 83
RO1
 134
 175
 132
 129
 149
RO2
 182
 190
 189
 189
 190
RO3
 183
 189
 185
 185
 189
H
References
Adam, F., 2014. Measuring National Innovation Performance: The Innovation Union
Scoreboard Revisited. Measuring National Innovation Performance:pp. 5–8 http://
dx.doi.org/10.1007/978-3-642-39464-5.
Ahn, B.S., Choi, S.H., 2012. Aggregation of ordinal data using ordered weighted averaging
operator weights. Ann. Oper. Res. 201 (1), 1–16.
Archibugi, D., Denni, M., Filippetti, A., 2009. The technological capabilities of nations: the
state of the art of synthetic indicators. Technol. Forecast. Soc. Chang. 76 (7):917–931.
http://dx.doi.org/10.1016/j.techfore.2009.01.002.
Borcherding, K., Eppel, T., Von Winterfeldt, D., 1991. Comparison of weighting judgments
in multiattribute utility measurement. Manag. Sci. 37 (12), 1603–1619.
Carayannis, E.G., Campbell, D.F.J., 2006. Mode 3: meaning and implications from a
knowledge systems perspective. In: Carayannis, E., Campbell, D. (Eds.), Knowledge
Creation, Diffusion, and Use in Innovation Networks and Knowledge Clusters,
pp. 1–25.
Carayannis, E.G., Campbell, D.F.J., 2009. “Mode 3”and ‘Quadruple Helix’: toward a
21st century fractal innovation ecosystem. Int. J. Technol. Manag. 46 (3–4),
201–234.
Carayannis, E.G., Campbell, D.F.J., 2012. Mode 3 knowledge production in quadruple helix
innovation systems. Mode 3 Knowledge Production in Quadruple Helix Innovation
Systems. Springer, pp. 1–63.
Carayannis, E.G., Grigoroudis, E., 2016. Using multiobjective mathematical programming
to link national competitiveness, productivity, and innovation. Ann. Oper. Res. 246:
635–655. http://dx.doi.org/10.1007/s10479-015-1873-x.
Carayannis, E.G., Meissner, D., 2016. Glocal targeted open innovation: challenges,
opportunities and implications for theory, policy and practice. J. Technol. Transf.
1–17.
Carayannis, E.G., Provance, M., 2008. Measuring firm innovativeness: towards a compos-
ite innovation index built on firm innovative posture, propensity and performance at-
tributes. Int. J. Innov. Reg. Dev. 1 (1), 90–107.
Carayannis, E.G., Rakhmatullin, R., 2014. The quadruple/quintuple innovation helixes and
smart specialisation strategies for sustainable and inclusive growth in Europe and be-
yond. J. Knowl. Econ. 5 (2), 212–239.
Carayannis, E.G., Barth, T.D., Campbell, D.F.J., 2012. The Quintuple Helix innovationmodel:
global warming as a challenge and driver for innovation. J. Innov. Entrep. 1 (1), 2.
Carayannis, E.G., Goletsis, Y., Grigoroudis, E., 2015. Multi-level multi-stage efficiencymea-
surement: the case of innovation systems. Oper. Res.:253–274 http://dx.doi.org/10.
1007/s12351-015-0176-y.
Carayannis, E.G., Campbell, D.F.J., Rehman, S.S., 2016a. Mode 3 knowledge production:
systems and systems theory, clusters and networks. J. Innov. Entrep. 5 (1):17.
http://dx.doi.org/10.1186/s13731-016-0045-9.
Carayannis, E.G., Ferreira, J.J., Ferreira, F.A.F., Peris-Ortiz, M., 2016b. Location and innova-
tion capacity in multilevel approaches: editorial note. J. Knowl. Econ. 7 (4), 837–841.
Carayannis, E., Grebeniuk, A., Meissner, D., 2016c. Smart roadmapping for STI policy.
Technol. Forecast. Soc. Chang. 110, 109–116.
Carayannis, E.G., Grigoroudis, E., Goletsis, Y., 2016d. Amultilevel andmultistage efficiency
evaluation of innovation systems: a multiobjective DEA approach. Expert Syst. Appl.
62:63–80. http://dx.doi.org/10.1016/j.eswa.2016.06.017.
Cherchye, L., Moesen,W., Van Puyenbroeck, T., 2004. Social Inclusion in the EU: Towards a
Synthetic Indicator With Endogenous Weights. The Open Method of Coordination
and Minimum Income Protection in Europe: Liber Memorialis Herman Deleeck.
Acco, Leuven [u.a.].
Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T., Saisana, M., Saltelli, A., ...
Tarantola, S., 2008. Creating composite indicators with DEA and robustness analysis:
the case of the Technology Achievement Index. J. Oper. Res. Soc. 59 (2):239–251.
http://dx.doi.org/10.1057/palgrave.jors.2602445.
Chesbrough, H.W., 2006. Open Innovation: The New Imperative for Creating and Profiting
From Technology. Harvard Business Press.
Cho, J., Lee, J., 2013. Development of a new technology product evaluation model for
assessing commercialization opportunities using Delphi method and fuzzy AHP ap-
proach. Expert Syst. Appl. 40 (13):5314–5330. http://dx.doi.org/10.1016/j.eswa.
2013.03.038.
Chung, S., 2002. Building a national innovation system through regional innovation sys-
tems. Technovation 22 (8):485–491. http://dx.doi.org/10.1016/S0166-
4972(01)00035-9.
Clinton, B.D., Webber, S.A., Hassell, J.M., 2002. Implementing the balanced scorecard using
the analytic hierarchy process. Manag. Account. Q. 3 (3), 1–11.
Cooke, P., 2001. Regional innovation systems, clusters, and the knowledge economy. Ind.
Corp. Chang. 10 (4):945–974. http://dx.doi.org/10.1093/icc/10.4.945.
Cooke, P., Gomez Uranga, M., Etxebarria, G., 1997. Regional innovation systems: institu-
tional and organisational dimensions. Res. Policy 26 (4–5):475–491. http://dx.doi.
org/10.1016/S0048-7333(97)00025-5.
Eisenführ, F., Langer, T., Weber, M., Langer, T., Weber, M., 2010. Rational decision making.
Springer.
Etzkowitz, H., Leydesdorff, L., 2000. The dynamics of innovation: from National Systems
and “Mode 2” to a Triple Helix of university–industry–government relations. Res. Pol-
icy 29 (2), 109–123.
Freeman, C., 1987. Technology and Economic Performance: Lessons from Japan. Pinter
Publishers, London.
Freudenberg, M., 2003. Composite Indicators of Country Performance: A Critical Assess-
ment. OECD Science, Technology and Industry Working Papers 16:p. 35. http://dx.
doi.org/10.1787/405566708255.
Gibbons, M., Limoges, C., Nowotny, H., Schawartzman, S., Scott, P., Trow, M., ... Trow, M.,
1994. The new production of knowledge: the dynamics of science and research in
contemporary societies. Contemp. Sociol. 24. http://dx.doi.org/10.2307/2076669.
Goletsis, Y., Chletsos, M., 2011. Measurement of development and regional disparities in
Greek periphery: a multivariate approach. Socio Econ. Plan. Sci. 45 (4):174–183.
http://dx.doi.org/10.1016/j.seps.2011.06.002.
http://dx.doi.org/10.1007/978-3-642-39464-5
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0010
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0010
http://dx.doi.org/10.1016/j.techfore.2009.01.002
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0020
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0020
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0025
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0025
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0025
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0025
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0030
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0030
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0030
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0035
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0035
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0035http://dx.doi.org/10.1007/s10479-015-1873-x
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0045
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0045
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0045
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0050
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0050
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0050
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0055
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0055
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0055
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0060
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0060
http://dx.doi.org/10.1007/s12351-015-0176-y
http://dx.doi.org/10.1007/s12351-015-0176-y
http://dx.doi.org/10.1186/s13731-016-0045-9
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0075
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0075
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0080
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0080
http://dx.doi.org/10.1016/j.eswa.2016.06.017
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0090
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0090
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0090
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0090
http://dx.doi.org/10.1057/palgrave.jors.2602445
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0100
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0100
http://dx.doi.org/10.1016/j.eswa.2013.03.038
http://dx.doi.org/10.1016/j.eswa.2013.03.038
http://dx.doi.org/10.1016/S0166-4972(01)00035-9
http://dx.doi.org/10.1016/S0166-4972(01)00035-9
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0115
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0115
http://dx.doi.org/10.1093/icc/10.4.945
http://dx.doi.org/10.1016/S0048-7333(97)00025-5
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0130
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0130
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0135
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0135
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0135
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0140
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0140
http://dx.doi.org/10.1787/405566708255
http://dx.doi.org/10.2307/2076669
http://dx.doi.org/10.1016/j.seps.2011.06.002
17E.G. Carayannis et al. / Technological Forecasting & Social Change 131 (2018) 4–17
Grigoroudis, E., Siskos, Y., 2010. Customer satisfaction evaluation: methods for measuring
and implementing service quality. Int. Ser. Oper. Res. Manag. Sci. 139:1–20. http://dx.
doi.org/10.1007/978-1-4419-1640-2.
Grupp, H., Mogee, M.E., 2004. Indicators for national science and technology policy: how
robust are composite indicators? Res. Policy 33 (9):1373–1384. http://dx.doi.org/10.
1016/j.respol.2004.09.007.
Grupp, H., Schubert, T., 2010. Review and new evidence on composite innovation indica-
tors for evaluating national performance. Res. Policy 39 (1):67–78. http://dx.doi.org/
10.1016/j.respol.2009.10.002.
Hajek, P., Henriques, R., Hajkova, V., 2014. Visualising components of regional innovation
systems using self-organizing maps—evidence from European regions. Technol. Fore-
cast. Soc. Chang. 84:197–214. http://dx.doi.org/10.1016/j.techfore.2013.07.013.
Hollanders, H., Es-Sadki, N., Kanerva, M., 2016a. European Innovation Scoreboard 2016
Methodology report. European Commission (Retrieved from http://ec.europa.eu/
DocsRoom/documents/17821).
Hollanders, H., Es-Sadki, N., Kanerva, M., 2016b. Regional Innovation Scoreboard 2016
Methodology report. European Commission http://dx.doi.org/10.2769/35418.
Hsieh, L., Chin, J., Wu, M., 2006. Performance evaluation for university electronic libraries
in Taiwan. Electron. Libr. 24 (2):212–224. http://dx.doi.org/10.1108/
02640470610660387.
Hwang, C.L., Yoon, K., 1981. Multiple Criteria Decision Making. Lecture Notes in Econom-
ics and Mathematical Systems 186.
Jacobs, R., Goddard, M., Smith, P., 2004. Measuring Performance: An Examination of Com-
posite Performance Indicators. A Report for the Department of Health (Vol.
Technical).
Joshi, R., Banwet, D.K., Shankar, R., 2011. A Delphi-AHP-TOPSIS based benchmarking
framework for performance improvement of a cold chain. Expert Syst. Appl. 38 (8):
10170–10182. http://dx.doi.org/10.1016/j.eswa.2011.02.072.
Keeney, Raiffa, H., 1976. Decisions with Multiple Objectives. John Wiley, New York.
Kline, S.J., Rosenberg, N., 1986. An Overview of Innovation. Eur. J. Innov. Manag. 38:
275–305. http://dx.doi.org/10.1108/14601069810368485.
Lundvall, B.-Å., 1992. National Systems of Innovation: Towards a Theory of Innovation
and Interactive Learning. National Systems Of Innovation Towards a Theory Of Inno-
vation and Interactive Learning. Pinter, London.
Meesapawong, P., Rezgui, Y., Li, H., 2014. Planning innovation orientation in public re-
search and development organizations: using a combined Delphi and analytic hierar-
chy process approach. Technol. Forecast. Soc. Chang. 87:245–256. http://dx.doi.org/
10.1016/j.techfore.2013.12.023.
Meissner, D., Carayannis, E., Sokolov, A., 2016. Key features of roadmapping for company
and policy strategy making. Technol. Forecast. Soc. Chang. 110, 106–108.
Metcalfe, J.S., 1995. The Economic Foundations of Technology Policy: Equilibrium and
Evolutionary Perspectives. Handbook of the Economics of Innovation and Technolog-
ical Change pp. 409–512.
Nelson, R.R., 1993. National Innovation Systems: A Comparative Analysis. National Inno-
vation Systems:p. 541 (Retrieved from http://books.google.co.uk/books?id=
YFDGjgxc2CYC).
OECD, 1999. Managing National Innovation Systems. J. Econ. Lit. 26. http://dx.doi.org/10.
1787/9789264189416-en.
OECD, 2008. Handbook on Constructing Composite Indicators: Methodology and User
Guide. Methodology 3. http://dx.doi.org/10.1787/9789264043466-en.
Paas, T., Poltimäe, H., 2010. A Comparative Analysis of National Innovation Performance:
The Baltic States in the EU Context. Tartu. Retrieved from. http://www.mtk.ut.ee/
sites/default/files/mtk/dokumendid/febawb78.htm.
Paredes-Frigolett, H., Pyka, A., Pereira, J., Gomes, L.F.A.M., 2014. Ranking the performance
of national innovation systems in the Iberian Peninsula and Latin America from a
neo-Schumpeterian economics perspective. Garbenstr. 15. Kommunikations-, Infor-
mations- und Medienzentrum der Universität Hohenheim, Stuttgart:p. 70593 (De-
cember 1 Retrieved from http://opus.uni-hohenheim.de/volltexte/2014/1028)
Parkan, C., Wu, M.L., 1997. On the equivalence of operational performance measurement
and multiple attribute decision making. Int. J. Prod. Res. 35 (11), 2963–2988.
Perez-Moreno, S., Rodriguez, B., Luque, M., 2016. Assessing global competitiveness under
multi-criteria perspective. Econ. Model. 53:398–408. http://dx.doi.org/10.1016/j.
econmod.2015.10.030.
Prainsack, B., Carayannis, Elias G., Campbell, David F.J., 2012. Mode 3 knowledge produc-
tion in Quadruple Helix Innovation Systems: 21st-Century democracy, innovation,
and entrepreneurship for development. Minerva 50 (1):139–142. http://dx.doi.org/
10.1007/s11024-012-9194-6.
Reisinger, H., Cravens, K.S., Tell, N., 2003. Prioritizing performance measures within the
balanced scorecard framework. Manag. Int. Rev. 43 (4):429–437 (Retrieved from
http://www.jstor.org/stable/40835943).
Saaty, T.L., 1990. How to make a decision: the analytic hierarchy process. Eur. J. Oper. Res.
48 (1):9–26. http://dx.doi.org/10.1016/0377-2217(90)90057-I.
Saaty, T.L., 1994. How to make a decision: The analytic hierarchy process. Interfaces 24
(6), 19–43.
Saaty, T.L., 2005. The analytic hierarchy and analytic network processes for the measure-
ment of intangible criteria and for decision-making. In: Fiigueira, J., Greco, S., Ehrgott,
M. (Eds.), Multiple Criteria Decision Analysis: State of the art Surveys. Springer, New
York, pp. 345–407.
Saisana, M., Tarantola, S., 2002. State-of-the-art Report on Current Methodologies and
Practices for Composite Indicator Development.Joint Research Centre. Italy: Europe-
an Commission:pp. 1–72 http://dx.doi.org/10.13140/RG.2.1.1505.1762.
Schubert, T., 2006. How Robust are Rankings of Composite Indicators when Weights are
Changed, Proposing a NewMethodology. Trest Conference: Neo-Schumpeterian Eco-
nomics: An Agenda for the 21st Century, pp. 86–110.
Tavana, M., Hatami-Marbini, A., 2011. A group AHP-TOPSIS framework for human space-
flight mission planning at NASA. Expert Syst. Appl. 38 (11):13588–13603. http://dx.
doi.org/10.1016/j.eswa.2011.04.108.
Tidd, J., Bessant, J., 2013. Managing Innovation: Integrating Technological, Market and Or-
ganizational Change. Wiley.
Weber, M., Borcherding, K., 1993. Behavioral influences on weight judgments in
multiattribute decision making. Eur. J. Oper. Res. 67 (1), 1–12.
Dr. Elias G. Carayannis is Full Professor of Science, Technology, Innovation and Entrepre-
neurship, aswell as co-Founder and co-Director of the Global and Entrepreneurial Finance
Research Institute (GEFRI) and Director of Research on Science, Technology, Innovation
and Entrepreneurship, European Union Research Center, (EURC) at the School of Business
of the GeorgeWashington University inWashington, DC. Dr. Carayannis' teaching and re-
search activities focus on the areas of strategicGovernment-University-Industry R&Dpart-
nerships, technology road-mapping, technology transfer and commercialization,
international science and technology policy, technological entrepreneurship and regional
economic development. Dr. Carayannis has several publications in both academic and
practitioner journals, including IEEE Transactions in Engineering Management, Research
Policy, Journal of R&DManagement, Journal of Engineering and TechnologyManagement,
International Journal of Technology Management, Technovation, Journal of Technology
Transfer, Engineering Management Journal, Journal of Growth and Change, Review of Re-
gional Studies among others.
Yorgos Goletsis is Assistant Professor in the Department of Economics of the University of
Ioannina. He holds a Diploma Degree in electrical engineering and the Ph.D. degree in Op-
erations Research, both from the National Technical University of Athens, Athens, Greece.
His research interests include operations research, decision support systems, multicriteria
analysis, quantitative analysis, data mining, artificial intelligence, efficiency evaluation,
project evaluation. He has participated in a series of ICT related projects especially in the
healthcare domain. He is member of IEEE, EMRBI-Euromed Research Business Institute,
EUROWorking Group on Multicriteria Aid for Decisions, International Society of Multiple
Criteria Decision Making. Currently, he teaches operations research, decision making and
entrepreneurship related subjects.
Evangelos Grigoroudis is Associate Professor on management of quality processes in the
School of Production Engineering and Management of the Technical University of Crete,
Greece (2002−). He has received distinctions from the Hellenic Operational Research So-
ciety, the Academy of Business and Administrative Sciences, the World Automation Con-
gress, the Foundation of Ioannis and Vasileia Karayianni, the Technical University of
Crete, and the State Scholarships Foundation of Greece. He acts as reviewer for more than
60 scientific journals, and he is associate editor and member of the Editorial Board of the
several scientific journals. He is author/editor of 17 books on the measurement of service
quality, the business strategy and management, and the multicriteria decision aid ap-
proaches, as well as of a significant number of research reports and papers in scientific
journals and conference proceedings.
http://dx.doi.org/10.1007/978-1-4419-1640-2
http://dx.doi.org/10.1016/j.respol.2004.09.007
http://dx.doi.org/10.1016/j.respol.2004.09.007
http://dx.doi.org/10.1016/j.respol.2009.10.002
http://dx.doi.org/10.1016/j.techfore.2013.07.013
http://ec.europa.eu/DocsRoom/documents/17821
http://ec.europa.eu/DocsRoom/documents/17821
http://dx.doi.org/10.2769/35418
http://dx.doi.org/10.1108/02640470610660387
http://dx.doi.org/10.1108/02640470610660387
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0195
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0195
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0200
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0200
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0200
http://dx.doi.org/10.1016/j.eswa.2011.02.072
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0210
http://dx.doi.org/10.1108/14601069810368485
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0220
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0220
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0220
http://dx.doi.org/10.1016/j.techfore.2013.12.023
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0230
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0230
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0235
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0235
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0235
http://books.google.co.uk/books?id=YFDGjgxc2CYC
http://books.google.co.uk/books?id=YFDGjgxc2CYC
http://dx.doi.org/10.1787/9789264189416-en
http://dx.doi.org/10.1787/9789264189416-en
http://dx.doi.org/10.1787/9789264043466-en
http://www.mtk.ut.ee/sites/default/files/mtk/dokumendid/febawb78.htm
http://www.mtk.ut.ee/sites/default/files/mtk/dokumendid/febawb78.htm
http://opus.uni-hohenheim.de/volltexte/2014/1028
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0265
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0265
http://dx.doi.org/10.1016/j.econmod.2015.10.030
http://dx.doi.org/10.1016/j.econmod.2015.10.030
http://dx.doi.org/10.1007/s11024-012-9194-6
http://www.jstor.org/stable/40835943
http://dx.doi.org/10.1016/0377-2217(90)90057-I
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190204511277
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190204511277
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190206571447
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190206571447
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190206571447
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf201703190206571447
http://dx.doi.org/10.13140/RG.2.1.1505.1762
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0295
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0295
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0295
http://dx.doi.org/10.1016/j.eswa.2011.04.108
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0305
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0305
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0310
http://refhub.elsevier.com/S0040-1625(17)30323-2/rf0310
	Composite innovation metrics: MCDA and the Quadruple Innovation Helix framework
	1. Introduction
	2. Innovation Scoreboards and the Quadruple Innovation Helix Framework
	3. Methodology
	4. Application and results
	5. Concluding remarks and future research
	Acknowledgement
	Appendix A
	References