Int. Journal of Business Science and Applied Management, Volume 7, Issue 1, 2012
Developing a performance measurement system for public
research centres
Deborah Agostino
Dipartimento di Ingegneria Gestionale, Politecnico di Milano
Piazza Leonardo da Vinci, 32 - 20133 Milano, Italy
Telephone: +39 02.2399.4073
Marika Arena
Dipartimento di Ingegneria Gestionale, Politecnico di Milano
Piazza Leonardo da Vinci, 32 - 20133 Milano, Italy
Telephone: +39 02.2399.4070
Giovanni Azzone
Dipartimento di Ingegneria Gestionale, Politecnico di Milano
Piazza Leonardo da Vinci, 32 - 20133 Milano, Italy
Telephone: +39 02.2399.3981
Martina Dal Molin
Dipartimento di Ingegneria Gestionale, Politecnico di Milano
Piazza Leonardo da Vinci, 32 - 20133 Milano, Italy
Telephone: +39 02.2399.4044
Cristina Masella
Dipartimento di Ingegneria Gestionale, Politecnico di Milano
Piazza Leonardo da Vinci, 32 - 20133 Milano, Italy
Telephone: +39 02.2399.4081
This study aims at developing a performance measurement system (PMS) for research and development (R&D)
activities carried out by public research centres. Public research institutions are characterized by multiple
stakeholders with different needs, and the management of R&D activities requires balancing the multiple goals
of different stakeholders. This characteristic is a key issue in the process of construction of the PMS. Empirical
evidence is provided by an Italian public research centre, where the researchers carried out a project aimed to
develop a PMS following action research principles. This project gave the possibility to researchers to interact
with different stakeholders and integrate their different information needs in a comprehensive set of key
performance indicators (KPIs). As a result, multidimensional framework for measuring R&D performance in a
public research centre is proposed and a set of Key Performance Indicators is developed, suggesting
implications for academics and practitioners.
Keywords: performance measurement, public research centres, stakeholders, accountability, decision-making
Int. Journal of Business Science and Applied Management /
In recent years public research systems in several industrialized countries, underwent an intense
transformation process due to the increasing organizational complexity of research & development (R&D)
activities and the hit of the worldwide financial crisis. Leading research institutions claim that many complex
problems of the society demand innovative solutions which combine knowledge from different scientific areas
that can be achieved through interdisciplinary research (National Academies, 2005). To achieve this, very often,
research centres carry out interdisciplinary R&D projects, which require collaborative and sometimes informal
behaviours (Cross, Borgatti and Parker, 2002; Allen, James, & Gamlen, 2007) and new forms of organization
and management (Welter, Jooß, Richert, Jeschke, & Brecher, 2012). At the same time, the financial crisis has
reduced the overall government spending and specifically the budget for research centres (e.g. Arena
&Arnaboldi, 2013). This reduction of funding has increased the competition between research institutions, while
at the same time they are required to demonstrate their value for money. To face these challenges, research
institutions need to consciously manage their core processes, the creation and development of their knowledge
assets (Rowley, 2000), and consistently redesign their support processes and managerial instruments (e.g.
Arena, Arnaboldi, Azzone & Carlucci, 2009; Arena, Arnaboldi & Azzone, 2010a;Welter et al. 2012).
In this context, our paper focuses on one specific managerial instrument - i.e. performance measurement
system (PMS). In the last decades, measuring R&D performance has become a fundamental concern for R&D
managers and executives. Scholars and practitioners have already recognized the relevance of performance
measurement in R&D in relationship to different purposes (e.g. Chiesa, Frattini, Lazzarotti&Manzini, 2009;
Kulatunga, Amaratunga&Haigh, 2011). A PMS can be useful for motivating researchers, evaluating R&D
projects’ profitability, supporting decision-making, communicating the centre’s results to external
constituencies, and stimulating organizational learning.
However, most of the current research about PMS in R&D focuses on private sector organizations (e.g.
Bremser&Barsky, 2004; Chiesa et al., 2009) and these results are not easily applicable to Government-funded
research centres, since they overlook some key characteristics of these organizations. In industrial firms, R&D
activities are primarily financed by the company itself and they represent one of the activities in their value
chain. In this context, the company does not have the need of searching for research funding, given that it is self
financed and research outputs represent the input for further processes of the firm’s value chain. The research
results are incorporated into products and, in the end, brought to the market and sold by the company,
‘hopefully’ resulting in an increase in the revenue and profits of the firm, and contributing amortize the R&D
investment. On the contrary, in Government-funded research centres, such as public research institutions, the
research activity represents the core mission of the organization and the R&D activity needs to be financed by
external parties. This structure has two main implications. On the one hand, the role played by the research
activity is central, because the output of the research represents the end itself. Research institutes contribute to
the early stages of the innovation process of various customers within the national innovation system and serve
thus as an important research infrastructure, (Shenker, 2001; Leitner & Warden, 2004). On the other hand, the
need for searching funding is associated with the pressure of demonstrating the ability of generating research
outputs that provide value for the society. The output of public research centres is expected to have a positive
impact on the wider society; their principal aim is to disseminate the research results and to have a return in
terms of scientific-technological progress, driving the national strength (Senker, 2001; Coccia, 2004). However,
once the funding are received, mainly by the public government (Senker, 2001; Coccia, 2004), the research
centre has to be accountable on how it spend public money. In this sense, Coccia (2005) argued that research
institutions are dependent on the government for carrying out their activities. As consequence of these
specificities, public research centres are subject to different and contrasting stakeholders pressures that are wider
compared to private sector organizations (Dixit, 2002; Coccia, 2004; Arnaboldi, Azzone & Savoldelli, 2004).
Public government, the wider society, the researcher, the administrative manager but also private partners, have
different objectives and their alignment can be extremely complex, especially in the current R&D landscape
characterized by reduced finance and higher competition between institutions.
To our best knowledge, relatively a few contributions have dealt with the issue of R&D performance
measurement in the public sector (e.g. Coccia, 2001a; 2004; Leitner & Warden, 2004; Secundo, Margherita, Elia
& Passiante, 2010) and a comprehensive framework for measuring performance of public research institutes is
still missing, setting the motivation for this work. This paper has the objective to develop a performance
measurement system (PMS) for a public research centre. The centre has to deal with the informative needs of a
diversified range of stakeholders that require accountability in relationship to the centre’s activities and, in
particular, the use of funding, but also aim to encourage collaboration between different research units and
support decision making processes. The PMS has been developed through action research, whereby the
researchers interacted with different stakeholders and integrated their different information needs to a
comprehensive set of key performance indicators (KPIs). Interviews and meetings with scientific and
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
administrative directors over a time horizon of ten months were useful for designing a dashboard for public
research institutions that integrates the information needs of a plurality of stakeholders.
The paper is organized as follows. The next section outlines the state of the art on the issue of performance
measurement for R&D activities. The third section details the methodology used for data collection and
analysis, describing the phases of the research project. The fourth section presents the PMS framework and the
set of indicators that have been developed. Finally, the last section clarifies contributions and limitations of this
R&D performance measurement in research centres
Various studies on public research centres show a growing interest in R&D performance measurement
(Luwel, Noyons & Moed, 1999; Senker, 2001;Coccia, 2004; Coccia & Rolfo, 2007). Originally, particular
attention has been given to the use of bibliometric and technometric indicators to evaluate public research centre
performances (e.g. Narin & Hamilton, 1996; Luwel et al., 1999; Noyons, Moed & Luwel, 1999). Bibliometric
involves the quantitative analysis of the scientific and technological literature. It relies on the assumption that
the frequency with which a paper or a patent is cited is a measure of the impact of the paper or patent. The most
common bibliometric indicators are literature indicators i.e. indicators that measure the scientific performance
based on the number of publications and the count of citations in the scientific literature, and, obviously, the
higher the better. Similar considerations are applicable to patents too. Those patents which are most highly cited
and most science linked are also the patents that tend to be most heavily licensed, and are therefore making the
largest contribution to the economy (Narin & Hamilton, 1996).However, bibliometric indicators provide a
partial and non-systemic picture of the performance of a research centre, because they overlook some relevant
elements such as the impact of the research activity on the society and they ignore the role played by
government funding (Coccia, 2004).
Moving from these considerations, Coccia proposed a weighted index to provide a synthetic measurement
of public research’s activities - Relev Model I and Relev Model II - (Coccia2001b; 2004, 2005). The Relev
Model is based on an input-output model, where inputs consist in public funds, personnel payrolls and cost of
labour, and outputs are represented by self-financing deriving from activities of technology transfer, training
(e.g. number of degree students, PhD students), teaching (measured as the number of courses held by
researchers), international and domestic publications, international and domestic conference proceedings. Based
on a weighted input output ratio, the Relev model attempts to synthesize the performance of public research
centres in a unique score. This score has the final objective to support external accountability, providing
evidence of the value for money generated by these institutions. Indeed, these studies generally adopt one
specific perspective. They reflect the aim of governments to define metrics that could be used to assess the
research centres performances in order to facilitate the identification of the most and the least productive
laboratories, and to support policy decisions on the level and the direction of the public funding of research
(Coccia, 2004). These kinds of metrics, instead, are less useful to support internal decision making and report a
centre’s performances to a broader range of stakeholders.
To respond to this need, a few studies started to focus on the use of key performance indicators for
assessing R&D results in public sector research centres (e.g. Leitner & Warden, 2004; Chu, Lin, Hsiung & Liu,
2006; Secundo et al., 2010). This stream of research is based on Intellectual Capital reporting models (e.g.
Stewart, 1997; Edvinsson & Malone, 1997). At a general extent, the Intellectual Capital encompasses three
components: human capital, structural capital, and relational capital. The human capital refers to the
characteristics of the entire organization’s staff and management. The structural capital refers to organizational
infrastructure. The relational capital refers to the establishment, development and maintenance of relationships
with external stakeholders. The combination and integration of these forms of capital determines the entity’s
results that include both economic performances and other intangible results, such as research- and society-
oriented results (Leitner & Warden, 2004). These approaches to performance measurement provide a more
complete picture of the resources available to research centres, however they seem to overlook the performances
of the centres in the transformation process of inputs in outputs (e.g. they overlook issues such as efficiency and
Finally, further evidence in relationship to R&D performance measurement is provided by the private
sector literature, where the topic has been investigated from two different perspectives. In particular, one stream
of research focuses on the choice of the performance dimensions and the performance indicators that are best
suited to the characteristics of R&D. Some of these works consist in the application of well-known PMS
frameworks, such as Balanced scorecard, Skandia navigator, Intangible asset monitor, to R&D activities (e.g.
Kerssens-Van Drongelen & Bilderbeek, 1999; Bremser & Barsky, 2004). Other works consist in the
development of ad-hoc frameworks and set of indicators to assess the performance of R&D (e.g. Pawar &
Driva, 1999; Kim& Oh, 2002; Marr, Schiuma & Neely, 2004; Mettänen, 2005). These papers can provide a
reference point for the identification of performance indicators for R&D, though they are not tailored to the
specific characteristics of public sector organizations. The second stream of research, instead, focuses on the
Int. Journal of Business Science and Applied Management /
strategic design of the PMS in R&D firms and addresses the relationship between the characteristics of the
PMS, the objective of the PMS and contextual variables, such as type of R&D activity, industry and company’s
size (e.g. Kerssens-van Drongelen & Cook, 1997; Chiesa & Frattini, 2007; Chiesa et al. 2009). These papers
highlight how the PMS can be configured in a different way on the basis of the objectives that it is supposed to
pursue (e.g. diagnostic, motivation and coordination), hence suggesting to pose particular attention to the
different objectives of the stakeholders of a research centre.
The above review has highlighted that there has been widespread dispute among researchers and
practitioners about which measures are more suitable in assessing the performance of research organizations,
and, at present, this debate is still open (Elmquist & Le Masson, 2009; Whelan, Teigland, Donnellan & Golden,
2010; Dumay & Rooney, 2011).
Choice of the research method
This paper reports the results of a project aimed at developing a PMS for an Italian technological research
centre. To carry out this goal we adopted action research (AR) as research methodology. Action research has
been selected given its aim “to contribute both to the practical concerns of people in an immediate problematic
situation and to the goals of social science by joint collaboration within a mutually acceptable ethical
framework” (Rapoport, 1970: 499). On the one hand, this methodology gives the possibility to focus on a
problematic practical situation, the need to introduce a PMS to deal with the current complex research
environment that represents at the same time a relevant academic concern. On the other hand, the collaborative
nature of the interaction between the researchers and the employees of the research centre allowed capturing the
specific requirements from each stakeholder supporting the definition of the most appropriate set of KPIs.
Specifically, we conducted AR drawing on the seminal work of Susman & Evered (1978). According to the
authors, AR is seen as a cyclical process articulated in five phases: diagnosing, action planning, action taking,
evaluating, and specifying learning (see Figure 1). At the centre of the cycle there is a client system that is the
social system in which the members face problems to be solved by action research. Different techniques were
used for data collection and analysis, including in-depth interviewing, direct observation and documental
analysis (data were drown from the records, memos, and reports that the client system routinely produces). The
interviews were recorded and transcribed, though the empirical material was not codified, but instead analyzed
textually, with each author highlighting emergent themes. Overall 64 interviews were performed and three
official plenary meetings took place; the process was highly participative (Reason & Bradbury, 2001) and
different stakeholders involved continued to provide comments about the researchers’ proposals.
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
Figure 1: The action research cycle (Susman and Evered, 1978)
Considering alternative
courses of action for
solving a problem
Select a course of action
Studying the
consequences of action
Identifying general findings
Identifying or defining a
of a client
The client system infrastructure
The client system is a public research centre in Italy, active in different fields of technological
development. The centre currently employs more than 1,000 people, with more than 900 researchers. The centre
is governed by an executive committee that defines the centre’s long term strategies and is in charge of its
management. It is characterized by a dual structure: the scientific director is responsible for all the research
activities of the centre; the administrative director, on the other hand, is responsible for the administrative
structure. The distinctive characteristic of the centre is represented by interdisciplinary R&D activities that cover
several fields, ranging from robotics, neuroscience, energy, to smart materials and drug discovery. The
interdisciplinary nature of the research activity is then reflected in the organizational structure that consists of 12
central research units, 11 research centres located at the premises of, and in collaboration with, other research
institutions both in Italy and abroad, and 3 facilities that provide support services to the research units (e.g.
animal facility).
At the beginning of the project, a working group was constituted within the client system, including the
Head of the Management Control Office and a member of its staff, the Head of the ICT Office, and two internal
consultants. The working group interacted with the researchers in all the phases of the AR cycles. In addition,
different employees of the centre were involved as informants and prospect users of the PMS (see Table1).
Int. Journal of Business Science and Applied Management /
Table1: The client system
Participants and informants
Organizational unit
Role in the project
Scientific Director
Central administration
Initiator and user
Administrative Director
Central administration
Initiator and user
Head of the Management Control Office
Central administration
Head of the client working group
Management and Control Office Staff
Central administration
Member of the client working group
Head of ICT Office
Administrative office
Member of the client working group
Consultant 1
Member of the client working group
Consultant 2
Member of the client working group
Head of RU 1
Research unit
Informant and user
Head of RU 2
Research unit
Informant and user
Head of RU 3
Research unit
Informant and user
Head of RU 4
Research unit
Informant and user
Head of RU 5
Research unit
Informant and user
Head of RU 6
Research unit
Informant and user
Head of RU 7
Research unit
Informant and user
Head of RU 8
Research unit
Informant and user
Head of RU 9
Research unit
Informant and user
Head of RU 10
Research unit
Informant and user
Head of RC 1
Research centre
Informant and user
Head of RC 2
Research centre
Informant and user
Head of RC 3
Research centre
Informant and user
Senior Scientist 1
Research unit
Informant and user
Senior Scientist 2
Research unit
Senior Scientist 3
Research unit
Senior Scientist 4
Research facility
Technician 1
Research facility
Technician 2
Research unit
Technician 3
Research unit
Team Leader 1
Research unit
Team Leader 2
Research unit
Head of RF 1
Research facility
Informant and user
Head of RF 2
Research facility
Informant and user
Head of RF 3
Research facility
Informant and user
Head of Research Office
Support office
Informant and user
Head of Project Office
Support office
Informant and user
Head of Technology Transfer Office
Support office
Informant and user
Head of Human Resource Office
Administrative office
Head of Engineering office
Administrative office
Head of Health and Safety
Administrative office
Head of Purchasing Office
Administrative office
Head of Legal Office
Administrative office
Head of Internal Audit Office
Administrative office
Head of Accounting Department
Administrative office
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
The AR Cycles
In this section we outline the phases of the two AR cycles, posing particular attention to specifying the
sources used for data collection in each step of the research.
AR Cycle I
The diagnosing phase started with a literature review concerning performance measurement in research
organizations, a series of project meetings and a brainstorming session on the role of the performance
measurement system in the specific setting. The outcome of this phase was the identification of the process-
oriented model (Pollanen, 2005) as a starting point for the development of the PMS and the acknowledgement
of the existence of different information needs in relationship to the PMS from a plurality of stakeholders.
Action planning consisted in mapping data and information needed to support the development of the PMS. A
list of informants and relevant documentation was prepared by action researchers in collaboration with a
working group of the research centre. In addition a list of critical issues (interview protocol) to be discussed with
the informants was prepared. In the action taking phase, 38 interviews were carried out with the heads of
scientific units and researchers of the centre, and also the administrative personnel. The analysis was
complemented with public documents and confidential reports, entering in detail the scientific activity carried
out by each department. Based on these data, a prototype of the PMS model and a preliminary set of KPIs were
developed. In the evaluating phase the proposed model and the preliminary set of KPIs were presented in two
plenary meetings, one involving the scientific director, the administrative director and a selected number of
other internal officers and one involving the heads of the research units. Feedback, comments and suggestions
for improvements were collected. Finally, specifying learning consisted in summing up the learning outcomes of
the AR cycle. All the comments and feedbacks deriving from the evaluation phase were integrated. The
outcome of this phase was twofold: the validation of the overall PMS model and an ‘explosion’ in the number of
KPIs due the receipt of many proposals from the participants. Overall 42 indicators and several variations to
each of them were suggested, posing the basis for the second AR cycle, aimed to the refinement of the set of
indicators. The following table outlines the techniques used for data collection and analysis in different steps of
the AR cycle, the role of the researchers and the client working group and the output produced (Table 2).
Table 2: AR Cycle I
Role of the researchers
Role of the client
working group
Challenge the
informants to identify
emerging issues
Contribute to the
Definition of the role of
the PMS in the research
Define the agenda and
design data collection
Support the researchers
in identifying potential
informants and provide
feed-backs and
Design of the data
collection tools to
support the preparation
of the PMS prototype
Perform interview and
data analysis
Contribute to the
discussion and
participate in data
Development of the
PMS overall model and
preliminary set of
Present the PMS
Contribute to the
Presentation of the PMS
prototype in two plenary
Collect and analyse
feed-backs and
Support researchers in
interpreting feed-backs
and suggestions
Integration of feed-backs
and suggestions
Int. Journal of Business Science and Applied Management /
AR Cycle II
The diagnosing phase started with a structured analysis of the comments received concerning the set of
indicators and the acknowledgement that the proposals received needed to be organized and selected. Action
planning consisted in the definition of shared selection criteria to reduce the number of the indicators, still
ensuring that the information needs of different stakeholders were met. To this aim, the following selection
criteria were adopted: relevance, measurability, cost and timeliness (Lynch and Cross, 1991; Neely et al., 2003).
In addition, to support the selection of the KPIs, a second round of 26 interviews was planned with selected
informants. Action taking consisted in carrying out the interviews with an application of the selection criteria to
define the new set of KPIs. For each KPI, an information protocol was prepared (see Table 4). It defines in
operational terms how the indicator should be computed, the unit of measure, the level of detail, the relevance of
the indicator, the frequency of data collection and the owner of these data (see also Arena & Azzone, 2010).
Table 3: The information protocol
Number of Patents
Number of patents distinguished by category
Computation procedure
Number of patents of the year distinguished by the following categories:
First filing
Extension abroad
Data source: Patents database
Data calculated over the following time horizon: year
Unit of measure
Level of detail
Scientific structure
Research unit
Research team
Relevance of the indicator
This KPI evaluated the output of the research centre and its related research units in
terms of patents, with respect to the different phases of a licensing process (first filing,
extension abroad, abandoned and granted)
Frequency of data collection
Owner of the measure
Technology Transfer Office
In the evaluating phase the results were presented in a new plenary meeting involving the scientific and
administrative directors. Feed-backs we received mainly dealt with some specifications in the information
protocols in the indicators, leading to new revisions. Hence, after this second round of evaluation the final set of
KPIs was proposed suggesting practical directions for the research institute, but also general guidelines at the
academic level (see Table 4).
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
Table 4: AR Cycle II
Role of the researchers
Role of the client
working group
Project meetings
Identify key issues based
on the output of the first
AR cycle
Contribute to the
Identification of the key
Literature review,
Project meetings
Define the selection
criteria and design data
collection tools
Support the researchers
in identifying potential
informants and provide
feed-backs and
Design of the selection
criteria and data
collection tools to
support the PMS
Project meetings,
Perform interview and
data analysis
Participate in data
analysis and support the
researchers in the
definition of the
information protocols.
Development of the final
set of indicators based
on the selection criteria
Plenary meetings
Present the final set of
Contribute to the
Final presentation
Project meetings
Collect and analyse
feed-backs and
Practical directions for
the research institute, but
also general guidelines
at the academic level
Results are organized in three sections. First, we introduce the overall PMS model; then, we outline
different performance dimensions; finally, we present the complete list of KPIs.
The PMS model
The starting point for the development of the PMS model is the production system process of a R&D
organization. It can be represented through a process-oriented model(Brown & Svenson, 1998; Pollanen, 2005),
in which research inputs are processed and transformed into output, such as publications or patents. Accordingly
to this general model, performance measures can be defined on the basis of three interrelated elements: input,
output and outcome. Input refers to the amount of resources used in performing a certain activity; output refers
to the result of a transformation process; outcome refers to the long-term impact of the output on the external
environment(Lettieri & Masella, 2009). Based on the three above elements, three performance dimensions can
be identified, namely effectiveness, efficiency and impact (Azzone, 2008). Effectiveness refers to the output
characteristics, both quantitatively and qualitatively; efficiency refers to the ratio between output and input;
impact is a measure of the outcome and it refers to the long-term effects of the output on the external context.
Interaction with research and administrative personnel was useful in recognizing the importance of the
aforementioned dimensions. The quality of the output, the impact of the output on the society and the amount of
resources required for generating the output emerged as crucial aspects for successfully manage the research
centre. However, two further aspects were highlighted as central and they were consequently added to the
traditional model: risk and network. The risk is associated with the uncertainty that characterizes the research
activity. It is widely acknowledged that the context in which research institutes operate is characterized by
continuous changes and high variability (Leitner & Warden, 2004). When discussing with the administrative
and the research director, a question continuously emerged was the following: do we have enough resources, in
terms of quantity and quality, in order to deal with the unpredictability of the events, such as a change in
technology or a budget reduction? The importance of posing the attention to the quantity and quality of
resources to deal with uncertainty, led us to complement the traditional model with the risk dimension. The
network dimension is specifically related to the research activity. The heads of research units underlined several
times the importance of working collaboratively in order to achieve results. Publications as well as projects or
patens are rarely the result of a single researcher; rather, they derive from the joint working of more teams,
belonging to the same department, but also to different departments or different research institutions. This
reason justifies the choice of adding a further dimension to the model, the network, in order to capture the ability
of working collaboratively and output of this collaboration.
The final model we obtained (Figure 2) is characterized by five dimensions, suggesting that the holistic
management of a public research centre requires to consider the quantity and the quality of the output
(effectiveness), the impact of the output on the society (impact), the ability in transforming input into output
(efficiency), the quantity and the quality of the available resources (risk) and the ability and effects of working
collaboratively (network). Following, each dimension will be discussed separately, posing the attention on the
different relative importance given to stakeholders.
Int. Journal of Business Science and Applied Management /
Figure 2: The proposed PMS model
Patents & licences
Technology transfer
Human resources
Financial resources
The performance dimensions
The effectiveness dimension allows to evaluate the output of the R&D activities defining the level of
achievement of research objectives (Garcia-Valderrama and Mulero-Valdigorri, 2005). This aspect is related to
a key issue that is what is the output of a research centre and how to measure it (Steiner and Nixon, 1997). In the
case at hand, the output is represented by publications, funded projects, patents and technology transfer
activities. Publications and projects were the ‘immediate answer’ when the informants were asked about the
output of the research centre. This answer is in line with extant studies on research institute that recognized that
publication, books and reports represent the explicit transfer of knowledge of research bodies (e.g. Coccia,
2001b). On the other hand, patents and technology transfer activities emerged in a more ‘fragmented’ way.
Some scientific directors and researchers were fully aware of the relevance of these activities, as emerges from
the following quotation:
“Technology transfer is a key pillar in the mission of our institution, otherwise,
what are we here for?” (Head of RU 8).
Whilst in the other case, patenting was seen somehow as a secondary activity that can be explained
considering that the institute was at that time measuring outputs in terms of number of scientific publications
and funding obtained through competitive projects. The Head of the Technology Transfer Office pointed out:
“Technology Transfer activity is a green field. It needs to be organized and then
evaluated because in the early years of the research centre, the attention has been
attracted by scientific publications only. Yet we are growing and we have now a
portfolio of 68 patents and several commercial projects. They need to be monitored
and managed” (Head of Technology Transfer Office)
Based on these considerations, seven KPIs were defined to measure the research centre effectiveness: four
of them aim to evaluate the quantity of the output and the other three the quality (see Table 5 for the details of
the KPIs).
It is interesting to highlight that there was a polarization in the perceived usefulness of different aspects of
effectiveness depending on the personnel qualification. The administrative personnel and the executive
committee were mainly interested in the quantity of the output generated in terms of ‘how much money’, how
much patents’, ‘how much publications’ (several informants). They considered these KPIs particularly useful to
support external accountability; in fact they have been included in the annual report and have been published on
the website of the research institute. Instead, the scientific personnel considered these measures too ‘generic’
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
and posed particular attention to the quality of the output of research activities, highlighting the importance to
account for ‘good publications’, ‘impact on the scientific community’ and ‘participation in breakthrough
innovation projects’ (several informants). From this perspective, the Impact Factor (IF) and Citation Index (CI)
distribution were evaluated particularly useful to support motivation and decision making processes of research
units. Finally, the scientific director underlined the importance of the integration of the two types of indicators,
for a decision making purpose, to have an overall idea of the productivity of each RU. At the same time, he also
recognized the limitations of these figures: the impossibility to compare them between RUs because their
average value varies widely from one research area to another one.
The efficiency dimension evaluates the amount of resources used to generate the output, (i.e. publications,
funded projects, patents, and technology transfer activities). It represents a relevant aspect in publicly funded
services as governments are interested in receiving feedback on how public resources are used (Coccia 2001a;
Moreno & Tadepalli, 2002; Lettieri, Masella & Nocco, 2009).
The diversity of the output of research centres requires to define different measures in order to assess the
value of the resources to generate a unit of the output, either a publication, a patent or a financed project, leading
to the identification of four KPIs (see Table 2).
The relative importance of these measures was different according to the stakeholder and its organizational
position. The administrative director and the scientific director were the stakeholders most interested in
efficiency indicators. They strongly underlined the need to keep under control how many resources the centre
employs for carrying out research activities and the time spent doing this.
“Monitoring costs is a key issue for us. We have to keep out costs under control and
we need to be able to show how much efficient we are. […] From time to time,
someone stands up and claim they we spend too much. We need to be able to prove
that it is not true, we spend money, but we produce more” (Administrative director)
In order to satisfy his requirements, a synthetic efficiency measure to evaluate the overall ability of the
administrative staff in supporting research activities was included: the ratio between the external budget and the
number of FTE (Full Time Equivalent). The fundamental resource of the institute is represented by people, and
this measure gives the possibility to evaluate the cost for managing a unit of personnel. The executive
committee considered efficiency relevant too, though from a different perspective. The committee poses
particular attention on two specific performance indicators, cost of a scientific publication and incidence of the
external budget on the internal budget, both considered useful in supporting the external accountability. These
numbers are already included in the annual report of the institute and they are explicitly depicted with the
purpose to increase the awareness of external stakeholders about the research activity. The heads of the research
units and the researchers, on the other hand, downplayed the importance of efficiency measures, following the
idea that ‘research is what counts’ no matter how much it costs. Some of them even argued that measuring the
‘cost per output’ is not relevant for the management of a department, ‘what they need to know is the budget’.
The impact dimension measures the outcome, i.e. the effects of the output of the research activity on the
external environment. This aspect has been highlighted several times as crucial for the research centre. It
represents the mission of the institute:
“The Foundation has the scope to promote the technological development of the
country and the technological education, in line with the scientific and technological
national policies with the final aim to support the development of the productive
Italian system” (Foundation Statute, 2012: 3).
The importance of measuring the outcome is closely connected with the need of being externally
accountable, providing information to external stakeholders on how public money is used. The administrative
director was clear on this point since the kick-off meeting of the project:
“The philanthropic mission of our Institute is to generate know how and
employment for our country. We are receiving public money and we have, not only
the duty to account about how we are using public funding, but also we want to
demonstrate that we are using this money to support the technological development
of our country”(Administrative Director).
Int. Journal of Business Science and Applied Management /
However, the selection of the most appropriate impact indicator required to balance the trade-off between
the relevance of measuring the outcome and the cost for collecting these data. In fact, the measurement of this
aspect is far from being straightforward given the intangibility of the output and the existence of external factors
that makes it difficult to isolate the impact of research activities on the territorial environment. The KPIs finally
selected were related to cash flows from patents, licenses and technology transfer (see Table 2). They are a
proxy of the outcome because they give the possibility to account for the first order effect of a research activity,
which are leading indicators of second orders effect about impacts on the environment.
The scientific and the administrative directors of the centre were particularly keen on starting measuring
these aspects, whilst part of the administrative and research staff were somehow worried. The motivation at the
basis of this situation was that at present that the cash flows generated by these activities are still modest and
they feared that numbers could lead to wrong interpretations. Directors of research units did not perceived the
importance of measuring impact, being more attracted by KPIs about the quantity and the quality of the
“Patents, for my department, represent a cost. They do not generate income; this
information can be useful at the overall institute level. It takes some years for a
patent before gene rating money; hence, it is not relevant to manage my activity.
I’ve already seen the result of my research!” (Head of SF3)
The risk dimension is associated with the need to deal with the uncertainty and the variability of the
research context, as stated by the scientific director:
“Research trends change every year. On the basis of what happens outside, I need to
reconfigure the research activity inside here. The organizational structure should be
flexible and follow these changes accordingly. I need to be able to modify timely
the structure of scientific units; acquire new competencies, understand what
happens if I close a department and open a new one” (Scientific Director).
In particular, the emphasis given to flexibility and adaptability drove the choice of posing particular
attention to how the centre can monitor its resources in order to ensure its ability to respond to unexpected
events. In highly uncertain contexts, resources and the capability to reconfigure them are conceived as the basic
mean through which an organization can face risks (Arena, Arnaboldi & Azzone, 2010b; Arena, Arnaboldi &
Azzone, 2011; Lettieri, 2009; Lettieri, Masella & Radaelli, 2009). Hence, monitoring the ‘state of resources’ can
provide relevant information about how the organization could face a transformation. Obviously, this also poses
the problem of identifying which are the key resources, i.e. which are the resources critical for the success of
research institute. In the case at hand, four main elements were identified: human resources, financial resources,
scientific equipment and reputation; and seven indicators were defined to monitor these elements (see Table 5).
Once again, a different level of awareness and sensitivity of the stakeholders towards different issues
became visible. In particular, the scientific director was very attentive to read the implications of any variation
in different types of resources, going beyond numbers.
[…]“I want to monitor the age of scientific personnel because it provides very
useful information to me. I want a young research centre with a high turnover and
many PhD: by monitoring the age of the staff I can account for this aspect avoiding
the risk of increasing the average age” […]“The trend of PhD students for me is a
signal. If their number falls down, it is an alarm bell to me. First, they are a
resource, they are our primary row material. Second, if they stop coming here, they
going somewhere else. This means that other institutions are more attractive than
us. Why? I mean, it’s something to keep under control” (Scientific Director).
On the other hand, the administrative director and the executive committee appeared more focused on
financial resources and reputation (in particular in relationship to the public opinion) that was sees as a
considerable source of risk.
“We are on the newspaper almost every day. Public opinion poses particular
attention on the way in which we are employing public money. If we do not justify
how the public funding is used, we are immediately accused of wasting financial
resources that could be used for doing something else” (Administrative Director).
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
Finally, directors of RUs showed more interest for the scientific reputation as it gives the possibility to
understand how individual researchers, not only the centre as a whole, are known and considered in a specific
field of research.
The network dimension refers to the ability of the research personnel working together with researchers
from internal teams or from other external institutes. This aspect finds a rationale in the organizational changes
that are affecting the research activities. Collaboration is nowadays considered an essential requirement (Saez et
al., 2002) to survive in a research landscape characterized by inter-disciplinary activities (National Academies,
2005). The network dimension accounts for this collaborative behaviour considering the structure of
relationships between actors, the quantity and the quality of the joint output and the cost of collaboration. This
aspect is emerging in the public sector literature as a relevant performance dimension to evaluate collaboration
(e.g. Provan & Milward, 2001), while it has received less attention in the R&D field (e.g. Allen at al., 2007).
Three KPIs were included in the set of indicators, focusing on the quantity and the quality of joint publications
(see Table 2). Even though collaborative activities relate also to patents and research projects, no network KPIs
were associated with these typology of output. With respect to patents, the decision was to not include indicators
because of the limited amount of patents compared with publications. Considering projects, measurement
difficulties were identified, mainly related to the complexity of accounting joint internal projects. The main
concern associated with this aspect was the following:
“The majority of financed projects is carried out in a collaborative way; it is
therefore not useful to know the level of collaborations as it is by definition that we
are collaborating. This is not true for internal research projects, but they are usually
carried out by the single research team”. (Head of RC 3).
Again, the importance of evaluating the network dimension was different depending on the type of
stakeholder. Collaborative activities emerged several times during interviews and meeting with the scientific
personnel and the scientific director.
“We are working several times with people from other research centres. This is
associated with a greater effort in terms of coordination but better results because of
different competences (Senior scientist 1).”
On the contrary, they were never mentioned by the administrative staff and the executive board.
Int. Journal of Business Science and Applied Management /
The final set of indicators
Table 5 outlines the final set of indicators related to the five performance dimensions previously introduced
and highlights the priorities assigned to each indicator by different stakeholders.
Table 5: The set of indicators and stakeholders priorities
Head of
Number of scientific publications
Number of patents
Number of technology transfer activities
Stock of competitive projects
IF distribution
CI distribution
Success rate of competitive projects
Revenue generated by patents, licenses and spin-off
Revenue generated by technology transfer activities
Cost of a scientific publication
Cost to manage and maintain a patent
External budget/Internal budget
Internal budget/FTE
Scientific reputation
Press coverage
Capex and Opex
Employees (category, nationality, age, degree, genre)
Organizational climate
Saturation of equipment
Number of joint publications
IF of joint publications
CI of joint publications
During the second phase of the action research cycle, the cost and validity of the proposed set of KPIs was
verified, providing practical guidelines to the research centre for the implementation of these measures. Results
of this phase vary according with the performance dimension.
KPIs related to effectiveness are diversified in terms of cost of measures. Most of them can be easily
calculated because they base on data that are already collected for different purposes. The most relevant
exceptions are the Impact factor and Citation index distribution. These indicators, though considered very
important for the assessment of the quality of the research by the scientific director, require ad hoc data
collection and further elaboration in order to ‘clean’ data. Effectiveness indicators are particularly relevant to
have an overall idea of how the centre is performing and to assess the performances of each RU with respect to
its research areas; however, they require some cautions in relationship to data interpretation, since different RUs
cannot be easily compared with each other based on these numbers (e.g. bibliometric indicators change
significantly in different research fields).
KPIs in the outcome dimension aim to provide a proxy of the impact of the research activity on the society.
Data about cash flows are already monitored for accounting purposes. However, they provide only a partial
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
information about the impact of the research activity on the society. Hence, it could be interesting to
complement them through qualitative information to have a more comprehensive idea of research outcomes.
KPIs related to the efficiency can be quite easily calculated because the institute already monitors cost
information very precisely. However, the interpretation of these data can be controversial. In particular, this is
the case of the cost per scientific publication that answer to the need of external accountability. Similar to
effectiveness indicators, it varies significantly across different research units depending on the specific research
field (equipment, materials, and other research instruments significantly vary in different research areas,
resulting in different costs per publication).
KPIs concerning the risk dimension are characterized by some differences in term of measurement costs.
Capex, opex, turnover and data about employees can be determined easily, instead this is not the case for
indicators about the reputation, saturation of key equipment and the organizational climate that entail ad hoc
data collection. Reputation and saturation of key equipment capture two aspects that are particularly relevant for
the research centre, but, at present, they cannot be calculated automatically, and the computation ask a great
effort to collect data and to elaborate them; in the next future, these indicators could benefit from the
exploitation of some supporting tools to collect data routinely. The organizational climate, on the other hand, is
assessed through a survey yearly and it suffers from the limitations that are typical of similar approaches (e.g.
risk of bias, limited timeliness).
Finally, KPIs in the network dimension exploit the same data base of those related to effectiveness, and
share their strengths and weaknesses. In addition, they provide an overall picture of the extent to which
researchers of different RUs collaborate, without entering into details of the contribution of different parts to the
research outputs.
This paper aimed at constructing a performance measurement system (PMS) for a public research centre
characterized by the presence of a diverse set of stakeholders with multiple objectives, ranging from making
decisions, motivating researchers to demonstrating external accountability. The PMS has been developed
through action research with respect to a specific context, an Italian technological research centre, where the
researchers recursively interacted with different stakeholders in the construction of the PMS. Based on the state
of the art literature and data collected from the interviews, a multidimensional framework for measuring R&D
performance was proposed and a set of KPIs was developed. The traditional process-oriented model (Pollanen,
2005) that encompasses the performance dimensions of effectiveness, efficiency and impact, was complemented
with two further dimensions of risk and network. The former aims to monitor the uncertainty that characterizes
the research activity, and the latter allows to capture the ability of working collaboratively. The introduction of
these two dimensions moves from the idea that the PMS should be aligned with the characteristics of the
organizations in which it is used and the trends of development they should deal with (e.g. Chiesa et al.,
2009).The set of KPIs puts together the information needs of different stakeholders that showed a definite
interest in relationship to a certain subset of measures, often disregarding the others. The integration of different
information needs in a comprehensive set of indicators aimed at providing a holistic picture of the centre
performances to all the relevant actors, raising their attention on the existence of potential trade-offs between
different measures.
This study contributes to the extant literature in two different ways. First, models, frameworks and
methodologies for measuring R&D performances have mostly focused at the firm level, with an economic or
strategic focus (Secundo et al., 2010). We deployed them in the context of a public research institute, where
attention of researchers, so far, concentrated on the one specific stakeholder the government that is mainly
interested in an overall assessment of research centres’ performances in relationship to funding allocation.
Second, we tried and integrated the concepts for performance measurement and performance management of
interdisciplinary collaborative research, which depicts new challenges for promoters and science managers in
the research environment of the twenty-first century (Agostino & Arnaboldi, 2012; Arena & Azzone, 2005;
Arnaboldi & Azzone, 2010).
Since the research was conducted in a specific organization, the possibility of generalizing the results to
similar institutions is a key issue. Hence, it is important to distinguish between what is general in scope and
what is case-specific. First, the multi-dimensional framework proposed aims to be general in scope. The use of
the process oriented model as a starting point and the integration of the two dimensions of risk and network aim
to make the PMS model more coherent with the pressures that public research organizations are currently
experiencing in different countries. Focus on efficiency and effectiveness, call for interdisciplinary research and
growing uncertainty are key challenges for these institutions. Moreover, risk and network represent two trends
of development in relationship to performance measurement in different types of organizations (e.g. Provan and
Milward, 2001), though, in most of cases these performances are not formally integrated in the PMS (e.g. Arena
et al., 2011). This paper represents a possible way to explicitly integrate these aspects in the PMS, opening the
path to future research to further adapt the proposed model to different contexts. The second result of general
Int. Journal of Business Science and Applied Management /
validity is the approach followed to develop the set of KPIs. The choice of the KPIs was guided by the priorities
of different stakeholders. Their requirements and the purpose of use in relationship to different information were
mapped and cross-checked in order to build a set of indicators that would be comprehensive and transversal to
the whole organization. The set of indicators, on the other hand, is case specific and reflects the characteristics
of the research centre under investigation, even though it could provide source of inspiration for similar
Finally we address the limitations of this work. The data were collected through a qualitative and
collaborative methodology, action research; hence the results suffer from the limitations that are typical of this
research methodology. In particular, action research is situational, hence, many of the relationships between
people, events, and things are a function of the situation as relevant actors currently define it and they change as
the definition of the situation changes (Susman & Evered, 1978). Accordingly, while the collaboration between
the researcher and the institute staff led to the identification of the KPIs, it may have also resulted in a degree of
bias. However, this methodology gave us the possibility of having lively, detailed and involved discussions with
members of the organizations. Our ideas about performance dimensions and KPIs were critically challenged in
order to satisfy their specific requirements. Moreover, the active participation of users and the integration of
their different informational requirement contributed to reducing resistance to the project and promoting the
development of the PMS. An important field of future research remains the applicability of the proposed model
and methodology to other public research organizations in order to verify its generalizability in different
institutions operating in different countries.
Agostino, D., & Arnaboldi, M. (2012). Design issues in Balanced Scorecards: the “what” and “how” of control.
European Management Journal, 30(4), 327-339.
Allen, J., James, A.D., &Gamlen, P. (2007). Formal versus informal knowledge networks in R&D: a case study
using social network analysis. R&D Management, 37(3), 179-196.
Arena, M., &Arnaboldi, M. (2013). How dealing with Spending Reviews? The case of Italy. Public Money and
Management, forthcoming.
Arena, M., Arnaboldi, M., & Azzone, G. (2010a). Student perceptions and central administrative services: The
case of higher education in Italy. Studies in Higher Education, 35 (8), 941-959.
Arena, M., Arnaboldi, M., & Azzone, G. (2010b). The organizational dynamics of Enterprise Risk Management.
Accounting, Organizations and Society,35(7), 659-675.
Arena, M., Arnaboldi, M., & Azzone, G. (2011). Is enterprise risk management real? Journal of Risk Research,
14(7), 779-797.
Arena, M., Arnaboldi, M., Azzone, G., & Carlucci, P. (2009). Developing a performance measurement system
for University Central administrative services. Higher Education Quarterly, 63(3), 237-263.
Arena, M., &Azzone, G. (2005). ABC, Balanced Scorecard, EVA™: An empirical study on the adoption of
innovative management accounting techniques. International Journal of Accounting, Auditing and
Performance Evaluation, 2(3),206-225.
Arena, M., &Azzone, G. (2010). Process based approach to select key sustainability indicators for steel
companies. Ironmaking and Steelmaking, 37(6), 437-444.
Arnaboldi, M.,& Azzone, G. (2010).Constructing performance measurement in the public sector. Critical
Perspectives on Accounting, 21(4),266-282.
Arnaboldi, M., Azzone, G., &Savoldelli, A. (2004).Managing a public sector project: The case of the Italian
Treasury Ministry. International Journal of Project Management, 22(3), 213-223.
Azzone., G. (2008). Controllo di gestione nelle amministrazioni pubbliche. Decisioni e accountability per
ministeri, enti locali e università. Milano:ETAS.
Bremser W.G., &Brasky, N.P. (2004).Utilising the Balanced Scorecard.R&D Management, 34(3), 229-238.
Brown, M.G.&Svenson, R.A. (1988).Measuring R&D productivity. Research-Technology Management, July
ChiesaV.,& Frattini F. (2007). Exploring the differences in performance measurement between research and
development: evidence from a multiple case study.R&D Management, 37(4), 283-301.
Chiesa, V., Frattini, F., Lazzarotti V., & Manzini R. (2009). Performance measurement of research and
development activities. European Journal of Innovation Management,12(1), 25-61.
Deborah Agostino, Marika Arena, Giovanni Azzone, Martina Dal Molin and Cristina Masella
Chu, P.Y., Lin, Y.L., Hsiung, H.H.,& Liu, T.Y. (2006). Intellectual Capital: an empirical study of ITRI.
Technological Forecasting & Social Change, 73, 886-902.
Coccia, M. (2001a). Scientific research performance within public institutes. Scientometrics, 65(3), 307-321.
Coccia, M. (2001b). A basic model for evaluating R&D performance: theory and application in Italy. R&D
Management, 31(4), 453-464.
Coccia, M. (2004). New models for measuring the R&D performance and identifying the productivity of public
research institutes. R&D Management, 34(3), 267-280.
Coccia, M.(2005). Scientometric model for the assessment of the scientific research performance within the
public institutes. Scientometrics,65(3), 297311.
Coccia, M., &Rolfo, S. (2007). How research policy changes can affect the organization productivity of public
research institutes. Journal of Comparative Policy Analysis, Research and Practices, 9(3), 215-233.
Coughlan, P. &Coghlan, D. (2002). Action Research for operations management. International Journal of
Operations & Production Management, 22(2), 220-240.
Cross, R., Borgatti, S.P.& Parker, A. (2002). Making invisible work visible: using social network analysis to
support strategic collaboration. California Management Review, 44(2), 2546.
Dumay J.,& Rooney, J. (2011). Measuring for managing? An IC practice case study. Journal of Intellectual
Capital, 12(3), 344-355.
Edvisson, L., & Malone, M.S. (1997). Intellectual Capital: Realizing Your Company’s True Value by Findings
its Hidden Brainpower. New York, NY:HarperBusiness.
Elmquist M., & Le Masson, P. (2009). The value of a “failed” R&D project: an emerging evaluation framework
for building innovative capabilities. R&D Management, 39(2), 136-152.
Garcia-Valderrama, T.,&Mulero-Mendigorri, E. (2005). Content validation of a measure of R&D effectiveness.
R&D Management, 35(3), 311-331.
Hult, M., &Lennung, S. (1980). Towards a definition of action research: a note and a bibliography. Journal of
Management Studies, 17, 241-250.
Kerssen-van Dronelen, I.C. &Bilderbeek, J. (1999). R&D performance measurement: more than choosing a set
of metrics. R&D Management, 29, 35-46.
Kim, B.,& Oh, H. (2002). An effective R&D PMS: survey of Korean R&D researchers. International Journal of
Management Science, 30, 1931.
KulatungaU., Amaratunga D., &Haigh R. (2011). Structured approach to measure performance in construction
research and development. Performance measurement system development. International Journal of
Productivity and Performance Management, 60 (3), 289-310.
Lepori, B., van den Besselaar, P.,Dinges, M.,Potì, B.,Reale, E.,Slipsaeter, S.,Thèves, J., & van der Meulen, B.
(2007). Comparing the evolution of national research policies: what patterns of change? Science and Public
Policy, 34(6), 372388.
Leitner, K.H., & Warden, C. (2004). Managing and reporting knowledge-based resources and processes in
research organization: specifics, lessons learned and perspectives. Management Accounting Research, 15,
Lettieri, E., (2009). Uncertainty inclusion in budgeting technology adoption at a hospital level: Evidence from a
multiple case study. Health Policy, 93(2):128-136.
Lettieri, E., Masella, C., (2009). Priority Setting for Technology Adoption at a Hospital Level: Relevant Issues
from the Literature. Health Policy, 90(1):81-88.
Lettieri, E., Masella, C., Nocco, U., (2008).Budgeting And Health Technology Assessment: First Evidence
Obtained From Proposal Forms Used To Submit The Adoption Of New Technology. International Journal
of Technology Assessment in Health Care, 24(4):502-510.
Lettieri, E., Masella C., Radaelli, G., (2009).Disaster Management: findings from a systematic review. Disaster
Prevention and Management, 18(2):117-136.
Luwel, M., Noyons, C.M. &Moed, F. (1999).Bibliometric assessment of research performance in Flanders:
policy background and implications. R&D Management, 29(2), 33141.
Lynch, R.L. & Cross, K.F. (1991). Measure Up ± the Essential Guide to Measuring Business Performance.
Mandarin, London.
Marr, B., Schiuma, G. & Neely, A. (2004). Intellectual capital: defining key performance indicators for
organisational knowledge assets. Business Process Management Journal, 10(5), 551-69.
Int. Journal of Business Science and Applied Management /
Mettänen, P. (2005). Design and implementation of a performance measurement system for a research
organization. Production Planning & Control, 16(2), 178-88.
Moreno, A.A. & Tadepalli, R. (2002). Assessing academic department efficiency at a public university.
Managerial and Decision Economics, 23, 385-397.
Narin, F., & Hamilton, K.S. (1996).Bibliometric performance measures.Scientometrics, 36(3), 293310.
National Academies (2005).Engineering and Public Policy. Facilitating Interdisciplinary Research.
Washington, DC:National Academies Press.
Neely, A., Marr, B., Roos, G., Pike, S. & Gupta, O. (2003). Towards the third generation of performance
measurement. Controlling, 15(3/4), 129-135.
Noyons, E.C.M., Moed, H.F. ,&Luwel, M. (1999). Combining mapping and citation analysis for evaluative
bibliometric purposes: A bibliometric study. Journal of the American Society for Information Science,
50(2), 115131.
Pawar, K.S., &Driva, H. (1999). Performance measurement for product design and development in a
manufacturing environment. International Journal of Production Economics, 60-61, 6168.
Pollanen, R.M. (2005). Performance measurement in municipalities Empirical evidence in Canadian context.
International Journal of Public Sector Management, 18(1), 4-24.
Provan, K.G.,&Milward, H.B. (2001). Do networks really work? A framework for evaluating publicsector
organizational networks.Public Administration Review, 61, 413-423.
Rapoport, R. (1970). Three Dilemmas of Action Research. Human Relations, 23(6), 499-513.
Rowley, J. (2000). Is higher education ready for knowledge management? The International Journal of
Educational Management, 14(7), 325-33.
Reason, P., & Bradbury, H. (Eds.). (2001). Handbook of Action Research: Participative inquiry and practice.
London : Sage Publications
Saez, C.B., Marco, T.G. &Arribas, E.H. (2002). Collaboration in R&D with universities and researchcentres: an
empirical study of Spanish firms. R&D Management, 32, 321341.
Secundo G., Margherita A., Elia G., &Passiante G. (2010). Intangible assets in higher education and research:
mission, performance or both? Journal of Intellectual Capital, 11(2), 140-157.
Senker, J. (2001). Changing organization of public research in Europe. Implications for benchmarking human
resources in RTD, Paper prepared for Human Resources. In:RTD session of the contribution of European
socio-economic research to the benchmarking of RTD policies in Europe, Brussels, March 15-16.
Steiner, A. & Nixon, B. (1997). Productivity and performance measurement in R&D. International Journal of
Technology Management, 13(5/6), 486-496.
Stewart, T.A. (1997). Intellectual Capital: The New Wealth of Organizations. New York: Doubleday/Currency.
Susman, G.I. &Evered, R.D. (1978). An Assessment of the Scientific Merits of Action Research. Administrative
Science Quarterly, 23(4), 582-603.
Whelan E., Teigland R., Donnellan B.,& Golden W. (2010). How Internet technologies impact information
flows in R&D: reconsidering the technological gatekeeper. R&D Management, 40(4), 400-413.
Welter, F., Jooß, C., Richert, A., Jeschke, S. &Brecher, C. (2012). Organisation and Management of Integrative
Research. In: C. Brecher (Ed.) Integrative Production Technology for High-Wage Countries. Berlin
Heidelberg: Springer-Verlag, pp. 6473.
Yin, R.K. (1994). Case Study Research: Design and Methods. Thousand Oaks: Sage Publications.