The definition of organizational guidelines is an extremely important action for the establishment of strategies that contribute to the achievement of objectives considered essential to a company. Brazilian Agricultural Research Corporation (Embrapa) is a public company under private law linked to the Brazilian federal government Ministry of Agriculture, Livestock and Food Supply (MAPA), whose mission is “to undertake research, development and innovation solutions that ensure the sustainability of agriculture, for the benefit of Brazilian society” (Embrapa, 2008: p. 18). Its guidelines indicate what the Corporation intends to achieve in a given period of time considering a strategic area, in order to achieve its mission and organizational objectives. The guidelines, therefore, represent the set of principles, standards and goals that help to ensure a more favorable development trajectory for the Corporation.
In public RD&I institutions, such as Embrapa, organizational performance is dependent on the results of the research it conducts and the benefits they bring to society. In this context, the process of quantitative method and scientific computation (QMSC) is one of the essential organizational innovation processes. This process supports the promotion and advancement of knowledge, promotes the application of quantitative methods, mathematical modeling and scientific computing techniques and tools, and helps Embrapa face the complex challenges of Brazilian agricultural research, especially those related to efficiency and agility of the analysis and extraction of information and knowledge from the data generated by agricultural research. Therefore, the establishment of organizational guidelines for the QMSC process becomes essential to Embrapa in achieving excellence in RD&I.
This article aims to establish organizational guidelines for advanced quantitative methods and scientific computing (QMSC) for research, development and innovation (RD&I) area at Embrapa. It is an exploratory qualitative study that was carried out from a strategic and systemic perspective and that contributes to making RD&I governance at Embrapa viable. The article is structured in four sections, in addition to this introduction. The second section presents a brief discussion of the theoretical basis of this study. The third section details the methodology used. The fourth section summarizes the results and the main findings of the study. In the fifth section, the results and their implications are discussed.
2. Theoretical Framework
Currently, country economic prosperity is linked to the development of the knowledge economy (Kearney, 2013) based on the production, distribution and use of knowledge and information (Organisation for Economic Co-Operation and Development, 1996). In this economy, people, knowledge and technologies cooperate to promote value to organizations, local communities and countries (Hadad, 2017), with knowledge as the main production input. Knowledge-based societies are at the center of the debate on globalization as well as the practices of major international organizations, as efforts to produce new products and technologies a strategy to promote competitive advantage.
Knowledge as the basis of the modern production process has the power to penetrate all social segments, favoring a change in government and business mentality and attitudes, changing all socioeconomic structures (Hadad, 2017). Kearney writes (2013: pp. 20-21):
Since the 1990s, the advance of the knowledge economy has been an unstoppable force worldwide [...]. It has spurred nations of varied economic strengths to invest more heavily in knowledge as a motor for growth and to reorient their policies to address the issues involved.
The global economic crisis that began in 2008 has raised doubts about the globalization process. This crisis has encouraged the mobilization of diverse actors (governments, private sector, research, development and innovation institutions, educational institutions and civil society) around relevant themes such as governance, knowledge management, research and innovation, technology, etc. for the implementation of effective stabilization and sustainable development policies. It also emphasized that socioeconomic growth is linked to the development of policies and guidelines on research, innovation and incentives to education (Kearney, 2013).
Today, innovation is a priority for organizations that strive to survive in a context of globalized and fierce competition. In this context, they invest efforts in the development of new products, production processes, management methodologies, services and, especially, research and development (R&D), fundamental to enhance the capacity for organizational growth, as they are sources of production for new knowledge and contribute to leveraging innovation.
In Brazil, data from the Innovation Survey reported by Frigeri (2017) and by the Brazilian Institute of Geography and Statistics (2016), for the period from 2012 to 2014, show that the average percentage of investment in R&D made by the 132,429 companies participating in the survey is 0.8% of revenue. This result reinforces that innovation is a major challenge to the implementation of R&D projects and that factors such as economic risk, scarcity of financing sources, lack of qualified personnel and organizational aspects such as structure, guidelines, culture and organizational learning are considered critical because they facilitate and/or hinder innovation (Romminger, 2017).
In the case of the agricultural sector, a prominent sector in the Brazilian economy, there are few studies that analyze how the innovation process is constituted. According to Romminger (2017: p. 8), “Agriculture has unique characteristics that hinder the development of new technologies and require greater participation by the State, either as a regulator or as an agent of innovation”.
Embrapa, since its creation in 1973, has acted as an innovation agent in Brazilian agricultural sector, as a result of an intense and effective support by the Brazilian government (Dahlman & Frischtak, 1993). With the mission of producing specific knowledge and skills, the Corporation has been leading research efforts in this sector, being one of those responsible for bringing science and development closer in Brazil (Salles-Filho & Bin, 2014).
Currently, research institutions in the Brazilian agricultural sector suffer a drastic reduction in public funding, which forces them to invest in innovation focused on product diversification and quality improvement, both for producers in agribusiness and for traditional producers whose demands are quite different (Romminger, 2017). In this context of unavailability of public funding “[...] agricultural research is at the mercy of fiscal volatilities, without a clear alternative provided by the other actors of the system. These actors, although existing, do not have the infrastructure, human resources, or interaction with each other comparable to Embrapa” (Romminger, 2017: p. 9). This reinforces the relevance of understanding how and what factors promote innovation in public R&D institutions.
Organizational innovation is one of the factors that leverage innovation. For Salles-Filho & Bin (2014), the organizational aspect of the innovation process is essential, since it is increasingly evident that public research institutions need to make the appropriation of their results more efficient for the society. The authors admit that novation is a process that takes place within the company and beyond it, therefore, it is a collective game whose “... assets and skills needed to transform research results into innovation are particularly varied and are far from laboratories and even the field” (Salles-Filho & Bin, 2014: p. 440).
R&D institutions are seen as entities that create knowledge and skills; therefore, they learn and evolve scientifically, technologically and organizationally (Garcia & Salles-Filho, 2009; Salles-Filho, Bonacelli, & Mello, 2000; Salles-Filho et al., 2000). In this context, the technological and organizational innovations that these institutions need to implement in order to meet the current requirements depend on their learning capacity. Organizational learning is, therefore, essential to innovation, since it relates to the company’s ability to manipulate information and combine existing and new knowledge through the interaction between people and/or markets to diagnose, create alternative solutions, correct distortions and/or improve organizational performance (Moraes, Oliveira, & Cazarini, 2011). Therefore, in order to build a successful trajectory, companies such as Embrapa need to continuously evaluate their internal organizational processes to build competencies and skills necessary for their adaptation to the external environment with which they interact (Garcia & Salles-Filho, 2009).
The concept of innovation was revised by the Organisation for Economic Co-operation and Development (2018: p. 22) and is currently defined as “[...] a new or improved product or process (or combination thereof) that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process)”. Note that the term unit has a generic meaning and refers to any social actor (whether individual or organizational) that is responsible for promoting innovation.
The term organizational innovation has also been replaced by innovation in business processes that means “a new or improved product or business process (or combination thereof) that differs significantly from the firm’s previous products or business processes and that has been introduced on the market or brought into use by the firm” (Organisation for Economic Co-operation and Development, 2018: p. 11).
At Embrapa, one of the essential business processes is the Quantitative Methods and Scientific Computing (QMSC) process, considered by the Company’s RD&I area as a core unit of its research. This process aims to promote and advance knowledge, boost the application of quantitative methods, mathematical modeling and scientific computation techniques and tools and provide inputs to Embrapa in facing complex challenges of Brazilian agricultural research, particularly linked to efficiency and agility in the analysis and extraction of information and knowledge from the data generated by agricultural research.
Since the creation of the Corporation, the QMSC process has been considered a strategic action that requires diversified initiatives in order to promote continuous revitalization of this area. A report produced in 2013 by a group of researchers recommended that Embrapa integrate professionals of this area with those in other research areas, in order to increase the efficiency of research through internal and external partnerships. This integration should culminate in the creation of a research network entitled Embrapa’s QMSC Network.
This Network would be responsible for enabling the use of new techniques of scientific computing, data and information analysis, aiming at improving the quality of agricultural research, and managing, leading and supporting the actions of this area in the Corporation. Therefore, the establishment of organizational guidelines for Embrapa’s QMSC area is perceived as an important organizational innovation that will contribute to the improvement and interpretation/extraction of knowledge from complex data sets (large volumes of data; hundreds or thousands of heterogeneous attributes/variables, distributed in different sources, etc.), while also providing inputs to research projects that involve the generation and evaluation of multiple hypotheses, enabling interdisciplinary analysis of various scientific issues. In addition, the innovation of this business process will be fundamental to the promotion of prospective, descriptive and impact assessment studies in the Brazilian agricultural research.
This study is an exploratory research (Collis & Hussey, 2005), that used a qualitative approach and is of a descriptive nature, specifically narrative review, falling within the scope of the interpretive paradigm (or qualitative paradigm) (Segura-Muñoz et al., 2002; Guedes, 2011). The documentary research technique (Bauer & Gaskell, 2000) was applied to managerial and organizational publications produced by Brazilian and international research institutions.
In the first phase of the study, a selection of Brazilian and international institutions with a predominant role in RD&I in agriculture, generating scientific and technological knowledge, was carried out. In Brazil, we sought to identify a similar size public institution with operations at the federal level, with strong focus on RD&I, although not in the agricultural sector (since this is Embrapa’s mission). At the international level, we selected the main internationally recognized RD&I institutions, dedicated to the generation of knowledge in Science and Technology (S&T) on the agricultural themes.
Based on this sample of companies, we sought to understand how the topic of QMSC was treated in these institutions based on the publicly available on their websites information. The organizational structures of these institutions were verified and, subsequently, those companies that indicated activities related to the use and development of QMSC were identified and selected.
The second phase of the study, focused on the general panorama of current QMSC research, was a bibliographic study focused on two dimensions. First, random consultations were carried out in databases of scientific journals, in order to identify the state of the art and trends in the area of QMSC, in terms of knowledge advancement. Second, a series of consultations was carried out in various systems of classification and categorization of subjects, in particular, those with a thematic scope encompassing Scientific Computing and Mathematics.
In order to carry out this study of the general panorama of QMSC research, the following methodological procedures were adopted:
1) Identification and selection of systems for the classification and categorization of knowledge including the areas of scientific computing and quantitative methods;
2) Identification and selection of keywords and categories of knowledge that include the thematic quantitative methods and scientific computing;
3) Identification and selection of a set of areas of knowledge, comprising Scientific Computing and Quantitative Methods;
4) Drawing up a table of areas of knowledge for quantitative methods;
5) Drawing up a table of areas of knowledge for scientific computing;
6) Identification and selection of Brazilian and international institutions that generate scientific knowledge on Scientific Computing and Quantitative Methods, especially those dedicated to RD&I in the above disciplines, in agricultural and related sciences;
7) Consultation in databases of scientific publications in order to identify articles that describe the state of the art and trends in RD&I on the QMSC;
8) Establishment of the general overview of QMSC research.
The two phases of the study were performed using document analysis technique. The following documents were analyzed: “VI Embrapa Master Plan: 2014-2034” and “Vision 2014-2034: the future of technological development in Brazilian agriculture” and “Vision 2014-2034: the future of technological development in Brazilian agriculture: synthesis” (Embrapa, 2015).
The results of this analysis generated valuable reports, providing inputs to the analysis of Embrapa’s internal and external context, and identifying trends in structuring QMSC for the Corporation. The documentary analysis also identified the points of connection, convergence and adherence that the documents studied contained thus contributing to the proposal of organizational guidelines for QMSC for Embrapa.
4. Partial Results
Benchmarking study was conducted in RD&I institutions aiming to get to know their organizational models, and especially identify the existence of research activities related to QMSC. Of all identified institutions only seven were selected due to their similarities with Embrapa: Commonwealth Scientific and Industrial Research Organization (CSIRO) (https://www.csiro.au/); Rothamsted Research (http://www.rothamsted.ac.uk/); French National Institute for Agricultural Research (INRA) (http://institut.inra.fr/en); Agricultural Research Service/United States Department of Agriculture (ARS/USDA) (https://www.ars.usda.gov/); Economic Research Service/United States Department of Agriculture (ERS/USDA) (https://www.ers.usda.gov/); National Agricultural Statistics Service/United States Department of Agriculture (NASS/USDA) (https://www/nass.usda.gov/index.php); Fundação Oswaldo Cruz (Fiocruz) (https://portal.fiocruz.br/pt-br).
There was a predominance of international institutions, since only one Brazilian institution was identified, specifically, the Oswaldo Cruz Foundation (Fiocruz). It is worth clarifying that with regard to Brazilian institutions, we sought to identify and select federal public institutions, with a strong performance in RD&I, although not specifically focused on the agricultural sector, but rather with some degree of similarity with Embrapa in terms of its institutional nature and mission.
A set of systematized information was produced for each of the selected institutions, organized into two categories: 1) Basic data on the institutions, including: name of the institution, country, year of creation, legal status, number of employees, number scientists, total budget, government budget/contribution, objectives; 2) Information on the structure of the organizations, containing identification of the institution, accompanied by a brief description, with emphasis on the areas of activity.
The analysis of the results of the benchmarking suggested that: 1) there is no clear similarity between the selected institutions and Embrapa, either in terms of the scope of institutional performance, or with regard to the organizational structure; 2) Quantitative Methods and Scientific Computing, as areas of expertise and delimitation of knowledge are widely disseminated in Brazil, especially at Embrapa, however we did not find equivalence and/or similarity in terms of organizational structure in international institutions; 3) there is evidence of the emergence of a new area and/or field of knowledge, which has been referred to in the literature as Data Science (Pedroso Júnior, 2018: p. 21).
Partial results of the study on the general overview of QMSC research allowed to elaborate two tables of areas and subareas of knowledge, one for Scientific Computing and the other for Quantitative Methods, together with a map in the form of a graph to represent and promote the visualization of the areas and subareas of Scientific Computing and Quantitative Methods.
The two tables were based on a careful reading and critical analysis of the various systems of classification and categorization of subjects in use in the main national and international institutions, involved in the organization and systematization of scientific and technological knowledge. Two of these international systems revealed greater relevance to the nature and the thematic scope of the areas of Scientific Computing and Quantitative Methods, namely: 2012 ACM Computing Classification System and 2010 Mathematics Subject Classification (Table 1). Both international systems reveal important aspects that describe particularities of the state of the art and trends in terms of research on QMSC, when confronted with the bibliographic databases search results. Figure 1 presents an extract of the two knowledge tables obtained as partial results of the second phase of the study.
Table 1. The main categories of topics in SC and QM identified in knowledge tables: ACM computing classification system and mathematics subject classification.
Figure 1. The main categories of subjects in SC and QM identified in the knowledge tables: ACM Computing Classification System and Mathematics Subject Classification.
Another result of this second phase of the study refers to the construction of a map in the form of a graph (Figure 1), containing the areas and sub-areas of Scientific Computing and Quantitative Methods, allowing an integrated visualization of the diverse knowledge domains involved.
5. Final Results
The final result achieved by the study is basically constituted by the partial results obtained in the two phases of the research. The results of both phases provided the basis for the formulation and establishment of the guidelines and organizational orientation to be followed, so as to enable the governance of scientific QMSC research, as well as to establish a QMSC network at Embrapa. In search of legitimacy and allignment of purposes, the partial results achieved were submitted to the analysis of the statements present in the strategic objectives and principles contained in the documents that guide Embrapa’s institutional policy, mentioned above. Based on this analysis, the organizational guidelines identified below are proposed.
Proposed Organizational Guidelines for Data Science Actions at Embrapa
Considered an important stage in strategic management implementation model, the definition of organizational guidelines concerns the establishment of goals for the organization (Zavaglia, 2013). To this end, the need to redefine the institutional mission and strategic objectives is considered.
In the context of the actions developed by the study, we assumed that the mission, vision and values defined by Embrapa in its IV Master Plan are up to date and consistent with the reality of 2018, eliminating the need for the redefinitions of these concepts.
The proposed QMSC organizational guidelines aim to strengthen this action at Embrapa. Such guidelines detail the conduct that will lead the Corporation to reach its organizational objectives, preparing it to face the great research challenges based on the institutionalization of Data Science actions perspective and the understanding that these actions must be the core of the research process.
In this sense, the organizational guidelines proposed below must be understood as a solid set of procedures that determine how the goals (Oliveira, 2018) related to the institutionalization of Data Science actions must be achieved at Embrapa.
Based on Zavaglia (2013), it appears that goals correspond to the efforts that the organization directs in order to meet the proposed organizational objectives, or else, the objective is the target or point that one intends to achieve, or even, a task performed by managers and employees.
The following three dimensions of organizational guidelines with their respective targets were defined aiming to establish the governance of QMSC actions based on the Data Science actions at Embrapa: strategic dimension; tactical dimension; and operational dimension. Although there is interdependence between the dimensions, the division was necessary to make the monitoring and evaluation of actions/goals more agile and to allow for the minor correction of course within the scope of each dimension.
The strategic dimension represents a set of essential corporate actions that affect, to a greater or lesser extent, other strategic actions of Embrapa. Thus, the strategic dimension indicates to the entire Corporation the actions that must be taken to establish Data Science as a priority action. It is an essential dimension because it defines strategic guidelines that help the entire Corporation to understand the relationship that Data Science has with other RD&I actions at Embrapa. In this dimension, two strategic guidelines (SG) were defined with their respective goals (GSG):
• SG-1—Prepare a priority corporate plan for the actions in Data Science at Embrapa:
o GSG-1.1—Establish an interdisciplinary working group with professionals from the Central and Decentralized Research Units to propose a corporate plan aimed at institutionalizing priority data science actions at Embrapa.
• SG-2—Align Embrapa’s Data Science actions, proposed by the working group and approved by Administrative Board at Embrapa, with those of quality assurance and management of the RD&I process:
o GSG-2.1—Establish an interdisciplinary working group with professionals from the Central and Decentralized Research Units to identify and propose alignment in Embrapa’s guidelines and standards for the RD&I processes that underpin the Data Science actions at Embrapa proposed in SG-1.
The tactical dimension represents a set of emerging themes that result from the institutionalization of Data Science actions. This dimension makes clear to the entire Corporation the interdependence between the actions of Data Science and the other actions/areas of Embrapa, especially those that will support and guide their institutionalization, such as RD&I, physical and computational infrastructure and people management actions. It is a dimension that defines tactical procedures that guide the entire Corporation in coordination with other themes that touch on Data Science actions at Embrapa. In this dimension, three tactical guidelines (TG) with their respective goals (GTG) were defined:
• TG-1—Identify and manage the human and organizational competences existing at Embrapa aiming at prioritizing Data Science actions by knowledge domain based on national research priorities and in line with the new research challenges:
o GTG-1.1—Identify the gaps between the knowledge and experience of professionals working in QMSC and the priority areas of research at Embrapa aiming to propose actions for continuing education and development of Data Science teams;
o GTG-1.2—Identify the gaps in the institutional agendas of the Decentralized Research Units with respect to QMSC actions and priority research areas at Embrapa aiming to suggest collaborative efforts to strengthen the institutionalization of data science actions at Embrapa.
• TG-2—Propose a continuing education program for human and organizational training in Data Science topics considered a priority in MT 1.1 and 1.2:
o GTG-2.1—Conduct a survey of training needs among QMSC professionals considering the results of MT 1.1 and 1.2;
o GTG-2.2—Structure effective, agile and low-cost ways to make the continuing education program at Embrapa viable.
• TG-3 —Allign corporate standards for RD&I, people management and physical and computational infrastructure to the priority actions of Data Science at Embrapa:
o GTG-3.1—To revise corporate norms related to the RD&I, people management (postgraduate, visiting researcher, recognition, performance evaluation and promotion) and physical/computational infrastructure processes that touch upon the priority actions of Data Sciences in Embrapa.
The operational dimension represents a set of practical actions derived from tactical and strategic actions that must be carried out to establish Data Sciences as an action essential to ensure excellence at Embrapa. This dimension confirms the achievement of the goals and strategies for the implementation of Data Science actions insofar as it is the dimension that transforms a priority theme into a routine management action, that is, it makes the action essential to Embrapa’s operation/functioning. , following the standards of excellence dictated by the environment. In this dimension, three operational guidelines (OG) with their respective goals (GOG) were defined:
• OG-1—Implement the corporate priority plan for the Data Science actions at Embrapa:
o GOG-1.1—Map, describe and outline simple and flexible improvement plans for the Data Sciences actions at Embrapa;
o GOG-1.2—Perform the mapped, described and improved processes of Data Science actions at Embrapa.
• OG-2—Monitor and systematically evaluate the actions of Data Sciences at Embrapa:
o GOG-2.1—Develop mechanisms for systematic monitoring and evaluation of Data Science actions at Embrapa focusing on management of human and organizational competencies, facing national research challenges and research excellence at Embrapa;
o GOG-2.2—Execute monitoring and evaluation actions defined for the actions of Data Sciences at Embrapa.
• OG-3—Invest in human and organizational skills identified as priorities to promote Data Sciences actions at Embrapa:
o GOG-3.1—Execute the continuing education program for human training and organizational development on priority themes of Data Science for Embrapa;
o GOG-3.2—Execute the corporate norms defined for the RD&I, people management (post-graduation, visiting researcher, recognition, performance evaluation and promotion) and physical/computational infrastructure processes that relate to the priority Data Sciences actions at Embrapa.
The study allowed analyzing important aspects related to the QMSC area, in terms of its internal and external environments. Internally, at Embrapa, we were able to analyze the QMSC status. In terms of the external environment, national and international institutions similar to Embrapa and protagonists of the research in this area, now called Data Sciences, were identified.
Another contribution worth mentioning came from the analysis of relevant aspects of the state of the art in QMSC, which led to compiling two tables of areas and sub-areas of knowledge domains for Scientific Computing and Quantitative Methods.
In this sense, there is another relevant contribution arising from the analysis of the expansion of the scope of the QMSC area at Embrapa, with the emergence of new fields of action and new lines of research, such as Data Science.
Pioneering experiences in this regard are already reported in countries such as Australia and China, in addition to the European Union. In Australia, the Research Council for Mathematical and Statistics Frontiers Center of Excellence (ACEMS) was created; in China there are key research laboratories in Data Science, as part of a priority agenda under the national strategy; and in the European Union, an action plan has been established to foster data-driven economy.
In Brazil, it would be up to Embrapa, as an institution of excellence in tropical agricultural research, to assume the role of protagonist in this area of knowledge, and to constitute and strengthen an action geared towards Data Science, which would allow fostering the development of inter- and trans-disciplinary research programs in agriculture sector.
It is essential for Embrapa to prioritize Data Science as a research action in data analysis using mathematical, statistical and computational methods, which intensify efforts to allow the transformation of data into new information and knowledge through the engineering of data use and reuse, that is, providing agile responses to complex society demands, supporting decision-making and the formulation and/or management of public policies.
The proposed organizational guidelines for undertaking data science actions, recorded in this article, fulfill the important role of promoting the coordination and internal/external dialogue between the different strategic actions of Embrapa and this emerging area of knowledge domain.
Beforehand, we recognize the existence of gaps and questions not answered by the study; therefore we propose a new multi-transdisciplinary study, mainly to ascertain the evidence of the emergence of a new area and/or field of knowledge, called Data Sciences.
It should also be noted that this study obtained an unforeseen additional result, which refers to the construction of structured knowledge tables through a web application (Figure 1), constituting an important input for the organization, classification, categorization and, mainly, visualization of the areas and sub-areas of Scientific Computing and Quantitative Methods for Embrapa.
 Dahlman, C. J., & Frischtak, C. R. (1993). National Systems Supporting Technical Advance in Industry: The Brazilian Experience. In R. Nelson (Ed.), National Innovation Systems: A Comparative Analysis (Chapter 13, pp. 441-449). New York: Oxford University.
 Garcia, A. E. B., & Salles-Filho, S. L. M. (2009). Trajetória institucional de um instituto público de pesquisa: O caso do Ital após 1995. Revista de Administracao Pública, Rio de Janeiro, 43, 661-693.
 Guedes, N. G. (2011). Revisao do diagnóstico de enfermagem estilo de vida sedentário: Análise de conceito e validacao por especialistas. 253 f. Tese (Doutorado)—Fortaleza, Universidade Federal do Ceará.
 Kearney, M.-L. (2013). Policy Imperatives in Research and Innovation Management. In A. Olssan, & L. Meek (Eds.), Effectiveness of Research and Innovation Management at Policy and Institutional Levels: Cambodia, Malaysia, Thailand and Vietnam (pp. 19-28). Kuala Lumpur: OCDE.
 Moraes, C. T., Oliveira, S. R., & Cazarini, E. W. (2011). Proposta de modelo de organizacao inovadora baseado na aprendizagem organizacional. Revista Ingepro—Inovacao, Gestao e Producao, 3, 95-102.
 Organisation for Economic Co-Operation and Development (1996). The Knowledge-Based Economy (45 p.). Paris.
 Organisation for Economic Co-Operation and Development (2018). Oslo Manual 2018: Guidelines for Collecting, Reporting and Using Data on Innovation (4th ed., 258 p.). Paris: OECD, Eurostat (The Measurement of Scientific, Technological and Innovation Activities).
 Salles-Filho, S., & Bin, A. (2014). Reflexoes sobre os rumos da pesquisa agrícola. In A. M. Buainain, E. Alves, J. M. da Silveira, & Z. Navarro (Eds.), O mundo rural no Brasil do século 21: A formacao de um novo padrao agrário e agrícola (Parte 3, Cap. 3, pp. 423-452). Brasília: Embrapa.
 Salles-Filho, S. L. M., Bonacelli, M. B., & Mello, D. (2000). Metodologia para o estudo da reorganizacao institucional da pesquisa pública. Parcerias Estratégicas, 9, 86-108.
 Segura-Munoz, S. I., Takayanagui, A. M. M., Santos, C. B. dos, & Sanchez-Sweatman, O. (2002). Systematic Literature Review and Meta-Analysis: Basic Notions about Its Design, Interpretation and Application in Health Research. In Proceedings of the 8th Brazilian Nursing Communication Symposium (p. 15). Sao Paulo: USP: Escola de Enfermagem de Ribeirao.
 Zavaglia, T. (2013). Planejamento e administracao estratégica. In R. Dias, M. Cassar, & T. Zavaglia (Eds.), Introducao à administracao: Da competitividade à sustentabilidade (3rd ed., pp. 117-179). Campinas: Alínea.