Perspective

Challenges in developing national orthopedic health research agendas in the Netherlands: process overview and recommendations

Annabelle RIKEN¹, Rudolf W POOLMAN^1,2, Rob G H H NELISSEN¹, Maaike G J GADEMAN^1,3; On behalf of the advisory board members of the Dutch Orthopedic Association’s first and second health research agenda

¹Department of Orthopaedics, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden; ²Department of Orthopaedic Surgery, Joint Research, OLVG, Amsterdam; ³Department of Clinical Epidemiology, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, The Netherlands

ABSTRACT — Growing demand for clinical research to improve evidence-based medicine in daily medical practice led to healthcare evaluation, which assesses the effectiveness of the existing care. The first step is identifying and prioritizing the most important evidence uncertainties. A health research agenda (HRA) can be valuable and helps determine funding and resource allocation, aiding researchers and policymakers to design successful research programs and implement the results in daily medical practice. We provide an overview of the development process of the first 2 HRAs within orthopedic surgery in the Netherlands and the following research process. In addition, we developed a checklist with recommendations for the future development of an HRA. This perspective guides the development of highquality and widely supported nationwide HRAs, including preparatory actions. This improves the uptake of evidence uncertainties in a successful research program and disseminates evidence-based literature in daily medical practice to improve patient care.

 

 

Over the last 2 decades, the growing demand for clinical research to assess health interventions has led to analytical and transparent approaches to set research priorities to improve patient outcomes by stimulating evidence-based clinical practice (1). Research prioritization helps determine funding allocation (2), effective use of resources, successful clinical research, and increased uptake of health research, which helps researchers and policymakers (3). A health research agenda (HRA), also called a knowledge agenda or research priority setting, is a valuable tool for setting research priorities. During the developmental process, evidence uncertainties in daily practice are evaluated to determine what research is needed to lead to cost-effective and evidence-based practice (4). In the Netherlands, establishing an HRA became an issue after 2000 (5). Caron-Flinterman et al. (6) were the first to report the setting of an HRA in 2006 to treat asthma and chronic obstructive pulmonary disease. Internationally, the first HRA was completed in 2007 by the James Lind Alliance for treating asthma. The Dutch Orthopaedic Association (NOV) launched the first HRA in 2015 (7), followed by the second in 2019 (8).

We used the NOV HRAs as a case study to provide an overview and reflect on the development process to improve understanding among patients, doctors, and researchers within orthopedics. In addition, we described and reflected on the ensuing research process. Furthermore, we provided a checklist and guidance in the decision-making process of an HRA. This helps establish interactions and dialogue between multi- and inter-disciplinary research groups and stakeholders at national and international levels to improve the research process following an HRA.

Setting

We used the reports of NOV’s first 2 HRAs to analyze the decision-making process (7,8). We analyzed the following research process using information from CORE (Collaboration in Orthopaedic Research, a network for initiating and conducting scientific research in orthopedics in the Netherlands) and the NOV’s research coordinator. We focused on the preparatory actions and the HRAs’ establishment using the methodological steps described in the Advisory Report Health Care Evaluation (3) as a guideline. Table 1 explains the terms used.

Table 1. Definitions of the terms used, various points of action, and phases of the development of an health research agenda

Health research agenda (HRA)
	An HRA is also referred to as a knowledge agenda or research priority setting and can help set research priorities within a medical discipline. It includes an action plan describing how the evidence uncertainties can be solved through clinical research and may also include evidence uncertainties in healthcare organizations and innovations (6)
Advisory board
	An advisory board guides a sustainable and high-quality priority-setting process, varying from providing advice, content review, program advocacy, and assistance to providing support and suggestions for developing an HRA (25)
Stakeholders
	Stakeholders are “individuals, organizations or communities directly interested in the process and outcomes of setting an HRA and high-quality scientific research” (26). They include patients, caregivers, and healthcare professionals with knowledge in the area of the medical specialty
Evidence uncertainties
	Evidence uncertainties are known as knowledge gaps, unanswered questions, or research priorities. The James Lind Alliance (JLA) defines evidence uncertainties as no up-to-date, reliable systematic reviews of research evidence addressing a medical topic in clinical practice (16)
The development process of an HRA
	The development process consists of 3 parts. First is the collection of evidence uncertainties. They are collected by analyzing the existing guidelines, medical literature, and study protocols. Furthermore, opinions among members of the association of medical specialists, members of the patient association(s), and the remaining stakeholders relevant to the scope of the HRA are gathered (13). Second, evidence uncertainties are selected using pre-defined criteria for the final prioritization (15). Finally, in the prioritization phase, evidence uncertainties are selected and listed as the most important to be added to the final HRA
Research process
	A research process consists of a series of steps or actions, such as formulating the research problem, extensive literature survey, developing a hypothesis, preparing the research design, determining sample size, collecting data, execution of the project, analysis of data, hypothesis testing, generalization and interpretation, and preparation of the report or presentation of the results (27)

Preparatory actions

The Figure describes the stages of the HRAs development process, including the research process.

Stages of the development process of a Health Research Agenda (HRA) and the following research process. It starts with the preparatory actions that need to be taken before establishing an HRA, followed by the actions to collect all the evidence uncertainties and the phases to reduce this to a manageable and researchable set of the most important evidence uncertainties in a prespecified area of interest. We included the phases of the research process following the HRA to highlight the different parts of the process to assess health interventions. We refer to Table 2 (see Appendix) for a detailed description of recommendations for setting up an HRA, including the requirement before undertaking the HRA (green boxes).

Step 1 starts with composing an advisory board to guide development. Participation of individual members should be independent, without burden, consultation, or relevant competing interests (3). The literature suggests including advisory board members with different backgrounds to ascertain various points of view (9), with a balance between academia/non-academia, specialist training/participation of residents, and stakeholders beyond the medical field (3,10).

The advisory board of HRA-1 consisted solely of orthopedic surgeons. For HRA-2, the board consisted of 7 orthopedic surgeons, 1 orthopedic resident, and 1 senior researcher. Both HRAs had face-to-face meetings, conference calls, and communication by e-mail. A Federation of Medical Specialists (FMS) consultant supported the decision-making process. The advisory board largely controlled the decision-making process, offered advice, and decided on the final HRAs. Neither patient associations nor other stakeholders were involved as board members in these HRAs

Step 2 is defining, forming, and involving a stakeholder group. Stakeholder engagement is crucial to ensure that evidence uncertainties align with patients’ and physicians’ needs and express the importance of study participation (5,11-13). Insight into stakeholders’ attitudes and perceptions improves the priorities set by institutions and healthcare organizations (14). Equal numbers of representatives from all key stakeholder groups increase transparency (6) and a diversity of evidence uncertainties (11). Increased response rate and engagement minimizes the chance of overlooking evidence uncertainties (10).

Within both HRAs, the orthopedic surgeons’ and residents’ involvement was significant (HRA-1: 32/49; HRA-2: 37/64). The patient’s influence seemed minimal (HRA-1: 2/49; HRA-2: 5/64). Other stakeholders (HRA-1: 15/49; HRA-2: 22/64) included various medical specialties (sports medicine, primary care physicians, emergency medicine, and geriatric medicine), researchers, a health insurance company, the Health and Youth Care Inspectorate, and the National Healthcare Institute.

Step 3 is to define and formulate the evidence uncertainties’ field, focus, scope, and broadness. Are they oriented in a specific healthcare problem, disease, subspeciality, department within a healthcare facility, or specific patient group? The characteristics of a researchable question must be defined, and consideration given to using the PICO (Population of interest, Intervention, Control, Outcome) or FINERMAPS criteria (feasible, interesting, novel, ethical, relevant, manageable, narrow, multidisciplinary, subjective, focused, specific, precise, or complex). Well-formulated research questions with a short and clear explanation are more likely to be successful. Reflecting on this was beyond the scope of our study.

Development process

Step 4 is the development process, which starts with collecting evidence uncertainties by analyzing the existing guidelines and current literature. Gathering evidence uncertainties from a diversity of contributors is advisable (13). Contributors can be members of the association of medical specialists, the patient association(s), and the remaining stakeholders.

In both HRAs, evidence uncertainties were collected by analyzing existing guidelines. NOV members, steering groups, and other stakeholders were asked to indicate the evidence uncertainties encountered in daily practice. In HRA-2, an overview of measures taken after HRA-1 was analyzed by an inventory of the ongoing research processes and added to the list of the collected items.

Step 5 is the point where the collected evidence uncertainties are selected by categorizing the submissions, grouping the duplicates or similar questions, and creating a summary. The aim is to retain the sense of what the respondent meant but in the form of a clear, well-formulated, researchable question. Defining criteria to select and prioritize research questions is essential and optimizes the prioritization process (15). Various literature suggests the following criteria: magnitude and urgency of the health problem, the present level of evidence, the impact of the research: (in)directly and long and short term, the feasibility of carrying out the research (technical, economic, political, sociocultural, ethical) and to combine the same or similar responses (12,16). Decision-making based on consensus awareness of arbitrary choices is recommended (13).

HRA-1’s list of collected evidence uncertainties was reduced by the advisory board using the following criteria: duplicates, the present level of knowledge or evidence, clinical relevance, researchability, and ongoing research. Decisions were made based on consensus awareness of arbitrary choices. Evidence uncertainties were divided into 7 orthopedic themes. In HRA-2, the advisory board divided the list into 12 themes. In addition, the Working Group Orthopaedics and Sciences (WOW)—consisting of researchers in the field of orthopedics—were consulted to assist. The criteria for selecting evidence uncertainties in HRA-2 differed somewhat from HRA-1: duplicates, the present level of knowledge or evidence, quality of the research question, the topic was not orthopedically related, not an evaluation or innovation question, ongoing research, and feasibility. 2 board members per sub-group assessed and selected the final evidence uncertainties for the prioritization process. It is unclear whether the criteria and motivations were evidence-based.

Step 6 is prioritizing the selected evidence uncertainties. Prioritizing is typically a “subjective,” “interpretative,” and “implicit” process (12). Experts assess the desirability of evidence uncertainties by discussing them interpretatively (1), traditionally an informal process led by power and influence. Several explicit criteria and systematic models have been developed recently (12), and a methodological, transparent, and anonymous approach is recommended to achieve an objective and consensus-based result using a predetermined, anonymous voting system.

In both HRAs, the advisory board led the prioritization process in a physical meeting using dot voting, which is not a specific evidence-based voting system. Anonymity cannot be guaranteed and might dilute the results.

Step 7 are finalizing and implementing the HRA. A widely accepted HRA increases the complexity of prioritizing research, particularly within medical specialties with various topics (17). Due to stakeholders’ differing priorities, an extensive list of evidence uncertainties should be considered (10).

In both HRAs, 10 evidence uncertainties were selected, excluding some subspecialties. Neither HRA was set up by an evidence-based method. The quality and generalizability of the prioritized evidence uncertainties can therefore be biased.

Research process

Examining the research process following the HRA can help assess the characteristics and quality of prioritized evidence uncertainties (18). Good questions do not necessarily produce good research, but poorly constructed questions negatively affect all subsequent study stages (19).

We evaluated the HRA-1’s research process at the end of 2021, allowing 7 years for the follow-up. The NOV aimed for at least 5 of the prioritized evidence uncertainties to be published in a leading scientific journal within 5 years. 3 evidence uncertainties of HRA-1 entered the empiric research phase, and 6 failed the feasibility assessment. 2 research protocols were published; however, there were no publications on results, dissemination into practice, or integrations into guidelines. 3 evidence uncertainties of HRA-2 entered the empirical research phase, 1 submitted a grant proposal, and 3 were assessed for feasibility. 1 research protocol was published. At the time of our review, there were no publications on results, practice dissemination, or guideline integrations. Due to the short period between HRA-2 and our evaluation, we could not assess the influence of the measures taken after HRA-1.

Checklist

The checklist guides the HRA’s development (Table 2) and is based on our overview, underlined with several articles that report on health research priority settings (10,20-24). Although this overview was performed in the Netherlands within orthopedics, it can contribute to a debate on establishing an HRA and its research process internationally.

Limitations

Not all information was documented and recalled. Specific information to reflect on the decision-making process may be missing.

Disclosures

The authors, MJGM, RGHHN, and RWP, were involved in the decision-making process in one or both HRAs, so caution should be paid to their independence in the evaluation. Completed disclosure forms for this article following the ICMJE template are available on the article page, doi: 10.2340/17453674.2023.12402

Perspective

We highlighted the importance of defining criteria for selecting and prioritizing evidence uncertainties to guide high-quality research questions’ quality, characteristics, and formulation. We underlined the importance of formulating a broad or narrow research question representing the field, focus, and scope. Consideration should be given to focusing on broad-ranging, complex problems to avoid narrowing down and decrease the possibility of altering this depending on the most recent developments in health research. Analysis of the impact on the research process goes beyond the scope of our review. We recommend further research to increase our understanding to implement evidence uncertainties in a successful research program to improve high-quality patient care. After analyzing both HRAs, we suggest using an analytical, anonymous, and transparent prioritization method, like the Delphi method, to reach a widely supported HRA.

1. Andronis L. Analytic approaches for research priority-setting: issues, challenges and the way forward. Expert Rev Pharmacoecon Outcomes Res 2015; 15(5): 745-54. doi: 10.1586/14737167.2015.1087317.
2. Brouwers M C, Kho M E, Browman G P, Burgers J S, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010; 182(18): E839-E42. doi: 10.1503/cmaj.090449.
3. Dutch Association of Medical Specialists, health care evaluation steering committee. Advisory report. Health care evaluation: from project to process. Utrecht; 2016. p. 48.
4. Brouwers M C, Kho M E, Browman G P, Burgers J S, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010; 182(18): E839-42. doi: 10.1503/cmaj.090449.
5. Abma T A, Pittens C A C M, Visse M, Elberse J E, Broerse J E W. Patient involvement in research programming and implementation. Health Expect 2015; 18(6): 2449-64. doi: 10.1111/hex.12213.
6. Caron-Flinterman J F, Broerse J E W, Teerling J, van Alst M L Y, Klaasen S, Swart L E, et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Science and Public Policy 2006; 33(4): 291-304. doi: 10.3152/147154306781778993.
7. Dutch Orthopaedic Association, steering committee Health Research Agenda. Dutch Health Research Agenda orthopaedics: towards value-based patient care. ’s-Hertogenbosch: Nederlandse Orthopaedische Vereniging; 2015. p 43.
8. Dutch Orthopaedic Association, steering committee Health Research Agenda. Health Care Evaluation Orthopaedics: On the way to value based health care. ’s-Hertogenbosch: Nederlandse Orthopaedische Vereniging; 2019. p 31.
9. Chambers N, Harvey G, Mannion R, Bond J, Marshall J. Towards a framework for enhancing the performance of NHS boards: a synthesis of the evidence about board governance, board effectiveness and board development. Southampton, UK: NIHR Journals Library; 2013. doi: 10.3310/hsdr01060.
10. Viergever R F, Olifson S, Ghaffar A, Terry R F. A checklist for health research priority setting: nine common themes of good practice. Health Res Policy Syst 2010; 8(36): 9. doi: 10.1186/1478-4505-8-36.
11. Abma T A. Patients as partners in a health research agenda setting. Eval Health Prof 2016; 29(4): 424-39. doi: 10.1177/0163278706293406.
12. Sassi F. Setting priorities for the evaluation of health interventions: when theory does not meet practice. Health Policy 2003; 63: 14. doi: 10.1016/s0168-8510(02)00061-1.
13. Burgers J S, Wittenberg J, Keuken D G, Dekker F, Hohmann F P, Leereveld D, et al. Development of a research agenda for general practice based on knowledge gaps identified in Dutch guidelines and input from stakeholders. Eur J Gen Pract 2019; 25(1): 19-24. doi: 10.1080/13814788.2018.1532993.
14. Sibbald S L, Singer P A, Upshur R, Martin D K. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res 2009; 9(1): 43. doi: 10.1186/1472-6963-9-43.
15. Ulehake M J. Criteria and methods for prioritizing knowledge gaps in healthcare setting in western countries: a systematic review. Nijmegen: Radboud UMC; 2018. p 16.
16. Cowan K, Oliver S. The James Lind Alliance guidebook. 10th ed. Alliance TJL, editor. Southampton, UK: National Institute for Health Research; 2021. p 96.
17. Hoddinott P, Pollock A, O’Cathain A, Boyer I, Taylor J, MacDonald C, et al. How to incorporate patient and public perspectives into the design and conduct of research. F1000Res 2018; 7: 752. doi: 10.12688/f1000research.15162.1.
18. Chalmers I, Bracken M B, Djulbegovic B, Garattini S, Grant J, Gülmezoglu A M, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383(9912): 156-65. doi: 10.1016/S0140-6736(13)62229-1.
19. Agee J. Developing qualitative research questions: a reflective process. Int J Qual Stud Educ 2009; 22(4): 431-47. doi: 10.1080/09518390902736512.
20. Tong A, Synnot A, Crowe S, Hill S, Matus A, Scholes-Robertson N, et al. Reporting guideline for priority setting of health research (REPRISE). BMC Med Res Methodol 2019; 19(1). doi: 10.1186/s12874-019-0889-3.
21. Jongsma K R, Milota M M. Establishing a multistakeholder research agenda: lessons learned from a James Lind Alliance Partnership. BMJ Open 2022; 12(5): e059006. doi: 10.1136/bmjopen-2021-059006.
22. Rudan I, Chopra M, Kapiriri L, Gibson J, Ann Lansang M, Carneiro I, et al. Setting priorities in global child health research investments: universal challenges and conceptual framework. Croat Med J 2008; 49(3): 307-17. doi: 10.3325/cmj.2008.3.307.
23. Aslam S, Emmanuel P. Formulating a researchable question: a critical step for facilitating good clinical research. Indian J Sex Transm Dis AIDS 2010; 31(1): 47-50. doi: 10.4103/0253-7184.69003.
24. Ratan S K, Anand T, Ratan J. Formulation of research question: stepwise approach. J Indian Assoc Pediatr Surg 2019; 24(1): 15. doi: 10.4103/jiaps.JIAPS_76_18.
26. Deverka P A, Lavallee D C, Desai P J, Esmail L C, Ramsey S D, Veenstra D L, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res 2012; 1: 181-94. doi: 10.2217/cer.12.7.
27. Gostin L O, Levit L A, Nass S J. The national academies collection: reports funded by national institutes of health in beyond the HIPAA privacy rule: enhancing privacy, improving health through research. Washington (DC); 2009. p 320. doi: 10.17226/12458.

Appendix

Table 2. Recommendations and attention points for setting up an HRA, including the preparatory measures to be taken before setting up an HRA. For abbreviations, see Footnote

PREPARATORY ACTIONS

Formation of an advisory board Describe the characteristics of the advisory board members  Institutional affiliations, country or region, demographics (e.g., age, sex), discipline, experience, expertise Consider the involvement of an external consultant Increase the independence and confidence in a transparent outcome They can support and advise in the decision-making process Describe the extent of responsibilities during the development and management of the process Describe the strategy or method for identifying, engaging, and recruiting the advisory board members Sources: community of healthcare specialists, patient associations, partnership with the board of medical specialists, recruitment through hospitals using a printed charter and recruitment letter Considerations Participation of individual members should be independent, without burden, consultation, and relevant competing interests A diverse group of advisory board members with expertise and interest in the final HRA is recommended

Defining and forming the stakeholder group Describe the characteristics of the stakeholder group Demographic characteristics, areas of interest and expertise, discipline, affiliations Examples: Clinicians, patient (associations), caregivers, researchers, policy makers, funders, healthcare professionals, members of the community, including specific groups (vulnerable and marginalized populations), (non)governmental organizations, and industry Describe the extent of responsibilities during the collection, selection, and prioritization process Indicate the number of stakeholders involved Describe strategy or method for identifying, engaging, and recruiting of stakeholders Sources: community of healthcare specialists, patient associations, partnership with the board of medical specialists, recruitment through hospitals using a printed charter and recruitment letter Considerations A diverse group of stakeholders with interest and expertise in the final HRA is recommended to increase support in the final HRA. Decide on the possibility of assigning a different respondents’ vote weight Committed and well-informed stakeholders are more likely to contribute to the prioritization process

Defining the characteristics of the evidence uncertainties Define the scope of the health area, field, focus or scope of the evidence uncertainties Oriented on a specific healthcare problem, disease, subspeciality, department within a healthcare facility, or specific patient group Formulate a researchable question regarding the identified evidence uncertainty Define the characteristics of a researchable question and consider using the PICO and FINERMAPS criteria for developing a research question. PICO: Population of interest, Intervention, Control, Outcome. FINERMAPS: Do you want the question to be feasible, interesting, novel, ethical, relevant, manageable, narrow, multidisciplinary, subjective, focused, specific, clear, complex? Considerations Well-formulated research questions with a short and clear explanation representing the evidence uncertainty are more likely to be successful in a research process Define whether the uncertainty of evidence is a broad or narrow-based question

DEVELOPMENT PROCESS OF THE HRA

Collection of evidence uncertainties Describe methods for collecting evidence uncertainties Systematic reviews, reviews of guidelines/other documents, health technology assessment, cost-effectiveness analysis or cost-benefit analysis, evidence mapping, trial registries, feedback from former established HRAs, impact analysis of previous HRAs, consultation experts ((Delphi) survey, interviews, focus groups, public health needs Determining scope of the outcome of the HRA Broad list to detailed list, mono mono-/multidisciplinary, long long-term or short term goals. Define the responsible party for collecting the evidence uncertainties Advisory board or external consultant Considerations Pay attention to response rate, engagement and influence of the different stakeholders involved Pay attention to the number of evidence uncertainties that the different stakeholder groups suggest

Selection of evidence uncertainties Describe methods to refine or translate the collected evidence uncertainties into research topics or questions questions Describe reasons for modifying (removing, adding, reframing) evidence uncertainties Examples of selection criteria: magnitude health problem, disease burden, cost-effectiveness, present level of knowledge, resource flows, equitability, sustainability, ethical aspects, local research capacity, public health benefit, feasibility, cost, relevance, innovativeness, ongoing research, possibilities for implementation, urgency, (social) impact Describe methods for collating and categorizing evidence uncertainties Taxonomy or other framework Define the responsible party for summarizing, editing and preparing the list of evidence uncertainties for the prioritization process (advisory board or external consultant) Define the number of evidence uncertainties for the prioritization process Considerations

Prioritization of evidence uncertainties Decide the method for deciding on the HRA Ensure a repeatable, validated, iterative, transparent, legitimized, and fair method Examples: COHRED; PSP (James Lind Alliance); CHNRI; REPRISE; CAM tool; Delphi method; nominal group technique; interviews; focus groups; meetings; workshops Consensus based, metrics based, or a combination Define criteria to consider and focus on developing strong research questions for prioritization Examples of prioritization criteria: magnitude of a health problem, likelihood of reducing disease burden, cost-effectiveness, present level of knowledge, current resource flows, the degree of equitability, sustainability, ethical aspects and local research capacity Determine the threshold for excluding research topics/questions Ranking scores, proportions, votes, other criteria Considerations Pay attention to response rate, engagement, and influence of the different stakeholders involved on final result

The final HRA Describe number of evidence uncertainties or topics Define the time frame needed to answer the prioritized evidence uncertainties Interim, short-term, long-term Define the time frame for follow-up in the research process of the evidence uncertainties Define the time frame for publishing and/or implementing the research results of the evidence uncertainties Define the time frame to revise and/or update the HRA Outline the strategy or action plans for implementing the HRA in the actual research process Identify (responsible) parties for research process (funders, researchers, sponsor, sponsor’s legal representative, chief investigator, principal investigator, data controller, research participants, policymakers, industry) Considerations Transformation of the broad list of research priority areas into a research portfolio Plans to revise and update the HRA Evaluate whether the prioritized questions were answered

IMPLEMENTATION AND EVALUATION OF THE HRA

Implementation of the HRA Outline the strategy or action plans for implementing the HRA in the actual research process Identify (responsible) parties for research process Funders, researchers, sponsor, sponsor sponsor’s legal representative, chief investigator, principal investigator, data controller, research participants, policymakers, industry) Considerations Transformation of the broad list of research priority areas into a research portfolio

Evaluation of the HRA and feedback Describe method to evaluate and gather feedback Describe methods for checking whether the set evidence uncertainties have been answered Systematic reviews; evidence mapping; consultation with experts, funders, patient (associations) and policies; evaluation research process of prioritized evidence uncertainties; publications; updates and implementation guidelines; evaluation of latest research Define criteria to evaluate the HRA Interface between the health systems; policy-making; research transformed into actual and measurable improvements in people people’s health, outcome and impact Formulation and discussion of implications for future establishment of HRAs Considerations Setting an HRA is a long long-term and iterative process, evaluation provides valuable information and increases the quality of the prioritized evidence uncertainties

Abbreviations: COHRED: priority setting guide of the Council on Health Research for Development; PSP: Priority Setting Partnerships; CHNRI: Child Health Nutrition Research Initiative; REPRISE: REporting guidelines for PRiority SEtting of health research; CAM tool: the Combined Approach Matrix tool.