Explore select projects
See human-centered decision making in action.
Visit our Insights page to learn more about our expertise and approach to our work.
An Observational Study of the Patient-Centered Outcomes Research Institute’s Switch from In-Person to Virtual Research Application Review Panels During the COVID-19 Pandemic
The Patient-Centered Outcomes Research Institute (PCORI) reviews research applications using a process that includes preliminary review by primary reviewers, followed by panel discussion where primary reviewers and other reviewers on the panel (panelists) discuss and score the most competitive applications. PCORI switched from in-person to virtual panel discussions during the COVID-19 pandemic. This study compared reviewers’ scores and perceptions of panel interactions for virtual and in-person review panels and assessed reviewers’ perspectives about virtual panels. Overall, this study suggests that virtual and in-person review panels were largely comparable on reviewer scores and some key aspects of reviewer experiences in a multi-stakeholder review process. Despite having some challenges, virtual review processes may be a viable approach for funders during times of necessity and beyond. Practical considerations and recommendations for virtual panels are discussed.
Definitions, Methods, and Strategies to Examine Engagement: Progress and Future Directions for Research on Evaluation
Engagement is regarded as a core practice that can enhance the quality, utility, and relevance of evaluation. However, more research is needed to provide empirical guidance on designing effective engagement approaches for various communities. Expanding on a prior systematic review, this scoping review examined the current empirical landscape and knowledge gaps of RoE on engagement to (a) identify how engagement has been defined to clarify key concepts, (b) describe methodological and contextual characteristics of studies used to examine engagement, and (c) characterize engagement strategies utilized. Twenty-seven studies were included and coded for synthesis. Based on the review's findings, recommendations to drive future RoE agendas should (a) develop standards for empirical examinations of engagement, (b) determine conditions for successful partner selection, (c) measure engagement outcomes, and (d) apply innovative RoE methods and designs to broaden the empirical basis. Advancing these focus areas can address persistent gaps in RoE on engagement and strengthen the impact of evaluations that can foster meaningful community collaborations.