Explore select projects
See human-centered decision making in action.
Visit our Insights page to learn more about our expertise and approach to our work.
An Observational Study of the Patient-Centered Outcomes Research Institute’s Switch from In-Person to Virtual Research Application Review Panels During the COVID-19 Pandemic
The Patient-Centered Outcomes Research Institute (PCORI) reviews research applications using a process that includes preliminary review by primary reviewers, followed by panel discussion where primary reviewers and other reviewers on the panel (panelists) discuss and score the most competitive applications. PCORI switched from in-person to virtual panel discussions during the COVID-19 pandemic. This study compared reviewers’ scores and perceptions of panel interactions for virtual and in-person review panels and assessed reviewers’ perspectives about virtual panels. Overall, this study suggests that virtual and in-person review panels were largely comparable on reviewer scores and some key aspects of reviewer experiences in a multi-stakeholder review process. Despite having some challenges, virtual review processes may be a viable approach for funders during times of necessity and beyond. Practical considerations and recommendations for virtual panels are discussed.
Engaging Policy Stakeholders to Inform a Future National Secure Data Service
This project seeks to identify the data needs of federal policy stakeholders as future users of a National Secure Data Service using a human-centered design approach. It will conduct a landscape analysis of the data needs within the federal policy ecosystem and conduct a detailed case study with the National Science Board. This project will result in recommendations for the navigation and data concierge services needed by policy stakeholders and a prototype service framework or policy toolkit.
Definitions, Methods, and Strategies to Examine Engagement: Progress and Future Directions for Research on Evaluation
Engagement is regarded as a core practice that can enhance the quality, utility, and relevance of evaluation. However, more research is needed to provide empirical guidance on designing effective engagement approaches for various communities. Expanding on a prior systematic review, this scoping review examined the current empirical landscape and knowledge gaps of RoE on engagement to (a) identify how engagement has been defined to clarify key concepts, (b) describe methodological and contextual characteristics of studies used to examine engagement, and (c) characterize engagement strategies utilized. Twenty-seven studies were included and coded for synthesis. Based on the review's findings, recommendations to drive future RoE agendas should (a) develop standards for empirical examinations of engagement, (b) determine conditions for successful partner selection, (c) measure engagement outcomes, and (d) apply innovative RoE methods and designs to broaden the empirical basis. Advancing these focus areas can address persistent gaps in RoE on engagement and strengthen the impact of evaluations that can foster meaningful community collaborations.
Engagement in Evaluation
For any program, there are various individuals and organizations who have an interest in its existence, resources, services, implementation, outcomes, and/or impact. Engaging diverse individuals in evaluation design, conduct, and dissemination is critical to incorporating context and perspectives that allow evaluation findings to be more useful, meaningful, and trustworthy. In turn, evaluation can both better serve the needs and interests of the people who are most affected and also lead to program and policy improvement. Engagement is not unique to evaluation research. In fact, the body of literature on engagement science and practice is rich in adjacent fields. We draw on our experiences as professional evaluators who have incorporated engagement in our evaluative activities, and as researchers who have studied the science and practice of engagement in other contexts, to better equip evaluation practitioners to incorporate robust engagement in their evaluation work.