A process evaluation accompanying an attempted randomized controlled trial of an evidence service for health system policymakers
Wilson, Michael G
Grimshaw, Jeremy M
Haynes, R Brian
Hanna, Steven E
Lavis, John N
Date of Issue2015
Lee Kong Chian School of Medicine
Background: We developed an evidence service that draws inputs from Health Systems Evidence (HSE), which is a comprehensive database of research evidence about governance, financial and delivery arrangements within health systems and about implementation strategies relevant to health systems. Our goal was to evaluate whether, how and why a ‘full-serve’ evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care as compared to a ‘self-serve’ evidence service. Methods: We attempted to conduct a two-arm, 10-month randomized controlled trial (RCT), along with a follow-up qualitative process evaluation, but we terminated the RCT when we failed to reach our recruitment target. For the qualitative process evaluation we modified the original interview guide to allow us to explore the (1) factors influencing participation in the trial; (2) usage of HSE, factors explaining usage patterns, and strategies to increase usage; (3) participation in training workshops and use of other supports; and (4) views about and experiences with key HSE features. Results: We terminated the RCT given our 15% recruitment rate. Six factors were identified by those who had agreed to participate in the trial as encouraging their participation: relevance of the study to participants’ own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors; and participation of colleagues. Most reported that they never, infrequently or inconsistently used HSE and suggested strategies to increase its use, including regular email reminders and employee training. However, only two participants indicated that employee training, in the form of a workshop about finding and using research evidence, had influenced their use of HSE. Most participants found HSE features to be intuitive and helpful, although registration/sign-in and some page formats (particularly the advanced search page and detailed search results page) discouraged their use or did not optimize the user experience. Conclusions: The qualitative findings informed a re-design of HSE, which allows users to more efficiently find and use research evidence about how to strengthen or reform health systems or in how to get cost-effective programs, services and drugs to those who need them. Our experience with RCT recruitment suggests the need to consider changing the unit of allocation to divisions instead of individuals within divisions, among other lessons.
Health Research Policy and Systems
© 2015 Wilson et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.