The Commonwealth Fund is a private foundation committed to improving the health care system with a focus on the most vulnerable populations. For almost a century, The Commonwealth Fund has been investing and engaging directly in research and policy analysis with an eye toward action. What do we know about what works to improve the complex U.S. health care system? How can we spread what works in one setting to other states, health systems, and provider groups? What can we do to move the policy discussion forward and ensure that they are grounded in evidence?
Given the role of federal and state governments in purchasing and providing health care services, as well as their significant legislative and regulatory authority over the health care system, policymakers at the federal and state levels have always been a key target audience for The Commonwealth Fund’s work. The Commonwealth Fund has made a concerted effort to connect policymakers with key research and analysis.
The Commonwealth Fund’s Federal and State Health Policy initiative (FSHP) was designed to enhance The Commonwealth Fund’s capacity to:
- provide access to evidence-based policy analysis relevant to federal and state policymakers,
- disseminate and share lessons learned in policy and practice,
- enable state-to-state and state-to-federal learning and discourse, and
- incorporate policymaker’s priorities and the broader policy and political landscape to inform The Commonwealth Fund’s programmatic strategy.
The FSHP strategy relies on an orchestrated series of grants targeting critical groups of policymakers, including Medicaid directors, governors, legislators, exchange directors, and insurance commissioners at the state level and both the executive and legislative branch at the federal level. Over the past three years, The Commonwealth Fund has developed strategic partnerships and made recurring grants to national organizations active in all 50 states and U.S. territories: the National Governors Association (NGA) Center for Best Practices, the National Association of Medicaid Directors (NAMD), the National Conference of State Legislatures (NCSL), and the National Academy of State Health Policy (NASHP).
Through various convening, communication, and technical assistance activities, FSHP projects provide federal and state policymakers with access to objective, evidence-based policy analysis, disseminate lessons learned, facilitate learning and discourse both among states and between states and federal officials, and serve as strategic policy resources for The Commonwealth Fund. While the strategies employed by all of our state-focused grantees are similar, each grantee targets a distinct policymaking audience.
The logic model underlying the FSHP work is simple: present data and analysis to policymakers with the expectation that repeated exposure to evidence-based resources will result in informed policymaking.
Simply tracking the number of times a publication has been cited in the media or requested from policymakers does not adequately assess the impact the work may be having on policymakers. Policymaking is rarely logical or linear – an issue may cross a legislator’s desk and then months or years of exploration, conversations, and research may go into the decision to craft legislation. Indeed, assessing the policy impact of projects has proven to be a difficult, but an extremely important, undertaking.
To gauge how The Commonwealth Fund’s work and FSHP dissemination activities contribute to policymakers’ deliberations and work, we track how often our research is cited by policymakers in news articles, government reports, and social media. We also track how often policymakers contact The Commonwealth Fund’s staff with requests for expertise. Finally, we glean insight into FSHP’s effectiveness from evaluations and feedback from those attending FSHP-convened events as well as from our biennial audience surveys.
Because we rely on our grantees to extend The Commonwealth Fund’s reach with policymakers, we equip them with resources to improve their own efforts to assess how their work contributes to policy deliberations. In 2014, the FSHP program convened all of its grantees for an all-day session with an evaluation expert with a specialty in policy-focused work (Beer et al. 2012). The in-person meeting was an invaluable way to bring together organizations with similar, but distinct target audiences and content areas that were actively partnering with The Commonwealth Fund to reach federal and state policymakers. Through our ongoing work with grantees, we were able to workshop scenarios where assessing the policy impact of a project was difficult or impossible and identify assessment strategies that generated more meaningful insight around the impact of a project.
Using the Right Tool for the Right Job
Central to measuring impact or policy contributions is having a clear sense of the outcomes desired and the strategies needed to reach your goal. For example, a project may be trying to build awareness around an issue or build the will to act. Or perhaps the project is looking to spur a specific action. Clarity around the goal will drive the use of appropriate tactics. Also important is articulating the audience whom your strategies will target. We used a framework developed by Tanya Beer and associates at the Center for Evaluation Innovation to help grantees refine their thinking about their target audiences and outcomes desired (Coffman and Beer 2015). Grantees focused on articulating a range of outcomes that could be measured given their strategy.
Modifying Data Collection
As part of the technical assistance process with our grantees, they were introduced to new data collection approaches to get more meaningful, actionable data. First, they were provided with concrete ideas about how to improve their existing event evaluation forms and survey questions. Most data collection efforts focus on output and activity as well as participant/audience satisfaction when what we really want to assess is the attitudes, knowledge, and our target audiences’ will to act on a given issue as a result of supported work. At the FSHP grantee workshop, grantees learned how to modify their evaluation post-meeting forms to gauge their audiences’ intent to use the information they learned in a given meeting. Sample questions include:
- Was the discussion a good match for current political and fiscal realities?
- How likely are you to use the information from this event to inform new or pending policy activities?
- How likely are you going to use what you’ve learned at this meeting to make the case with other stakeholders?
- How likely are you to apply the models presented here in your own state?
New Ideas
In addition to clarifying the right tactics for the right audience and modifying existing data collection efforts, we use other tools to gauge whether the research or resources from a given project are being used by the intended audience. Follow-up surveys and intercept interviews are examples of how The Commonwealth Fund and our grantees are working to assess if participants are more knowledgeable or receptive to an issue or a set of policy solutions. At our annual retreat for members of Congress, for example, we have added intercept interviews to collect more in-depth information about the ways in which the retreat has been helpful and has led to new bipartisan collaborations. These interviews serve as a complement to evaluation forms and follow-up outreach.
Looking Forward
Foundations can play a crucial role in informing the policy process by identifying timely and relevant issues that align with their areas of expertise and by strategically using grantees to extend their reach to this audience. At The Commonwealth Fund, we employ strategies to engage federal and state policymakers and as such, use a number of ways to assess our contributions to policy change. We are continuously working to identify additional ways to strengthen our internal capacity as well as to build grantee capacity around measuring policy contributions. Finally, foundations can support their grantees by providing technical assistance and tools for measuring project impact, as well as financial resources to fund this work.
References
Beer, Tanya, Pilar Stella Ingargiola, and Meghann Flynn Beer. Advocacy & Public Policy Grantmaking: Matching Process to Purpose. Denver, CO: The Colorado Trust, August 2012. https://www.evaluationinnovation.org/sites/default/files/Advocacy%20Public%20Policy%20Grantmaking.pdf.
Coffman, Julia, and Tanya Beer. The Advocacy Strategy Framework: A tool for articulating an advocacy theory of change. Washington, DC: Center for Evaluation Innovation, March 2015. https://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework.pdf.