June 2018
Laura C. Leviton, Ph.D., Senior Adviser for Evaluation, Robert Wood Johnson Foundation
Each year, GIH asks health funders to share their thoughts on our annual conference theme. This year, we’ve asked evaluators from foundations to reflect on our 2018 conference theme, Navigating Currents of Change.
Twenty-seven years ago, Will Shadish, Tom Cook, and Iasserted that the primary—if not the only— justification for evaluation is its usefulness. Yet how can evaluations be useful if they are mostly looking backwards? In 2018, rapid social change requires philanthropy to be nimble, to anticipate challenges and opportunities, and to be able to adjust its course. At the Robert Wood Johnson Foundation (RWJF), the challenge is even greater, given RWJF’s time horizon for building a Culture of Health. As stated on our website: “Our vision is that we, as a nation, will strive together to build a Culture of Health enabling all in our diverse society to lead healthier lives, now and for generations to come.” This is a long-term aspiration for which the ultimate outcomes will not be seen for decades. What to do?
These days, RWJF is using more leading indicators in evaluation, those that predict future performance. In contrast, most outcome and many process measures are what is termed lagging indicators—measures of past performance that are difficult to use for course adjustments in the near term. We are not abandoning evaluations of outcomes—not at all, as we still need high quality program evaluation to build wisdom over the longer term. But evaluation using leading indicators will help RWJF anticipate challenges and adapt strategy to rapid social change.
- In a logic model, leading indicators often include measurable activities, outputs, or intermediate outcomes that are closer to decision points in time than measuring the final goal. But they are not the equivalent of “a million burgers sold.” The ones that matter will help us predict what happens next—we care about trends, not single time points.
- In a theory of change, leading indicators often include trends in the mechanisms that are supposed to drive changes: “X is increasing so that Y can be achieved.”
Evaluation of progress in building a Culture of Health will focus on three kinds of leading indicators: signs of progress on powerful levers of change; understanding people’s mindsets; and evaluating ways to move from “retail” to “wholesale” social change.
Powerful levers of change. RWJF is seeking to make positive changes in the upstream social and economic determinants of health. We believe that specific improvements in policies, systems, and environments (PSE) offer the most leverage. Some PSE changes are directly in line with health, such as integrating health and social services for vulnerable populations, or planning cities that are inclusive and activity friendly. We have learned that other less conventional PSE changes affect health too, such as affordable housing. RWJF will work directly in some of these areas, and we hope that partners will address other issues, such as improving relations between marginalized communities and the police. We need evidence about these PSEs. RWJF’s four “pillar” research programs are building that evidence, just as the MacArthur Foundation did with affordable housing and the Wallace Foundation with social emotional learning in schools.
Information is never enough to enact policy change, however. RWJF supports advocates of healthful policies at federal, state and local levels. RWJF evaluated advocacy related to coverage of the uninsured and childhood obesity prevention. On that basis, we can propose some leading indicators of progress on advocacy: the measured capacity of community coalitions, the quality of technical assistance to advocates, Power Analysis© for community organizations, and various ways to take the “temperature” of state legislators and regulators through surveys, participant observation, and relationship-building. Policy advocacy offers a classic case of effort that is not linear—it operates in an adaptive system, but its many complexities can be monitored and course corrections can be offered. It can be evaluated!
Understanding mindsets. The Action Framework for building a Culture of Health includes working to ensure that each person’s health is valued by all. The framework also emphasizes health equity in everything RWJF does. As described in Habits of the Heart, the United States is in the midst of a dialectic between narratives about individual responsibility (“you are on your own”) and reliance on community (“we are in this together”). Information will not matter to action or attitudes unless it first deals with these narratives. RWJF commissioned work from NORC to further understand the various mindsets that Americans bring to the issues related to health. In addition, we have conducted substantial message pretesting and harvested media messages in various ways to examine trends in people’s attitudes that may offer opportunities for PSE changes. At least on many children’s issues, work in press by Liana Winett and colleagues indicates that a cooperative, communitarian mindset prevails – which is a leading indicator for change.
Moving from retail to wholesale social change. In recent decades, RWJF supported many demonstration projects on practice innovations—what former Executive Vice President Jim Marks termed “retail social change.” However, many of the innovations were complex and multifaceted, meaning that they were less likely to be adopted by others with fidelity. And although the innovations were often sustained for a few years by a few organizations, there was little impact on population health.
Today, RWJF tries for “wholesale” social change by motivating action in many places, then guiding action through technical assistance of many kinds, and working toward sustainability. For the most part, these are changes at an organization, community, or state level, not an individual level. We are in the process of developing leading indicators of communities’, organizations’, and states’ motivation to act, and their ability to use technical assistance. Abe Wandersman and his colleagues have shown how a measure of readiness (developed with RWJF support) predicts whether communities and organizations will adopt an innovation successfully. The format of technical assistance clearly matters to uptake—it needs to be geared to the complexity of the desired changes and the organization’s capacity. Finally, there are key predictors of sustainability.
What we are really trying to do with the Culture of Health, of course, is to help inspire a social movement. We do not need to be in charge of or brand it. We will happily partner with those of like mind, and we do. To the readers of this essay I would pose two questions:
- What are some leading indicators of successful partnerships? The literature on business partnerships offers some useful candidates, including articulating a shared value proposition and multiple relationships between the staff of the partners at various levels and functions.
- What are some leading indicators of successful social movements? Persistence and durability? Their appeal to the public? Turnout at the ballot box? Are there key lessons philanthropy has learned from its support of past movements?