Foundation Operations: Evaluation-> Program Evaluation-> Evaluation Basics
Learning from Your Grantmaking: The Basics of Program Evaluation
All evaluations begin with questions – questions for which the foundation wants answers. Ideally, an evaluation will only have a few questions, so think about the most important things you want to know and pose the questions. Do you want to know if one program you are funding is more effective in reducing childhood obesity than another? Or do you want to understand if your current method of grantmaking is the most effective for solving a specific problem? These questions will shape the work of the evaluation consultant, so only ask what you need to know.
There are two types of evaluation: process (or implementation) evaluation and outcomes evaluation. Some funders want to understand a process rather than a specific outcome. For example, a program that gives money to communities to identify and solve a specific problem might pose an evaluation question such as, “How do the communities come together and identify the program and work together to try and solve it?” This type of question will result in a very descriptive evaluation report but may not address whether a community solved the problem it identified (which can be accomplished by conducting an outcomes evaluation and looking at the indicators), so be very careful in framing the questions and be as clear and specific as you can. The most robust evaluation process will incorporate both types of evaluation.
Many grantmakers use a request for proposals (RFP) or request for qualifications (RFQ) to select an evaluation consultant. The RFP or RFQ can be released publicly or sent to selected evaluators. In response to an RFP, an evaluator submits a proposed plan for the evaluation; with an RFQ, the evaluator submits his proposed team’s qualifications to conduct the evaluation. Some funders find an RFQ preferable to an RFP because an RFQ allows them to work with the evaluator to create the evaluation plan. Some funders do not include a budget but ask the evaluators to set a cost for their proposed work. Others have a budget in mind and ask the evaluators to plan their work for that cost. Some grantmakers use a percentage of the total program cost; often 10 percent is given as the percent of total budget cost for evaluation. Using a competitive process to select an evaluator helps ensure fairness and also provides a diverse pool of evaluators from which to select.
The following pieces of information in an RFP or RFQ are very important to evaluators in determining whether they have the capacity to do the work:
description of the initiative or program to be evaluated, including goals and anticipated outcomes;
purpose of the evaluation (for example, to evaluate and learn about an implementation process, to determine the program or initiative’s outcomes, to help the foundation refine its grantmaking strategies);
timeframe for the initiative or program and for the evaluation;
number of grantees or sites, if applicable;
primary and secondary audiences for the evaluation;
ways in which the evaluation findings will be used; and
budget (this is preferable for evaluators to help them determine if the design is feasible within the available resources).
Innovation Network, "Glossary: Nonprofit Planning & Evaluation" (Washington, DC: 2005). The glossary provides definitions of terms related to the evaluation process.
Innovation Network, "Point K Tools and Resources." The Point K Learning Center provides tools and resources to build nonprofits' ability to plan and evaluate their own programs. Recommended readings include:
The California Wellness Foundation, Evaluations and Lessons Learned from our Grantmaking. This is a Web-based series in which foundation staff, grantees, and contractors share lessons learned and information gleaned from grantmaking programs and strategies. The foundation presents these publications three or four times a year.
The California Wellness Foundation, Reflections: On Evaluating Our Work (Woodland Hills, CA: 2004). This issue of Reflections discusses the evaluation experiences of The California Wellness Foundation, organizational milestones for evaluation, specific grants that exemplify aspects of evaluation grantmaking, and some general conclusions about foundation evaluation.
Robert Wood Johnson Foundation, “A Guide to Evaluation Primers.” This report is a guide to various handbooks and primers on program evaluation and examines areas such as collaborative evaluation, in-house evaluation, and independent evaluation. Available at http://www.rwjf.org/files/publications/RWJF_ResearchPrimer_0804.pdf