Measurement and evaluation play an important role in how we at Democracy Fund approach our work. We believe that we have a responsibility to be transparent about the progress we are making against our goals and that we must be open to new information that may point us in new directions. To this end, we use a variety of tools to measure our work and better understand whether and how we are making progress. We also often provide our grantees with evaluation resources to help them take a deeper look at how they can be more effective.
An important part of our approach to evaluation is the idea that accountability, by definition, requires feedback. Our board and advisors provide valuable feedback and direction throughout the year, but we also need to hear from our grantees and partners to gain a more complete picture of how we are doing. In order to ensure that this feedback is as honest and critical as possible, we recently hired the Center for Effective Philanthropy (CEP) to survey our grantees and produce a Grantee Perception Report that assesses Democracy Fund across a wide range of factors. CEP was able to gather anonymous feedback from 25 grantees – 83% of our portfolio at the time.
CEP’s report compared Democracy Fund to 300 foundations as well as to a custom cohort of 16 peer funders. As a relatively young foundation, with a little over three years of grant making under our belts, we were eager to see these first results. There is a lot of data, which we will be exploring periodically here on the blog, but the following three themes stood out.
- Expertise and Impact: Grantees rated Democracy Fund staff highly for understanding grantees’ strategies (77th percentile) and the fields in which we work (82nd percentile). At the same time, we were ranked in the 40th percentile for having an impact on the field. Anonymous comments suggest our expertise is a strong foundation for future success but it’s too early to judge the effects of our work.
- Selection and Evaluation Processes: Grantees find our selection process to be more valuable than most (97th percentile in strengthening their organizations), and the Democracy Fund is in the 84th percentile for grant dollars awarded per hour of application work. Concurrently, grantees feel high pressure to change organizational priorities to receive funding (98th percentile).
- Relationships with Grantees: Overall, Democracy Fund grantees said that we are fair and highly responsive. They also indicated that we provide our grantees with more non-monetary assistance – from strategic planning advice to assistance in securing other funding – than most organizations. Yet, grantees showed they feel low levels of comfort in approaching us with problems during the life of a grant, compared to how grantees feel in approaching other foundations (29th percentile).
While many of the results indicate that Democracy Fund is on the path to the kind of foundation we hope to be, we asked CEP to lead confidential focus groups at our October grantee meeting to explore areas of concern. The focus groups addressed two questions. First, why do our grantees feel less comfortable approaching us with problems, compared to the grantees of other foundations? Second, how can we reduce any burdens associated with our grant selection process? Both focus groups were meant to help us better understand what was going on and hear solutions from our grantees.
The Democracy Fund team left the meeting for these focus groups, so that the conversations could be candid and fruitful conversations. One of the more revealing pieces of information uncovered by the CEP facilitators was the connection between Democracy Fund’s focus on metrics and our grantees’ discomfort in approaching us with problems. It revealed a need for more communication and expectation setting around how we approach, develop, and use metrics in evaluating grants. I’ll explore these findings more in a follow up post, but we are already experimenting with some ways to address this need. For example, we’re piloting a new metrics and reporting template with a few grantees, and we’re thinking about new ways to demonstrate our commitment to smart risk-taking and experimentation. On another front, we’re also sending out our first user survey on our new grants management software.
In 2015, we will review our overall grant making process and while we don’t expect significant changes, it will be a moment to think further about our relationships with grantees and about how we can improve our partnerships while maintaining the practices and values that rank Democracy Fund so highly in other areas.
We would like to thank all our grantees that participated in the survey and offer our thanks to CEP for the thoughtful report. We’re looking forward to continuing the discussion about these results and to having a baseline for future surveys.