Measuring the impact of community development: A conversation with Paul Mattessich of Wilder Research

Community development practitioners want to be more effective. Their funders want to gauge the value of their investments. Policymakers want to demonstrate cost savings. And everyone wants to know: How did the community really benefit?

The ability to measure results has become increasingly important for community development organizations and other entities that rely on government and philanthropic funding to carry out their work. Those of us in the community development field know our work is valuable, but how can we effectively measure its impact? How can we keep the costs and complexities of a measurement system from getting out of hand? And how does our work affect broader community trends? To explore these questions, Community Dividend spoke with Paul Mattessich, Ph.D., executive director of Wilder Research.

A division of the Amherst H. Wilder Foundation in St. Paul, Minn., Wilder Research is one of the largest and most prominent nonprofit research institutions in the United States. Since 1917, Wilder Research has conducted hundreds of studies on community conditions and the effectiveness of health and human service programs. Mattessich has headed Wilder Research since 1982. In addition to leading projects and studies, he frequently lectures and writes on the topics of social trends, community building, and organizational effectiveness.

Community Dividend: You've led a number of projects at Wilder Research that involve measuring program effectiveness and impact. What does it mean to measure the impact of community development work, and why is it an important thing to do?

Paul Mattessich: To understand the term "measurement of community development," it's important to understand the difference between measuring the process of development and measuring the outcomes of development. In some cases, measurement involves simply tracking an organization's activities, such as the number of individuals it trained. That is process. Measuring outcomes involves tracking whether a program or initiative actually had the intended effect and whether it improved the community in some way. Did it change residents' behavior? Did it produce any social or economic benefits?

Measuring the impact of community development work is important because the issues that the Federal Reserve and community development organizations are working on are truly significant for enabling communities to succeed. Communities need a strong economic infrastructure, and the only way to know if we've built it is to measure whether it's happened. Also, we want to be able to do our work as effectively and efficiently as possible. The only way to achieve that is to measure—to understand what we're doing and where it works and doesn't work, so that we can strive to improve it.

If organizations don't measure their impact, they risk not being able to demonstrate the effectiveness of their programs. They risk not being able to motivate the community by showing people what a difference their programs really make. They risk losing out on the opportunities to bring in more human and financial capital and to identify their strengths and weaknesses. Organizations risk losing a great deal if they don't have good measurement.

CD: What do you think is the best way to measure the impact of a community development program or initiative?

PM: When it comes to measuring impact, I think it's valuable for practitioners and policymakers to think in terms of a logic model. It's a visual model that uses a sequential point of view to measure the results of an effort. It starts out with the inputs that go into a community development effort, such as resources and materials. Then it moves on to activities: What is it that an organization actually did? Then to outputs: the immediate, countable results from those activities. And then to outcomes: What happened as a result of the effort, both in the short term and the long term? Examples of outputs include the number of small business loans granted, the number of housing units built, and the number of residents who attended training seminars. In terms of outcomes, we can ask questions, such as, Did residents' access to retail and services increase? Did access to affordable housing increase? Did crime rates decrease? If we clearly identify our intended outcomes and measure them, we can demonstrate whether our efforts really did have a positive impact.

CD: Given the complexities of community conditions, how can an organization be sure that its effort was truly responsible for an observed change in a community?

PM: That's an important question. If there is a change, was it because of us or because of the many other things happening in the community at the same time? Going back to the logic model framework that I mentioned, outputs are relatively easy to attribute to a specific program. If an organization provides training to a hundred people, you can obviously attribute that to the program. It becomes more complex when we move to outcomes. In most cases, it's possible to establish whether or not a program actually contributed to a particular outcome or set of outcomes, but it requires having a good comparison. By that, I mean that we need to look at several locations that have a given program in place and compare them to similar locations that don't have that program.

For example, let's say we want to measure the impact of a community development initiative in rural communities that have populations under 20,000. We can find other rural communities of similar size and condition that aren't part of the initiative and use them as a comparison. If, after a few years, the communities with the initiative have stronger, higher-capacity organizations, more commercial activity, populations that are declining less rapidly, or whatever the desired outcomes are, we can be reasonably sure that it's because of our community development initiative. And the reason we can be sure is because we have a good, solid comparison.

CD: What about the cost? Is it possible to measure impact without spending a lot of money?

PM: It's important to understand that it does cost money to evaluate impact. There's no way around it. That being said, there are some low-cost, creative ways to do at least certain parts of your evaluation. One way is to avoid the cost of collecting data by using measures that are already being tracked. For many communities, there are existing data on housing conditions, crime rates, work force participation rates, poverty rates, and other demographic characteristics that are available through the U.S. Census Bureau or local government agencies.

Another approach is to identify data that are easily obtainable at very little cost. For example, if the intended outcome is to increase the number of businesses on blocks that have been largely abandoned, it's very easy to walk down those blocks and count up how many storefronts are vacant and boarded up, and then walk down those same blocks one year later and see if that number has changed. A third way to save money is to team up with similar organizations and use a single evaluation or survey. An evaluation that is cost-prohibitive for a single organization can be affordable if three or four organizations are willing to share the cost of the design. A nonprofit can also establish its own logic model by using free or low-cost guides that are published by institutions that do this type of work. That doesn't mean an evaluation can always be done without professional help. Sometimes it does require hiring a social scientist or an economist, but there are many situations where it does not.

CD: A number of states and cities have established community indicators projects. Can you tell readers more about this method of community measurement?

PM: The purpose of community indicators projects is to identify the critical ingredients of community success, measure them, and promote action in order to push social and economic trends in a positive direction. We have a lot of experience working with community indicators at Wilder Research. We're currently responsible for the oversight of a web-based indicators project called Minnesota Compass. It's been under way for over five years now and has about ten focal areas in which we do measurement, including housing, health, education, public safety, and the economy. For each area, we identified three or four key measures that together provide a reading on whether communities in Minnesota are getting better or worse and how the state compares to the U.S. overall.

CD: How does the work of community development organizations relate to those sorts of community measures? And how important is that work when it comes to moving trends in a positive direction?

PM: I'll give you some examples of how it relates. One measure of community well-being is housing affordability, which can be quantified by the number of households that face a housing-cost burden. A household is considered "cost-burdened" if its housing costs equal 30 percent or more of its monthly gross income. Community development efforts that increase the supply of affordable housing can decrease the proportion of cost-burdened households. In the area of public safety, crime rates are a widely used measure. Efforts that bring neighbors together and support small business development can have a positive impact on crime rates and residents' perceptions of neighborhood safety. Small business development can also lead to increased work force participation, which is a key measure of economic health. For example, Wilder Research's evaluation of the Neighborhood Development Center in St. Paul has shown the impact that neighborhood entrepreneurs can have on job creation and other countable measures of economic success. Community development organizations can have a major impact on trends when they target specific locations, identify the strengths and weaknesses of those locations, and then try to build on those strengths and remedy those weaknesses. I think their work is extremely significant, especially at the micro level.

CD: How can community development organizations build on the momentum of existing community measurement efforts?

PM: They can commit to the use of measures that fit within a logic model and demonstrate how the use of those measures over time can lead to continuous improvement. They can choose established, readily available measures that anyone can understand and accept, and use those measures to celebrate progress and motivate people to do even more. Also, organizations in Minnesota can get involved in Minnesota Compass by visiting the project online at www.mncompass.org. In addition, we plan to expand Compass into North Dakota and South Dakota within the next year, and we're approaching our expansion in a way that isn't strictly proprietary. We plan to provide the infrastructure, but community leaders in the Dakotas will contribute to the development of their respective sites and will be empowered to continue Compass in a way that meets their needs locally.

CD: For other geographic areas in the Ninth District, what advice do you have for starting a community indicators project or simply starting a conversation about community measurement?

PM: States all have slightly different characteristics, but each state can apply the common framework that underlies community indicators projects. This framework can work for smaller geographic areas as well. The only exception would be if there is a lack of population that makes the numbers too small to be reliable.

If I were to start from scratch, I'd identify the key stakeholders who have a reason to want the measurement to occur, and then I'd find someone from a local university, extension service, or research organization—whichever is most accessible—who could bring some technical advice to the table. The next steps would be to define what the community wants to achieve, identify how it might measure its progress toward those goals, and determine what's actually feasible in terms of cost.

CD: What would you say is the biggest challenge community groups face when they get together to have this conversation?

PM: I think sometimes there's a tendency for the conversation to become overly complex. My advice would be to keep things simple, focus on a few key measures, and tie those measures to what you feel are the most important goals you want to achieve. Don't try to measure absolutely everything, because you are likely to get bogged down.

CD: Based on your experience, what types of community measures resonate best with funders and policymakers?

PM: First, measures that funders and policymakers have had a hand in developing. No measure is perfect, but people will buy into measures that they feel they own, so whenever possible, you should include these stakeholders in the selection of your measures before you gather your data. Second, the more a measure relates to a positive cost-benefit, the more appealing it will be to policymakers.

CD: Systems for community measurement are constantly evolving. Where do you see the field headed in the next five years?

PM: Data that can be used to measure community conditions are becoming more accessible—not just for professional statisticians and demographers, but for people who work on the ground—and I think that trend will continue. It used to be that we could only get data at the time of the census. Now, data for small geographic areas with populations of less than 20,000 are available annually. Within the next five years, I think it'll be much easier for people to get data for geographies that they define, not just those that are predetermined. Also, I think we'll continue to see organizations gravitate toward using more standardized measures of community well-being.

CD: What would you say to readers who are skeptical about the feasibility of measuring impact or are hesitant to commit resources to a measurement effort?

PM: I'd say that programs or interventions may have some unmeasurable impacts, but it is always possible to measure something. There isn't any community development effort that doesn't have at least some measurable effects, in terms of desired outcomes and in terms of cost-benefit.

The real value of quantitative measurement is that if you can get people to agree ahead of time on what the goals are and what the best measures are, then after you've tried an intervention, everyone will also agree on whether it worked. It puts everyone on the same page in terms of knowing what to expect, what to hope for, and what to work toward. And measuring our outcomes not only allows us to set goals to improve our impacts over time, it helps ensure that we are having the maximum impact for the dollars we're spending. All in all, it just makes great common sense.

For a visual summary of the impact-measurement process, see the diagram below.

Click on diagram to view larger image