Author
In 2007, Greater Twin Cities United Way (United Way) invested $5 million in 36 job training and employment assistance programs. At the same time, the organization launched a years-long project to analyze the impact of the programs. Five years later, the analysis is starting to yield impressive and meaningful results.
As the largest nongovernmental funder of job training and employment assistance in the Minneapolis-St. Paul area, United Way had a strong incentive to understand the impact of its investments. Typical impact indicators used at the time were outputs, such as the number of people who completed a program or the number of job placements. Outputs tell you something, but they do not measure outcomes, which are the long-term effects of programs. Encouraged by volunteers on its Basic Needs Committee, many of whom were business professionals, United Way began pursuing a different way to quantify outcomes, one adapted from the business world: Return on Investment, or ROI.
Getting ROI under way
An ROI analysis compares a program’s costs to the benefits it creates: total benefits in the numerator, total costs in the denominator. In the case of United Way’s job training programs, the difference between participants’ pre-program earnings and post-program earnings is the key benefit. Another benefit is avoided costs, which are measured by state-provided data on participants’ reduced use of public services such as food or cash assistance. The number resulting from the benefits-costs comparison is the ROI. The breakeven point is $1.00; anything above $1.00 represents a positive return.
For its ROI analysis, United Way needed data on the wages that graduates of its job training programs earn. The Minnesota Department of Employment and Economic Development (DEED) already collected some components of the necessary wage information, so United Way established a partnership with DEED to collect the rest of the needed data about the graduates—such as demographic information, wage and industry of most recent employment, and receipt of public benefits—from its funded agency partners.
Starting in 2007, all agencies receiving United Way funding for job training programs agreed to collect key data from consenting program participants. Agencies entered the data into spreadsheets and sent them to DEED, where analysts matched the program participants to DEED’s wage information. The agencies added in dates for program completion and job placement as participants reached those milestones. Data on more than 20,000 program participants have been collected so far.
When the data collection first began, a few agencies were hesitant to participate. Would United Way implement a strict cutoff score and slash funding for any programs that fell below it? Could one number, the ROI estimate, be a useful means of evaluating so many different programs? Would the use of ROI encourage programs to pick only the easiest-to-serve clients and stop serving populations that have high barriers to employment? To alleviate the concerns, United Way staff made it clear that the ROI project was iterative and informational and would not replace other tools used to evaluate programs.
The results so far
Early results of United Way’s ROI initiative suggest that for the majority of the analyzed programs, the community’s investment is recouped within a year. This means that for every one dollar that goes in, approximately one dollar in return—in the form of higher income taxes paid and lower utilization of public services—is generated within one year.
Estimating long-term returns has proven to be more difficult, but given the strong one-year ROI and the positive effects of steady employment and earnings over time, United Way’s ten-year ROI, even with conservative assumptions, will likely be much greater than $1.00.
The long-term ROI estimate suggests that early investments in job training and employment assistance pay off. The analysis also suggests that higher-intensity programs—for example, those that include an education or skills component in addition to job placement services—create longer-lasting improvements in participants’ earnings trajectories. This would support the idea that for some participants, education and training bring more benefit than immediate new employment.
The process of constructing the ROI analysis enabled United Way staff to understand the costs, structures, and other aspects of each of the 36 job training programs in far greater detail than they had before, and to recognize the tremendous variety across programs, in terms of services delivered, intensity of services, and to whom costs should be attributed when program participants access services at several different agencies. United Way staff came to the realization that, in some respects, ROI is best thought of as a tool for continuous improvement, to be used repeatedly over years.
“ROI should be part of a broad set of measurement tools,” says Luke Weisberg, a workforce development consultant for United Way and former director of the Governor’s Workforce Development Council (GWDC), a nonpartisan state workforce office housed within DEED. “It is much better at measuring an organization’s own change over time than at measuring across programs.”
Participating agencies are seeing the benefit of taking part in the ROI project. For example, the longitudinal data allow staff at the International Institute of Minnesota (IIM) to show students their progress over the course of a program. While United Way is one of only a few funders requiring such rigorous data collection and analysis, other funders appreciate the data and pay attention to the results. Thus, the ability to clearly demonstrate the program’s value can give the IIM and other agencies a leg up in competitive grant applications.
Next steps for ROI
United Way’s methodology continues to grow more sophisticated through collaborations with partners such as the GWDC and the Wilder Foundation, and the data-collection process is becoming more automated and user-friendly. In the beginning, agencies entered client information into a spreadsheet, which had to be manually entered into a different spreadsheet by DEED. Each quarter, DEED’s master spreadsheet was manually updated with the previous quarter’s results. Now analysts at DEED have automated the data transfer and update processes. ROI will soon be incorporated into other reports, thus reducing double-entry for staff. As the system becomes increasingly automated, agencies should find it easier to generate meaningful analyses that inform program design and demonstrate program value to prospective funders.
In the future, United Way and the GWDC will take on the question of shared attribution: When an agency offers multiple services, how do you allocate overhead expenses among them? Similarly, how do you apply ROI when a participant may be served by several agencies?
“Outcomes produced by job training programs are often reliant and contingent upon adequate investment in safety net programs, childcare, early learning programs, parenting skills resources, adequate housing and nutrition, etc.,” explains Brian Paulson, United Way’s director of innovation strategies. “We can’t be so myopic as to suggest that the job training program is solely responsible for the outcome.”
By enabling before-and-after comparisons, the data set underlying the ROI project will allow researchers to examine the effects of policy changes. With enough data, United Way and the GWDC hope to understand the value added by each of the different services a participant can access, from childcare assistance to adult basic education to housing support. In a similar vein, United Way and the GWDC will look to understand which components of a typical job training program are the most important to a positive outcome.
Both Weisberg and Nick Maryns, senior policy analyst at the GWDC, say that a primary goal of the ROI project is to establish a system that agencies can use to analyze their own performance over time.
There is no doubt that this collaborative work around ROI is benefiting the state, nonprofit agencies, philanthropic funders, participants, and the community—even if the measurement systems can’t quantify it all quite yet.
Sarah Radosevich recently served as a social finance intern for Greater Twin Cities United Way and is currently a public policy research associate at the Minnesota Chamber of Commerce. For more information on the United Way’s ROI project, contact Brian Paulson at 612-340-7601 or brian.paulson@unitedwaytwincities.org.
