The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IV, Number 1, 1998
Issue Topic: Performance Measurement
Evaluations to Watch
National Learning Project Evaluation
Jay Bell of James Bell Associates describes the National Learning Project Evaluation of the United Way.
Background
Primary responsibility for addressing social needs is presently undergoing a dramatic shift in the United States. Local communities—and particularly the nonprofit human service organizations within those communities—are being asked to assume an increasingly larger share of the burden. At the same time, human service programs are under increased scrutiny by government, philanthropic, and corporate funders, all of whom are demanding demonstrable results. United Ways (UW) are also being challenged to show their added value to local community improvement efforts.
Supported by a grant from the Lilly Endowment, Inc., the United Way of America (UWA) has initiated the National Learning Project (NLP) to assess the use of program outcome information at seven UW sites that have compiled program outcome information.
In addition to site selection, UWA has supported the NLP by convening the seven NLP UWs in July and December 1996; creating a private electronic board on UW Online for project managers at NLP UWs; conducting focus groups at six of the seven sites; and contracting for the development of “theories of change” or “logic models” for each of the participating UWs.
The UWs participating in the NLP expect to gain the following:
- National technical assistance and consultation based on the needs they identify
- Support for networking among themselves
- National recognition as leaders in measuring program outcomes and achieving community impact
In particular, NLP participants expect to benefit from a national evaluation that provides—on a real-time basis—ongoing documentation of progress and sharing of lessons learned. Knowledge gained from these sites will be used to assist the larger group of UWs in advancing faster and further along the road to full implementation of an outcomes approach. In turn, the expected improvements in UW-sponsored programs will contribute to an improved quality of life in the local communities receiving these services.
Evaluation Description
NLP SitesAsheville, NC |
To help realize useful information from the NLP, the evaluation is a qualitative assessment that embodies several operational purposes: 1) to analyze how sites differ in approaches and objectives in relation to their past progress and current environment; 2) to monitor each site's implementation of work-plans, including shifts in approaches and objectives, and reasons for those shifts; 3) to assess each site's achievement of activities, outputs, and outcomes; 4) to compare the results of different strategies used to attain similar objectives; 5) to assess the role of external and internal factors that affect each UW's progress and shifts in plans; 6) to compare progress across sites to find insights that might benefit sites' efforts to advance toward their objectives; and 7) to share (transfer) the cumulative learnings with other, non-NLP UWs.
Work on the evaluation project began in May 1997, and will extend over 32 months to a completion date of December 1999. Since the seven sites are at different stages of outcome implementation, baseline reports in the form of case studies are being completed for each site, with a baseline synthesis report on all sites to follow. Over the course of the project, annual reports for each site will provide updated information on implementation progress. The evaluation will conclude with a final cross-site report that looks across all phases of the project.
Selected Points and Next Steps
At a December 1997 meeting of UWs, selected points about outcome measurement implementation experiences derived from baseline case studies across the seven NLP sites were presented.
These points included the following:
- Implementation took more effort and a longer period of time than expected
- Every site employed a phased/incremental approach
- Although improvements in the results of programs and program design were undoubtedly occurring, there were few concrete examples and no systematic tallies
- NLP sites had to invent many new tools in the area of defining expected results of individual programs
- The demands placed on volunteers and staff changed dramatically
- Bridging to community impact remained an unresolved issue for sites seeking improvements in the status of the community.
At that same meeting, the following were listed as the next steps for NLP sites:
- Planning for and demonstrating community impact
- Enlisting collaboration of funding partners, especially government agencies that are the largest funders in the community priority areas
- Monitoring changes in the perceptions of UW by the community/public
- Demonstrating improvements in programs/ systems (independent of whether changes in community status occur).
The NLP evaluation is being conducted by Jay Bell of James Bell Associates and Joe Wholey and John McLaughlin of the USC Washington Center. For information, contact Jay Bell (see contact information below).
Jay Bell
Project Director, National Learning Project Evaluation
James Bell Associates, Inc.
2111 Wilson Blvd. Suite 1120
Arlington, VA 22201