The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> Teen Outreach Program
> Evaluation #3 (1995-1998): Lessons Learned From the National Replication Project
A Profile of the Evaluation of Teen Outreach Program
Profile Information
- Full Text (HTML)
- How to cite
- About the OST Database
Program Description
| Overview | Teen Outreach Program (TOP) is a school-based program involving young people ages 12–17 in volunteer service in their communities. The program connects the volunteer work to classroom-based, curriculum-guided group discussions on various issues important to young people. Designed to increase academic success and decrease teen pregnancy, TOP helps youth develop positive self-image, learn valuable life skills, and establish future goals. |
| Start Date | 1978 |
| Scope | national |
| Type | after school, weekend |
| Location | urban, suburban, rural |
| Setting | public school, community-based organization, religious institution, private facility, recreation center, other |
| Participants | middle school and high school students |
| Number of Sites/Grantees | 176 sites nationwide (2003) |
| Number Served | 13,000 students (2003) |
| Components | TOP encompasses three interrelated elements: (1) supervised community volunteer service, (2) classroom-based discussions of service activities, and (3) classroom-based discussions and activities related to key social-developmental tasks of adolescence. Volunteer activities are selected by participants under supervision of trained staff and adult volunteers, sensitive to the needs and capacities of both participants and the local communities. Examples of volunteer activities include: working as aides in hospitals and nursing homes, participation in walkathons, peer tutoring, and various other activities. Sites were required to provide a minimum of 20 hours per year of volunteer experience to participants, with the median participant in the study performing 35 hours of service. Classroom discussions are designed (via the Teen Outreach Curriculum) to provide structured discussions, group exercises, role-plays, guest speakers, and informational presentations. Discussions are geared toward either maximizing learning from students' volunteer experiences, or toward helping cope with important developmental tasks such as managing family relationships, new academic and employment challenges, handling close friendships and romantic relationships, etc. Facilitators are given wide latitude in directing discussion and specific material about sexuality actually comprises less than 15% of the written curriculum (some of which was often not used due to site leaders' discretion). Service learning discussions focused on issues such as dealing with a lack of self-confidence, social skills, assertiveness, and self-discipline (in order to help prepare for their service experiences). Participants are also encouraged to think about and discuss what they learned while volunteering, and hear others do the same. Classroom discussions are led by trained facilitators (youth professionals, teachers, or very rarely guidance counselors). Program staff often work in conjunction with various community organizations, faith groups, and businesses in order to provide the volunteer activities to participants. |
| Funding Level | $100–$600 per participant per year |
| Funding Sources | TOP currently has no national funding. Program sponsors find their own funding and often institutionalize the program for sustainability. |
Evaluation
| Overview | The TOP program has been regularly evaluated for over 10 years. Throughout the early 1990s, researchers conducted numerous nonrandomly designed evaluations of TOP's impacts in reducing teen pregnancy and school failure. It was thought, however, that a true experimental evaluation would be more influential to policymakers, practitioners, and researchers in order to investigate the potential for broad, developmentally focused interventions targeting the prevention of diverse problem behaviors. Thus, evaluators collected data with a randomly designed study carried out between 1991 and 1995. A subsequent evaluation focused on whether the program has differential impacts across different populations. Finally, Cornerstone Consulting Group, Inc. (CCG), the company that owns and operates the program, drafted a report reflecting on the lessons they learned during their efforts to expand and replicate their program on a national scale since 1995. |
| Evaluators | Joseph P. Allen, University of Virginia Susan Philliber and Scott Herrling, Philliber Research Associates Gabriel P. Kuperminc, Yale University Cornerstone Consulting Group, Inc. |
| Evaluations Profiled | Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Approach Who Benefits Most From a Broadly Targeted Prevention Program? Differential Efficacy Across Populations in the Teen Outreach Program The Replication Challenge: Lessons Learned From the National Replication Project for the Teen Outreach Program (TOP) |
| Evaluations Planned | None at this time |
| Report Availability | Allen, J. P., Philliber, S., Herrling, S., & Kuperminc, G. P. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729–724. Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach Program. Journal of Community Psychology, 29(6), 637–655. Available at www.cornerstone.to/top/prevent.pdf (Acrobat file). Cornerstone Consulting Group, Inc. (1999). The replication challenge: Lessons learned from the National Replication Project for the Teen Outreach Program (TOP). Houston, TX. Available at www.cornerstone.to/what/rep.pdf (Acrobat file). Allen, J. P., Kuperminc, G., Philliber, S., & Herre, K. (1994). Programmatic prevention of adolescent problem behaviors: The role of autonomy, relatedness, and volunteer service in the Teen Outreach Program. American Journal of Community Psychology, 22, 617–638. Allen, J. P., Philliber, S., & Hoggson, N. (1990). School-based prevention of teenage pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach Program. American Journal of Community Psychology, 8, 505–524. Philliber, S., & Allen, J. P. (1992). Life options and community service: Teen Outreach Program. In B. C. Miller, J. J. Card, R. L. Paikoff, & J. L. Peterson (Eds.), Preventing adolescent pregnancy: Model programs and evaluations (pp. 139–155). Newbury Park, CA: Sage. |
Contacts
| Evaluation | Joseph P. Allen, Ph.D. Department of Psychology, Gilmer Hall University of Virginia Charlottesville, VA 22903 Tel: 434-982-4727 Email: allen@virginia.edu |
|
| Program | Sharon Lovick Edwards Cornerstone Consulting Group One Greenway Plaza, Suite 550 Houston, TX 77042 Tel: 713-627-2322 Email: sedwards@cornerstone.to |
|
| Profile Updated | February 21, 2003 | |
Evaluation 3: The Replication Challenge: Lessons Learned From the National Replication Project for the Teen Outreach Program
Evaluation Description
| Evaluation Purpose | In October of 1995, Cornerstone Consulting Group, Inc. (CCG) assumed primary responsibility for and ownership of the TOP program from the Association of Junior Leagues International. After doing so, they began an effort to reshape the replication and dissemination of TOP on a national level. The purpose of this report, while not a formal evaluation per se, was designed to “look back” over the first three years of CCG's TOP replication efforts in order to see what lessons could be learned that would be of importance for the education-health-social service field. |
| Evaluation Design | Non-Experimental: CCG's approach is to tell the story of the TOP replication effort. The authors recollect and reconstruct the story of TOP's replication and extract lessons from that reconstruction. They also spoke extensively with program staff and administrators in various TOP sites in order to gain awareness of how replication and expansion unfolded in these settings. |
| Data Collection Methods | Interviews/Focus Groups: Site-level staff and administrators were asked about their experiences expanding individual TOP programs at the local level. Observation: CCG personnel combined their personal experiences with and recollections of TOP's replication efforts, in effect telling a narrative of the program's history since CCG assumed responsibility for the program. |
| Data Collection Timeframe | The information contained in this report pertains to the time period between 1995 and 1998. |
Findings:
Formative/Process Findings
| Costs/Revenues | There is no single, sustained TOP funding stream. Therefore, CCG employed a “hitchhiking strategy,” taking advantage of opportunities not specifically geared toward TOP for funding new programs. For example, in California, state funding for teen pregnancy prevention (e.g., the California Teenage Pregnancy Prevention Grant Program) enabled the program to rapidly expand into several communities as selected school district grantees chose TOP to fulfill their requirements of implementing a “proven program” of teen pregnancy prevention. TOP had significant funding to act as the “champion” of the program, in terms of outreach and marketing, but lacked funding to ease start-up costs and ongoing operational costs, which CCG thought would have greatly improved their ability to replicate the TOP program. |
| Systemic Infrastructure | After taking over TOP in 1995, Cornerstone considered transferring the program to a single, permanent, all-purpose home (for example, a large youth organization such as Boys and Girls Clubs of America or 4-H), but discovered that such organizations were either managing similar programs or were not in position to take on overall management of TOP. CCG then decided that the challenge of growing TOP into a larger-scale national project was theirs alone. Early on in the replication effort, CCG assessed TOP's primary strengths and weaknesses in order to formulate a strategy for replication. They found that the TOP program was sound and well developed, had an unusually extensive and credible evaluation, successful example sites, numerous committed funders, and an extensive network of philanthropic and human resources contacts. On the other hand, the curriculum was somewhat cumbersome, unattractively packaged, and unwieldy, funding was unavailable to ease the installation or operational costs of new TOP providers, and there was no readily available and stable distributional system to rapidly create new TOP sites. CCG made the decision early on to make the number of youth served the benchmark for judging the program's success. Given CCG's resources, TOP was never run as a “franchise replication,” a replication done exactly the same way at every site. Rather, CCG found that TOP had to be “massaged” to fit a wide variety of settings, target populations, and situations. For instance, both boys and girls participate, as do participants of different ethnic groups and economic backgrounds. Programs are most often run in middle or high schools, but also have been adapted to run in community-based settings. Therefore, program features had to be both adaptable and flexible. Given TOP's long history of evaluation, the evaluators (Philliber Research Associates), taught new sites methods of self-assessment in order to reduce administrative burdens. CCG also decided to focus on providing access to TOP training, developing user-friendly curricula, ensuring more local control in program design, and providing high-quality, up-to-date program materials. To strengthen existing TOP sites, CCG developed a newsletter and offered technical assistance by telephone. To expand into new sites, they changed their strategy from being a retail provider to being a wholesale provider, engaging school districts and communities, rather than individual classrooms, as providers of the continued development, replication, and institutionalization of TOP. TOP's marketing strategy emphasized a brief and consistent message, namely that TOP is proven to work, is affordable, and provides opportunities for youth development. CCG developed promotional materials telling the TOP story, while at the same time aggressively seeking venues in which to promote the program. These efforts included: creating and distributing a newsletter, publishing brochures, fact sheets, and hand-outs; presenting at conferences and meetings; making TOP materials available on HandsNet, an online health and human services information center; and using contacts to put TOP in front of key decision makers. CCG also contracted with the communications and public relations firm Podesta Associates in order to increase TOP's visibility, an effort timed to coincide with the release of new research from the University of Virginia on a random assignment study of TOP's effectiveness in preventing teen pregnancy and academic failure. CCG and TOP maintained an evaluation system to monitor achievements, especially through relationships with the University of Virginia and Philliber Research Associates. The positive evaluation results were found to be one of the program's greatest assets. CCG also maintained relationships with foundation program officers, TOP providers, and program sponsors. These “friends” of the program often served as TOP's most effective sales force. |
Summative/Outcome Findings
| Academic | TOP participants demonstrated only 60% the risk of course failure as the risk in the comparison group (p<.001). The program had significantly greater efficacy in preventing course failure for females and for members of racial/ethnic minority groups (p<.001). Program effects were also significantly stronger for students who had a history of prior suspension (p<.001), although the program still demonstrated positive program effects for students without histories of suspensions (p<.05). Program effectiveness did not vary by familial risk factors, such as parental education or household composition (one- vs. two-parent households). TOP participants demonstrated 52% the risk of academic suspension as demonstrated by comparison group students (p<.001). Program effects did not significantly differ among different demographic groups, behavioral risk factor groups, or familial risk factor groups, indicating that the program was more or less equally effective for these various groups, on this measure. Multilevel modeling procedures did not significantly alter the results, with the exception of the interaction between minority status and participation in predicting course failure, which dropped from statistical significance to the trend level, and the interaction between gender and course failure in predicting course failure, which rose to the .01 level of significance. |
| Systemic | CCG took eight lessons away from their efforts to replicate TOP on the national level. These are:
|