The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
A Profile of the Evaluation of Teen Outreach Program
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Program Description
| Overview | Teen Outreach Program (TOP) is a school-based program involving young people ages 12–17 in volunteer service in their communities. The program connects the volunteer work to classroom-based, curriculum-guided group discussions on various issues important to young people. Designed to increase academic success and decrease teen pregnancy, TOP helps youth develop positive self-image, learn valuable life skills, and establish future goals. |
| Start Date | 1978 |
| Scope | national |
| Type | after school, weekend |
| Location | urban, suburban, rural |
| Setting | public school, community-based organization, religious institution, private facility, recreation center, other |
| Participants | middle school and high school students |
| Number of Sites/Grantees | 176 sites nationwide (2003) |
| Number Served | 13,000 students (2003) |
| Components | TOP encompasses three interrelated elements: (1) supervised community volunteer service, (2) classroom-based discussions of service activities, and (3) classroom-based discussions and activities related to key social-developmental tasks of adolescence. Volunteer activities are selected by participants under supervision of trained staff and adult volunteers, sensitive to the needs and capacities of both participants and the local communities. Examples of volunteer activities include: working as aides in hospitals and nursing homes, participation in walkathons, peer tutoring, and various other activities. Sites were required to provide a minimum of 20 hours per year of volunteer experience to participants, with the median participant in the study performing 35 hours of service. Classroom discussions are designed (via the Teen Outreach Curriculum) to provide structured discussions, group exercises, role-plays, guest speakers, and informational presentations. Discussions are geared toward either maximizing learning from students' volunteer experiences, or toward helping cope with important developmental tasks such as managing family relationships, new academic and employment challenges, handling close friendships and romantic relationships, etc. Facilitators are given wide latitude in directing discussion and specific material about sexuality actually comprises less than 15% of the written curriculum (some of which was often not used due to site leaders' discretion). Service learning discussions focused on issues such as dealing with a lack of self-confidence, social skills, assertiveness, and self-discipline (in order to help prepare for their service experiences). Participants are also encouraged to think about and discuss what they learned while volunteering, and hear others do the same. Classroom discussions are led by trained facilitators (youth professionals, teachers, or very rarely guidance counselors). Program staff often work in conjunction with various community organizations, faith groups, and businesses in order to provide the volunteer activities to participants. |
| Funding Level | $100–$600 per participant per year |
| Funding Sources | TOP currently has no national funding. Program sponsors find their own funding and often institutionalize the program for sustainability. |
Evaluation
| Overview | The TOP program has been regularly evaluated for over 10 years. Throughout the early 1990s, researchers conducted numerous nonrandomly designed evaluations of TOP's impacts in reducing teen pregnancy and school failure. It was thought, however, that a true experimental evaluation would be more influential to policymakers, practitioners, and researchers in order to investigate the potential for broad, developmentally focused interventions targeting the prevention of diverse problem behaviors. Thus, evaluators collected data with a randomly designed study carried out between 1991 and 1995. A subsequent evaluation focused on whether the program has differential impacts across different populations. Finally, Cornerstone Consulting Group, Inc. (CCG), the company that owns and operates the program, drafted a report reflecting on the lessons they learned during their efforts to expand and replicate their program on a national scale since 1995. |
| Evaluators | Joseph P. Allen, University of Virginia Susan Philliber and Scott Herrling, Philliber Research Associates Gabriel P. Kuperminc, Yale University Cornerstone Consulting Group, Inc. |
| Evaluations Profiled | Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Approach Who Benefits Most From a Broadly Targeted Prevention Program? Differential Efficacy Across Populations in the Teen Outreach Program The Replication Challenge: Lessons Learned From the National Replication Project for the Teen Outreach Program (TOP) |
| Evaluations Planned | None at this time |
| Report Availability | Allen, J. P., Philliber, S., Herrling, S., & Kuperminc, G. P. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 64(4), 729–724. Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach Program. Journal of Community Psychology, 29(6), 637–655. Available at www.cornerstone.to/top/prevent.pdf (Acrobat file). Cornerstone Consulting Group, Inc. (1999). The replication challenge: Lessons learned from the National Replication Project for the Teen Outreach Program (TOP). Houston, TX. Available at www.cornerstone.to/what/rep.pdf (Acrobat file). Allen, J. P., Kuperminc, G., Philliber, S., & Herre, K. (1994). Programmatic prevention of adolescent problem behaviors: The role of autonomy, relatedness, and volunteer service in the Teen Outreach Program. American Journal of Community Psychology, 22, 617–638. Allen, J. P., Philliber, S., & Hoggson, N. (1990). School-based prevention of teenage pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach Program. American Journal of Community Psychology, 8, 505–524. Philliber, S., & Allen, J. P. (1992). Life options and community service: Teen Outreach Program. In B. C. Miller, J. J. Card, R. L. Paikoff, & J. L. Peterson (Eds.), Preventing adolescent pregnancy: Model programs and evaluations (pp. 139–155). Newbury Park, CA: Sage. |
Contacts
| Evaluation | Joseph P. Allen, Ph.D. Department of Psychology, Gilmer Hall University of Virginia Charlottesville, VA 22903 Tel: 434-982-4727 Email: allen@virginia.edu |
|
| Program | Sharon Lovick Edwards Cornerstone Consulting Group One Greenway Plaza, Suite 550 Houston, TX 77042 Tel: 713-627-2322 Email: sedwards@cornerstone.to |
|
| Profile Updated | February 21, 2003 | |
Evaluation 1: Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Approach
Evaluation Description
| Evaluation Purpose | To answer the following questions: (1) Does the program impact teen pregnancy rates of participants? (2) Does the program have an impact on school failure and suspension rates? and (3) Do any such impacts vary according to student gender, parental education level, household composition, or racial/ethnic minority status? |
| Evaluation Design | Experimental: TOP was evaluated at 25 different sites nationwide from 1991 to 1995. Sites randomly assigned students by selecting students individually from a waiting list (either out of a hat or choosing every other name on an alphabetized list) or less frequently by offering TOP to randomly selected classrooms in lieu of regular curricular offerings in Health or Social Studies. The total sample size included 342 TOP students and 353 control students. Attrition rates were 5.3% among TOP students and 8.4% among control students. No significant demographic differences were found between TOP and control groups, although the control group demonstrated significantly higher levels of prior course failure and teen pregnancy. Three sites were eliminated from the analysis due to evaluators' investigations into either groups' differential attrition rates or deviations between TOP and control groups on program entry characteristics. Program entry differences and demographic characteristics were also statistically controlled for during the subsequent data analysis. Given that program and control groups differed somewhat at program entry, the evaluators broke down the remaining 22 sites into those where the control group exhibited more problematic behavior at program entry and those where the control group exhibited less problematic behavior than the TOP group at program entry. This was done to investigate whether any program effects were merely an artifact of the control group demonstrating, on average, more problem behavior prior to program entry. The evaluators conducted separate analyses breaking down the data in this way. The evaluators also examined the data by (1) reanalyzing the data with the three previously dropped sites re-included, and (2) dropping sites where the control group were most different from the TOP group on reported problem behaviors. Neither scenario altered the findings reported below in any significant way. |
| Data Collection Methods | Secondary Source/Data Review: Archival data from one nonrandom site, containing information regarding history of course failure and school suspension, were used to assess the validity of students' self-reports of these behaviors in the surveys/questionnaires. Surveys/Questionnaires: Students complete surveys at the beginning and end of each program year. Surveys asked about demographic characteristics such as age, grade-level, race, household composition, and parents' education level. Surveys also asked whether students had ever either been pregnant or caused a pregnancy, failed any courses in the past year, or had been suspended in the past year. |
| Data Collection Timeframe | Data were collected during the 1991–1995 school years. |
Findings:
Formative/Process Findings
| Costs/Revenues | The program can be offered for a full academic year to a class of 18–25 students for approximately $500–$700 per student. If facilitator and site-level coordinator costs are provided as an in-kind contribution by schools and volunteer service organizations, this cost can be kept at approximately $100 per student. |
| Recruitment/Participation | Eighty-six percent of TOP participants were female and 14% were male. Over 67% of participants were black, while 17% were white, and another 13% were Hispanic. Sixty-nine percent of participants were in either ninth or tenth grade. The average age of TOP participants was approximately 16. The average TOP participant's parent was a high school graduate. A total of 45.6% of TOP participants came from two-parent households. |
Summative/Outcome Findings
| Academic | Risk of school suspension in the TOP group was only 42% as large as the risk of suspension in the control group (p<.001). Risk of course failure in the TOP group was only 39% as large as the risk of course failure in the control group (p<.001). TOP's effects on course failure and risk of suspension did not differ depending on whether the control group exhibited more problem behavior than the treatment group prior to the evaluation. The archival data on school suspension and course failure revealed that most students were accurate in their reporting of such behavior. There were no significant differences between TOP and control groups in the accuracy of this self-reporting. The evaluators report that the slight but statistically insignificant differences that did exist between groups in accuracy of reporting would not have resulted in a bias in favor of either group. Students who worked more volunteer hours were at lower risk for course failure during the program. This interaction was statistically significant (p<.04). No interaction effects of program participation with household composition, parental education levels, racial/ethnic minority status, student grade in school, or history of prior problem behaviors were found in predicting program outcomes. |
| Prevention | The effects of the program in preventing teen pregnancies were highly significant for females (p<.001); a separate analysis of the program for males was not possible due to the small number of males and male reports of causing a teen pregnancy. No other interaction effects were found by household composition, racial/ethnic minority status, student grade in school, or history of prior problem behaviors, indicating that the program works in the same ways for these various populations. For females, risk of teen pregnancy was only 41% as large for TOP participants as in the control group (p<.05). Pregnancy prevention effects for females were no larger or smaller depending on whether the control group differed in problem behavior prior to program entry; the program's effects on pregnancy prevention outcomes were not significantly smaller or larger for either set of control group sites. |
Evaluation 2: Who Benefits Most From a Broadly Targeted Prevention Program? Differential Efficacy Across Populations in the Teen Outreach Program
Evaluation Description
| Evaluation Purpose | To assess whether a national prevention program works equally well among various populations. More specifically, the evaluation examines the following questions: (1) Is a broad-based competence-enhancing intervention most efficacious when serving higher risk adolescents, in terms of both familial risk factors and behavioral risk factors? and (2) Does the program appear equally effective across different sociodemographic groups of youth? |
| Evaluation Design | Experimental and Quasi-Experimental: Data were collected from both TOP participants and control group students over a four-year period across over 60 sites nationwide. Comparison students were selected in a number of ways. Twenty percent of the sample (N=660) were randomly assigned to either the treatment or control group. For the remainder of the sample, some TOP participants nominated other students who they thought “would fill out the entry questionnaire about the same way [they] did,” while other control students were selected by facilitators and guidance counselors based on the similarity of sociodemographic backgrounds. The total sample included 1,673 TOP participants and 1,604 comparison students. Attrition was 8.9% among TOP students and 7.8% among comparison students. Although students not completing the study were more likely to have had or caused a pregnancy, been suspended, and failed courses previously, there were no significant differences between treatment and control youth due to attrition. Comparison group students were slightly more likely, however, to be female (29.1% vs. 24.6%, p<.01) and black (46.1% vs. 44.3%, p<.05) than were TOP participants. Given that the sample size would have been relatively small if the data analysis were limited to only the randomized subset of participants, all findings reported are for the full sample. The evaluators performed checks to see whether any of the effects significantly differed between random and nonrandom subsamples, but no such differences were discovered. Each problem behavior was assessed separately using hierarchical logistic regression analysis. Interaction factors were included in the model to test whether program effects varied among various populations. Given the nested nature of the data (students within classrooms) multilevel modeling procedures were also employed to see whether the nested nature of the data affected any of the general results. A small number of participants (5.9%) had been previously involved with TOP. |
| Data Collection Methods | Surveys/Questionnaires: Students complete surveys at the beginning and end of each program year. Surveys asked about demographic characteristics such as age, grade-level, race, household composition, and parents' education level. Surveys also asked whether students had ever either been pregnant or caused a pregnancy, failed any courses in the past year, or had been suspended in the past year. |
| Data Collection Timeframe | Data were collected 1992–1996. |
Findings:
Formative/Process Findings
| Costs/Revenues | The program can be offered for a full academic year to a class of 18–25 students for approximately $500–$700 per student. If facilitator and site-level coordinator costs are provided as an in-kind contribution by schools and volunteer service organizations, this cost can be kept at approximately $100 per student. |
| Recruitment/Participation | Students entered the program either as part of their “health curricula,” as an academic elective, via teacher/guidance counselor suggestion, or as an after school activity. Roughly 40% of TOP participants were in the ninth grade, with another 28% in tenth grade, 18% in eleventh grade, and 14% in twelfth grade. Nearly one-quarter of the participants were female and more than three-quarters were male. Approximately 44% of participants were black, 38% were white, and 12.6% were Hispanic. The average participant's mother had a high school diploma. Over half (54.1%) of the participants lived in two-parent households. Nineteen percent of TOP participants had had a prior suspension. Thirty-two percent of participants had had a prior course failure Eight percent of participants had had a prior pregnancy and 3.1% had already been teenage parents. |
Summative/Outcome Findings
| Academic | TOP participants demonstrated only 60% the risk of course failure as the risk in the comparison group (p<.001). The program had significantly greater efficacy in preventing course failure for females and for members of racial/ethnic minority groups (p<.001). Program effects were also significantly stronger for students who had a history of prior suspension (p<.001), although the program still demonstrated positive program effects for students without histories of suspensions (p<.05). Program effectiveness did not vary by familial risk factors, such as parental education or household composition (one- vs. two-parent households). TOP participants demonstrated 52% the risk of academic suspension as demonstrated by comparison group students (p<.001). Program effects did not significantly differ among different demographic groups, behavioral risk factor groups, or familial risk factor groups, indicating that the program was more or less equally effective for these various groups, on this measure. Multilevel modeling procedures did not significantly alter the results, with the exception of the interaction between minority status and participation in predicting course failure, which dropped from statistical significance to the trend level, and the interaction between gender and course failure in predicting course failure, which rose to the .01 level of significance. |
| Prevention | TOP participants demonstrated 53% of the risk of pregnancy as the risk demonstrated by comparison group students (p<.001). The program effectiveness did not significantly differ between students of different grade levels, genders, or racial/ethnic groups. While the program continued to demonstrate effectiveness for both teen parents and non-teen parents, the analysis revealed that program effects were significantly stronger for teen parents (p<.001). Non-teenage parents were two-thirds as likely as comparison students to have a pregnancy (p<.05), while teen parents were only 18% as likely as comparison students to have a pregnancy (p<.001). Program effects did not differ by familial risk factors, such as parents' education or one- vs. two-parent families. Multilevel modeling procedures did not significantly alter the results in regards to pregnancy prevention. |
Evaluation 3: The Replication Challenge: Lessons Learned From the National Replication Project for the Teen Outreach Program
Evaluation Description
| Evaluation Purpose | In October of 1995, Cornerstone Consulting Group, Inc. (CCG) assumed primary responsibility for and ownership of the TOP program from the Association of Junior Leagues International. After doing so, they began an effort to reshape the replication and dissemination of TOP on a national level. The purpose of this report, while not a formal evaluation per se, was designed to “look back” over the first three years of CCG's TOP replication efforts in order to see what lessons could be learned that would be of importance for the education-health-social service field. |
| Evaluation Design | Non-Experimental: CCG's approach is to tell the story of the TOP replication effort. The authors recollect and reconstruct the story of TOP's replication and extract lessons from that reconstruction. They also spoke extensively with program staff and administrators in various TOP sites in order to gain awareness of how replication and expansion unfolded in these settings. |
| Data Collection Methods | Interviews/Focus Groups: Site-level staff and administrators were asked about their experiences expanding individual TOP programs at the local level. Observation: CCG personnel combined their personal experiences with and recollections of TOP's replication efforts, in effect telling a narrative of the program's history since CCG assumed responsibility for the program. |
| Data Collection Timeframe | The information contained in this report pertains to the time period between 1995 and 1998. |
Findings:
Formative/Process Findings
| Costs/Revenues | There is no single, sustained TOP funding stream. Therefore, CCG employed a “hitchhiking strategy,” taking advantage of opportunities not specifically geared toward TOP for funding new programs. For example, in California, state funding for teen pregnancy prevention (e.g., the California Teenage Pregnancy Prevention Grant Program) enabled the program to rapidly expand into several communities as selected school district grantees chose TOP to fulfill their requirements of implementing a “proven program” of teen pregnancy prevention. TOP had significant funding to act as the “champion” of the program, in terms of outreach and marketing, but lacked funding to ease start-up costs and ongoing operational costs, which CCG thought would have greatly improved their ability to replicate the TOP program. |
| Systemic Infrastructure | After taking over TOP in 1995, Cornerstone considered transferring the program to a single, permanent, all-purpose home (for example, a large youth organization such as Boys and Girls Clubs of America or 4-H), but discovered that such organizations were either managing similar programs or were not in position to take on overall management of TOP. CCG then decided that the challenge of growing TOP into a larger-scale national project was theirs alone. Early on in the replication effort, CCG assessed TOP's primary strengths and weaknesses in order to formulate a strategy for replication. They found that the TOP program was sound and well developed, had an unusually extensive and credible evaluation, successful example sites, numerous committed funders, and an extensive network of philanthropic and human resources contacts. On the other hand, the curriculum was somewhat cumbersome, unattractively packaged, and unwieldy, funding was unavailable to ease the installation or operational costs of new TOP providers, and there was no readily available and stable distributional system to rapidly create new TOP sites. CCG made the decision early on to make the number of youth served the benchmark for judging the program's success. Given CCG's resources, TOP was never run as a “franchise replication,” a replication done exactly the same way at every site. Rather, CCG found that TOP had to be “massaged” to fit a wide variety of settings, target populations, and situations. For instance, both boys and girls participate, as do participants of different ethnic groups and economic backgrounds. Programs are most often run in middle or high schools, but also have been adapted to run in community-based settings. Therefore, program features had to be both adaptable and flexible. Given TOP's long history of evaluation, the evaluators (Philliber Research Associates), taught new sites methods of self-assessment in order to reduce administrative burdens. CCG also decided to focus on providing access to TOP training, developing user-friendly curricula, ensuring more local control in program design, and providing high-quality, up-to-date program materials. To strengthen existing TOP sites, CCG developed a newsletter and offered technical assistance by telephone. To expand into new sites, they changed their strategy from being a retail provider to being a wholesale provider, engaging school districts and communities, rather than individual classrooms, as providers of the continued development, replication, and institutionalization of TOP. TOP's marketing strategy emphasized a brief and consistent message, namely that TOP is proven to work, is affordable, and provides opportunities for youth development. CCG developed promotional materials telling the TOP story, while at the same time aggressively seeking venues in which to promote the program. These efforts included: creating and distributing a newsletter, publishing brochures, fact sheets, and hand-outs; presenting at conferences and meetings; making TOP materials available on HandsNet, an online health and human services information center; and using contacts to put TOP in front of key decision makers. CCG also contracted with the communications and public relations firm Podesta Associates in order to increase TOP's visibility, an effort timed to coincide with the release of new research from the University of Virginia on a random assignment study of TOP's effectiveness in preventing teen pregnancy and academic failure. CCG and TOP maintained an evaluation system to monitor achievements, especially through relationships with the University of Virginia and Philliber Research Associates. The positive evaluation results were found to be one of the program's greatest assets. CCG also maintained relationships with foundation program officers, TOP providers, and program sponsors. These “friends” of the program often served as TOP's most effective sales force. |
Summative/Outcome Findings
| Academic | TOP participants demonstrated only 60% the risk of course failure as the risk in the comparison group (p<.001). The program had significantly greater efficacy in preventing course failure for females and for members of racial/ethnic minority groups (p<.001). Program effects were also significantly stronger for students who had a history of prior suspension (p<.001), although the program still demonstrated positive program effects for students without histories of suspensions (p<.05). Program effectiveness did not vary by familial risk factors, such as parental education or household composition (one- vs. two-parent households). TOP participants demonstrated 52% the risk of academic suspension as demonstrated by comparison group students (p<.001). Program effects did not significantly differ among different demographic groups, behavioral risk factor groups, or familial risk factor groups, indicating that the program was more or less equally effective for these various groups, on this measure. Multilevel modeling procedures did not significantly alter the results, with the exception of the interaction between minority status and participation in predicting course failure, which dropped from statistical significance to the trend level, and the interaction between gender and course failure in predicting course failure, which rose to the .01 level of significance. |
| Systemic | CCG took eight lessons away from their efforts to replicate TOP on the national level. These are:
|