Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of Quantum Opportunities Program

Program Description

Overview

The Quantum Opportunities Program (QOP) is designed to help at-risk youth make a “quantum leap” up the ladder of opportunity through academic, developmental, and community service activities, coupled with a sustained relationship with a peer group and a caring adult, offered to them over their 4 years of high school. The QOP framework strives to compensate for some of the deficits found in poverty areas by (a) compensating for both the perceived and real lack of opportunities, which are characteristic of disadvantaged neighborhoods, (b) providing interactions and involvement with persons who hold prosocial values and beliefs, (c) enhancing participants’ academic and functional skills to equip them for success, and (d) reinforcing positive achievements and actions.

The program was implemented on a pilot basis by the Ford Foundation and Opportunities Industrialization Centers of America (OICA), a nonprofit organization serving the poor, unemployed, underemployed, and youth. The pilot was implemented in five cities: San Antonio, Texas; Philadelphia, Pennsylvania; Milwaukee, Wisconsin; Saginaw, Michigan; and Oklahoma City, Oklahoma. The United States Department of Labor (DOL) and the Ford Foundation also tested QOP on a larger scale via a demonstration with two sites under private management and administration (Philadelphia, Pennsylvania, and Yakima, Washington) and five sites under federal management and administration (Cleveland, Ohio; Fort Worth, Texas; Houston, Texas; Memphis, Tennessee; and Washington, D.C.).

Start Date	1989–1993 for the pilot and 1995–2001 for the demonstration—six of the seven sites began the demonstration in 1995, while the Washington, DC site began in 1996
Scope	national
Type	after school, mentoring, summer/vacation, before school, weekend, comprehensive services
Location	urban, rural
Setting	public schools and community-based organizations
Participants	high school students
Number of Sites/Grantees	There were five sites for the pilot and seven sites for the demonstration. A local community-based organization (CBO) in each city implemented and operated a QOP program. Each CBO teamed with one to three high schools.
Number Served	125 in the pilot and 50, 80, or 100 at each site for a total of 580 for the demonstration
Components	Both the pilot and demonstration offered services to single cohorts of entering high school freshman throughout their time in high school. The pilot offered continuing services to these youth for 4 years, while the demonstration offered services for 5 years. The pilot targeted youth from families receiving welfare, while the demonstration targeted youth with low grades who attended high schools with high dropout rates. Eligible youth were enrolled in QOP and served even if they transferred to other schools, dropped out of school, became incarcerated, or became inactive in QOP for long periods of time. QOP used an “anti-attrition” program design in which program staff kept in touch with participants even during periods when these youth were disinterested in the program or moved. Services were provided year-round to enrollees, and were designed to be intensive and to address barriers to success. Although services were meant for youth up until graduation, enrollees who graduated received limited additional services focused primarily on the transition from high school to post-secondary education or training. The three primary activities offered to participating youth were supplemental academic activities (e.g., tutoring, computer-based instruction), development activities (e.g., activities designed to instruct youth about health, alcohol, drug abuse, sex, family planning, arts, career, and college planning), and community service activities aimed at improving conditions in the communities in which youth live. The QOP model prescribed an annual participation goal of 250 hours of each of these activities, for a total of 750 hours per year. Secondary aspects of the program model included financial incentives (stipends, accrual accounts, enrollee bonuses), staff bonuses (tied to youth participation), and supportive services (snacks, transportation assistance, and other services as needed). In addition, youth received intensive case management and mentoring to help them graduate from high school and enroll in postsecondary education or training. Each youth’s program was coordinated by a caring adult, a “case manager,” who served as mentor, role model, disciplinarian, broker, and problem solver. QOP case managers referred enrollees to community health and mental health services, summer jobs programs, and local agencies that provide housing, food, income support, or child care. The program mantra was “once in QOP, always in QOP,” which led case managers to devote time to helping their youth, even if the youth was no longer interested in QOP. The program model specified roughly 15 to 25 enrollees per case manager.
Funding Level	$1.1 million was spent for the pilot, $5 million for the DOL demonstration sites, and $4.1 million for the Ford Foundation demonstration sites plus Ford-funded OICA technical assistance to all sites. In addition, DOL sites were required to provide $200,000 per year per site in matching funds for the first 4 years.
Funding Sources	Ford Foundation, U.S. Department of Labor

Evaluation

Overview	The pilot was evaluated by Brandeis University from 1989 to 1993, with the goal of understanding the program’s impacts. The demonstration was evaluated by Mathematica Policy Research and Berkeley Policy Associates from 1995 to 2004, with the goal of understanding both the program’s implementation and its impacts on participants.
Evaluators	Andrew Hahn, Brandeis University Allen Schirm, Myles Maxfield, Nuria Rodriguez-Planas, Laura Castner, and Christina Tuttle, Mathematica Policy Research, Inc. Mary Vencill, Berkeley Policy Associates Vida Maralani, Department of Sociology at the University of California at Los Angeles
Evaluations Profiled	Evaluation of the Quantum Opportunities Program: Did the Program Work? The Quantum Opportunities Program Demonstration: Implementation and Short-Term Impacts The Quantum Opportunities Program Demonstration: Initial Post-Intervention Impacts
Evaluations Planned	To measure longer term impacts, data will be collected in winter 2005. The final report will be produced in late 2005.
Report Availability	Hahn, A., Leavitt, T., & Aaron, P. (1994). Evaluation of the Quantum Opportunities Program: Did the program work? Waltham, MA: Brandeis University. Lattimore, C. B., Grotpeter, J. K., & Taggart, R. (1998). Blueprints for violence prevention, book four: Quantum Opportunities Program. Boulder, CO: Center for the Study and Prevention of Violence. Schirm, A., Rodriguez-Planas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunities Program demonstration: Short-term impacts. Washington, DC: Mathematica Policy Research. Available at www.mathematica-mpr.com/education/qop.asp. Maxfield, M., & Castner, L., Maralani, V., & Vencill, M. (2003). The Quantum Opportunities Program demonstration: Implementation findings. Washington, DC: Mathematica Policy Research. Available at www.mathematica-mpr.com/education/qop.asp. Maxfield, M., Schirm, A., & Rodriguez-Planas, N. (2003). The Quantum Opportunities Program demonstration: Implementation and short-term impacts. Washington, DC: Mathematica Policy Research. Available at www.mathematica-mpr.com/education/qop.asp. Schirm, A., & Rodriguez-Planas, N. (2004). The Quantum Opportunities Program demonstration: Initial post-intervention impacts. Washington, DC: Mathematica Policy Research.

Contacts

Evaluation	Andrew Hahn, Ph.D. Center for Human Resources Heller Graduate School, Brandeis Univeristy Waltham, MA 02254-9110 Tel: 617-736-3774 Fax: 617-736-3851 Allen Schirm, Ph.D. Senior Fellow Mathematica Policy Research, Inc. 600 Maryland Avenue SW, Suite 550 Washington, DC 20024-2512 Tel: 202-484-4686 Fax: 202-863-1763 Email: aschirm@mathematica-mpr.com
Program	C. Benjamin Lattimore Opportunities Industrialization Centers of America, Inc. 1415 Broad Street Philadelphia, PA 19122 Tel: 215-236-4500 Fax: 215-236-7480
Profile Updated	December 15, 2004

Evaluation 1: Evaluation of the Quantum Opportunities Program: Did the Program Work?

Evaluation Description

Evaluation Purpose	To examine the impact of the QOP pilot on participants.
Evaluation Design	Experimental: Prior to program implementation, researchers selected 50 eighth grade students from families receiving public assistance and who were entering ninth grade at each of the five pilot sites. From these 50 students, 25 from each site were selected at random to be eligible to participate in the program (QOP members) and were recruited into the program. The remaining 25 youth at each site formed the control group. Evaluators eliminated one site (Milwaukee) from the analysis because there was no evidence that Milwaukee QOP members received a substantial amount of services, and the Milwaukee site was not able to retain contact with enough members of their experimental and control groups to provide for sound analysis. Of the remaining 100 participants and 100 controls, evaluators were able to reach 88 QOP members and 82 controls, who were included in the follow-ups and retained in the final sample used in the analyses. Analysis of the two groups at sample entry indicated that groups were largely free of systematic differences and there was no evidence of attrition bias.
Data Collection Methods	Surveys/Questionnaires: Questionnaires were administered in the fall of 1989 (9th grade), 1990 (10th grade), 1991 (11th grade), and 1992 (12th grade) to QOP members and control group members to gather information about demographic characteristics, work experience, school experiences, health knowledge, and personal attitudes and opinions. Two additional questionnaires that focused on future plans and post-secondary school outcomes were administered in the spring and fall of 1993, after youth were scheduled to have completed high school graduation. Tests/Assessments: QOP members and control group members administered tests assessing their academic skill levels (Test of Adult Basic Education Form 5 Level), and functional skill levels (APL 40 Item Version Survey – CCP Tier Mastery Test). Tests were administered in the fall of 1989, 1990, and 1991 and in the spring of 1993.
Data Collection Timeframe	Data were collected between 1989 and 1993.

Findings:
Summative/Outcome Findings

Academic	QOP members across sites were significantly more likely (p < .10) than control group youth to have graduated from high school (63% vs. 42%), although the data by site indicate these differences reached statistical significance only in the Philadelphia site. QOP members across sites were significantly more likely (p < .10) than control group youth to have enrolled in some type of post-secondary school 6 months after high school graduation (42% vs. 16%). These differences were most pronounced in the Philadelphia site, were still statistically significant in Oklahoma City, but were not large enough to reach statistical significance in San Antonio or Saginaw. Post-secondary enrollment effects held for both 2-year and 4-year college enrollment. The rate of 4-year college attendance among QOP members was more than three times higher than the control group rate (18% vs. 5%) and their rate of 2-year college attendance was more than twice as high (19% vs. 9%) 6 months after high school graduation. Both of these differences were significant (p < .10). Both sets of effects were again most pronounced in the Philadelphia site. Results from the 1st year (freshman year in high school) showed that test scores for many of the academic and functional skills tested declined for both the experimental and control groups. After 2 years, the experimental group’s average scores for all 11 academic and functional skills were higher than control group scores, and 5 of these differences were significant (p < .10). By the time most of the sample members were leaving high school in the spring of 1993, average experimental group scores on all 11 skills were higher than control group skills, and all of these differences were statistically significant (p < .10). There were variations in these effects by site during the high school years, with the Philadelphia site showing strong effects, Oklahoma City and Saginaw showing slightly positive effects, and San Antonio showing no positive effects. For orientation toward and expectations for post-secondary education, there were no significant differences between the two groups after 1 year. After 2 years, however, the experimental group demonstrated significantly higher (p < .10) educational expectations than the control group, and by the time most youth were leaving high school, this difference was even more pronounced. There were variations in these effects by site during the high school years, with the Philadelphia site showing strong effects, Oklahoma City and Saginaw showing slightly positive effects, and San Antonio showing no positive effects. There were no significant differences between the experimental and control groups during the high school years for the likelihood of being reported a school dropout. However, QOP group members were significantly less likely (p < .10) than control group youth to have dropped out of high school (23% vs. 50%) at the time of the fall 1993 survey (which counted dropouts as those who had not finished high school and were not currently in school). This difference was less pronounced according to the earlier spring 1993 survey, in which youth were asked whether they had ever dropped out of school. The dropout differences were most pronounced at the Philadelphia site, and were still statistically significant in Oklahoma City, though not large enough to be statistically significant in either San Antonio or Saginaw despite trends in the positive direction in these latter two sites. There were no significant differences between the experimental and control groups for self-reported school grades or assessments of their need for reading and math help.
Community Development	During the 6 months since finishing QOP, significantly more experimental group members than control group members (p < .01) served as volunteer tutors, counselors, or mentors (28% vs. 8%) and gave time to nonprofit, charitable, school, or community groups (41% vs. 11%). Also during this time period, more experimental group members than control group members participated in a community project (21% vs. 12%), although these differences were not significant. Differences in these indicators tended to be largest in Philadelphia and Saginaw.
Prevention	Though there was no evidence of program effects on the likelihood of having children during the high school years, there was evidence that QOP members were less likely to have children than control group members by the time of the post-high-school follow-up. By this time period, 24% of QOP members had children compared to 38% of control group members (p < .10). The QOP effect here appeared to be smallest in the Philadelphia site, and was largest in San Antonio, followed by Oklahoma City and Saginaw, and then Philadelphia, though none of the single site differences reached statistical significance. Control group members were significantly (p < .10) more likely to express a need for help with an alcohol or drug problem in the fall 1993 survey (no QOP members expressed such a need), but the actual number saying they had this need was very small. No significant program effects were found for contraceptive knowledge or AIDS knowledge during the high school years.
Workforce Development	There were no significant differences between the experimental and control groups’ assessments of their need for help in training for or finding a good job.
Youth Development	Significantly more QOP members than control group members (p < .01) had received honors or awards (34% vs. 12%) in the past 12 months at the time of the fall 1993 survey. The difference was greatest in the Philadelphia and San Antonio sites. Significantly more QOP members than control group members (p < .10) agreed or strongly agreed with the statements, “I am hopeful about the future” (98% vs. 86%) and “my life has been a success” (74% vs. 51%) at the time of the fall 1993 survey. Though not statistically significant, self-assessments by members of both the QOP and control groups at the time of the fall 1993 survey were highly positive. Ninety-three percent of QOP members and 82% of control group members strongly agreed or agreed with the statement that their family life was happy. Only 9% of QOP members and 17% of control group members strongly agreed or agreed with the assertion that they were lonely. More than half of both groups disagreed with the statement that they were bothered about things. A lower percentage of QOP members (5%) than control group members (13%) reported uncertainty regarding what steps to take in the future at the time of the fall 1993 survey, but a slightly higher percentage of control group members professed to know their future steps exactly (37% vs. 35%). These differences were not statistically significant.

A Profile of the Evaluation of Quantum Opportunities Program

Profile Information