You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Teen REACH (Responsibility, Education, Achievement, Caring, and Hope) program is an effort to provide positive youth activities during nonschool hours for Illinois youth. Teen REACH program providers are funded by the Illinois Department of Human Services (DHS). Funding goes to community-based agencies across Illinois such as local health departments, social service agencies, youth agencies, faith-based organizations, and community coalitions to implement the program. Teen REACH seeks to expand the range of choices and opportunities that enable, empower, and encourage youth to achieve positive growth and development, improve expectations and capacities for future success, and avoid or reduce negative risk-taking behavior. Its goals are to improve youth’s academic performance; provide opportunities for learning positive social skills, demonstrating positive social interactions, and building positive social relationships; encourage the adoption of positive decision-making skills that discourage harmful risk-taking behaviors; and strengthen parent-child bonds and community involvement.
Start Date October 1998
Scope state
Type after school, comprehensive services, weekend
Location urban, suburban, rural
Setting community-based organization, religious institution, recreation center, public school
Participants kindergarten through high school students (ages 6 through 17)
Number of Sites/Grantees 37 organizations with 73 program sites in 1998; 109 organizations with 250 program sites in 2004
Number Served over 50,000 from 1998 to 2004, approximately 30,000 annually
Components Programs are required to provide the following core services: (1) academic assistance, including homework time, basic skills tutoring, and enrichment programs to encourage creativity; (2) recreation, sports, cultural, and artistic activities to provide safe outlets for trying new skills and interests, building friendships and sense of belonging, and gaining developmentally relevant experiences; (3) positive adult mentors who maintain positive, sustained relationships with youth participants through one-on-one interactions; (4) life skills education that promotes abstinence from risky behaviors such as substance use, crime, violence, and sexual activity; and (5) parental involvement activities aimed at fostering parent/staff discussion and relationships, parent/child bonds, and community involvement. In addition, the Illinois DHS suggests that program participants should be given the opportunity to participate in at least one community service activity each year.
Funding Level $8.4 million in 1999; $18.4 million in 2000; over $19 million annually from 2001 to 2004
Funding Sources Illinois Department of Human Services (DHS)


Evaluation

Overview DHS’s Bureau of Community and Youth Services contracted with the Center for Prevention Research and Development at the University of Illinois to conduct an outcome evaluation for Teen REACH. During the pilot phase, the evaluation sought to understand youth outcomes at a small subset of program sites, in order to improve the overall evaluation process and provide data for continuous improvement. Second- and third-year evaluations have built on this pilot evaluation, adding a staff survey and including more programs in the evaluation sample. In the third year, work began on the development of program benchmarks and parent and teacher surveys that ultimately will be used to assess Teen REACH programs.
Evaluators Center for Prevention Research and Development at the University of Illinois
Evaluations Profiled Teen REACH: A Summary of the Pilot Evaluation

Teen REACH: Annual Evaluation Report 2002

Teen REACH: Annual Evaluation Report 2003
Evaluations Planned Teen REACH evaluations will continue to be conducted annually.
Report Availability University of Illinois, The Center for Prevention Research and Development. (2001). Teen REACH: A summary of the pilot evaluation. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.

University of Illinois, The Center for Prevention Research and Development. (2002). Teen REACH: Annual evaluation. Champaign: Author.

University of Illinois, The Center for Prevention Research and Development. (2004). Teen REACH: Annual evaluation report. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.


Contacts

Evaluation Kay Erwin Mulhall, M.Ed.
Coordinator of Research Programs
Institute of Government and Public Affairs
Center for Prevention Research and Development
510 Devonshire Drive
Champaign, IL 61820
Tel: 217-333-3231
Fax: 217-244-0214
Email: kemulhal@uiuc.edu
Program Karrie Rueter
Illinois Department of Human Services
Office of Prevention
535 W. Jefferson, 3rd Floor
Springfield, IL 62702-5058
Tel: 217-557-2943
Email: dhspab@dhs.state.il.us
Profile Updated December 1, 2004

Evaluation 1: Teen REACH: A Summary of the Pilot Evaluation


Evaluation Description

Evaluation Purpose To assess the relationship between program participation and youth educational and socio-behavioral outcomes, to develop local understanding and capacity for program sites to effectively participate in the evaluation, and to provide opportunities for grantees to understand, utilize, and continuously improve their programs through data-based decision making.
Evaluation Design Quasi-Experimental and Non-Experimental: The Illinois Department of Human Services (DHS) selected a sample of five agencies administering programs DHS judged to be well functioning and likely to cooperate with the evaluation process. The sample was selected to represent four of the five DHS regions in the state, include a range of types of provider agencies (e.g., community-based organizations, health departments), represent both rural and urban communities, and assure cultural and racial diversity among participants. Providers from each selected agency were asked to select one site based on sample size and on strength of the program. Across the five agencies, seven program sites were asked to participate in the evaluation. At the completion of data collection, evaluators had pretest and posttest data from five sites (two sites did not submit complete data due to the absence of a consistent staff member).

Evaluators used a single-group pretest/posttest design to assess changes in youth outcomes. Data were examined by comparing pretest and posttest data and by exploring the relationship between posttest data and program attendance levels (i.e., high, medium, and low attendance) or dosage (defined in this evaluation as the total number of days youth attended). Attendance levels were measured as follows: high (241 programs days or more attended over the past two years), medium (145–240 programs days), and low (1–144 programs days).

Of the 349 youth initially attending the selected program sites, 275 completed pretest surveys (response rate of 79%). At posttest, 283 youth completed surveys, and a total of 197 youth completed both surveys.

Of those who completed both, the average age at pretest was 13, with 41% in Grades 4–6, 46% in Grades 7–9, and 13% in Grades 10–12. This sample was evenly split between girls and boys. The majority was African American (68%), with the remainder Latino (21%), white (12%), and other (11%). The vast majority (89%) participated in the free/reduced-price lunch program at school. Over a third (35%) lived in single-parent households, 46% lived in two-parent households, and the remaining 19% lived with non-parent adults (e.g. grandparents, foster parents, etc.).

Of the 167 youth who answered questions about program participation at posttest, 48% had been involved for 2 years, 32% for 1 year, and 20% for less than 1 year, with an average of 1.28 years. Of these youth, 9% participated about 6–7 days a week, 38% participated 5 days, 18% participated 4 days, and 35% participated 1–3 days, with an average weekly attendance of 4.08 days. In addition, 47% were classified as high attenders, 26% as medium, and 27% as low. The average total dosage was 267 days, ranging from 16 to 672 days.

Program staff and youth were selected for interviews using a theoretical sampling approach, which involves selecting informants based on the likelihood that they will provide a rich source of information about the phenomena under study. Specifically, local staff selected youth on the basis of age, gender, and length of program experience.
Data Collection Methods Interviews/Focus Groups: Staff and youth interviews were conducted during visits at five of the sites. Through group interviews at each site visited, evaluators spoke with every director and coordinator and at least two other staff members, for a total of 25 staff members across sites. Staff who could not participate in the group interview were interviewed individually. All staff were asked about their program goals, whether the program was being implemented as originally planned, program changes they would like to make, implementation obstacles encountered and how they were handled, program successes, evidence that the program was working, whether the program differed in its effectiveness for particular groups, program improvement suggestions, and program strengths and weaknesses. Evaluators conducted five youth focus groups, one at each site visited. At least 7 youth participants were interviewed at each site, with 8–12 youth in most focus groups. Youth participants, ages 10–18, were asked about their best experience in the program, what they would keep in the program, what they would add or take away to improve the program, and anything else that would help explain how the program runs.

Observation: Evaluators conducted 4–6-hour site visits in March and April 2001 at the five sites that completed surveys. Evaluators observed site facilities, staff-youth interactions, program implementation, and selected Teen REACH programming and activities.

Surveys/Questionnaires: Youth in Grades 4 and above at the selected sites were surveyed in December 2000/January 2001 (pretest) and again in May/June 2001 (posttest). The survey included items to assess youth background/demographic information, perceived connections with staff, perceived program impacts (the frequency with which specific topics are addressed in Teen REACH and perceptions of program benefits), and youth outcomes. Youth outcome areas included frequency of homework completion, school attendance (days absent in the past month and reasons for these absences), academic performance (self-reported grades), academic aspirations (e.g., “how important is to you to graduate from high school?”), quality of school life (e.g., frequency with which youth are happy when they are in school), engagement in prosocial and delinquent behaviors (e.g., “in the last 6 months, how often did you tell lies or cheat?”), substance use (reports of alcohol, tobacco, and drug use during the past 30 days), leadership (e.g., “I like to set a good example for other young people”), self-concept (e.g., “I am happy with myself as a person”), adult connections (e.g., “I know people/places in my community where I can get help with a personal problem”), peer-group cohesion (e.g., “I enjoy the time I spend with my friends”), problem-solving skills (e.g., “when I face a new problem, I first try to find out what it is”), and parent involvement in child’s education and Teen REACH (e.g., the frequency with which parents visit the Teen REACH program or their child’s school).
Data Collection Timeframe Data were collected during the 2000–2001 school year.


Findings:
Formative/Process Findings

Activity Implementation According to site observations and interviews with staff and youth, program goals and objectives were often met through recreation activities, which provided opportunities for youth to learn positive social skills, demonstrate positive social interactions, and build positive social relationships. For example, staff at two sites organized basketball and softball games with other Teen REACH programs or youth serving agencies and found these activities to be valuable in providing youth with opportunities for positive interaction with youth outside their town or neighborhood and from other racial and cultural groups.

Every program had staff interacting with, supervising, and moderating many of the recreation activities. Staff continually taught about appropriate social interaction through praise, modeling, direct rule giving, or enforcing. Although at times youth were observed doing this for each other, the quality of recreational programming seemed to be stronger and the goals more clearly met when staff were engaged directly with youth in the activity.

Youth frequently expressed enjoyment of experiences involving traveling beyond their neighborhood or community (e.g., amusement parks, plays, museums, conferences, movies, overnight trips out of town). Other experiences that staff cited as successful in engaging youth were mentoring trips, programs at local universities or colleges, special interagency youth nights, running a lawn service for the elderly, and using facilities at other collaborating agencies (e.g., computers, swimming pools, bowling lanes).

Most of the energy and focus for program planning and delivery observed by evaluators or noted by staff and youth were directed toward homework help and basic skills tutoring.

One area of perceived inconsistency across sites was the structuring and scheduling for academic assistance/tutoring/homework help. Some sites thought they had to enforce a definite homework-only time every day. Other sites were very flexible about their academic component, making homework help available every day, but not required.

Staff and youth at four sites continually cited homework help as a program strength. Staff at all sites said that many of the participants’ parents work and often do not have the time or energy—and sometime even the ability or experience—to help with homework.

There was little evidence indicating more than a cursory use of best practice tools that programs were supposed to implement (e.g., project-based learning, service learning, problem-solving approaches in innovative programming).

Many staff and some youth (mainly teenagers) stated that it was difficult to offer services for youth in the entire 6–17 age range. Staff recognized that services for youth aged 6–12 needed to be different than services for youth aged 13–17, but noted that it was difficult to implement programming for both age groups simultaneously, especially with limited staff.

Many staff reported sharing their own career development decisions with youth to help youth open up to planning for their futures and to highlight the steps and skills needed to “make it” (e.g., school success, avoiding drugs and gangs). One staff member developed a weekend basketball tournament that was interspersed with career talks and information.

Staff at some sites felt that evidence-based programming (i.e., programming that is based on the results of research using scientific methods) was encouraged by DHS, but not supported with appropriate training or resources.

Some sites felt that short-term or one-shot program sessions worked best at their sites, because of the difficulty of keeping a cohesive group involved across multiple sessions.
Parent/Community Involvement Most sites saw parental involvement as the most difficult component to implement successfully. According to the evaluators, only one site seemed to actively work with families to achieve the targeted program outcomes.

Many staff felt they faced the difficulty of youth going home to practices and values that conflicted with those the program supported; staff often saw their role as providing relationships, opportunities, and environments as an alternative youth’s home experiences.
Program Context/Infrastructure Many staff and youth described Teen REACH as a safe place. Staff at some sites felt that the program offered an alternative to gangs by providing youth learning experiences in discipline and knowing right from wrong. Some sites provided youth with safe rides home at the end of the day to ensure safety and lessen exposure to the streets and gangs.

Program staff and youth noted transportation as a challenge. Where provided, both youth and staff cited transportation as a great asset. Staff said that it provided safe passage to youth in areas with gang-related problems. A related concern is that many youth and staff commented on the poor condition of vans used for transportation.

Evaluators reported feeling a sense of safety and staff supervision when onsite, especially at the three sites with staff ratios of six youth to every one staff member. At these sites, there was a sense that youth were “wrapped in a net” of adult concern and availability.

At four of the sites visited, focus group youth frequently expressed their sense of belonging in the program. At the fifth site, strong connections were made between youth and individual staff members, but were not as strong overall. A sense of belonging was reflected in some youth’s descriptions of the program (e.g., “it’s like home”) and the positive peer relationships established in the program (e.g., “the first day I came here, I met a lot of friends”). Staff also noted youth’s sense of belonging, reporting that youth often came back to check in and that those who had graduated from the program often returned to work with younger youth.

Many program staff and youth cited a need for improved availability of appropriate space. Staff and youth at one site said it was difficult to build their program because they only had one room that was exclusively theirs, and that room had to store their materials. Another site often could not use its computer room because it was not adequately air-conditioned.

Staff and youth mentioned physical site maintenance as an area in need of improvement; evaluators noted only two of the sites as being clean and well kept. Some staff said that a well-maintained site sent a message to youth that they were valued, and that a poorly maintained or unclean facility sent the message that they were not worth nice space.

When asked what needed to be added to Teen REACH to make it better, many youth asked for recreation equipment, especially more basketballs or soccer balls “that worked.”

Four sites had access to a gym, and the boys at these sites continually cited basketball as one of the parts of Teen REACH that was important to them.
Recruitment/Participation According to youth focus groups at all five sites, Teen REACH’s recreational component was a strong draw, and was often the initial and sometimes sole reason for participation.

Some youth said that without transportation, the distance to Teen REACH from their schools or homes would make program participation difficult.

Staff at every site thought that they could serve more youth with sufficient staff and resources.
Staffing/Training Youth at all sites noted staff’s care, concern, and involvement as a core program strength.

Each site visited had at least one, and often many, staff whom youth perceived as caring and important adults in their life. These staff varied in their age, style, culture, ethnicity, expertise, and interests, but all were able to connect with a segment of their site’s youth. Examples include a mother with grown children who helped the “young ones” with their homework and a young man who grew up “on the wrong side of town” who motivated African-American boys to finish their homework so they could play basketball with him.

Both youth and staff at every site identified at least one program champion, a person who was the visible inspiration and advocate for the program and those involved, as the key to their program’s success. The position the champion held varied from site to site.

The evaluators found evidence that programs may have been understaffed, and that staff may have been overtaxed or under-supported. For example, two sites were unable to complete the youth survey process because they lacked a consistent staff member to follow through. In addition, some programs were limited in providing recreational programming because they had too few staff for the number of youth served.

Part-time staff repeatedly said that they might not be able to continue in their jobs due to Teen Reach’s lack of benefits, a need to find full-time work because they were not making enough money on their salaries from Teen REACH plus one or more other part-time jobs, and an inability to see a future for themselves there (e.g., no clear way to get raises, no career path).

Youth at two sites reported frustration that certain staff members had left and had not been replaced. This was frustrating because they missed these particular staff members, certain program aspects could not function without them, and remaining staff were now spread over more youth and did not have as much time for specific relationships as before.

In addition to working with youth, some staff were expected to take on the janitorial and maintenance work at their sites. As a result, staff sometimes had to choose between having a clean and attractive site and working with youth, or doing both and risking burn out.

Some staff felt that they did not have the kind of ongoing training needed. Specifically, staff engaged in tutoring and homework assistance for learning and behavior disabled youth requested training.

Staff at some sites expressed concern over the perceived time drain in fulfilling DHS paperwork and reporting requirements. One site stated that requirements increased each year, demanding more staff time, which was seen as time taken away from direct service to the youth, and as serving no purpose for them or for DHS.
Systemic Infrastructure Staff members from all sites noted difficulties in maintaining, sustaining, and expanding their programs.

Sites that had more success in securing funding beyond DHS funds were those that most successfully implemented the full complement of core services and best practices.

Most staff felt isolated from other Teen REACH sites. Some staff expressed an interest in implementing site-linked programming in which participants from multiple sites would be brought together for certain program components.


Summative/Outcome Findings

Academic No significant differences were found from pretest to posttest on academic measures, nor were academic outcomes correlated with attendance levels or dosage.
Family No significant differences were found from pretest to posttest on parent involvement measures, nor were these outcomes significantly correlated with program dosage. However, the pattern of results indicated that the high-attendance group reported more parental involvement than the other groups.
Prevention No significant differences were found from pretest to posttest on measures of delinquent behavior, nor were these outcomes significantly correlated with program dosage. However, the pattern of results indicated that the high-attendance group reported less delinquency.

At four sites, both youth and staff said that Teen REACH was an alternative to being on the streets and involved with gang activity.
Youth Development No significant differences were found in youth survey responses from pretest to posttest on items of self-concept, leadership skills, problem-solving skills, peer-group cohesion, adult connections, or prosocial behavior.

Program dosage was positively correlated with problem-solving skills (p < .05), self-concept (p < .01), peer-group cohesion (p < .05), and adult connections (p < .01). In addition, attendance levels were positively associated with adult connections (p < .01) and self-concept (p < .05).

Youth at every site noted that the program had a positive effect on their future aspirations. For example, one youth commented that Teen REACH “changed my whole attitude. We get direction here from staff. They tell us how to be, what we’re going to be in for in life.”

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project