You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Teen REACH (Responsibility, Education, Achievement, Caring, and Hope) program is an effort to provide positive youth activities during nonschool hours for Illinois youth. Teen REACH program providers are funded by the Illinois Department of Human Services (DHS). Funding goes to community-based agencies across Illinois such as local health departments, social service agencies, youth agencies, faith-based organizations, and community coalitions to implement the program. Teen REACH seeks to expand the range of choices and opportunities that enable, empower, and encourage youth to achieve positive growth and development, improve expectations and capacities for future success, and avoid or reduce negative risk-taking behavior. Its goals are to improve youth’s academic performance; provide opportunities for learning positive social skills, demonstrating positive social interactions, and building positive social relationships; encourage the adoption of positive decision-making skills that discourage harmful risk-taking behaviors; and strengthen parent-child bonds and community involvement.
Start Date October 1998
Scope state
Type after school, comprehensive services, weekend
Location urban, suburban, rural
Setting community-based organization, religious institution, recreation center, public school
Participants kindergarten through high school students (ages 6 through 17)
Number of Sites/Grantees 37 organizations with 73 program sites in 1998; 109 organizations with 250 program sites in 2004
Number Served over 50,000 from 1998 to 2004, approximately 30,000 annually
Components Programs are required to provide the following core services: (1) academic assistance, including homework time, basic skills tutoring, and enrichment programs to encourage creativity; (2) recreation, sports, cultural, and artistic activities to provide safe outlets for trying new skills and interests, building friendships and sense of belonging, and gaining developmentally relevant experiences; (3) positive adult mentors who maintain positive, sustained relationships with youth participants through one-on-one interactions; (4) life skills education that promotes abstinence from risky behaviors such as substance use, crime, violence, and sexual activity; and (5) parental involvement activities aimed at fostering parent/staff discussion and relationships, parent/child bonds, and community involvement. In addition, the Illinois DHS suggests that program participants should be given the opportunity to participate in at least one community service activity each year.
Funding Level $8.4 million in 1999; $18.4 million in 2000; over $19 million annually from 2001 to 2004
Funding Sources Illinois Department of Human Services (DHS)


Evaluation

Overview DHS’s Bureau of Community and Youth Services contracted with the Center for Prevention Research and Development at the University of Illinois to conduct an outcome evaluation for Teen REACH. During the pilot phase, the evaluation sought to understand youth outcomes at a small subset of program sites, in order to improve the overall evaluation process and provide data for continuous improvement. Second- and third-year evaluations have built on this pilot evaluation, adding a staff survey and including more programs in the evaluation sample. In the third year, work began on the development of program benchmarks and parent and teacher surveys that ultimately will be used to assess Teen REACH programs.
Evaluators Center for Prevention Research and Development at the University of Illinois
Evaluations Profiled Teen REACH: A Summary of the Pilot Evaluation

Teen REACH: Annual Evaluation Report 2002

Teen REACH: Annual Evaluation Report 2003
Evaluations Planned Teen REACH evaluations will continue to be conducted annually.
Report Availability University of Illinois, The Center for Prevention Research and Development. (2001). Teen REACH: A summary of the pilot evaluation. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.

University of Illinois, The Center for Prevention Research and Development. (2002). Teen REACH: Annual evaluation. Champaign: Author.

University of Illinois, The Center for Prevention Research and Development. (2004). Teen REACH: Annual evaluation report. Champaign: Author. Available at www.cprd.uiuc.edu/trdocs.html.


Contacts

Evaluation Kay Erwin Mulhall, M.Ed.
Coordinator of Research Programs
Institute of Government and Public Affairs
Center for Prevention Research and Development
510 Devonshire Drive
Champaign, IL 61820
Tel: 217-333-3231
Fax: 217-244-0214
Email: kemulhal@uiuc.edu
Program Karrie Rueter
Illinois Department of Human Services
Office of Prevention
535 W. Jefferson, 3rd Floor
Springfield, IL 62702-5058
Tel: 217-557-2943
Email: dhspab@dhs.state.il.us
Profile Updated December 1, 2004

Evaluation 2: Teen REACH: Annual Evaluation Report 2002


Evaluation Description

Evaluation Purpose To assess the relationship between program participation and youth educational and socio-behavioral outcomes, to develop local understanding and capacity for program sites to effectively participate in the evaluation, and to provide opportunities for grantees to understand, utilize, and continuously improve their programs through data-based decision making.
Evaluation Design Experimental and Non-Experimental: The Illinois Department of Human Services (DHS) selected 30 providers with programs they judged to be well functioning and likely to cooperate with the evaluation process. The sample was selected to provide geographic representation from all regions in the state, include a range of provider agency types (e.g., community-based organizations, health departments), represent both rural and urban communities, assure cultural and racial diversity among participants, and include newly funded and more established programs.

Qualitative data were collected from 15 of the 30 sites through site observations and staff and youth interviews. Sites visited during the previous year were not visited again. Program staff and youth were selected for interviews using a theoretical sampling approach, which involves selecting informants based on the likelihood that they will provide a rich source of information about the phenomena under study. Thus, a variety of individuals with differing roles in Teen REACH were asked to participate: program directors, program coordinators, various paid program staff, some unpaid volunteer staff, and youth between the ages of 10–18 who were currently participating in Teen REACH. Local staff selected youth on the basis of age, gender, and length of program experience.

Implementation was also assessed through a total of 192 staff surveys completed at 29 of the 30 evaluation sites (56% of the total number of staff surveys requested by each site). Of staff who responded to a survey item about their position in Teen REACH (n = 183), 55% were direct care program staff, 17% were site coordinators, 12% were project directors, 8% were volunteers, and 8% described their role as “other.”

Evaluators used a single-group pretest/posttest design to assess changes in youth outcomes. Four of the 30 sites did not submit completed posttest youth surveys and thus were not included in the data analyses and findings. Data were examined by comparing pretest and posttest data and by exploring the relationship between posttest data and program attendance levels (i.e., high, medium, and low attendance) or dosage (defined in this evaluation as the total number of days youth attended). Program attendance level was measured as high (433 days or more attended over the past three years), medium (81–432 days), and low (1–80 days).

Of the 1,648 youth attending the selected program sites at the time of the pretest, 1,046 youth at 30 sites completed the pretest (64% response rate), and 689 youth at 26 sites completed the posttest. A total of 468 participants (45%) completed both the pretest and posttest (the longitudinal sample). Findings are based on either the pre-post sample or, for program dosage analyses, the full posttest sample.

For the longitudinal sample, the average age at pretest was just over 12, with 50% in Grades 4–6, 30% in Grades 7–9, and 20% in Grades 10–12. This sample was evenly split between girls and boys. The sample was 36% African American, 33% white, 22% Latino, and 12% other. The majority (80%) participated in the free/reduced-price lunch program at school. Forty-one percent lived in single-parent households, 46% lived in two-parent households, and the remaining 13% lived with non-parent adults (e.g., grandparents, foster parents, etc.).

Of the 568 youth who answered questions about program participation at posttest, 20% had been involved for 3 years, 19% for 2 years, 27% for 1 year, and 34% for less than 1 year, with an average of 1.24 years. Of these youth, 10% participated about 6–7 days a week, 34% participated 5 days, 16% participated 4 days, and 40% participated 1–3 days, with an average of 3.9 days. In addition, 19% were classified as high attenders, 56% as medium, and 25% as low. The average total dosage was 259 days, with a range of 4 to 1,008 days.
Data Collection Methods Interviews/Focus Groups: During site visits, evaluators conducted staff interviews and youth participant focus groups to gain a deeper understanding of the program’s impact. At each site, 4–6 staff members and 5–10 youth participated in interviews. Group interviews were scheduled with the program director, program coordinator, and selected staff at each site. Staff who could not participate in the group interview were interviewed individually. All staff were asked about their program goals, whether the program was being implemented as originally planned, program changes they would like to make, implementation obstacles encountered and how they were handled, program successes, evidence that the program was working, whether the program differed in its effectiveness for particular groups, program improvement suggestions, and program strengths and weaknesses. Youth participants, ages 10–18, were asked about their best experience in the program, what they would keep in the program, what they would add or take away to improve the program, and anything else that would help explain how the program runs.

Observation: Evaluators conducted 4–6-hour site visits in May and June 2002 at 15 of the 30 evaluation sites. Evaluators observed selected programming and activities.

Surveys/Questionnaires: Youth in grades four and above at the selected sites were surveyed in October 2001/January 2002 (pretest) and again in May/June 2002 (posttest). The survey assessed youth background/demographic information, program exposure/dosage, perceived connections with staff, perceived program impacts, and youth outcomes. Youth outcome areas included frequency of homework completion, school attendance (days absent in the past month, and reasons for these absences), academic performance (self-reported grades), academic performance aspirations (e.g., “how important is it to you to graduate from high school?”), quality of school life (e.g., frequency with which youth are happy when they are in school), engagement in prosocial and delinquent behaviors (e.g., “in the last 6 months, how often did you tell lies or cheat?”), substance use (reports of alcohol, tobacco, and drug use during the past 30 days), leadership (e.g., “I like to set a good example for other young people”), self-concept (e.g., “I am happy with myself as a person”), adult connections (e.g., “I know people/places in my community where I can get help with a personal problem”), peer-group cohesion (e.g., “I enjoy the time I spend with my friends”), problem-solving skills (e.g., “when I face a new problem, I first try to find out what it is”), and parent involvement in child’s education and Teen REACH (e.g., the frequency with which parents visit the Teen REACH program or their child’s school).

Administered in April/May 2002 at 29 of the 30 sites, the staff survey assessed staff roles, characteristics, and responsibilities; organizational issues; perceived programming focus; and program impacts on youth.
Data Collection Timeframe Data were collected during the 2001–2002 school year.


Findings:
Formative/Process Findings

Activity Implementation Staff rated their programs as having the highest focus on academics (providing help with homework, stressing the importance of education, tutoring in certain subjects) and life skills (getting along with peers, conflict resolution, general problem solving). Parent- and family-focused programming was not rated as highly as a program focus.

Only 1 of the 15 sites visited had a mentoring program that (1) lasted for the whole year; (2) formally matched one mentor with one youth; (3) provided training, ongoing support, and recognition to mentors; and (4) provided space and time for the mentor program each week.

Staff and youth often cited homework help as a program strength. Staff at all sites said that many participants’ parents worked and often did not have the time, energy, or educational background to help with homework. So according to many respondents, if homework was not completed at Teen REACH, it would not get done.

Some youth noted that the program exposed them to new and different activities, such as participating in statewide youth leadership conferences, attending and producing plays, and visiting major museums. Staff and youth indicated that participants’ families rarely provided such experiences. Further, some urban youth felt they could not safely leave their neighborhood, while some rural youth had never been to a large city prior to the program.
Parent/Community Involvement Most sites saw parent involvement as the most difficult area to implement. Impediments included parents’ inability to attend events due to work schedules and childcare needs for their other children. Some families did not have phones, so staff could not easily contact them. In addition, some families did not cooperate with staff; one site staff member said that the program staff would phone parents when youth got in trouble, but parents did not address the problems.

Many sites described strong relationships with other groups in their communities, such as structured relationships with local police, 4-H, the park system, University of Illinois Extension, local hospitals, community colleges, universities, and local businesses.

In response to an open-ended question about program impacts, some staff noted improved parent involvement.
Program Context/Infrastructure A major program strength observed across the 15 sites was the adaptability to local needs and resources. The program’s core areas were similar throughout the state, but local staff seemed to embed the program in their particular communities. For example, staff at a particularly diverse inner city program used their youth’s different cultural groups as themes around which they created activities to promote appreciation and understanding of differences.

Many youth and staff reported that the program was a safe place, particularly among inner-city sites (but also mentioned at some rural sites). For most youth and staff, safety meant adult supervised activities. For others it meant keeping youth away from gangs and gang influence.

The provision of appropriate space, site maintenance, transportation, and materials varied greatly across the sites visited.
Program-School Linkages Staff reported strong relationships with schools, particularly sites that worked closely with local schools that had an enthusiastic and supportive school administration.
Staffing/Training Of staff that responded to a survey item asking them to rate the extent to which their site’s staff agreed on the overall program vision (n = 182), 45% reported complete agreement, 50% reported some agreement, 5% reported very little agreement, and one respondent indicated no agreement. Sites with the highest average agreement levels were also those with the highest staff job satisfaction levels.

The majority of surveyed staff had some postsecondary education (ranging from some college with no degree to graduate/professional degree), including 86% of program directors, 94% of site coordinators, and 72% of direct care staff.
Sixty-three percent of program directors, 80% of site coordinators, and 45% of direct care staff had at least 5 years of experience working with youth.

Of staff responding to the survey item on length of employment at Teen REACH (n = 183), 35% had worked there for more than 2 years, 22% for 1–2 years, and 43% for less than 1 year. The average was 1.52 years.

Just over one-fourth of staff responding to the survey (28%) indicated that staff turnover was a moderately serious or serious issue at their site.

Project directors viewed their roles as primarily administrative, indicating that tasks such as monitoring the numbers of youth served by the program and supervising staff were the most important parts of their roles. Also rated as highly important were liaison roles, such as educating the community about Teen REACH, and communicating with schools. Site coordinators viewed communicating with schools, working directly with youth, and generating program and activity ideas as their most important roles. Direct care staff indicated that their most important roles involved direct contact with participants, i.e., working with youth in groups and individually and establishing meaningful one-on-one relationships with youth. Staff in all three types of positions indicated that increasing their knowledge through ongoing professional development was a key responsibility.

Staff questioned their level of preparedness in several areas that they considered very important for their jobs. These areas were generating program and activity ideas (site coordinators and direct care staff), program evaluation (program directors), working with families (site coordinators), and establishing/maintaining an advisory board (program directors).

The areas of training that the majority of staff reported needing more of included working with special needs youth, increasing parent involvement, and using recreation to manage and teach behavior. Project directors, in particular, noted the need for additional training in curriculum design and research-based best practices.

Staff tended to report their colleagues and pride in their work as aspects of their jobs they liked best. Staff were least satisfied with salary, benefits, and promotion opportunities.

In response to an open-ended question asking what might make their jobs more satisfying, some staff indicated a need for increased funding and increased salary or benefits. Other themes included the need to hire additional staff (especially staff that could relate to youth), improve staff communication/coordination, increase parent involvement, increase program rules/structure/consistency, provide more staff development/training, serve more youth/expand the program, serve different groups of youth, decrease paperwork, increase cooperation from schools, increase the length of the funding cycle, and increase community involvement.

In 14 of the 15 sites, there was a clear “cornerstone” staff member for that site. These staff had a passion for reaching out to youth and had dedicated themselves to making the program work. Specifically, they worked to obtain additional program resources, create new and interesting projects for participants, and meet with school personnel or parents.

Staff repeatedly mentioned the increasing amounts of time needed to complete paperwork required by their various funding sources. They felt that these administrative requirements took away valuable staff time with youth. Staff also commented that the staff who were more effective with youth tended to also be the ones best qualified to complete paperwork.
Systemic Infrastructure Staff at all sites noted difficulties in maintaining, sustaining, and expanding their initiatives. In particular, program directors and coordinators noted the amount of time and creativity they spent to find enough resources to grow or even just maintain the program.

Sites that had more success in securing funding beyond the DHS grant were those that most successfully implemented the full complement of core services and best practices.


Summative/Outcome Findings

Academic Program dosage and attendance level were significantly correlated with both youth perceptions of the amount learned from program participation and with school absences (p < .01 each for perceived learning; p < .01 and p < .05, respectively for school absences). For amount learned from the program, correlations were in the predicted (positive) directions. However, for school absences, the relationship was in the opposite direction expected, with increases in program dosage and attendance levels relating to increases in absences. There were no other significant correlations between program dosage or attendance levels with other academic outcomes assessed (i.e., homework completion, self-reported grades, academic aspirations, and perceived school environment).

In response to an open-ended question about their experiences in Teen REACH, some youth mentioned the benefits they got from help with homework/better grades (e.g., “I’ve been getting higher grades because they help me”).

In response to an open-ended question about program impacts, many staff focused on improved academic performance (e.g., “I have watched and assisted several of our participants increase their grades from C’s, D’s, and F’s to A’s and B’s and making the school honor roll in the year or two [that they were] in our program”).
Family Both program dosage and program attendance level were significantly positively correlated with parent involvement in their child’s education (p < .01 and p < .05, respectively).
Prevention Neither program dosage nor program attendance level was significantly correlated with negative/delinquent behavior.
Youth Development Program dosage was significantly positively correlated with leadership skills (p < .05) and with adult connections (p < .01). There were no other significant correlations between dosage and youth development outcomes assessed (i.e., self-concept, problem-solving skills, and peer-group cohesion).

Program attendance level was positively correlated with leadership skills, peer-group cohesion, and adult connections (p < .05 for each). There were no other significant correlations between attendance levels and youth development outcomes assessed (i.e., self-concept and problem-solving skills).

In response to an open-ended question about program impacts, many staff noted improved participant attitudes and behavior, e.g., “One of our students was new to our school this year; she was very unhappy, unmotivated, and a discipline problem. By the end of the year she was happy, had made a lot of friends, and was absolutely not a discipline problem.” Staff also noted that youth had improved self-concept, e.g., “We have a girl in the sixth grade, who was very shy; she now has more self-esteem, because she is singing in our yearly talent show by herself and without music. She has really come out of her shell.”

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project