Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of BELL After School Instructional Curriculum

Program Description

Overview	The BELL (Building Educated Leaders for Life) After-School Instructional Curriculum (BASICs) is one of the BELL's two ongoing initiatives during nonschool hours (along with the BELL Accelerated Learning Summer Program [BELL Summer]). These out-of-school time educational programs are designed to: increase children's knowledge and mastery of reading, writing, and mathematics; raise children's academic expectations and self-esteem; empower parents; and develop effective mentoring relationships between children and positive adult role models. Basics is a 30-week extended-day tutorial operated in Boston, Massachusetts; New York, New York; and Washington, D.C. The program aims to improve youth's academic performance (in literacy and math), self-concept (including academic self-concept, effort, and motivation to learn), and social/community skills (including engagement as positive members of the community and the development of leadership skills).
Start Date	1992
Scope	national
Type	after school
Location	urban
Setting	public schools

Participants	kindergarten and elementary students (K–6)
Number of Sites/Grantees	15 schools in 2001–2002 and 17 schools in 2002–2003
Number Served	850 in 2001–2002
Components	Basics meets five days per week during the school year for approximately three hours each session. The program targets students of color living in low-income urban communities in grades K–6 who are performing below grade level in school. The program mobilizes hundreds of high school students, college students, and other community role models to work closely with elementary school children in small group tutoring and mentoring relationships. These tutors are placed at sites to be reflective of scholars' race and ethnic backgrounds, so that scholars may see role models similar to themselves. After being recommended for the program by teachers, principals, or parents, Basics scholars' parents then must elect to enroll their children into Basics and some parents pay a fee based on a sliding scale and provide transportation home when the program ends. Scholars receive curriculum that is aligned to state and national learning standards. For scholars in both Basics and BELL Summer, six literacy themes are covered, of which the first five are taught during the Basics program and the final theme is taught during BELL Summer. In addition, the mathematics curriculum is used in both the Basics program and BELL Summer. Scholars spend 1.5 hours of program time on instruction in basic reading, writing, and mathematics skills, four days each week. Scholars are instructed in small groups of no more than five per tutor. In addition to the basic instruction time, scholars are given 45–60 minutes to build skills related to classroom content areas by working on school homework assignments. On Fridays, special art, music, and drama projects allow scholars to demonstrate their strengths in ways other than traditional academic performance. Fridays are also devoted to the development of quality relationships between scholars and tutors and sometimes include guest speakers. Twenty minutes at the start of each day is devoted to Community Time, in which scholars come together to discuss issues relevant to their community. Scholars also participate in at least two community service projects per year. Each site employs an Educational Advisor (EA), who develops goals for each scholar using preprogram diagnostic assessments and various other forms of input. Tutors are then coached by the EA to offer appropriate and effective instruction aligned with scholars' goals. When needed, scholars are given one-on-one instruction by the EA. Basics encourages parents to read nightly with their children, participate in special events and program celebrations, and develop leadership and advocacy skills through participation in monthly workshops. Scholars' homework is also left unfinished during the program so that parents maintain this link between school and home. Program site managers engage school-day teachers in Basics by sharing and soliciting assessment information from them and by attending faculty meetings so that the Basics program is coordinated with school-day lessons as much as possible.
Funding Level	$4.1 million (2003)
Funding Sources	Arthur M. Blank Family Foundation, Boston Foundation, Boston's After School for All Partnership, Clipper Ship Foundation, Edna McConnell Clark Foundation, Educational Foundation of America, FleetBoston Financial Foundation, Fund for Nonprofit Partnerships in the Boston Public Schools, Harvard After School Initiative, Hyams Foundation, Inc., Nellie Mae Foundation, New Profit, Inc., Putnam Senior Executives Foundation, Smith Family Foundation, State Street Global Philanthropy Program, Verizon Communications, parent fees (on a sliding scale)

Evaluation

Overview	BELL undertakes evaluation to ascertain the extent to which its programs meet the needs of the children served, to improve its programs, and to disseminate best practices to the community. The evaluation aims to answer questions about the Basics program outcomes of interest to all program stakeholders.
Evaluator	BELL
Evaluations Profiled	Basics Afterschool Program 2001–2002 Academic Year Evaluation Report
Evaluations Planned	The program is evaluated annually.
Report Availability	BELL. (2002). Basics afterschool program 2001–2002 academic year evaluation report. Dorchester, MA: Author.

Contacts

Evaluation	Tiffany M. Cooper Director of Evaluation BELL National 60 Clayton Street Dorchester, MA 02122 Tel: 617-282-1567 ext. 108 Fax: 617-282-2698 Email: tcooper@bellnational.org
Program	Tiffany M. Cooper Director of Evaluation BELL National 60 Clayton Street Dorchester, MA 02122 Tel: 617-282-1567 ext. 108 Fax: 617-282-2698 Email: tcooper@bellnational.org
Profile Updated	August 5, 2003

Evaluation: Basics Afterschool Program 2001–2002 Academic Year Evaluation Report

Evaluation Description

Evaluation Purpose	To answer the following questions: (1) Was the Basics program associated with an increase in scholars' reading and math skills, academic self-concept, and/or attitude toward learning? (2) Was the Basics program associated with improvements in scholars' social competency and classroom performance? (3) What did parents think of the program?
Evaluation Design	Quasi-Experimental: The evaluators collected pre- and post-measures of various student outcomes from all students enrolled in the program sites at both the beginning and end of the program and compared differences in these measures to national norms on the measures of interest. Social skill development data were collected from eight sites out of 15 (given the burden of data collection in this regard).
Data Collection Methods	Document Review: Evaluators reviewed scholars' “portfolios” for indications of improvement in scholars' work products. Portfolios include scholars' writing samples, book reports, quizzes, and math and literacy exercises collected in the program. The portfolios also have each scholar's individual goals, teacher and parent assessments of their weaknesses, and tutors' observations of the scholar's progress. Site managers and tutors compile progress reports based on these portfolios for each scholar at the end of the first semester, in January, and again at the end of the program year. Secondary Source/Data Review: School Report Cards are collected from principals and/or teachers to examine whether Basics translates into school improvement. Grades are reported for English language arts, mathematics, and conduct/social development/social growth. These grades were compiled and analyzed for the three marking terms that scholars were in the program. Family income data from scholars at the Boston sites were examined. Surveys/Questionnaires: The Basics Report Card is Bell's survey for parents. At the end of the program, parents were asked to grade the Basics program on its structure and content, program staff, and effectiveness at reaching objectives for scholars and at partnering with parents. They were also asked to grade the overall program as an indication of their overall satisfaction. Tests/Assessments: Basics scholars in grades one to six were administered one of four grade levels of the Stanford Diagnostic Reading Test-IV (SDRT-IV) and the Stanford Diagnostic Mathematics Test-IV (SDMT-IV) at the beginning of the program, and again during its final weeks. Scholars in first and second grade were administered the same level, as were fifth and sixth grade scholars. First through fourth grade scholars were assessed in Phonetic Analysis, and Vocabulary and Comprehension, while fifth and sixth graders were assessed in Vocabulary, Comprehension, and Scanning techniques. The math assessment covers Concepts and Applications, and Computation for all grades. In addition, scholars' tutors and/or Site Managers completed the School Social Behavior Scales (SSBS) (Merrell, 1993) for each scholar within four to six weeks of a scholar's start date and again at the end of the program. The SSBS contains four Social Functional Levels, “high functioning,” “average,” “at-risk,” and “high risk.” Levels are determined by placing students scores along scores of a national norm group (i.e., “high functioning” includes scores above 80% of the national norm group). The instrument contains information on both social competence and antisocial behavior patterns, but it was the social competency component that was of interest to BELL, specifically its measures of interpersonal skills, self-management skills, and leadership skills (the latter added to the instrument by BELL). Finally, the Perception of Ability Scale for Students (PASS) (Boersma & Chapman, 1992) was administered at both the beginning and end of the Basics program in order to assess improvement in areas of academic self-concept. BELL was primarily interested in three components of the scale: perception of math ability, perception of reading ability, and school satisfaction. References Merrell, K. (1993). School social behavior scales. Eugene, OR: Assessment-Intervention Resources. Boersma, F. J., & Chapman, J. W. (1992). Perception of ability scale for students. Los Angeles: Western Psychological Services.
Data Collection Timeframe	Data were collected during the 2001–2002 school year.

Findings:
Formative/Process Findings

Recruitment/Participation

Program participants were approximately equally distributed along gender lines.

One quarter of 2001–2002 Basics scholars had some previous involvement in a BELL program.

The average family income of scholars at Boston sites was roughly $16,000.

Of the 309 scholars observed for social functioning using the SSBS scale, 94% were demonstrating less than “high functioning,” with 18% functioning at an “at-risk” or “high-risk” level.

The majority of scholars indicated that they enrolled in Basics to develop their literacy, math, or some other academic skills.

Satisfaction

The parent report card indicated that parents gave the program a B+ in program structure and content, program effectiveness, and partnership with parents.

Parents rated the program an A- in program staff's professionalism and competency, and for the overall grade.

Parents' additional comments on the report card revealed feelings that the program helped their children and that parents' valued the services provided by Basics

Summative/Outcome Findings

Academic

Scholars' reading and math performance was significantly higher in the spring than in the fall (p<.001). When gains were measured in terms of normal curve equivalent units (NCEs), which look at growth compared to the national norm group, scholars outpaced the norm group for both reading and math, although the difference was only significant for math NCE scores (p<.05).

On average, Basics scholars began Basics five months behind grade level in reading and three months behind in math. By the end of the program, scholars were one month above grade level in reading and three months above grade level in math.

Roughly 78% of scholars were at one of the two highest performance levels in reading at the pretest, as compared to approximately 84% at the posttest.

Approximately 47% of scholars were at one of the two highest performance levels in math at the pretest, as compared to 69% at the posttest.

Scholars' report cards indicated improvement over the school year. Scholars' grades moved from a C to a C+ in English Language Arts and from a C to a B- in Math. These grade point average gains were significant (p<.05) in English Language Arts.

Basics Progress Reports indicated significant improvements in reading, writing, and math, as evidenced in the quality of the work in their portfolios.

Scholars' perception of their math ability increased significantly from pretest to posttest (p<.001). Perceptions of reading ability increased from pretest to posttest, but were not significant, while satisfaction with school decreased slightly across the program year, but also was not significant.

Youth Development

Social skill development, as measured by the SSBS scale, was positive, although not substantial or significant. Scholars who attended more than 90% of the program session had significantly higher gains in Interpersonal Skills and Leadership Skills than scholars with low attendance (p<.05).

Scholars' report cards indicated significant improvements over the school year in Social Growth (p<.05). Scholars' grades in this area moved from a C+ to a B-.

A Profile of the Evaluation of BELL After School Instructional Curriculum

Profile Information