The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> Fifth Dimension/University-Community Links
A Profile of the Evaluation of Fifth Dimension/University-Community Links
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Program Description
| Overview | The Fifth Dimension/University-Community (UC) Links is an approach to after school programming used by Boys and Girls Clubs, YMCAs and YWCAs, recreation centers, and public schools across America, Mexico, Brazil, Australia, Denmark, Sweden, Spain, and Russia. It provides a way to increase the educational programming of such institutions without substantially increasing the costs of operation. Beginning in 1986, the overarching goals of the Fifth Dimension approach were to: (1) create sustainable activity systems that increase understanding of the cultural mediation of mind and the processes of cognitive and social development, (2) provide contexts for children to master knowledge and skills mediating changes in their everyday practices, (3) deepen understanding of how the social and individual create each other, and to (4) provide a context in which undergraduates from disciplines such as teacher education, developmental psychology, and communications have opportunities to connect theory with practice and deliver services to children. In 1996 the University of California Office of the President provided seed funding to expand this model throughout California. The UC Links network was established to promote the university eligibility and academic preparedness of underserved youth throughout California using the practices developed by Fifth Dimension programs worldwide. The intent was to broaden the base of K–12 learning, by providing access to educational resources to those youth who are not yet on the path to higher education. The statewide program has three major objectives: (1) to improve the educational opportunities and aspirations for diverse California youth from low-income communities throughout the state, so that more of them will become eligible for the University of California in years to come, (2) to improve undergraduate education by providing living laboratories where students can integrate theory and practice, and (3) to pioneer new ways of collaborating that bring together institutions like universities, communities, and K–12 schools in long-term, sustainable programs for underserved K–12 youth. |
| Start Date | 1986 |
| Scope | international |
| Type | after school |
| Location | urban, suburban, rural |
| Setting | community-based organization, public school, recreation center |
| Participants | elementary, middle, and high school students (ages 5–18) |
| Number of Sites/Grantees | over 50 worldwide sites (2003) |
| Number Served | The number of children per site will vary according to setting. In a public school setting, as many as 80 children may be served in any week. |
| Components | Fifth Dimension programs take place in a computer club that may or may not be associated with the child's school and are located near university campuses since most are affiliated with colleges and universities. Participants are often from low-income homes and many attend on a regular basis through high school. At some sites, high schoolers who attended return and act as mentors for the younger children. Children typically visit a Fifth Dimension on a drop-in basis, although in most public school Fifth Dimensions children attend on a daily basis. Some children spend four to six hours per week of after school time participating in the Fifth Dimension, while others may come once a week for a few hours. Opportunities and constraints vary across locations, seasons, populations, and sites. Many children enter the Fifth Dimension directly after school or homework sessions. The main activities center around off-the-shelf educational programs selected for appropriateness, appeal, and educational value. In a prototype Fifth Dimension system (local names vary), a dozen or more children encounter a large variety of computer games, noncomputer games, and telecommunications activities on a regular basis over the course of a school year. As a rule, Fifth Dimensions contain a variety of kinds of computers at a ratio of one computer for every two to three children. Computer games, such as Carmen San Diego and the Secret Island of Dr. Brain, the Magic School Bus series, and noncomputer games, such as Origami, chess, and Boggle are a part of a make-believe activity system. Task cards that accompany each game or activity enable children and their mentors to negotiate goals for playing games and coordinate activities. Learning is collaborative with children often working together or with adult mentors called Wizard's Assistants. Students with extensive experience are given the opportunity to achieve the status of Young Assistant to the Wizard. The child's participation is voluntary and self-paced; there is no traditional teacher-student structure. Each Fifth Dimension has a site coordinator who is trained to recognize and support Fifth Dimension's pedagogical ideals and curricular materials. Undergraduate students are there to learn and play with the children. The site coordinator monitors the balance of education and play in interactions between children and undergraduates. |
| Funding Level | varies by local program |
| Funding Source | The Andrew W. Mellon Foundation provided funding for field-testing and evaluation of the original three programs begun in 1986. In 1996 the University of California Office of the President began funding the UC Links network of after school programs in California. Each local Fifth Dimension program negotiates operating resources from the university, community participants, and other outside funding agencies. |
Evaluation
| Overview | Evaluators developed collaborations for evaluation at three Fifth Dimension sites: Appalachian State University (ASU), California State University at San Marcos (CSUSM), and University of California at Santa Barbara (UCSB). The ASU team evaluated Fifth Dimension sites that were operated as part of an after school program at four elementary schools in Boone, North Carolina. The CSUSM team evaluated a Fifth Dimension site operating at the Boys and Girls Club in Escondido, California. The UCSB team evaluated a Fifth Dimension site operating at the Boys and Girls Club in Goleta, California. Evaluators wanted to assess changes in children's literacy that can be attributed to participation in the Fifth Dimension, specifically: (1) does learning to use educational software on computers improve a child's mind? and (2) what kinds of cognitive changes, if any, occur over the course of a year in which computer-naïve children learn to use a series of educational programs in an informal and nonthreatening environment? The outcome measures tapped changes in student literacy, broadly defined, including changes in computer literacy, language comprehension, problem-solving strategies, and academic achievement. The primary research method was to assess relevant cognitive skills of students before they began the Fifth Dimension program and after they had extensive experience in the program (e.g., more than 10 or 20 visits over the course of the academic year), and to compare their pretest to posttest changes with those of similar students who did not participate. Separate evaluations were also conducted of the Expedition after school program, a Fifth Dimension program with an archaeological learning framework in Oakland, California. Expedition involves University of California at Berkeley (UCB) faculty, staff, and students directly with sixth graders through a service learning course, Anthropology 128, Archaeological Practice in a Sixth Grade After-School Program. These evaluations sought to measure the program's success in achieving a number of youth outcomes, as well as UCB undergraduate students' goals. |
| Evaluators | William E. Blanton, Gary B. Moorman, Bobbie A. Hayes, and Mark L. Warner, Appalachian State University Miriam W. Schustack, Rachelle Strauss, and Patricia E. Worden, California State University at San Marcos Richard E. Mayer, Richard Durán, Amy Lavezzo, Roxana Moreno, Jill Quilici, David Sanchez, Rebecca Simon, and Scott Woodbridge, University of California at Santa Barbara Tamara Lynn Sturak, UC Links Program, Graduate School of Education, University of California, Berkeley |
| Evaluations Profiled | Effects of Participation in the Fifth Dimension on Far Transfer (ASU) Learning About Technology in a Non-Instructional Environment (CSUSM) Cognitive Consequences of Participation in a Fifth Dimension After-School Computer Club (UCSB) What is Learned in an After-School Computer Club? (UCSB) Evaluation of Expedition – Computers and Archaeology After School (UCB) Expedition – Computers and Archaeology After School: Year-End Report, 2000–2001 (UCB) |
| Evaluations Planned | Evaluations are ongoing. |
| Report Availability | Note: This is just a sampling of reports available. Many of these reports, as well as others, are available at: www.education.miami.edu/ blantonw/5dclhse/publications/pub1.html. Summary Reports Mayer, R. E., Blanton, W. E., Durán, R., & Schustack, M. W. (1999). Using new information technologies in the creation of sustainable afterschool literacy activities: Evaluation of cognitive outcomes. Available at www.psych.ucsb.edu/~mayer/fifth_dim_website/ HTML/res_reports/research_reports.html. Mayer, R. E., Schustack, M., & Blanton, W. (1999). What do children learn from using computers in an informal collaborative setting? Educational Technology, 39(2), 27–31. Mayer, R. E. (1997). Out-of-school learning: The case of an after-school computer club. Journal of Educational Computing Research, 16, 333–336. Underwood, C., Welsh, M., Emmons, C., Lerner, D., & Sturak, T. (2002). University-Community Links to higher learning: Program impact report. Berkeley: University of California at Berkeley, Office of the President, Educational Outreach Department. Available at: www.uclinks.org. ASU Blanton, W. E., Moorman, G. B., Hayes, B. A., & Warner, M. L. (1997). Effects of participation in the Fifth Dimension on far transfer. Journal of Educational Computing Research, 16, 371–396. CSUSM Schustack, M. W., Strauss, R. & Worden, P. E. (1997). Learning about technology in a non-instructional environment. Journal of Educational Computing Research, 16, 337–352. UCSB Mayer, R. E., Quilici, J., Moreno, R., Durán, R., Woodbridge, S., Simon, R., et al. (1997). Cognitive consequences of participation in a Fifth Dimension after-school computer club. Journal of Educational Computing Research, 16, 353–369. Mayer, R. E., Quilici, J. H., & Moreno, R. (1999). What is learned in an after-school computer club? Journal of Educational Computing Research, 20, 223–235. UCB Sturak, T. L. (2000). Evaluation of Expedition – computers and archaeology after school. Berkeley: Interactive University Project, University of California at Berkeley. Available at www.mactia.berkeley.edu/aop/activity/expedition.pdf (Acrobat file). Sturak, T. L. (2001). Expedition – computers and archaeology after school: Year-end report, 2000–2001. Berkeley: Interactive University Project, University of California at Berkeley. |
Contacts
| Evaluation | UC Links Mara Welsh Mahmood UC Links Statewide Office 615C University Hall Berkeley, CA 94720-1040 Tel: 510-643-7349 Email: mwelsh@uclink4.berkeley.edu ASU William E. Blanton, Ph.D., Professor of Teaching and Learning University of Miami, School of Education Merrick 324-A 5202 University Drive Coral Gables, FL 33146 Tel: 305-284-5053 Email: blantonw@miami.edu CSUSM Miriam W. Schustack, Ph.D., Professor Department of Psychology California State University 320 University Hall San Marcos, CA 92096-0001 Tel: 760-750-4095 Email: mschusta@csusm.edu UCSB Richard E. Mayer, Ph.D., Professor Department of Psychology University of California, Santa Barbara Santa Barbara, CA 93106 Tel: 805-893-2472 Email: mayer@psych.ucsb.edu UCB Tamara Lynn Sturak, After School and Community Coordinator UC Links Program School of Education University of California at Berkeley 615 C. University Hall Berkeley, CA 94720-1670 Tel: 510-643-7349 Email: tamara@uclink.berkeley.edu |
| Program | UC Links UC Links Statewide Office 615C University Hall Berkeley, CA 94720-1040 Tel: 510-643-7349 Email: uclinks@socrates.berkeley.edu ASU Walter P. Oldendorf, Ph.D., Coordinator Reich College of Education Appalachian State University Edwin Duncan Hall: EDH 04 730 Rivers Street Boone, NC 28608 Tel: 828-262-7279 Email: oldendrfwp@appstate.edu CSUSM Miriam W. Schustack, Ph.D., Professor Department of Psychology California State University 320 University Hall San Marcos, CA 92096-0001 Tel: 760-750-4095 Email: mschusta@csusm.edu UCSB Lupe Arteaga, Club Coordinator Club Proteo 5701 Hollister Avenue Goleta, CA 93117 Tel: 805-967-1612 Email: marteaga@education.ucsb.edu UCB Tamara Lynn Sturak, After School and Community Coordinator UC Links Program School of Education University of California at Berkeley 615 C. University Hall Berkeley, CA 94720-1670 Tel: 510-643-7349 Email: Tamara@uclink.berkeley.edu |
| Profile Updated | May 20, 2003 |
Evaluation 1 (ASU): Effects of Participation in the Fifth Dimension on Far Transfer
Evaluation Description
| Evaluation Purpose | To determine the effects of children's participation in the Fifth Dimension on measures of their ability to transfer the problem-solving skills learned in the program to what they learned in the subject matter areas of reading and math. |
| Evaluation Design | Quasi-Experimental and Non-Experimental: Participants were 52 children from grades three, four, five, and six from four elementary schools in Boone, North Carolina. The experimental group was made up of 26 children who volunteered to participate in the Fifth Dimension program at Appalachian State University. The control group consisted of 26 children from the same schools who did not attend the Fifth Dimension. The two groups were matched on gender and classroom. The control group did not participate in any organized computer activity after school, although many of the children were engaged in a wide range of socially organized activity, ranging from sports to school band. |
| Data Collection Methods | Observation: Observations were made during 30 to 35 one-hour visits to the Fifth Dimension. Preservice teacher interns who worked with the children wrote field notes after each session. Field notes included information on children's activities, use of strategies, interactions with other children, etc. Tests/Assessments: The North Carolina End-of-Grade Tests were administered to students as a pretest and posttest. These reading and mathematics achievement tests are statewide standardized tests of basic academic skills administered at the end of each academic year. These tests were developed for the Course of Study prescribed by the North Carolina Department of Public Instruction (DPI), administered independently by the school system, scored by the DPI and reported as standard scores. |
| Data Collection Timeframe | Data were collected during the 1994 to 1995 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Descriptions in the field notes indicate the following elements of Fifth Dimension activity implementation, among others: the integration of basic skills; the rules and consequences involved; the embedding of tools appropriation; the freedom children have in making choices; the intensity of children's engagement in problem solving; the prevalence of playful experiences; the importance of the opportunity for children to earn the title of Young Golem Assistant; evidence of value and attitude acquisition by the children; children's contextualization of activities; strong relationships built between teacher interns and children through joint activity; successful accomplishment of tasks through joint activity with peers; the role of children in teaching one another and the interns; and the role of children in directing their own learning, both in terms of how and what they learn. |
| Staffing/Training | An intern's understanding of the teaching methodology behind Fifth Dimension is illustrated in one of the field notes. The intern comments on the pressure she felt to teach in the Fifth Dimension, although direct teaching is discouraged; instead staff are instructed to provide children with help when needed and only as much as is needed. She explains her realization that the children learn more by enjoying the activity than if they think they are doing the activity solely for the sake of learning. This idea is in line with the goals of the Fifth Dimension program. Interns use inferential questions during activities with participants. |
Summative/Outcome Findings
| Academic | Participants had significantly higher adjusted posttest reading and math achievement scores than the control group (p<.001 for both reading and math). The mean unadjusted posttest score for reading achievement on the North Carolina End-of-Grade Test for the treatment group was 156.9 for the experimental group and 148.7 for the control group. The mean unadjusted posttest score for math achievement for the treatment group was 164.1 for the treatment group and 155.6 for the control group. Differences in pretest scores accounted for 55% of the posttest variance on reading scores and 78% on math scores. Group membership as a dummy variable in the equation accounted for an additional 23% of the variance on the reading test and 9% on the math test. Adding gender to the equation did not account for a significant portion of the variance for either the reading or math test, indicating that gender did not interact significantly with group in predicting posttest scores. |
Evaluation 2 (CSUSM): Learning About Technology in a Non-Instructional Environment
Evaluation Description
| Evaluation Purpose | To measure the participants' ability to remember computer and noncomputer Fifth Dimension terminology as an indirect measure of the respective domain knowledge. The logic of this study relies on a finding from literature on expertise—even in tasks that do not require domain knowledge for success, people who have greater familiarity with the meanings and relationships among technical terms show improved performance. That is, the tasks that require simple processing of word-level stimuli show consistently better performance by domain experts—for whom these terms are rich and meaningful—than by domain novices for whom the words denote less elaborated concepts. |
| Evaluation Design | Quasi-Experimental: Participants were children ages 8–13 who were members of the Boys and Girls Club of Escondido, and who received parental permission to participate in the Fifth Dimension. One group consisted of children considered Fifth Dimension experts, who had completed 11–23 sessions in the Fifth Dimension prior to testing (N=13, mean age 9.8 years). The other group was Fifth Dimension novices—children who had just signed up for the Fifth Dimension, but had not yet attended any sessions (N=15, mean age 9.3 years). The groups were based on voluntary involvement and thus were not randomly selected. Participants entered the study on a revolving basis, as they qualified for inclusion in one of the two groups: the novices became eligible when they turned in signed permission slips (before participating in any Fifth Dimension activities) and the experts became eligible as of their eleventh Fifth Dimension visit. Each child was tested as soon as possible after entering the study, often within a few days, although there was sometimes a delay due to testing time constraints and child availability. The novices were all tested before engaging in any Fifth Dimension activities. The experts could continue to attend the program after they reached the criterion of 11 visits, so their mean number of visits before the time they were tested was 15.3. Because the researchers were unable to have randomly selected groups, they used controls internal to the design. Specifically, in their assessment, they included a condition of neutral words so that differences in overall memory skill can be excluded as an explanation for group differences—essentially, each participant served as his/her own control. |
| Data Collection Methods | Tests/Assessments: Children were tested on their recognition memory for a word list, comparing memory for words that are neutral with respect to the Fifth Dimension environment to words specific to the Fifth Dimension environment. The presence of the neutral items allowed researchers to determine whether any group differences on these measures might be due to improved general memory ability, or whether they could reasonably infer that the children were gaining familiarity with the domain of computers through their participation in the program. The test consisted of three sections run in succession on a computer: an initial word presentation, a recognition task, and a recall task. In the first two sections, target words presented consisted of 10 words each from 3 different categories: (1) words related to computer technology, (2) words related to common Fifth Dimension activities, and (3) neutral/unrelated words (words of moderate to low frequency, comparable to the two other categories, that the children would normally encounter in other environments, but not especially likely to occur in the Fifth Dimension). The 30 target words were presented in a different random order for each child. The words were selected by initially generating a much larger set, and then selecting the 20 words that seemed to best capture each of the three categories. Of the 20, 10 were randomly selected to be used as target words and the other 10 were used as recognition foils. Buffer items were additional neutral/unrelated words that were presented in fixed positions. In the initial word presentation, words to be remembered were presented one at a time for one second each. Thirty-six words were presented—thirty target words and six buffer words. Buffer words always appeared in the first and last three positions of the list to minimize primacy and recency effects on the target items. Children then engaged in a nonverbal distraction activity for a filled delay interval before moving on to the recognition task. The recognition task required participants to look at pairs of words and select the one that had appeared on the memory list. There were 34 word pairs, each containing one previously presented word and one foil word not previously presented. Thirty of these pairs consisted of critical target words paired with a foil from a different category. The remaining four pairs contained previously presented buffer words paired with new neutral/unrelated words as foils. The buffer pairs were always presented at the beginning of the task to minimize practice effects on the critical pairs. There was no time limit, but the computer program recorded the time for each response. Each participant's responses were scored for accuracy and the number of items correctly recognized was computed. For the recall task, participants were asked to say out loud as many of the words from the initial word presentation as possible. Responses were tape recorded and did not have a time limit. The number of items correctly recognized in each condition was computed by counting the number of originally presented target words and the number of recognition foil words that appeared in the recall on each participant's transcribed tape. |
| Data Collection Timeframe | The data were collected January through April 1995. |
Findings:
Summative/Outcome Finding
| Academic | Experts and novices had almost identical overall correct recognition, with each group recognizing 67% of the items. The experts recognized a mean of 20.2 items out of the 30 targets (SD=4.4), and the novices recognized 20.1 items (SD=3.5). This small difference was not statistically reliable (t(26)=.11, p>.90). The mean correct recognition for each category of item showed no main effect of word type—over all the subjects taken together, the three categories of words were approximately equally well recognized, F(2,52)<1. There was, however, a highly reliable interaction between the grouping variable (expert vs. novice children) and the word type, F(2,52)=6.72, p<.003. The children with more Fifth Dimension experience showed an advantage on both computer and Fifth Dimension words, as predicted. The experts showed a corresponding disadvantage on the neutral words, supporting the notion that their success with the domain-relevant words is due to specific experience with those concepts rather than to a general improvement in their memory skill. The disadvantage on the neutral words is almost exactly equal in magnitude to the sum of the advantages on the other words, leading to no overall difference in recognition accuracy between the groups. Response times for correct responses in the recognition task were highly variable within and between children, possibly because speed was not mentioned as a goal in the task. Statistical analysis showed no significant effects of group on condition. The trend of the response times, though, was conceptually consistent with the pattern of recognition accuracy and shows no evidence of a speed-accuracy tradeoff. For novices, the neutral category words (their best recognized word type) were recognized more quickly than the other categories, while for the experts the neutral words (their least-recognized word type) were the slowest to be recognized. Recall performance was quite low, consistent with the limited memory strategies of the average nine year old. For recall of target words, novices recalled an average of 1.9 target words out of the 30 presented (SD=1.5), and the experts recalled nearly twice as much: 3.4 target words (SD=2.2). Despite the very small number of items recalled, this difference (6% correctly recalled vs. 11%) was quite statistically reliable, F(1,26)=5.78, p=.02. The advantage of the experts appears to be primarily in their much greater recall of the computer words; this superiority of the experts in recall of the computer words was statistically significant, t(26)=2.176, p<.04. There were no reliable differences between the groups on the other two types of items: for Fifth Dimension words t(26)=1.351, p=.19, for neutral words t(26)=.165, p=.87. Because of the low recall rate, researchers also analyzed the recall data including both the originally presented words and the recognition foils. The reasoning behind this alternative analysis was that the children might have had difficulty distinguishing the words originally presented from the words seen only in the recognition task, or might not have been sure which group of words they were supposed to recall. Recall was somewhat higher under this more lenient scoring, suggesting that the children may have been confused about which words to report, or about whether a remembered word was seen in the initial phase or only in the later recognition task. Overall, the novices recorded 2.5 words (targets and foils combined), and the experts 4.2 words. There was a main effect of the group, with experts recalling significantly more words overall than novices, F(1,26)=4.62, p=.04. In the recall under this lax scoring, the pattern of differences seen in the standard scoring reappeared-the experts had a significant advantage for the computer words, t(26)=2.29, p=.03, but there were no differences for the other two types of words: for Fifth Dimension words t(26)=1.64, p=.11 and for neutral words t(26)=.03, p=.97. |
Evaluation 3 (UCSB): Cognitive Consequences of Participation in a “Fifth Dimension” After School Computer Club
Evaluation Description
| Evaluation Purpose | To answer the following research questions: (1) Do students who participate in the Fifth Dimension show greater improvement than nonparticipants in their comprehension of arithmetic word problems? (2) Do the benefits of participation in the Fifth Dimension persist over time? (3) Do the benefits of the Fifth Dimension remain for alternative methods of comparison? (4) Do the benefits of the Fifth Dimension persist over time for alternative methods of comparison? Each of the four research questions was handled as a separate “study” of arithmetic word problem improvements, which are, respectively: (1) a matched-by-student characteristics study, (2) a matched-by-student characteristics study of long-term gains, (3) a matched-by-pretest scores study, and (4) a matched-by-pretest scores study of long-term gains. |
| Evaluation Design | Quasi-Experimental: Approximately 120 students were tested, 16 were treatment students, and 16 were selected to constitute a comparison group. The treatment group consisted of third and fourth grade students who attended Club Proteo, a Fifth Dimension Program run by University of California at Santa Barbara faculty with the Goleta Boys and Girls Club, at least 10 times during the 1994–1995 school year. The comparison group consisted of the treatment group's classmates who did not attend Club Proteo and who were matched to the treatment group for grade and English language proficiency school district test scores. In addition, the comparison and treatment groups were matched for schoolteacher and gender in the matched-by-characteristic studies and for pretest scores in the matched-by-pretest studies. |
| Data Collection Methods | Tests/Assessments: The achievement assessment tools were pretest and posttest versions of a mathematics comprehension test called the Word Problem Comprehension Test (WPCT), provided in English and Spanish. The WPCT consisted of 12 problems broken down into three types, with four questions each. The first type presented a sentence from a word problem and asked the student to determine which equation corresponded to it. The second type presented a word problem and asked the student to determine which numbers were needed to solve the problem. The third type presented a word problem and asked the student to determine which arithmetic operations were needed to solve the problem. Tests were scored by tallying the number of correct answers selected by each student. A bilingual teacher administered the tests in both in English and Spanish in the children's regular classroom during the school day. The majority of the students in the study had learned Spanish as their first language. Spanish-speaking students who did not know English well were encouraged to complete as much of the English forms of the tests as possible. Similarly, English-speaking students who did not know Spanish well were encouraged to complete as much of the Spanish forms of the tests as possible. In the matched-by-characteristics studies for the English and Spanish WPCT, 16 treatment students and 16 comparison students took the English and Spanish WPCT as a pretest and posttest. In the matched-by-characteristics study of long-term gains on the English WPCT, 11 treatment students and 11 comparison students took the English WPCT as a pretest, posttest, and delayed posttest. These treatment and comparison students are a subset of the original group of 16 treatment students and 16 comparison students; five of the original treatment students were not among those tested in fall 1995 so they and their five matched comparison students were excluded from analysis. A similar analysis could not be conducted for the Spanish form of the WPCT because half of the students declined to take the test in Spanish in fall 1995. In the matched-by-pretest study of the English WPCT, 15 treatment students and 45 comparison students took the English WPCT as a pretest and posttest. In the matched-by-pretest study on the Spanish WPCT, 14 treatment students and 57 comparison students took the Spanish WPCT as a pretest and posttest. In the matched-by-pretest study of long-term effects, the participants were 12 treatment students and 36 comparison students who took the English WPCT as a pretest, posttest, and delayed posttest. |
| Data Collection Timeframe | The first test was administered at the beginning of the program in fall 1994 (the pretest). The testing was repeated in spring 1995, after treatment students had attended the club at least 10 times, using the posttest versions of each test (the posttest). The testing was again repeated in fall 1996 using the pretest versions of each test (the delayed pretest). |
Findings:
Summative/Outcome Findings
| Academic | Findings of the matched-by-characteristics studies for the English and Spanish tests indicate that Club Proteo participants scored significantly higher than the comparison group during the course of the academic year. The pretest-to-posttest gain was greater for the treatment group than for the control group on the English WPCT, although the difference was only marginally significant, t(15)=1.68, p<.06. The pretest-to-posttest gain was also greater for the treatment group than for the control group on the Spanish test, t(15)=2.85, p<.01. The pretest-to-posttest gain was greater for the treatment group than for the control group on the combined scores for both English and Spanish versions, t(15)=3.19, p<.01. In the matched-by-characteristics study of long-term gains on the English WPCT, results provide evidence that the cognitive benefits of participation in the club persist beyond the end of the school year. The groups did not differ significantly on the pretest, t(10)=.22, p=ns and the treatment group outperformed the comparison group on the posttest, t(10)=2.23, p<.025. This difference was still present on the delayed posttest, t(10)=2.36, p<.025. The matched-by-pretest study results indicate that the matched-by-characteristic study finding that the treatment group had significantly higher test scores over the course of the academic year than the comparison group holds up under an alternative method of comparison. An analysis of variance conducted on the Spanish and English posttest scores with group and pretest score as factors revealed a significant effect (albeit marginally for the English form) for group in which the mean posttest score of the treatment group (M=4.9 for the English form, M=3.8 for the Spanish form) was greater than the mean posttest score of the comparison group (M=3.6), F(1,52)=3.81, MSE= 0.9, p<.06 for the English form, and (M=2.1), F(1,63)=5.98, MSE=3.53, p<.05 for the Spanish form. In addition, the pattern of superiority of the treatment group was consistent across all pretest scores as is indicated by the lack of a significant group x pretest interaction, F(3,52)=1.321, MSE=3.790, p=Ns for the English form and F(3,63)=1.77, MSE=3.53, p=Ns Finally, the ANOVA revealed an effect for pretest score in which posttest scores were greater for students who scored higher on the pretest, F(3,52)=2.87, MSE=8.23 p<.05 for the English form, and F(3,63)=3.11, MSE=3.53, p<.05 for the Spanish form. The matched-by-pretest study of long-term gain results indicate that the matched-by-characteristic study of long-term gains finding that the treatment group had significantly higher test scores that persisted beyond the end of the school year than the comparison group holds up under an alternative method of comparison. Of students who took the pretest, posttest, and delayed posttest for the English form of WPCT, the treatment group performed better than the comparison group on both the posttest and on the delayed posttest. This observation was examined via an ANOVA on the English WPCT scores with group (treatment vs. comparison) and pretest score as between subject factors and time of test (posttest vs. delayed posttest) as a within subjects factor. Consistent with this observation, there was a main effect for group in which the treatment group outscored the comparison group across both the posttest and the delayed posttest, F(1,40)=7.51, MSE=3.53, p<.01; also consistent with this observation, there was no significant interaction between group and time of test, F(1,40)<1, indicating that the superiority of treatment over comparison group did not differ between posttest and delayed posttest. This finding suggests that the benefits of Fifth Dimension participation persist over the summer, such that the superiority of the treatment group over the comparison group can be seen at both the posttest and the delayed posttest. |
Evaluation 4 (UCSB): What Is Learned in an After-School Computer Club?
Evaluation Description
| Evaluation Purpose | To examine the cognitive consequences of participating in an after school computer club. The study tests the hypothesis that exposure to certain kinds of educational computing environments can transfer to improved cognitive processing in different situations. Specifically, researchers were interested in whether students who had participated in a computer club in which they learned to use a variety of educational programs were better able to learn a new piece of educational software that was presented as a paper-and-pencil mathematics test than were matched nonparticipants. The children's ability to transfer their learning to a new task, i.e., a Puzzle Tanks game, was examined by analyzing the frequency of errors and the quality of solution methods. |
| Evaluation Design | Quasi-Experimental: The participants were 50 children from an elementary school in southern California: 25 treatment group members and 25 comparison group members. At the beginning of the school year, several students from each participating class in the school were invited to join Club Proteo, a Fifth Dimension program run by the University of California at Santa Barbara faculty with the Goleta Boys and Girls Club. Students were selected based on teacher recommendations to represent the diversity of children at the school, with a special focus on language minority children. Ninety-two percent of the participants spoke Spanish as their first language and were designated as “limited English proficient” by their school. A total of 25 students (i.e., the treatment group) accepted the invitation and attended the computer club at least 10 times during the school year. The comparison students came from the same classrooms and had the same general characteristics as the treatment students, but had never attended the computer club. Each treatment student was matched with a comparison student who had the same English language proficiency classification, was in the same grade level, and was the same gender. In some cases, the matching was slightly imperfect, but these minor variations did not favor one group over the other. |
| Data Collection Methods | Tests/Assessments: Children were tested using a mathematical learning task. This task was a paper-and-pencil version of a mathematics computer game called Puzzle Tanks which none of the children had ever used. Children worked individually with the experimenter by reading instructions and attempting to solve a series of three puzzles (with immediate feedback after each “move” they made). Experimenters tallied the error rate on the three puzzles to provide an indication of the children's difficulty in learning to follow the directions of the Puzzle Tanks game and their ability to use feedback to help them learn to solve mathematical puzzles that they had never seen before. Experimenters also performed a more fine-grained analysis of the solution strategies that children used in learning to play the Puzzle Tanks game. On the last two learning problems students could use a more sophisticated solution method, which experimenters called a subtraction strategy, or a less sophisticated method, which they called an addition strategy. Based on students' patterns of moves, experimenters tallied the number of times students used either of these two strategies. |
| Data Collection Timeframe | Each student was tested during the last three weeks of the 1997–1998 school year. |
Findings:
Summative/Outcome Findings
| Academic | In learning to play the Puzzle Tanks game over the course of three learning problems, the treatment group failed to solve 24% (SD=28) of the problems within 18 trials whereas the comparison group failed on 43% (SD=35) of the problems. The moderate error rates indicate that the Puzzle Tanks task was an appropriate diagnostic instrument for the participant pool, with neither group displaying either a ceiling or floor effect. A paired t-test revealed that the difference in error rates is statistically significant t(24)=2.347, p<.03. The effect size is .53. In learning to play the Puzzle Tanks game over the course of the final two learning problems, children in the treatment group used the subtraction strategy (the more sophisticated strategy) on 42% (SD=42) of the problems whereas the comparison group used the subtraction strategy on 20% (SD=32) of the problems. A paired t-test revealed that this difference is marginally significant, t(24)=1.901, p<.07. The effect size is .68. In contrast, children in the treatment group used the addition strategy (the less sophisticated strategy) on 20% (SD=24) of the problems whereas the comparison group used it on 24% (SD=28) of the problems, reflecting a nonsignificant difference, t(24)<1, p>.20. |
Evaluation 5 (UCB): Evaluation of Expedition – Computers and Archaeology After School
Evaluation Description
| Evaluation Purpose | To examine how some of the program's objectives were met for both the sixth grade and UCB undergraduate participants during the first year of the Expedition program. Specifically, the evaluation measured success of youth on a series of desired outcomes. These outcomes stated that students will: (1) have fun and want to participate; (2) have a safe place to learn and play after school; (3) learn basic computer skills, such as mouse, interface, and file manipulation, and how to play games; (4) increase their reading and writing skills; (5) learn how to work cooperatively in groups; (6) learn how to teach others; and (7) have hands-on experience with archaeology materials. The evaluation also looked at goals of the program for the UCB students. These goals state that undergraduates will be provided with opportunities to: (1) participate in service that directly impacts the educational experience of sixth graders and contributes to neighborhood development; (2) gain experience in alternative educational/pedagogical approaches; (3) gain experience in the development of teaching tools; (4) develop and hone skills in participant observation, the creation of field notes, and the development of research questions to be answered with field data; (5) build teaching and communication skills, particularly within a strongly multicultural environment; and (6) positively impact children's lives at a crucial developmental stage by providing them with college-age role models/mentors. |
| Evaluation Design | Non-Experimental: UCB undergraduate students served as participant observers, collecting ethnographic data on participants in the Expedition program, a Fifth Dimension program based at Roosevelt Middle School in Oakland, California's San Antonio neighborhood. |
| Data Collection Methods | Observation: UCB undergraduates wrote extensive field notes following each day of their after school experience. These observations provided a view of the behaviors and accomplishments of the sixth grade participants over time, as well as the reflections and learning objectives of the UCB students. The UCB students kept “Expedition Logs” that documented each child's activities throughout the school year. Secondary Source/Data Review: Demographic data were collected for the sixth grade participants on their ethnicity/nationality and the primary language(s) spoke at home. However, these data were not available for about half of the participants because the survey asking parents to specify ethnicity and home language was not completely filled out by all parents. Program attendance data were also collected on the sixth graders. |
| Data Collection Timeframe | Data were collected during the 1999–2000 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Thirty after school activities were designed and put into practice by the 43 UCB students who completed an Anthropology course over two semesters. Some of the activities involved computer games, some focused on CD-ROM resources exploring a particular country or culture, such as Peru or the Maya, and others were hands-on archaeology activities. Some activities were improvised in order to meet children's individual needs. Of the 30 activities, 25 involved writing, 18 incorporated reading, 16 integrated computer games with ancient history/archaeology, 7 involved hands-on archaeology skills, 4 were web-based, and 3 explored computer skills. During the first year, sixth graders completed a total of 592 activities over 18 weeks. Each activity incorporated three levels of difficulty: beginner, good, and expert. Of the activities completed during the first year, 259 were beginner level, 208 were good level, and 125 were expert level. The children's expedition logs documented the completion of a total of 149 writing tasks for all participants. The children were required to read daily in the program, and some received individual help with reading. One UCB student noted that the program provided the sixth graders with “a lot of focused, individual attention.” |
| Program Context/Infrastructure | Literacy is a major challenge at Roosevelt Middle School, with 57% of the students in LEP (limited English proficiency) status. More than 50% of the families in Oakland's San Antonio neighborhood are at or below the poverty level. |
| Recruitment/Participation | A total of 61 sixth graders, ages 10–13, were served by this program over a total of 902 hours. Aside from a few children who moved away over the course of the year and a few who dropped out, the evaluator found overall retention to be good, with about 25 children participating for the full year. New children joined the program throughout the year. Aside from the 25 yearlong regulars, a large number of latecomer kids stayed with the program to the end. Of the participants for whom data are available, the ethnicity/nationality of participants are as follows: 11 were African American/Black, 7 were Chinese/Chinese American, 4 were not specified, 2 were Mexican American, and there was one student from each of the following groups: Vietnamese, Bosnian, Latin American, Tongan, Mien, and Cambodian. This diversity reflects the diversity of the neighborhood; San Antonio is the most culturally diverse region of Oakland, with a large population of recent immigrants. Of the participants for whom data are available, the language(s) participants spoke at home are as follows: 12 speak English, 6 speak Cantonese, 4 speak Spanish, 2 speak Spanish and English, and one student each speaks each of the following: Bosnian, Laotian, Vietnamese, Tongan and English, Mien and English, and Cambodian and English. Data on the participants' computer access and experience indicate that 50 children had used a computer before, 19 had a computer at home (31 did not), and 5 had never used a computer. |
| Staffing/Training | A UCB student indicated that the process of writing field notes was beneficial, “The field notes served as the most rewarding aspect of the class in terms of connecting all the components. While writing the field notes, I was forced to question the motivations behind my actions.” |
Summative/Outcome Findings
| Youth Development | Field notes provide evidence that students learned how to work cooperatively in groups and how to teach others. For example, a UCB student describes one such interaction between two of the sixth graders, “After [Sheila] had completed the game, we joined up with her friend, Rita, and played Carmen San Diego. Rita has had more experience playing this game and made most of the decisions. However, she was not bossy. Rita explained all of the moves to Sheila, and took Sheila's ideas and thoughts into consideration.” Field notes provided evidence that the sixth graders gained hands-on experience with archaeological materials and were highly engaged in the activities. A UCB student describes this phenomenon, “We could not choose a task card fast enough for them—they wanted to handle and play with whatever was placed before them … They envisioned possibilities for our task cards that even I could not have imagined. They asked us if they could bring their own bag of artifacts, or their own garbage and analyze them…” Field notes suggest that youth were learning empowerment and independence through the Expedition activities. One UCB student noted that “free-form activities such as writing to the Supernova and choice-based activities such as exploring the study in Cartoon History of the Universe, helped develop feelings of empowerment and independence in many of the sixth graders. In order to better foster these feelings, mentors should (and often did, according to my observations) allow their young partners to make mistakes and discover solutions on their own.” Field notes suggest that the mentoring relationship was beneficial to both youth and mentors, as this field note illustrates, “My experience with Loretta has been positive for both of us. Loretta is a girl who truly loves to learn … I think she loved having a nonauthoritative mentor to help her one-on-one. At first, she was a little shy about asking questions and reading aloud. Towards the end, however, it was questions, questions, questions! … My experience with her has only strengthened my desire to teach community college someday.” Another UCB student noted that, “being able to interact with the futures of our world and play a substantial role in their development was very exciting.” |
Evaluation 6 (UCB): Expedition – Computers and Archaeology After School Year End Report, 2000–2001
Evaluation Description
| Evaluation Purpose | To examine how program goals were met during the second year of the Expedition program. The goals are to serve low-income sixth graders in Oakland, California by: enhancing their educational opportunities; providing them with a safe and enriching after school environment; helping develop their critical thinking and literacy skills; providing them access to computer technology; and using computer-based tools (with archaeology as a learning framework) to facilitate and motivate them to create their own stories and artifacts and to explore their immediate community in the broader context of the world beyond their neighborhood. |
| Evaluation Design | Non-Experimental: Data were collected from the Expedition program, a Fifth Dimension program based at Roosevelt Middle School in Oakland, California's San Antonio neighborhood. UCB undergraduate students served as participant observers, collecting ethnographic data on program participants. |
| Data Collection Methods | Interviews/Focus Groups: Interviews were conducted with three teachers at Roosevelt Middle School who had students in the Expedition program. Teachers were asked about how long they had been aware of the Expedition program, what feedback they had gotten about the program from participants, their view of program impacts on students, whether they taught archaeology and whether students were interested in archaeology, whether there was a particular student who was influenced by the program and in what ways, and suggestions for improvements. Teachers were also asked if they had seen changes in participants' self-esteem, confidence, behavior, academic achievement, attitudes toward social studies or history, computers skills and/or attitudes toward computers, and interest in going to college or academic goals. Observation: UCB undergraduates were required to write up detailed field notes after each Expedition session. Their observations focused on individual students and activities, their interactions with other students and adults, as well as their interactions with computers and other activity tools. Secondary Source/Data Review: Data on participants' ethnicity/nationality and primary language(s) spoke at home were supplied by parents on program intake records. Surveys/Questionnaires: Participants were asked to fill out an informal (non-required) survey. The survey asked for basic background information on the child, including computer and email use, knowledge of history and archaeology, favorite class, interests/hobbies, friends in the Expedition program, reason for coming to the program, and hopes for the program. A total of 46 participants completed the survey, 27 of which were “regular” participants (those who stayed with the program over the course of the year). |
| Data Collection Timeframe | Data were collected during the 2000–2001 school year. |
Findings:
Formative/Process Findings
| Activity Implementation | Thirty after school activities were designed and put into practice. They included computer games, computer skills, CD-ROM resources for various countries and cultures, hands-on archaeology activities, and improvised activities to meet children's individual needs. |
| Program Context/Infrastructure | Literacy is a major challenge at Roosevelt Middle School, since the school has a large population of recent immigrants, with 64% of the students in LEP (limited English proficiency) status. Of Roosevelt's 1,090 students, 82% qualify for a school-provided lunch program. A major challenge to the quality of the Expedition program has been the lack of Internet access in the computer labs, and most of the school in general. Expedition's staying power lead to a major breakthrough in this area—in spring 2001, several Information Systems & Technology staff offered to help finish the Internet network at Roosevelt, most of which was complete as of June 2001. Evaluators note that this assistance generated a tremendous amount of goodwill toward UCB at this school and in the community. Furthermore, the evaluators comment that teachers have renewed enthusiasm for using computer labs in future teaching practices, and that the availability of the Internet will enable the program to more readily expand computer lab access to families. |
| Recruitment/Participation | During 2000–2001, 75 children came to Expedition at least once and 36 were regular participants. These 36 provided most of the demographic data. Ethnic profiles for participants for whom data were available were: African American/Black (11), Chinese and Chinese American (6), Mexican American (5), Asian (1), Latin American (1), Native American (1), Honduran (1), white (1), and Salvadoran (1). Languages participants spoke at home included English (17), Spanish (13), Spanish and English (2), Chinese (2), Cantonese and Mandarin (2), Chinese and English (1), and Mien (1). Of the 27 regular participants who filled out a survey, 15 indicated that they had a computer at home, and 12 did not. In addition, 8 had used email and 19 had not. Asked at the end of the school year if they planned to attend college in the future, 14 of the 15 participants who were asked said “yes” or “definitely yes” and one said “not sure.” Expedition had a large number of nonnative English speakers during the 2000–2001 school year. Evaluators conjecture that this was primarily because LEP participants brought friends to the program, since Expedition staff included several bilingual mentors. |
| Staffing/Training | UC Berkeley undergraduates staffed Expedition two days each week for 22 weeks, resulting in 1,558 hours spent with the child participants. |
| Systemic Infrastructure | The Roosevelt Village Center (RVC) Collaborative at Roosevelt Middle School is the site and administrative home for Expedition. The village centers are collaborative ventures of community organizations and schools to implement coordinated comprehensive programs and services for youth at school sites after school and on weekends. According to evaluators, this model of “plugging in” the after school program to the larger RVC program has proven successful, sustainable, and efficient for UCB participants. The RVC has been helpful in institutionalizing the after school relationship with the school district, school staff, and parents. In addition, RVC provides administrative framework and data collection for Expedition activities, staffing for the school site, security, screening, structure, and transportation for Expedition field trips. |
Summative/Outcome Findings
| Academic | One-on-one practice with an adult increased literacy in students who were below grade level in reading, writing, and English speaking skills, as this field note illustrates, “After talking to [a participant] for a while he realized that I was actually going to help him read and understand the questions. First he would say that he did not know how to read English. Then he admitted that he knew, but he did not understand what he was reading. After a while, I got him to read everything and he would actually ask me why some words have different meanings sometimes.” In interviews, teachers were quick to mention steady improvements in English language proficiency of student participants. One student brought his math grade up from a C- to a B during the year, and his math teacher attributes his increased effort to his participation in and devotion to the Expedition program. Another Expedition student was described as having made great progress in English, writing, and computer skills. Native English speakers who were below grade level in reading skills were supported and tended to be more willing to practice reading in one-on-one situations, as this field note indicates, “[During] times when I was working only with Taneesha [one-on-one], she was persistent in her reading and even though she may have felt challenged, she was comfortable in taking the time to sound out words, and we wouldn't go on until we had read it correctly. Since we have been working in groups, she has been afraid to read, and often the others are so anxious to read, that she ends up not reading at all. And when she does read, she is definitely not as persistent in sounding out the difficult words because there is an audience of her peers listening to her, and perhaps she is embarrassed that her reading ability is not as high as theirs.” According to field note data, for some students, having another child or two in a supportive small group setting was helpful in developing literacy skills, “They slowly read the instructions together, one sentence each. Most of the time they were able to read what it was, but not really understand, in which case I would have to explain it to them … Their comfort shows in that they are less afraid of situations and even more willing to try. They are not embarrassed by reading in English and they are beginning to be more vocal with the English they do know.” Undergraduate students working with the children found that hands-on activities and problem-solving games were very effective learning tools, especially for children who have difficulty with the dominant paradigm for learning in schools, “We took more notes on the artifacts that we found and used our archaeological sketches to reference the objects to the book. The girls were excited about this game and wanted to explore every room as it magically transitioned to the past and filled with life and color. They specifically were interested in the grinding of the corn, and Anita was delighted to know that she could make baskets the way the ancient Pueblo people could.” When given the option to type on the keyboard, students were eager to write and use Microsoft Word to improve their writing and computer skills, as this field note illustrates, “When he noticed that a line would appear under a word he had typed, he would realize he had spelled it wrong. I never saw Microsoft Word as a learning tool, but he was able to spell the words correctly because immediately when he misspelled the words a line would appear.” |
| Family | Two participants were given their own computers by their mothers during the year in recognition of their intense interest in computers and their dedication to the after school activities. |
| Youth Development | Field note observations suggest that participants' self-esteem and self-confidence flourished and increased markedly when they were permitted to be the experts, with adults who had abandoned the authoritative role, “Like many other students, Anita seems to show more confidence in her abilities, taking a risk in trying them out, with the more challenges she successfully overcomes ¼ It is at this level that I am starting to see a definite change in Anita's approach to the game. She is now more independent, challenging the ideas that I put forth. I have often detected a sense of doubt in her after I have answered some of the questions that she has asked, but she has never been so forward as to openly challenge my ideas. I am happy to see the change.” |