Non-Experimental: A staff member within each agency (referred to as a “core agency member”) was responsible for overseeing the overall implementation of MiM at his or her various program sites, and for selecting which sites would be evaluated through site visits. This selection process resulted in a sample of four sites from three agencies: one each from BELL and Girls Inc., and two sites within the YMCA. Although the St. Louis Science Center was a pilot agency participating in previous years of the evaluation, it was not included in the final evaluation due to staff turnover within the agency and lack of capacity to support site visits. Sites were selected based on the core agency members’ interpretation of the program’s successful implementation of MiM and whether the site had pre-existing math programming that MiM could supplement. MiM curriculum supplemented BELL’s math block, Girls Inc.’s SMART (science, math, and relevant technology) curriculum, and YMCA’s unstructured math activities.

Each site was visited once over the course of the school year, with the exception of Girls Inc., which was visited twice. During site visits, interviews were conducted with program directors, assistant program directors, and lead teachers; program staff participated in focus group discussions; and MiM activities were observed. In addition, evaluators conducted phone interviews with core group members from each agency and held focus group interviews with program directors from all participating sites of each organization to gain a better understanding of MiM implementation beyond the four evaluated sites.

Interviews/Focus Groups: Interview and focus group questions focused on the nature and extent of training and technical assistance, as well as the impact of MiM on the participating program, staff, and youth. Interviews also included questions about sustainability, such as “How likely are the participating afterschool sites to continue using the activities as the grant period comes to an end?” Core agency members were asked about dissemination efforts in conjunction with challenges publicizing MiM.

Observation: The observation protocol examined high quality implementation, which was defined as including the promotion of math content, linking the activity to real world settings, clear staff instruction, the enjoyment levels of the youth participating, and reflection on the math concepts in the activity.

While observations revealed that the process of implementation did not differ appreciably across agencies, program directors and staff reported that MiM activities were used in a variety of ways at the different agencies, including during transition time, community time, math blocks, informal times, and as a treat for the youth.

In interviews, directors and staff reported that MiM activities were easy to use in a “grab and go” manner, required limited supplies and preparation, helped with transition processes, and engaged children in a fun way. In addition, core agency members reported being especially appreciative that MiM could be adaptable to their program.

The level of MiM implementation varied across agencies. BELL reported an increased level of implementation compared to previous program years due to the staff’s familiarity with MiM and the integration of MiM into the program. Girls Inc. reported that MiM usage remained consistent from previous years, but in a less formal way. The YMCA, however, reported that sites decreased levels of MiM implementation, most likely due to less TERC direct involvement in the last program year, and less accountability at the TERC and agency levels.

Higher quality implementation was observed during site visits at programs that communicated expectations about MiM implementation to staff and designated individuals (core agency members, program directors, or lead teachers) to frequently check-in with staff to make sure implementation was going smoothly. Observations showed that less experienced staff members especially benefitted from such accountability.

Even though TERC’s former accountability system of feedback forms met resistance with program staff and directors during the first 2 years of the evaluation, it also proved to be helpful by providing a tool for staff to reflect on the success or challenge of a MiM activity; some program directors reported interest in re-implementing this component.

Observation sites produced more documentation on accountability than regular program sites, but agencies reported that this higher level of accountability was due to the greater buy-in and pride of being selected as an evaluation site.

Higher quality implementation was found at sites where staff conducted MiM activities with a specific focus on the math content. Their successful methods included giving clear activity instructions, asking guiding questions, helping children problem-solve, reviewing the math content with the children, checking the children’s understanding of math principles in the activity, connecting the content to everyday life, and allowing the children to play active leadership roles in the activities.

Directors from the BELL sites felt that MiM provided a stronger math program than the one they regularly implemented, especially given that the MiM curriculum was better aligned with the TERC-developed curriculum used during the school day in the children’s classrooms.

Some core members and site directors felt that, in some cases, activities were not effective because they were not being used with the correct age and/or skill group, youth already possessed the math skills being promoted in the activity, or MiM activities were used to fill time rather than promote math skills. Site visits suggested that minimal program effects could also be attributed to the fact that some staff may not have been promoting the math component of the MiM activity as much as they could have been. Instead, some staff made activities into a competitive game, used only simple math concepts, did not promote math thinking during the activity through questioning or reflection, or led the activity in such a way that youth had only minimal opportunities to use their math skills and knowledge.

Some directors reported that MiM benefited their site because it promoted movement after children were sedentary in school all day and helped to develop a strong connection between math and everyday life.

Staff expressed a desire for future MiM implementation to look slightly different: some reported that MiM activities should be incorporated more informally as back-up or transition activities and not the core curriculum, while some staff reported that they wanted to be able to link MiM activities more directly to what the children were learning at school.

Program directors expressed that future MiM implementation could benefit from more opportunities to discuss and brainstorm ideas about MiM with staff members. The two forms of support cited as least helpful by program directors at the evaluation sites were promoting parent involvement in MiM and having a more detailed MiM training guide.

The issue of engaging youth emerged as a common challenge for program sites, especially around meeting the needs of children of different ages, abilities, and interests.

Some staff reported implementing MiM activities as an option during "choice time," in which children have the opportunity to engage in a variety of classroom activities according to their preference. In turn, certain youth were exposed to MiM more than others based on their preference to participate.

Staff reported that when they implemented MiM activities, some children refused to participate, responded negatively, or felt anxious. To address this struggle, not all staff presented the MiM activities as math lessons; observations revealed that some staff opted to give alternate labels to the math principles. Other staff members presented MiM activities as competitions to promote youth engagement.

Higher quality implementation was found at sites with strong leadership, as evidenced by the successful active involvement of program directors and lead teachers from the evaluated sites. Interviews with staff revealed that they believed implementation would benefit from even more support from the program director. Staff also expressed that having a leader to champion MiM was a major factor in encouraging its continued use and longevity.

Staff turnover was a challenge across programs, and staff reported that it would be challenging to keep implementing MiM with continued staff turnover. When trained staff left a site, new staff members reported being less likely to implement MiM activities since they felt less comfortable with the curriculum content and scope, seeing it as an optional resource or a way to fill time instead of a means to achieve learning goals. Similarly, staff with limited teaching backgrounds appeared to struggle to implement MiM activities in a way that promoted math skill-building through the material. These staff reported choosing MiM activities based on what was easiest to implement or what engaged youth the most.

Interviews with core agency members and program directors revealed that training and professional development for staff varied according to the agency. For example, Girls Inc. held MiM training during the program orientation for staff and provided constant and regular communication about the curriculum, whereas BELL provided MiM training “on the job” and not necessarily as a separate didactic component.

Sites reported challenges in training staff because meeting time was very limited, and staff also needed time to be trained in non-MiM curricula. In addition, Girls Inc. noted that it was a struggle to maintain the momentum of training due to the high rate of staff turnover and regularly needing to get new staff “up to speed.”

Interviews revealed that site staff felt their training was sufficient. They reported that activities were self-explanatory and only minimal training was necessary to use the material.

Some program directors reported that it would be beneficial to have more comprehensive MiM trainings so that staff could work on drawing connections between the activities and their underlying math lessons. Directors’ suggestions included presenting fewer activities at a time so staff could fully understand each one and providing role modeling of activities and guided questions.

TERC provided technical assistance in the form of online resources and a newsletter, and staff reported that they felt this was adequate support. Specifically, staff members reported that they benefitted from these resources because they offered new ideas for activities and allowed them to hear what other agencies were doing.

Directors believed that MiM encouraged staff to use math more frequently and mix math into various activities, as well as increased staff’s confidence because they were able to observe the children learning during MiM. Moreover, directors felt that staff were able to gain more control of the group during activities, run more organized activities, and see the importance of math skills and the link between math and everyday life.

BELL staff did not associate any changes in themselves to their participation in the MiM initiative, while staff from Girls Inc. and the YMCA only saw minimal impact. Staff who did identify an effect reported improvement with behavior management skills, seeing math in everyday life, mixing math into more activities, and developing an overall MiM mindset within the program.

TERC intentionally provided less technical assistance to sites during the 2006–2007 program year compared to the previous year, and core agency members reported appreciating this shift. Interviews with core agency members showed that TERC’s minimized involvement allowed the programs to implement MiM with more flexibility and in a way that was more conducive to the agency’s specific needs, time constraints, and program structure. Programs did not have to document MiM activities in feedback forms as in previous years, a change that reportedly eliminated a great deal of stress for staff.

Core group members reported that MiM dissemination efforts were successful within their own agencies and with affiliate sites but that they experienced challenges publicizing MiM at a broader level. The Girls Inc. core agency member expressed that it was challenging to promote MiM as a web-based, stand alone program, and that MiM was difficult to implement at multiple sites because of the differing sizes and structures of affiliate programs.

TERC staff reported receiving approximately two new requests each week to join their mailing list in the 6 months preceding the release of the report. TERC’s extensive outreach included efforts to disseminate MiM materials, publicize the MiM program, promote the MiM website, conduct MiM trainings and workshops, and implement MiM activities. Outreach efforts involved partnering with a number of educational organizations including universities, zoos, libraries, afterschool organizations, museums, and public schools.