Non-Experimental: The sample consisted of 130 Thinking SMART participants over the first 4 years of the project. Surveys were administered to all participants each year before the start of program activities (pretest), at the beginning of the program year (baseline), and at the end of the program year (posttest). Out of a total 225 possible respondents, 130 girls completed both pretest and posttest surveys, resulting in a 58% response rate. Out of the non-respondent group (participants who completed only pretest or posttest surveys), 6 participants responded to only the posttest survey and 89 participants responded to only the pretest survey; these groups were not include in the data analysis.

A bias analysis compared the characteristics of respondents and non-respondents and found that the groups were similar with respect to key demographics such as race/ethnicity, type of household, and income levels. The average age of non-respondents (12.69) was about half a year older than the average age of respondents (12.09). The bias analysis found that the group of respondents was representative of the total participant population, and that the evaluation findings were not biased by the exclusion of non-respondents.

Surveys/Questionnaires: Baseline surveys collected basic demographic information on participants, including program year, grade, age, race/ethnicity, household type, disability, and income level, as well as data on previous involvement in math, science, or computer activities; math and science course taking; access to and use of computers; academic and career aspirations and plans; and involvement with Girls Inc. and other community or school activities.

Pretest and posttest surveys consisted of the following: a question about plans to attend college with response choices of “2-year,” “4-year,” “don’t know,” and “no” (i.e., does not plan to attend); 18 questions assessing participation in both in- and out-of-school STEM activities, such as “looking at the stars or using a telescope,” rated on a 4-point scale from “often” to “never”; 4 questions about assessing knowledge about STEM career paths (math, science, computer science, and engineering), rated on a 4-point scale from “a lot” to “nothing at all”; 5 questions  assessing interest in STEM careers (general, math, science, engineering, and computer science), rated on a 5-point scale from “very interested” to “never considered it”; 12 statements assessing confidence in performing in STEM-related activities, such as “It wouldn’t bother me at all to take more math,” rated on a 4-point scale from “strongly agree” to “strongly disagree”; and 11 statements assessing attitudes toward math and science as reflected in reactions to media messages, such as “Science is for boys,” rated as either “agree” or “disagree.”

Of program participants with available demographic data, 43% were non-White, almost 40% came from homes where one or both parents were absent, 36% received free or reduced-price lunches, and 10% had a disability.

Participants were in grades 4 through 10 and ranged in age from 10 to 18 years old. The majority of participants were ages 11 (35%) or 12 (33%) and were in grade 5 (27%) or 6 (41%).

Across the 4 years of the program, the retention rate was at least 58% (not including participants who may have completed the program but did not fill out the posttest survey).

The program sustained the interest of almost three-fifths of participants in one program year.

Little change was seen in girls’ interests in STEM activities from pretest to posttest; pretest survey responses indicated that the majority of girls were already engaged in STEM-related activities before entering Thinking SMART, leaving little room for improvement. Specifically, surveys showed that girls reported a slight increase in participating “often” in only two activities: “Repair things that use electricity” (from 7% to 8%) and “Use meters or gauges” (from 5% to 8%). Fewer participants reported never participating in the following activities at posttest compared to pretest (meaning that they were more likely to participate in these activities at posttest): “Watch science programs on television” (from 23% to 22%), “Look at the stars or use a telescope” (from 19% to 18%), “Repair things that use electricity” (from 50% to 46%), and “Use meters or gauges” (from 49% to 45%). Otherwise, the findings showed that girls’ engagement in most STEM-related activities declined or stayed the same following participation in Thinking SMART.

Even though pretest survey responses showed that the majority of respondents already planned to attend a 4-year college or university (69%), there was an increase in the percentage of girls reporting such plans following their participation in Thinking SMART (77%). In addition, fewer girls were undecided about their college-going plans from pretest to posttest (from 21% to 15%) while the percentage of girls planning to attend 2-year colleges decreased slightly (from 9% to 8%).

Girls' views of math improved slightly from pretest to posttest, with a higher percentage of girls “strongly” or “somewhat” agreeing with four out of the six positive statements about their math competence, including “It wouldn’t bother me at all to take more math” (from 63% to 72%), “I almost never get upset during a math test” (from 62% to 64%), “I am usually comfortable during a math test” (from 67% to 69%), and “Math doesn’t scare me at all” (from 60% to 64%),. There were slight decreases for the other two items: “I am sure that I can learn math” (from 94% to 93%) and “I can get good grades in math” (from 87% to 86%).

Fewer girls held negative viewpoints of math in posttest surveys, as well, with an increase in girls who reported that they “strongly” or “somewhat” disagreed with all six negative statements, including “Even though I study, math seems hard for me” (from 47% to 54%), “Most subjects I can handle OK, but I seem to goof up math” (from 55% to 62%), “My mind goes blank and I can’t think clearly when working math” (from 59% to 66%), “I get a bad feeling when I think of trying hard math problems” (from 63% to 64%), “I’m not the type to do well in math” (from 64% to 76%), and “A math test would scare me” (from 40% to 72%).

Overall, Thinking SMART participants increased girls’ knowledge about STEM careers after completing the program. There was a slight increase in the percentage of girls who reported learning “a lot” or “some” about careers in science (from 71% to 77%) and math (from 71% to 74%), and a slight decrease for computer science careers (from 59% to 58%). The majority of participants reported being “very interested in STEM careers” at both pretest (85%) and posttest (86%). Declines were seen from pretest to posttest in the percentage of girls reporting being “somewhat” or “very” interested in careers in math (from 53% to 44%), science (from 71% to 60%), or computer science (from 62% to 59%).

Results showed a large increase in girls reporting they gained “a lot” of knowledge about engineering careers (from 10% to 20%) and reporting being “somewhat interested” in engineering careers (from 21% to 32%).

Slight pretest to posttest increases were found in girls’ agreement with positive messages heard about STEM in the media, including “Girls can do anything” (from 92% to 97%), “Girls need to plan for a career” (from 71% to 73%) and “Most girls are good at science” (from 67% to 68%). Increased percentages of girls disagreed with negative media messages, including “Science is for boys” (from 91% to 95%), “Girls need to work with people” (from 48% to 52%), and “Girls can’t read maps” (from 92% to 93%).

Despite the very large percentages of girls who disagreed with the negative messages, a pattern emerged in the pretest and posttest findings that showed an increase in the percentage of girls who agreed with certain negative messages: “Computers are too hard for girls” (from 3% to 5%), “Boys understand machines better” (from 10% to 18%), “Girls who like math are weird” (from 3% to 7%), “Nice girls don’t get dirty” (from 6% to 7%), and “Girls don’t really need math” (from 4% to 7%).