You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼


Program Description

Overview Founded in 1999 in Chicago, Illinois, Project Exploration (PE) is a nonprofit science education organization that works to ensure communities traditionally overlooked by science—particularly minority youth and girls—have access to personalized experiences with science and scientists. Programs for youth are supported by public programs that provide a window into science in action through exhibits, online initiatives, and contributions to the field of science in out-of-school time (OST). PE offers four main programs for youth: Junior Paleontologist (JP), Sisters4Science (S4S), Dinosaur Giants (DG), and All Girls Expedition (AGE).
Start Date 1999
Scope local
Type afterschool, summer/vacation, weekend, comprehensive services, mentoring
Location urban
Setting public school, community-based organization, private facility, recreation center
Participants middle school through high school students
Number of Sites/Grantees one
Number Served From 1999–2006, 84 youth participated in JP, 274 youth participated in S4S, 150 youth participated in DG, and 30 youth participated in AGE. These numbers are unduplicated and do not reflect that many youth participate in multiple programs.
Components JP is a 3-week summer fieldwork immersion program in which 12- to 17-year-old youth participate in intensive classroom sessions in anatomy, geology, and paleontology. These sessions help prepare youth for a week-long paleontology field expedition, where youth collect raw data in situ alongside scientists.

DG allows high school students to fulfill community service graduation requirements by serving as docents for science exhibits at local museums. As preparation, DG participants receive training in exhibit facts, history, and interpreting information for the public.

S4S is a weekly afterschool program for minority middle school girls combining science exploration with leadership development. In addition to providing hands-on science activities, S4S introduces girls to a variety of women scientist role models.

AGE is a 2-week classroom and fieldwork experience in which middle and high school girls learn practical geology, biology, evaluation, and field skills alongside scientists.

PE maintains a long-term relationship with participants in all programs, providing ongoing mentoring and leadership development opportunities until they graduate from high school.
Funding Level $1,483,615 (fiscal year 2009 total assets)
Funding Sources Major funders include The Whitten-Newman Foundation, The Chicago Community Trust, Motorola Foundation, and Abbott.


Evaluation

Overview An evaluation was conducted of PE’s OST science education programs to provide a snapshot of program functioning and to address the extent to which programs demonstrate the ability to achieve their objectives. The evaluation further helped identify program strengths and weaknesses in order to inform decisions aimed at making the programs more effective. Four areas of analysis were emphasized: program purpose and design, strategic planning, program management, and program results/accountability.
Evaluator Adam Tarnoff, Independent Evaluation Consultant
Evaluations Profiled Project Exploration Youth Programs Evaluation
Evaluations Planned In 2010, PE completed a 10-year longitudinal evaluation, to be published in late 2010. This retrospective review of PE alumni investigated patterns of participation and subsequent educational and career/life choices. The goals were to describe PE’s influence on past participants and to explain the organizational practices that support science learning for youth traditionally underrepresented in science.
Report Availability Project Exploration. (2006). Project Exploration: Youth programs evaluation. Chicago: Author. www.projectexploration.org/PDF/program-eval-v2-final.pdf


Contacts

Program Mary Elizabeth Perez
Manager of Program Operations and Logistics
950 East 61st Street

Chicago, Illinois 60637
Tel: 773-834-7622
Fax: 773-834-7625
Email: meperez@projectexploration.org
Evaluation Not available.
Profile Updated January 10, 2011


Evaluation: Project Exploration Youth Programs Evaluation



Evaluation Description

Evaluation Purpose To assess four performance areas of PE programs: program purpose and design, strategic planning, program management, and program results and accountability.
Evaluation Design

Non-Experimental: The evaluation design was guided by the Program Assessment Rating Tool (PART), a publicly available performance evaluation instrument developed by the White House Office of Management and Budget. Supporting evidence was derived from document review, five interviews of PE program alumni (current participants each interviewed one PE college-age alum of their choosing), two youth participant focus groups (each consisting of five participants from various grades, schools, and PE youth programs), and secondary source/data review.

Supporting evidence also included an output/outcome analysis of a sample of 101 participants from JP and Advanced Paleontologists (AP), a similar field expedition program that is no longer offered by PE. JP and AP were the only PE youth programs that had baseline or outcome-driven evaluation data at the time of the evaluation, so existing student records from these programs were used to draw conclusions about field expedition alumni outputs (e.g., student characteristics, school demographics) and outcomes (e.g., student graduation rate, college enrollment patterns).

Data Collection Methods

Document Review: Evaluators reviewed existing agency performance documents and financial information including documentation about personnel management, board descriptions, program goals and mission, strategic planning, work plans, training, and funding proposals and reports.

Interviews/Focus Groups: These focus groups used a 10-question protocol aimed to capture youth’s thoughts about their PE experience through queries such as, “What interested you in participating in PE?” and “How—if at all—has PE changed the way you think about yourself and your future?”

Interviews with PE alumni asked about their current activities, challenges of college life, impact of PE after high school and financial concerns about college.

Secondary Source/Data Review: PE documentation collected as evidence for PART responses included program reports, prior evaluation data, annual reports, participant feedback surveys, and student records and statistics (including demographic data).

To compare PE to other STEM education programs, PART data was obtained from NASA Education Enterprise and the U.S. Department of Education 21st Century Community Learning Centers, two federal programs selected because they shared similar key goals with PE.

To compare PE youth to their peers, assumed characteristics of a comparison cohort were collected from public databases of Chicago Public Schools (CPS) student data. Outcome measures included high school graduation rate, college enrollment rate of high school grads, and the percentage of college students who choose to major in science.

Test/Assessments: The PART consists of a series of questions that assess different aspects of program performance split into four sections:

  1. Program Purpose and Design assesses whether the program’s purpose and design are clear and sound through five yes/no questions (e.g., “Does the program address a specific and existing problem, interest or need?”).
  2. Strategic Planning assesses whether the program has valid long-term and annual measures and targets through eight yes/no questions (e.g., “Does the program have baselines and ambitious targets for its annual measures?”).
  3. Program Management rates the program’s administration, including financial oversight and program improvement efforts through seven yes/no questions (e.g., “Does the program use strong financial management practices?”).
  4. Program Results/Accountability rates program performance on measures and targets reviewed in the Strategic Planning section and through other evaluations through five questions (e.g., “Has the program demonstrated adequate progress in achieving its long-term performance goals?”), with possible answers of “yes,” “large extent,” “small extent,” “no,” and “not applicable.”

Section scores are tallied and translated into qualitative ratings for the overall PART score: effective, moderately effective, adequate, ineffective, or results not demonstrated. A “results not demonstrated” rating is given, regardless of overall score, when the program does not have agreed-upon performance measures or lacks baseline and performance data.

References White House Office of Management and Budget. (2005). Program Assessment Rating Tool. Washington, DC: Author. Available at: www.omb.gov/part/
Data Collection Timeframe Data were collected in 2005–2006.


Findings:
Formative/Process Findings

Costs/
Revenues

A review of grant documents provided evidence that PE’s budget requests were explicitly linked to the accomplishment of annual and long-term performance goals, and that resource needs were presented in a complete and transparent manner. Specifically, the Strategic Planning section of the PART found that historical budget requests were tied to primarily output-oriented goals and measures (namely, the numbers of students served), and that future budgets were tied to new program goals.

Reports to funders, budget documents, and an external audit indicated that PE used strong financial management practices. Specifically, the Program Management section of the PART determined that program funds were obligated in a timely manner and spent for the intended purpose.

The evaluation did not find any cost efficiency measures with baseline or target information, which resulted in a “No” response to the PART Program Management question regarding the existence of procedures for achieving cost effectiveness in program execution.

Parent/
Community Involvement
PE demonstrated a track record of long-term, ongoing relationships with museums, external scientists, and CPS and administrators. However, partner organizations (specifically museums and schools) did not appear to participate on a level at which it would be appropriate for them to measure and report performance related to PE program goals.
Recruitment/
Participation

From 1999–2006, 101 individual youth had been on paleontology field expeditions and PE had provided 136 total field expedition experiences.

Program records included racial and ethnic composition of the youth, and of the 101 field alumni, 55 were Black, 7 White, 28 Latino, and 11 Other. In addition, 46 were male and 55 were female.

Youth participated in their first field experience at a mean grade of 9.47, ranging from grades 7 to 12. On average, each participant had 1.35 field experiences; 73% of participants had one field experience while the remaining 27% participated multiple times.

From 1999–2006, PE maintained active relationships with 81% of former field participants, averaging 2–3 years of contact before high school graduation. Active relationships were maintained through such means as offering year-round activities and events (e.g., one-day activities where youth could meet and interact with scientists, leadership opportunities in PE programs, college readiness workshops); writing letters of recommendation; nominating youth for scholarships; and providing feedback on college essays. The mean grade level of pre-college students in active contact with PE was 10.73.

PE participants attended 37 different schools: 29 high schools and 8 middle schools.

Systemic Infrastructure

PE earned a Program Purpose and Design PART score of 100%. PE’s Strategic Plan documents provided evidence for the clarity of the program’s purpose and design, and demonstrated an approach to youth programs that was closely aligned with education research while being unique from other science outreach programs.

PE earned a Strategic Planning PART score of 57%. A review of PE’s long-term and annual measures and targets revealed that insufficient attention was paid to setting clear and measurable performance goals. Strategic plan documents revealed existing baseline data on outputs, namely program participation and audience characteristics; however, documentation revealed a lack of outcome-oriented goals that addressed youth programs’ impact on participants, as JP was the only youth program with a baseline of outcome-driven evaluation data.

PE received a PART score of 71% for its performance in Program Management. A description of program management responsibilities in grant application documents demonstrated successful accountability for cost, schedule, and performance results; managers were held directly accountable to the PE executive for the results of their programs. Although PE successfully used program feedback to adjust priorities, allocate resources, and take management actions, its program management practices did not fully meet the standard of the PART evaluation. A lack of supporting evidence, namely baseline performance data and cost efficiency measures, prevented PE from demonstrating meaningful performance targets for all of its youth programs.

PE received a PART score of 40% in Program Results/Accountability. Due to the lack of performance measures, PE was able to demonstrate only a “small extent” of adequate progress in achieving long-term goals, even though existing measures pointed to very substantial progress. PE achieved its annual performance goals to a “large extent,” based on the historical data on program enrollment, participant demographics, and unanalyzed participant feedback.

PE was found to compare favorably on the PART Program Results/Accountability measure to other programs such as NASA Education Enterprise and Department of Education 21st Century Community Learning Centers, based on PE’s explicit steps to address deficiencies common in comparable programs and to develop effective solutions.


Summative/Outcome Findings

Academic

PE program participants were found to graduate high school at a higher rate (92%) than their non-participating peers who attended the same schools (78%).

The average CPS city-wide high school graduation rate from 1999–2004 was 68% while the average graduation rate for high schools attended by PE participants was 78%; 69% of PE high schools were above the average graduation rate and 17% of PE high schools were below the average graduation rate. These calculations were weighted by the number of expected PE graduates from each school; that is, the calculations included students who the evaluator was reasonably certain had graduated (e.g., they were on track to graduate, or other students reported that they did) but who the evaluator was unable to reach during the evaluation period to confirm.

Thirty-two youth enrolled in college following graduation (79% of graduated PE seniors), with twenty-five of those students going to four-year degree programs (57% of graduated PE seniors). Subsequently, the net impact of participating in PE’s field program was that a PE participant was 3.4 times more likely to enroll in a 4-year college degree program than a typical CPS student.

Of PE field alumni who graduated from high school, 25% of all students and 35% of all female students went on to major in science in college.

Of PE alumni who went on to attend college, 34% of PE alumni and 42% of female PE alumni chose science as a major. Female alumni chose science majors at 5.3 times the national average rate, and male alumni chose science at 2.2 times the national rate.

Systemic

PE strategic planning documents recognized the need for baseline data and a long-term, ongoing evaluation strategy.

In response to the PART evaluation, PE established three strategic goals—curiosity, continuity, and capacity—to create a clear connection between program activities and the broader PE mission, and to assist future program planning.

In response to the PART evaluation, PE also drafted three outcome-based performance goals as initial steps to define an appropriate set of outcomes and measurements: access and equity, efficacy and engagement, and concepts and skills. The access and equity goal aimed to provide science opportunities to expand life and career options for youth who would otherwise not have such access. The efficacy and engagement goal aimed to enhance youth’s self-efficacy in, and positive attitudes toward, science through programs that offered meaningful opportunities for youth to engage in the practices of the scientific community. The concepts and skills goal referred to improving youth’s mastery of concepts and skills critical to the practice of science. Program staff expressed the need for goals to be closely aligned to the program design model to ensure that these goals and objectives were meaningful to PE and their work.

Based on the evaluation’s findings that PE generally lacked baseline data and measures, PE drafted efficiency-based performance measures for “efficacy and engagement” outputs and outcomes as a starting point for all performance goals to eventually be linked to at least one efficiency measurement.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project