The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

• Permission to Use Materials: In consideration for your agreement to the terms and conditions contained here, the President and Fellows of Harvard College, c/o the Harvard Graduate School of Education, grant you a personal, non-exclusive, non-transferable license to access and use Harvard Family Research Project publications hosted on the Global Family Research Project Site.
• User may download material from the Site only for User's own personal, non-commercial use. User may not otherwise copy, reproduce, retransmit, distribute, publish, commercially exploit or otherwise transfer any material. The burden of determining that use of any information, software or any other content on the Site is permissible rests with User.
• User will seek permission or licensing for any commercial use requests from the Harvard Graduate School of Education Office of Sponsored Projects.

• Home
• Research Areas
  • Complementary Learning
  • Family Involvement
  • Out-of-School Time
  • Early Childhood Education
  • Evaluation
  • Other Research Areas
• Publications & Resources
  • Search Publications
  • Browse Our Publications
  • Publications Series
  • Copyright and Reprint Permissions
• HFRP News
  • News & Announcements
  • Blog: New Directions in Family Engagement
  • In the Media

• Publications & Resources
  • Search Publications
  • Browse Our Publications
  • Publications Series
  • Copyright and Reprint Permissions

• Large text size
• Normal text size
• Small text size

Evaluation terminology can be confusing to even the most seasoned of evaluators. This resource provides commonly accepted definitions to evaluation terms frequently used in the out-of-school time field. For “real life” examples of how these terms are used, check our Out-of-School Time Evaluation Database, currently offering detailed evaluation profiles of over 20 out-of-school time programs nationwide.

Accountability:
Accountability means that a public or private agency, such as a state education agency, that enters into a contractual agreement to perform a service, such as administer 21st Century Community Learning Center programs, will be held answerable for performing according to agreed-on terms, within a specified time period, and with a stipulated use of resources and performance standards.

Benchmark:
(1) An intermediate target to measure progress in a given period using a certain indicator. (2) A reference point or standard against which to compare performance or achievements.

Data collection methods:
Document Review:

• This is a review and analysis of existing program records and other information collected by the program. The information analyzed in a document review was not gathered for the purpose of the evaluation.
• Sources of information for document review include information on staff, budgets, rules and regulations, activities, schedules, attendance, meetings, recruitment, and annual reports.

Interviews/Focus Groups:

• Interviews and focus groups are conducted with evaluation and program/initiative stakeholders. These include, but are not limited to, staff, administrators, participants and their parents or families, funders, and community members.
• Interviews and focus groups can be conducted in person or over the phone. Questions posed in interviews and focus groups are generally open-ended and responses are documented in full, through detailed note-taking or transcription.
• The purpose of interviews and focus groups is to gather detailed descriptions, from a purposeful sample of stakeholders, of the program processes and the stakeholders' opinions of those processes.

Observation:

• Observation is an unobtrusive method for gathering information about how the program/initiative operates.
• Observations can be highly structured, with protocols for recording specific behaviors at specific times, or unstructured, taking a more casual, “look-and-see” approach to understanding the day-to-day operation of the program.
• Data from observations are used to supplement interviews and surveys in order to complete the description of the program/initiative and to verify information gathered through other methods.

Secondary Source/Data Review:

• These sources include data collected for other similar studies for comparison, large data sets such as the Longitudinal Study of American Youth, achievement data, court records, standardized test scores, and demographic data and trends.
• Like the information analyzed in a document review, these data were not gathered with the purposes of the evaluation in mind; they are pre-existing data that inform the evaluation.

Surveys/Questionnaires:

• Surveys and questionnaires are also conducted with evaluation and program/initiative stakeholders. These are usually administered on paper, through the mail, in a highly structured interview process in which respondents are asked to choose answers from those predetermined on the survey, or more recently, through email and on the Web.
• The purpose of surveys/questionnaires is to gather specific information—often regarding opinions or levels of satisfaction, in addition to demographic information—from a large, representative sample.

Tests/Assessments:

• These data sources include standardized test scores, psychometric tests, and other assessments of the program and its participants.
• These data are collected with the purposes of the evaluation in mind. For example, the administration of achievement tests at certain intervals to gauge progress toward expected individual outcomes documented in the evaluation.

Evaluation Design:
Experimental Design:

• Experimental designs all share one distinctive element: random assignment to treatment and control groups.
• Experimental design is the strongest design choice when interested in establishing a cause-effect relationship. Experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.

Non-Experimental Design:

• Non-experimental studies use purposeful sampling techniques to get “information rich” cases.
• Non-experimental evaluation designs include: case studies, data collection and reporting for accountability, participatory approaches, theory based/grounded theory approaches, ethnographic approaches, and mixed method studies.

Quasi-Experimental Design:

• Most quasi-experimental designs are similar to experimental designs except that the subjects are not randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment.
• Like the experimental designs, quasi-experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.
• Types of quasi-experimental designs include: comparison group pre-test/post-test design, time series and multiple time series designs, multiple time series designs, non-equivalent control group, and counterbalanced designs.

Formative/Process Evaluation:
Formative evaluations are conducted during program implementation in order to provide information that will strengthen or improve the program being studied—in this case, the out-of-school time program or initiative. Formative evaluation findings typically point to aspects of program implementation that can be improved for better results, like how services are provided, how staff are trained, or how leadership and staff decisions are made.

Indicator:
An indicator provides evidence that a certain condition exists or certain results have or have not been achieved. Indicators enable decision-makers to assess progress towards the achievement of intended outputs, outcomes, goals, and objectives.

Performance Measurement (also called Performance Monitoring):
“The ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals”¹ (sometimes also called outcomes). Performance measurement is typically used as a tool for accountability. Data for performance measurement is often tied to state indicators and is part of a larger statewide accountability system.

Summative/Outcome Evaluation:
Summative evaluations are conducted either during or at the end of a program's implementation. They determine whether a program's intended outcomes have been achieved—in this case, the out-of-school time program or initiative's outcomes. Summative evaluation findings typically judge the overall effectiveness or “worth” of a program based on its success in achieving its outcomes, and are particularly important in determining whether a program should be continued.

¹ U.S. Government Accounting Office, April 1998.

A Few Questions Explained

1. What is the difference between performance measurement and program evaluation?

2. What are the main features and trade-offs in design choice?

3. What is the difference between quantitative and qualitative data?