Volume IX, Number 1, Spring 2003

Issue Topic: Evaluating Out-of-School Time

Beyond Basic Training

Doing What Works: Scientifically Based Research in Education

In the midst of recent education reform policies, a new phrase has taken center stage. Since its numerous appearances in the No Child Left Behind Act of 2001, the phrase “scientifically based research” has sparked fervent questions among researchers, educators, and policymakers. What exactly is scientifically based research? What are the implementation challenges? How can research and evaluation respond to these new demands? Suzanne Bouffard, consultant for Harvard Family Research Project, examines the new science-based research standards.

According to the No Child Left Behind Act (NCLB), all federally funded education programs, including out-of-school time programs that are funded under Title I,¹ must be based on research studies that meet scientific standards. Government publicity materials (see www.nochildleftbehind.gov) refer to this as “doing what works.” These user-friendly versions of the law cite the metaphor of medical research to explain how practice should be informed by experimental studies (i.e., studies in which participants are randomly assigned to treatment and control groups).

The primary goal of scientifically based research (SBR) is to ensure that programs for children are based on methods that have been proven effective and are therefore more likely to benefit other children, with a corollary goal of increasing the overall quality of education research. According to NCLB legislation, the following principles define scientific quality:

• Use of the scientific method with an emphasis on experimental control (or comparison) groups
• Replication of results, using multiple studies by different investigators
• Ability to generalize results from one sample to other children in the general population
• Fulfillment of rigorous standards with an emphasis on peer review
• Convergence (or consistency) of results between studies

Challenges to SBR Implementation
There is consensus that education research should follow the same general principles as other sciences, yet experts note some unique challenges to implementing SBR in educational contexts, including the realm of out-of-school time. For example, diverse values and goals for education provoke debate, and randomizing students to treatment and control groups poses ethical problems (Shavelson & Towne, 2002). Experts also question whether scientifically based research necessitates a specific research methodology, with many believing that scientific quality is determined not by a particular methodology, but by the appropriateness of the methodology for the research question. In fact, a recent report on SBR in education by the National Research Council (NRC) explicitly discourages the exclusion of non-experimental studies, with the explanation that descriptive studies may first be necessary in order to design effective experimental interventions (Shavelson & Towne); furthermore, random assignment to experimental groups is necessary only to establish causality. For some questions, for example how family relationships affect school achievement, random assignment may be neither desirable nor ethical (Raudenbush, 2002).

What Can Researchers and Evaluators Do?
The call for scientifically based research has been met with a number of responses. The NRC report recommends choosing a research method based on the kind of question being asked (Shavelson & Towne, 2002). Appropriate methods might include surveys, observations, or experiments. Other experts have pointed out that improving education research requires more than a strong methodology, and underscore a need for the collaboration of professionals from various jobs and disciplines. Some state that the best research is conducted by people with direct experience in the educational system (Gardner, 2002), with teachers providing valuable insight into questions concerned with activity in the classroom (Olson & Viadero, 2002). Others highlight how research from cognitive and developmental psychology can inform classroom instruction (Hirsch, 2002).

Experts also emphasize the role of a strong research community in increasing the quality of education research. Debate, discussion, and the peer review process are important, but researchers additionally stress the need for the accumulation of studies in order to establish scientific certainty. They caution that this process will require public patience and support, and should neither provoke discouragement nor overshadow existing research (Raudenbush, 2002). The challenge is to increase the quality of work while using the research that is currently available.

Toward this goal, the U.S. Department of Education’s Institute of Education Sciences has created a new “What Works Clearinghouse” of high-quality research studies (see www.w-w-c.org) and the independent, nonprofit Education Quality Institute will publish a collection of policy briefs on education research (see www.eqireports.org). These initiatives have resulted from an increased dialogue between researchers, practitioners, and policymakers. Some feel that this dialogue, along with the chance to improve the quality of education research and practice, may prove to be one of the greatest opportunities resulting from the new focus on scientifically based research.

Suzanne Bouffard, Consultant, HFRP
Email: suzanne.bouffard@duke.edu

For more information on scientifically based research in an out-of-school time context see Evaluating the 21st Century Community Learning Centers Program—A View From the States.

References

Gardner, H. (2002, September 4). The quality and qualities of educational research. Education Week, 22(1), 49, 72. [Available at: www.edweek.com/ew/ewstory.cfm?slug=01gardner.h22.]

Hirsch, E. D., Jr. (2002, October/November). Classroom research and cargo cults [Electronic version]. Policy Review, 115. [Available at: www.policyreview.org/OCT02/hirsch.html.]

Olson, L., & Viadero, D. (2002, January 30). Law mandates scientific base for research. Education Week, 21(20), 1, 14–15. [Available at: www.edweek.com/ew/ewstory.cfm?slug=20whatworks.h21.]

Raudenbush, S. (2002, February). Scientifically based research. Paper presented at the Scientifically Based Research Seminar, Washington, DC.

Shavelson, R. J., Towne, L., & the Committee on Scientific Principles for Education Research. (Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press.

¹ Title I refers to the set of programs in NCLB that relate to improving the academic achievement of the disadvantaged. As the largest federal program supporting elementary and secondary education (funded at $10.4 billion in FY 2002), Title I targets these resources to the districts and schools where the needs are greatest. Title I provides flexible funding that may be used to provide additional instructional staff, professional development, extended-time programs, and other strategies for raising student achievement in high-poverty schools.