Evaluation Update: November 2004

The Evaluation of Educational Technology Interventions got under way in fall 2003. This congressionally mandated study is analyzing the effectiveness of technology in improving learning in math and reading for students across the U.S.

The first year began with solicitations of applications from developers of educational technology software. The research team reviewed more than 160 software products, selecting 16 to be part of the study (click here for a list of the packages selected and summaries of their features). Of these products, 15 are being used in participating school districts. About half of the interventions being tested focus on improving reading in the first and fourth grades; the other half are geared toward boosting math skills, particularly in algebra, in grades six and nine.

After the selection process was completed, the next step was to identify school districts interested in testing the technology interventions. In addition to receiving nominations from companies whose products were selected for study, the research team also contacted many school districts and received inquiries from others. Ultimately, 32 districts implemented one or more of the software products.

In mandating this study, Congress required that it use scientific methods and control groups or conditions, which in this case involve an experimental research design based on random assignment. Teachers who volunteered to test the products were assigned by lottery to one of two groups: (1) one using the technology interventions (the program group), or (2) one not using the interventions (the control group). Of the 450 or so teachers participating in the study, 250 are in the program group, and 200 are in the control group. The bulk of these teachers (about 300) are testing the reading interventions in grades 1 and 4; about 150 teachers are piloting the math interventions in grades 6 and 9. However, the numbers of students testing each type of software is approximately equal, since most of the math teachers are using the interventions in more than one classroom.

To evaluate learning gains, the research team is administering achievement tests to students at the beginning and end of the school year. The team is also observing classrooms three times and interviewing teachers over the course of the year to look at the context and conditions that are related to effectiveness of educational technology. Both efforts got under way in October 2004. First-, fourth-, and sixth-grade students are taking the SAT 10 (known as the SESAT in first grade). Ninth-grade students are taking a customized test developed by Educational Testing Service, based on the ETS end-of-year algebra test, to assess their algebra skills.

During summer and fall 2005, the team will analyze the data and prepare a report to be submitted to Congress. Consideration is being given to adding another year of data collection with the same teachers and new students. These data would be gathered in 2005-2006.