Methodology
As described above, SETDA has generated conditions for, policies in support of, and indicators of the impact of effective technology uses related to NCLB, Title 2, Part D. This section recommends methods for assessing states' progress on these changes and describes the rationale behind these recommendations.
Three constraints were of primary importance as SETDA generated approaches to assessing progress:
1. First, SETDA recommends that states collect survey data at the state, district, and school levels, with one individual (a "point person") at each level responsible for collecting and reporting this data. This strategy has the following benefits: Surveys are an excellent strategy for collecting large amounts of data as long as the issues being surveyed are ones about which respondents have accurate information and as long as there is little reason for respondents to misreport that information (or as long as there are checks in place to ensure that respondents do not misreport information). Using a single point person to collect and report data places minimal burden on teachers and students.
Ideally, to gauge states' progress toward NCLB goals, survey methods would be used to collect data from every building and every district in each state. Although states can require schools receiving NCLB Title II funds to comply with data-collection efforts, SETDA recommends providing incentives to those schools not receiving funds to ensure statewide compliance. Furthermore, to ensure that the data collected is comparable across different schools, districts, and states, point-persons at each level should be provided with detailed instructions designed to guide the gathering and reporting of information.
Although survey methods will be used to assess some indicators within each of the five conditions outlined above, not all variables of interest are amenable to this type of study. Where either survey methods or the use of a single point person to gather information is likely to compromise data quality, data collection at the classroom level would be used to augment and complement survey data.
2. In addition to survey data, SETDA recommends that states collect data at the classroom level through site teams at schools identified through a stratified random sampling. This method would incorporate multiple data collection strategies, including student and teacher surveys, classroom observations, and artifact reviews. These methodologies, while able to provide more detailed information, are time, cost, and resource intensive. Thus, these methods are suggested only when survey methods cannot provide sufficient information about technology conditions or impacts, and they would be employed with small, representative samples of schools within each state rather than statewide.
Data collection methods at the classroom level would be used primarily to gather data about Educator Proficiency, Effective Practice, and Student Impacts, as these variables cannot be adequately measured solely through the use of surveys completed by a single point person. Again, to ensure that data is comparable across schools, districts, and states, it is critical that states employ rigorous data collection methodology. SETDA recommends that each institution be required to follow standard procedures for selecting representative samples, administering surveys, and collecting observation and artifact data. And particularly because these efforts place a greater burden on participants, SETDA recommends putting incentives in place to encourage the participation of all schools selected to take part in this process.
The charge of the SETDA taskforce is to generate a set of critical or core data elements that gauge a school, district, or state's progress toward NCLB goals. The notion is that these core elements would serve as the minimum set of data needed to gauge a state's standing on technology integration. While classroom level data will include multiple data sources, these elements should be treated as part of the minimum set of data. Schools that agree to participate should be required to commit to all classroom level data collection efforts, rather than participating in some subset of them.
Two final points: Prior to implementing either of the recommended data collection methods, it is important that all collection instruments be piloted and field-tested in a small subset of schools across the nation. Methods and instruments should only be implemented at a national level after it is determined that they generate data that accurately represents (within a reasonable margin of error) conditions in schools and districts.
It is important to emphasize that none of the data collection efforts proposed here are intended to profile schools or districts as "failing" or "succeeding" in their technology efforts. Rather, these efforts are intended to gauge progress within a state on technology integration, specifically integration that is likely to advance teaching, learning, and academic achievement. Furthermore, these efforts provide information about the state's status on conditions - contextual factors such as professional development opportunities, infrastructure, and school culture - that contribute to effective technology use. This information can be used to strategically deploy states' resources to address conditions where needs are greatest.
Early in the process, the SETDA taskforce realized that no one strategy for data collection could satisfy all of these constraints. In fact (and not surprisingly), adhering to one constraint often resulted in compromising another. For this reason, SETDA recommends a two-pronged approach to data collection.
