  |
 |
Ideas for a Pilot Project Proposal for Evaluating
the Impact and/or Effectiveness of Technology
Scope
The Work Group brainstormed components of a project plan (framework), possibly using stratified random sampling, to evaluate the impact and/or effectiveness of technology.
What Constitutes a Successful Pilot Study?
- Good base-line data
- Consensus on data indicators and elements
- Survey data validated with authentic assessments
- Adoption of results by all states
- Indicators are deemed valuable and useful by districts and schools
- Collected data provide valid descriptions of the school
- There is a representative sample of LEAs that ensures the validity of results
- Results are generalizeable to all states
- Data correlates with student achievement (i.e., scientifically-based research/assessment)
- All school levels are represented (i.e., district, school, classroom; K-12 elementary and secondary)
- Data captures technology integration in curriculum
- There is research-based correlation to curriculum content standards
- Data collection addresses teacher and student behaviors
- Survey results provide contextual information
- Reporting formats integrate results reporting formats integrate policies, practices and proficiencies
- Data addresses leadership in both teaching practices and administrative practices
- Data address database-driven decision-making
- Local data is aggregateable to state and national levels
- Incentives are provided for school participation, such as:
- Value-added incentives for teachers, principles, and district administrators
- School improvement grants
Data adds to the knowledge of scientific research
Uses both survey and authentic assessment data (performance measures, portfolio observations)
- There is clear correlation between technology/intervention and achievement

Tasks/Components of a Successful Pilot Study
Focus
User base of effective practices
Outcomes emphasizing technology literacy and student achievement
Model emphasizes differential study components
Data Sources
Significant amount of data is generated from data regularly collected by schools (low data burden)
Lesson plans (samples of multiple instructional lessons
Authentic assessments, e.g., performance measures, portfolios, observations of teachers and students
Attitudes/beliefs surveys, interviews
Sampling
Define representative samples
Select a number of states to participate (requires OMB clearance if over 9 states)
Range of Technology Use
See the EnGauge Model below
Training
Evaluators
School personnel (teachers and administrators)
Selection of Organization to Conduct Study
RFP competition
Study conducted over 18 to 24 month period
Other Components
School commitment and buy-in
Incentive funding to participate in study
Data analysis methodologies
|
 |
|