SETDA National Leadership Institute Toolkit


Ideas for a Pilot Project Proposal for Evaluating the Impact and/or Effectiveness of Technology

Scope
The Work Group brainstormed components of a project plan (framework), possibly using stratified random sampling, to evaluate the impact and/or effectiveness of technology.

What Constitutes a Successful Pilot Study?

  • Good base-line data
  • Consensus on data indicators and elements
  • Survey data validated with authentic assessments
  • Adoption of results by all states
  • Indicators are deemed valuable and useful by districts and schools
  • Collected data provide valid descriptions of the school
  • There is a representative sample of LEAs that ensures the validity of results
  • Results are generalizeable to all states
  • Data correlates with student achievement (i.e., scientifically-based research/assessment)
  • All school levels are represented (i.e., district, school, classroom; K-12 elementary and secondary)
  • Data captures technology integration in curriculum
  • There is research-based correlation to curriculum content standards
  • Data collection addresses teacher and student behaviors
  • Survey results provide contextual information
  • Reporting formats integrate results reporting formats integrate policies, practices and proficiencies
  • Data addresses leadership in both teaching practices and administrative practices
  • Data address database-driven decision-making
  • Local data is aggregateable to state and national levels
  • Incentives are provided for school participation, such as:
  • Value-added incentives for teachers, principles, and district administrators
  • School improvement grants ­ Data adds to the knowledge of scientific research ­ Uses both survey and authentic assessment data (performance measures, portfolio observations)
  • There is clear correlation between technology/intervention and achievement

Tasks/Components of a Successful Pilot Study
Focus

  • User base of effective practices
  • Outcomes emphasizing technology literacy and student achievement
  • Model emphasizes differential study components
    Data Sources
  • Significant amount of data is generated from data regularly collected by schools (low data burden)
  • Lesson plans (samples of multiple instructional lessons
  • Authentic assessments, e.g., performance measures, portfolios, observations of teachers and students
  • Attitudes/beliefs surveys, interviews
    Sampling
  • Define representative samples
  • Select a number of states to participate (requires OMB clearance if over 9 states)
    Range of Technology Use
  • See the EnGauge Model below

    EnGauge Model

    Training

  • Evaluators
  • School personnel (teachers and administrators)
    Selection of Organization to Conduct Study
  • RFP competition
  • Study conducted over 18 to 24 month period
    Other Components
  • School commitment and buy-in
  • Incentive funding to participate in study
  • Data analysis methodologies

  • Scientifically Based Research
    Technology Literacy Assessment
    Common Data Elements
       
    Chapter Summary

    An Assessment Framework

    Gauging Progress

    The NCLB Matrix

    Methodology

    Key Questions

    Common Data Elements for Education Technology Assessment

    SETDA Common Data Elements Project

    Ideas for a Pilot Project

    Effective Teaching with Technology Assessment
    National Education Technology Plan