This article was downloaded by: [La Trobe University] On: 16 January 2013, At: 18:29 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Computers in the Schools
Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wcis20

Instruments for Assessing the Impact of Technology in Education
Rhonda Christensen & Gerald Knezek a a b

Institute for the Integration of Technology into Teaching and Learning (IITTL), University of North Texas, P.O. Box 311337, Denton, TX, 76203, USA b University of North Texas, P.O. Box 311337, Denton, TX, 76203, USA Version of record first published: 11 Oct 2008.

To cite this article: Rhonda Christensen & Gerald Knezek (2001): Instruments for Assessing the Impact of Technology in Education, Computers in the Schools, 18:2-3, 5-25 To link to this article: http://dx.doi.org/10.1300/J025v18n02_02

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.tandfonline.com/page/termsand-conditions This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages

whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Downloaded by [La Trobe University] at 18:29 16 January 2013

INSTRUMENTS AND TESTING
Downloaded by [La Trobe University] at 18:29 16 January 2013

Rhonda Christensen Gerald Knezek

Instruments for Assessing the Impact of Technology in Education

SUMMARY. Ten years of instrument development are summarized and placed within a framework for assessing the impact of technology in education. Seven well-validated instruments spanning the areas of attitudes, beliefs, skills, competencies, and technology integration proficiencies are presented, along with data analysis examples. These instruments are proposed for use in modeling the process of technology integration, which is believed to be an important intermediary step in effective use of technology in teaching and learning. [Article copies available for a fee from
The Haworth Document Delivery Service: 1-800-HAWORTH. E-mail address: Website: © 2001 by The Haworth Press, Inc. All rights reserved.] RHONDA CHRISTENSEN is Instructor, Institute for the Integration of Technology into Teaching and Learning (IITTL), University of North Texas, P.O. Box 311337, Denton, TX 76203 (E-mail: rhondac@tenet.edu). GERALD KNEZEK is Professor of Technology and Cognition, University of North Texas, P.O. Box 311337, Denton, TX 76203 (E-mail: gknezek@tenet.edu).
[Haworth co-indexing entry note]: “Instruments for Assessing the Impact of Technology in Education.” Christensen, Rhonda, and Gerald Knezek. Co-published simultaneously in Computers in the Schools (The Haworth Press, Inc.) Vol. 18, No. 2/3, 2001, pp. 5-25; and: Evaluation and Assessment in Educational Information Technology (ed: Leping Liu et al. ) The Haworth Press, Inc., 2001, pp. 5-25. Single or multiple copies of this article are available for a fee from The Haworth Document Delivery Service [1-800-HAWORTH, 9:00 a.m. - 5:00 p.m. (EST). E-mail address: getinfo@haworthpressinc.com].

 2001 by The Haworth Press, Inc. All rights reserved.

5

6

Evaluation and Assessment in Educational Information Technology

KEYWORDS. Technology, assessment, teachers, instruments, pre-service, in-service, attitudes, competencies, stages, integration
Downloaded by [La Trobe University] at 18:29 16 January 2013

CALL FOR ACCOUNTABILITY Significant resources have been expended to place computers in the schools over the past two decades. Recently, the call for accountability has become strong. According to a study by the Educational Testing Service, the total cost of technology in U.S. schools as of the late 1990s was about $3 billion, or $70 per pupil (Coley, Cradler, & Engal, 1998). Many educators have reported their opinions concerning the effects of this influx of technology on student learning; and, especially since the early 1990s, scholars in the field have pointed out the need to address the issue of accountability. As plans are made for the increased use of technology, it is important for policymakers, educators, and researchers to understand how teachers and children relate to this technology (Martin, Heller, & Mahmoud, 1992). Small Allocation for Teacher Training A major corollary of the accountability issue has to do with the proportion of technology funds that should be spent on training. Data from a 1995 national survey of school district technology budget allocations revealed that approximately 55% of technology money was being spent on hardware and 30% on software. Teacher education accounted for only 15% of the allocated funds (U.S. Congress, 1995). The U.S. Department of Education has recommended that districts allocate 30% of their technology budgets to staff development activities (U.S. Congress, 1995), but a CEO Forum report suggests that, as of 1998-99, schools were still spending less than 10% of their budget on training (CEO Forum, 1998). Projected Training Needs The 1998 CEO Forum study also estimated that the $3 billion spent on technology in U.S. schools represents just over 1% of total education spending, and that it will cost about $15 billion to make all our schools “technology rich.” This is about $300 per student, or about 5% of total education spending, and about five times what we now spend on technology (CEO Forum, 1998). If the recommended 30% of this $300 per

Instruments and Testing

7

Downloaded by [La Trobe University] at 18:29 16 January 2013

student is allocated to teacher training, then about $90 per pupil, or $1,800 per elementary school teacher with a class of 20, should be budgeted for training. This large proposed increase in funds for technology staff development necessitates studies to determine what types of teacher education and re-education lead to effective use of technology in the classroom. Indicators of Success Although there is a demonstrated need for investment in teacher training to make effective use of information technology in the classroom, there are no clear indicators of which prescribed training produces the desired outcomes. This paper focuses on indicators of technology integration and the association of these indicators with positive student learning outcomes. The Search for Positive Impact on Students Much recent research in educational technology has focused on impact. Researchers have been generally successful in demonstrating the positive impact of technology infusion with appropriate teacher training on teacher attitudes toward information technology, student attitudes toward information technology, and other learning-related student dispositions such as motivation (Christensen, 1997; Collis, Knezek, Lai, Miyashita, Pelgrum, Plomp, & Sakamoto, 1996; Woodrow, 1992). Instruments have been developed by several researchers to aid in this process (Christensen & Knezek, 1998; Knezek & Christensen, 1998; Knezek & Christensen, 2000; Ropp, 1999). Researchers have been less successful in identifying positive effects of technology infusion on student achievement, although some studies have met with success. For example, a review of 130 studies by Bailo and Sivin-Kachla (1995) concluded that using technology to support instruction improves student outcomes in language arts, math, social studies, and science. An evaluation of the West Virginia Basic Skills/Computer Education Program concluded that “the effective use of learning technology has led directly to significant gains in math, reading, and language arts skills in West Virginia” (Mann, Shakeshaft, Becker, & Kottkamp, 1999). More commonly, however, findings are mixed. For example, Pierce (1998) summarized a large-scale study of the impact of technology on mathematics achievement, conducted by the Educational Testing Service (ETS) and

8

Evaluation and Assessment in Educational Information Technology

sponsored by the Milken Exchange on Education Technology and Education Week:
Downloaded by [La Trobe University] at 18:29 16 January 2013

Students who spent the most time at a computer in school actually scored lower than their peers on a national math test, the study found. Students who used “drill and practice” software also scored lower. But students who used computers for simulations and real-life applications of math concepts scored higher, especially those students in middle school. The study suggests that school districts should focus attention on professional development for teachers to make sure they know how to use computers with their students effectively. (p. 1) Apparently, positive impacts of technology on achievement require wide-scale, long-term initiatives that include sufficient access to technology for students and teachers, and teacher training in appropriate technology integration techniques to be used in the classroom. Effective study of this kind of initiative could benefit from longitudinal data, and would likely be multivariate in nature. A research method that is believed to be well suited to these constraints is described in the following section. TECHNIQUES, PROCEDURES AND INSTRUMENTS Two theoretical approaches that form a logical basis for selecting assessment instruments of technology integration are presented in this paper. The first is diffusion of innovation, while the second is a structural model of technology integration. Early research by Rogers (1983) found that adoption of innovations is an active process that involves much reinvention. Adopters must reinvent the innovation and make it their own if they are to continue using it. Hall and Rutherford (1974) developed the Concerns-Based Adoption Model (CBAM) in the early 1970s for the study of adoption of any educational innovation. CBAM Stages of Concern (1974) and Levels of Use of the Innovation (Loucks, Newlove, & Hall, 1975) classifications have been adapted for information technology innovations and used by researchers on several continents over the past three decades. The Levels of Use form, which is described in more detail in the following section, is based on CBAM work.

Instruments and Testing

9

Downloaded by [La Trobe University] at 18:29 16 January 2013

The second approach is a structural model of technology integration developed by Knezek and Christensen (Knezek, Christensen, Hancock, & Shoho, 2000). This model is based on the assumption that educator will (positive attitude), skill (competency, ability to perform tasks), and access to technology tools are all required for successful technology integration. The model is grounded in educational psychology principles that determine key variables influencing the school learning environment (Klausmeir & Goodwin, 1975), and it relies on the multivariate technique of Structured Equation Modeling (Schumacker, 1996) to guide measurement techniques. The instruments presented in the booklet Instruments for Assessing Educator Progress in Technology Integration are generally organized along these lines (Knezek, Christensen, Miyashita, & Ropp, 2000). Descriptions of key instruments from this book, and related instruments relevant to the content of this paper, follow. Selected Instruments for Assessing the Impact of Technology in Education Attitude Instruments 1. The Teachers’ Attitudes Toward Computers Questionnaire (TAC version 5.1) (Christensen & Knezek, 1998) measures seven major indices regarding teacher attitudes. These scales are: F1-Enthusiasm/Enjoyment, F2-Anxiety, F3-Avoidance/Acceptance, F4–E-mail for Classroom Learning, F5-Negative Impact on Society, F6-Productivity, and F7-Semantic Perception of Computers. The reliabilities for these subscales typically range from .87 to .95 with K-12 teacher data. 2. Teachers’ Attitudes Toward Information Technology Questionnaire (TAT version 2.0) (Knezek & Christensen, 1998) is a semantic differential instrument that measures attitudes toward new information technologies including e-mail (variable coded as EMAILT), the World Wide Web (WWWT), multimedia (MMT), technology for teacher productivity (PRODT), and technology for classroom learning (PRODCL). Reliabilities for these scales typically range from .91 to .98 for K-12 teachers. Beliefs and Needs The Snapshot Survey by Norris and Soloway (Norris, Box, & Soloway, 1999; Norris, Soloway, Knezek, Topp, Young, & Box, 2000) addresses

10

Evaluation and Assessment in Educational Information Technology

how prevalent technology is in education today, and what an educator believes about the technology. This survey includes scales for beliefs and needs of the educators as well as classroom use of computers.
Downloaded by [La Trobe University] at 18:29 16 January 2013

Skill/Competency 1. Technology Proficiency Self-Assessment (Ropp, 1999) measures educator proficiency in e-mail, the World Wide Web (WWW), Integrated Applications (IA), and Teaching with Technology (TT). 2. Technology in Education Competency Survey (Christensen, 1999) is a self-assessment rating form covering teacher competencies in nine major areas addressed by the National Council for the Accreditation of Teacher Education (NCATE) standards for the United States. Level of Proficiency 1. Stages of Adoption of Technology (Christensen, 1997) is a self-assessment instrument of a teacher’s level of adoption of technology, based on earlier work by Russell (1995). 2. Level of Use (Griffin & Christensen, 1999) is a self-assessment instrument adapted from the Concerns-Based Adoption Model (CBAM) designations for adoption of an educational innovation. EVIDENCE OF SUCCESSFUL USE OF INSTRUMENTS Relationship of Attitudes to Stages of Adoption Data were gathered from 1,135 K-12 teachers from 13 school districts in north central Texas during 1998. This region of Texas contains predominantly rural schools. Educators were asked to complete the Stages of Adoption of Technology form (see Appendix A), the TAC, and the TAT during March 1998. The average Stage of Adoption value across these 1,135 educators was 4.13. High correlations were found between stages of adoption and computer anxiety (r = .67, p < .0005, N = 973), in the direction of higher stages being associated with reduced anxiety. Higher stages of adoption were also strongly associated with increased computer enjoyment (r =

Instruments and Testing

11

Downloaded by [La Trobe University] at 18:29 16 January 2013

.60, p < .0005, N = 973). Background information was gathered regarding home access to computers and the Internet. To determine whether use of a computer at home or access to the WWW at home made a difference in teachers’ stage of adoption, t-tests were carried out. Findings were that both home use of a computer and home access to the Internet were very strong discriminators for high or low stages of adoption (Knezek & Christensen, 1999). The nature of the relationship appears to be that home access makes a large contribution to technology integration advancement at the higher stages. A discriminant function analysis, which is a form of regression analysis in which the dependent variable (stage of adoption) is assumed to be only categorical, was carried out on the data to determine if teacher attitudes toward information technology are sufficiently strongly related to advances in stages of adoption to be useful as predictors of stage. The seven major attitude subscales on the TAC, plus computer enjoyment and anxiety subscales that were completed by teachers and students in the study, were used as predictors for stage of adoption. Based on their reported attitudes toward information technology, 46% of the teachers in the sample could be correctly classified into their own reported stage. The expectation for correct classification would be 1/6 = 16.7% by chance. Roughly 90% accuracy is possible if one is willing to accept plus or minus one stage as a success. This is often acceptable in a situation where the average stage of an entire school, rather than that of an individual teacher, is the goal (Knezek & Christensen, 1999). Reliability of Stages of Adoption as an Outcome Measure Because the Stages of Adoption instrument is a single-item survey, internal consistency reliability measures cannot be calculated. However, a high test-retest reliability estimate (.91) was obtained from a sample of 525 K-12 teachers from a metropolitan north Texas public school district for the period of August through November 1999. The Stages of Adoption form was included on two attitudinal questionnaires that were completed by educators as near to each other in time as within the same hour, or separated in time by as much as several days. During this process, educators never had access to the information provided on one survey while completing the other. A Pearson Product-Moment Correlation Coefficient was calculated between the two reported stage measures as a form of test-retest reliability. The resulting value of .91 indicates high consistency for these educators on reported stages within the recognized limitations (remembering and contextual cues) that un-

12

Evaluation and Assessment in Educational Information Technology

doubtedly inflated the estimate, compared to a standard reliability index.
Downloaded by [La Trobe University] at 18:29 16 January 2013

Relationship to CBAM Level of Use A study was conducted from August through September 1999 to determine if ratings on the CBAM Levels of Use of an Innovation scale (see Appendix B) were closely related to the ordered categories on the Stages of Adoption of Technology instrument. Responses from the same 525 educators cited in the previous section indicate that these two scales are related (r = .64). Future research is needed to determine if certain levels of use correspond with precise stages of adoption. Validation of Association with Curricular Sequence The first three University of North Texas courses listed in Table 1 have for more than a decade been approved for a Texas Education Agency Information Processing Technologies (IPT) endorsement, which can be added to a teaching certificate. Beginning in 1997, the third course (CECS 4100) was modified to include a major module on the Texas Essential Knowledge and Skills (TEKS), and funds awarded through a U.S. Department of Education Preparing Tomorrow’s Teachers to Use Technology (PT3) grant were used to add the fourth course beginning in spring 1999-2000. Completion of this four-course sequence entitles undergraduates to a University of North Texas certificate in curriculum and technology integration. During fall semester 1998, participants in the first course of the sequence (CECS 1100–Computer Applications) began on the average at stage 3.1 and exited the course on the average at stage 3.9. Students in the second course in the sequence (CECS 3440–Teacher Productivity Tools) began on the average at stage 4.1 (posttest data were not available), while students in the third course (CECS 4100) began on the average at stage 4.4 and exited from the course at stage 5.2. This can be compared to a typical teacher, as measured by a sample of more than 1,000 from the same geographic area, during the same period. The average rating across practicing teachers was 4.13, a value lying near the center of the indices reported across the three-course technology integration sequence. These results are graphically displayed in Figure 1 (Christensen & Knezek, 2000a).

Instruments and Testing

13

TABLE 1. Computer Education and Cognitive Systems (CECS) Courses in the Technology Applications Sequence

Downloaded by [La Trobe University] at 18:29 16 January 2013

Course Number CECS 1100

Course Title Computer Applications in Education Technology and the Teacher Computers in the Classroom

Course Description This is a tool-based course in which students learn to use an integrated word processing, spreadsheet, and database package. This course includes the use of presentation software and theories as well as the instructional design of presentation materials. Students in this class learn to find appropriate instructional software for the classroom, develop a multimedia project, find Internet resources for classroom use, develop a technology-infused lesson plan, and make a Web page to link to their instructional resources. This course provides field-based technology integration experiences to pre-service educators.

CECS 3440

CECS 4100

CECS 4800

Technology Integration Mentoring

FIGURE 1. Stages of Adoption of Technology for Three Classes of Pre-Service Teachers versus In-Service Teachers (1998)
6 5.2 5 4.13 4 3 2 1 0 Typical CECS 1100 CECS 3440 CECS 4100 Students Teacher Students Students (n = 1135) 3.1 3.9 4.1 Pretest Posttest 4.4

Profiles of Attitudes Toward Information Technology Among Educators at Various Stages of Adoption Results of analysis of the data from 1,135 rural Texas educators are presented in Table 2 and graphically illustrated in Figure 2 (Christensen & Knezek, 2000b). Based on analysis of data, it appears that teachers who are in Stage One (awareness) also rated themselves lower in computer enjoyment, computer avoidance, e-mail, productivity, and overall per-

14

Evaluation and Assessment in Educational Information Technology

TABLE 2. Means, Standard Deviations, and Levels of Significance for Profile of Rural Educators’ Attitudes Toward Technology by Stages of Adoption (1998 Data)

Downloaded by [La Trobe University] at 18:29 16 January 2013

Factor F1-Enjoyment F2-Anxiety (lack of) F3-Avoidance (acceptance) F4-E-mail F5-Negative Impact (lack of) F6-Productivity F7-Semantic Perception E-mailT WWWT MMT ProdT ProdCL

Stage 1 2.97 (.51) 2.59 (.65) 3.36 (.41) 3.14 (.50) 3.13 (.66) 2.88 (.42) 4.58 (.98) 4.18 (1.22) 4.64 (1.34) 4.93 (1.11) 4.73 (1.40) 4.89 (1.47) 18

Stage 2 3.15 (.59) 2.67 (.69) 3.62 (.39) 3.08 (.57) 3.11 (.58) 2.98 (.44) 4.55 (1.03) 4.73 (1.30) 5.17 (1.22) 5.31 (1.06) 5.00 (1.20) 5.63 (1.05) 72

Stage 3 3.39 (.52) 3.22 (.60) 3.79 (.34) 3.13 (.44) 3.40 (.50) 3.10 (.37) 5.02 (1.03) 4.77 (1.18) 5.30 (1.28) 5.52 (1.00) 5.55 (1.09) 5.96 (.96) 120

Stage 4 3.62 (.51) 3.58 (.63) 3.93 (.39) 3.23 (.61) 3.62 (.60) 3.28 (.38) 5.44 (1.03) 5.18 (1.17) 5.60 (1.22) 5.84 (1.02) 5.95 (.93) 6.02 (1.06) 248

Stage 5 3.85 (.50) 4.01 (.52) 4.08 (.37) 3.30 (.59) 3.70 (.52) 3.39 (.37) 5.77 (.82) 5.51 (1.12) 5.93 (1.04) 6.02 (.99) 6.18 (.83) 6.26 (.86) 217

Stage 6 4.12 (.51) 4.44 (.47) 4.28 (.32) 3.62 (.80) 3.96 (.55) 3.62 (.37) 6.13 (1.10) 5.90 (1.28) 6.33 (1.01) 6.44 (.88) 6.56 (.79) 6.62 (.73) 127

Sig F .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000

N

Note. Anxiety, Avoidance and Negative Impact are coded in a positive direction. Higher means represent a lack of anxiety, lack of avoidance and lack of perceived negative impact.

ception of computers. They rated themselves as being more anxious toward computers and more negative in their feelings about the impact of computers. Teachers who reported being in the sixth stage of technology adoption had the highest mean scores among the six stages of adoption category groupings in computer enjoyment, e-mail, productivity, semantic perception of computers, e-mail for teachers, WWW for teachers, multimedia for teachers, productivity for teachers, and productivity for classroom use. This subset of teachers also rated themselves the lowest of all the groups of teachers in anxiety, computer avoidance, and a negative feeling toward the impact of computers.

Instruments and Testing

15

FIGURE 2. Profile of Attitude by Stage of Adoption for Rural Texas Educator Data (1998) Attitude Profiles by Stage

Downloaded by [La Trobe University] at 18:29 16 January 2013

7 6.5 6 5.5 5 4.5 4 3.5 3 2.5

F1-Enjoyment

F3-Avoidance

F2-Anxiety

F4-Email

F7-Semantic Perception

WWWT

MMT

2

F5-Negative Impact

F6Productivity

Stage 1

Stage 2

Stage 3

Stage 4

Stage 5

Stage 6

Associations Between Increased Technology Proficiency and Self-Reported Level of Technology Integration Teachers from a suburban metropolitan area in north Texas received weekly local site training with the technology purchased through a school district bond election. Educators were invited to complete pretest and posttest online assessments of their technology skills and perceived needs during the fall 1999. One hundred eight (108) teachers responded to the request for pretest data, while 262 provided online posttest data. The Technology Proficiency Self-Assessment Instrument (TPSA) was included to gather data on teacher competencies (Ropp, 1999). This instrument was selected because it has high reliability and measures skills recommended by the International Society for Technology in Education for all K-12 teachers. Four of Ropp’s measurement scales (with five items each) were included: E-mail, Integrated Applications (IA), Teaching with Technology (TT), and the World Wide Web (WWW). On a scale of 1 (little knowledge) to 5 (great knowledge), e-mail proficiency increased from a group mean average of 3.34 at the time of the

ProdCL

EmailT

ProdT

16

Evaluation and Assessment in Educational Information Technology

Downloaded by [La Trobe University] at 18:29 16 January 2013

pretest in early fall, to 4.20 at the time of the posttest in December 1999. Proficiency with integrated applications increased from a group mean average of 2.85 on the pretest to 4.18 by the posttest. Teachers’ perceived proficiency in teaching with technology improved from an average rating of 3.25 at the time of the pretest to 4.16 at posttest time. Proficiency with the World Wide Web improved from 3.60 at the pretest to 4.53 at the posttest. All of these changes are highly significant in the statistical sense (p < .0005). Furthermore, as shown in Figure 3, technology skill profiles for the groups with different reported stages of adoption were quite distinct, for all groups except Stage Five versus Stage Six. This implies that some other attributes besides technology skills (such as teacher attitudes) may be the distinguishing factor for Stage Five versus Stage Six level of technology integration. Changes in Pre-Service Educator Skills Data were gathered from UNT pre-service teachers using the Technology Proficiency Skills Assessment (TPSA) for the first time during fall 1999. Initial findings, based on one CECS 4100 class of 21 students, are graphically displayed in Figure 4. Students enrolled in the Computers in the Classroom course reported significant skill gains (p < .001)
FIGURE 3. Profile of Confidence in Skills by Stage of Adoption for Suburban Texas Educator Data (1999) Skill Profiles by Stage
5 4.5 4 3.5 3 2.5 2 TPTT TPIA TPWWW TPEMAIL

Stage 2 Stage 3 Stage 4 Stage 5 Stage 6

Instruments and Testing

17

FIGURE 4. Pre- and Posttest Technology Integration Skills Comparison for CECS 4100 Pre-Service Students TPSA Pre-Service Gains

Downloaded by [La Trobe University] at 18:29 16 January 2013

6 5 4 3 2 1 0

5.6 4.5 4.54

5.84 5.39

5.56

4.03

3.96 Pretest Posttest

E-mail

WWW

Integrated App.

Tech. in Teaching

in four areas measured by the TPSA. The effect sizes of these gains ranged from .86 to 1.39. The average gain in level of technology proficiency was approximately one standard deviation. Pre-post changes in ISTE competencies gathered from a class of pre-service teacher education students during fall 1999 are shown in Figure 5. This direct appraisal, nine-item form developed by Christensen, has yielded an internal consistency reliability estimate of .92 across 188 pre-service educators and 40 university faculty (Christensen & Knezek, 2000c). Items for this instrument are listed in Appendix C. WST Model Collaborative work by the authors dating back to 1991 has recently been consolidated into a new model for integrating technology into the classroom. This model relies on multiple indicators to show that technology investment can contribute to student achievement (see Figure 6). The Will, Skill, Tool (WST) Model includes three key elements for successful integration of technology: will (attitude) of the teacher, skill (technology competency), and technology tools (access to technology tools). The model also postulates that technology integration in the classroom contributes to higher student achievement. Preliminary analysis of this model has yielded promising results. Two studies based on this approach are summarized in the following section.

18

Evaluation and Assessment in Educational Information Technology

FIGURE 5. Pre- and Post-Competencies for CECS 4100 Pre-Service Students
CECS 4100 Pre/Post Competencies Fall 1999 5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 MEAN 4100 pretest 4100 posttest

Downloaded by [La Trobe University] at 18:29 16 January 2013

IMPACT OF WILL, SKILL, AND TOOLS ON TECHNOLOGY INTEGRATION Survey responses gathered during fall 1999 from teachers in a public high school in the Dallas/Fort Worth metroplex area of northern Texas were used to test some components of this model (Knezek, Christensen, Hancock, & Shoho, 2000). Regression analyses were carried out on data from 39 teachers who completed a battery of instruments. A brief summary of the findings follows. Impact of Will. Approximately 40% of the variance in stage of adoption was found to be attributable to “will” measures for these teachers. The R-squared for Stage of Adoption predicted from the TAT attitude scales (E-mailT, WWWT, MMT, ProdT, ProdCL) was .39. Impact of Will and Skill Combined. Adding skill measures to the equation increased the predictability of Stages of Adoption of Technology from roughly 40% to 70%. The R-squared for stages predicted from TAT attitude measures and TPSA skill measures was .69. Combined Impact of Will, Skill and Access to Technology Tools. Adding measures of access to technology tools for teachers increased the predictability of Stages of Adoption from 70% to 84% for this set of data. The R-squared for TAT attitudes, TPSA skills, and the three tool variables of current hours per week using technology in the classroom

Instruments and Testing

19

FIGURE 6. Will, Skill, Tool (WST) Model of Technology Integration
SAT

Downloaded by [La Trobe University] at 18:29 16 January 2013

TAC-7F

Achievement

TAAS

TAT-5F

Will

Other standardized scores

TPSA-EMAIL CBAM-LOU TPSA-WWW TSPA-IA STAGES TPSA-TT Skill Classroom Integration

CHOURS HOMECOM HOMEWWW

Tools

FACILITIES

(CHOURS), access to a computer at home (HOMECOM) and access to the WWW at home (HOMEWWW) was .84. Goodness of Fit for Individual Educators. Ideally, structured equation modeling (Joreskog & Sorbom, 1998; Schumacker, 1996) would have been employed to determine how well the model fit the data, but its use was not attempted in the spring 2000 analysis due to the small data set and large number of degrees of freedom in the model (Schumacker, 1999). Instead, discriminant function analysis was chosen to explore how well will, skill, and technology tools could place an individual in the proper Stage of Adoption. Twenty-five of 39 secondary school teachers provided complete data and were used in the analysis. The self-reported Stage of Adoption for these 25 ranged from Stage 2 (learning the process) to Stage 6 (creative applications to new contexts). The combined prediction abilities of the four discriminant functions derived for the data set yielded 100% accuracy in placing educators into

20

Evaluation and Assessment in Educational Information Technology

proper stages of adoption, based solely on the will, skill, and tool parameters indicated in the model (Knezek, Christensen, Hancock, & Shoho, 2000). Impact of Technology Investment on Student Achievement. In a related test of the WST Model focused on student achievement, Hancock (Knezek, Christensen, Hancock, & Shoho, 2000) randomly selected 100 of approximately 1,046 Texas school districts for detailed study of available variables representing both ends of the technology infusion-to-student-achievement continuum. The Texas Assessment of Academic Skills (TAAS) test pass rate and district-wide average SAT score were used as dependent variables in a regression analysis. District-wide hardware expenditures and district-wide expenditures on software and technology supplies were the two independent variables used to indicate extent of support for information technology (tool) facilities. Major findings were that hardware and software expenditures together accounted for approximately 11% of the variance in SAT scores across the sample of school districts (R squared = .11498). This relationship was found to be significant at the .05 level (F = 3.25, 2 3 50 df, p = .0472). In addition, hardware and software expenditures together accounted for approximately 11% of the variance in TAAS pass rates across school districts (R squared = .11). This relationship was also significant at the .05 level (F = 3.70, 2 3 58 df, p = .03). In summary, approximately 11% of Texas schools districts’ standardized student achievement scores could be predicted from level of investment in technology (Knezek, Christensen, Hancock, & Shoho, 2000). These findings should certainly be interpreted with caution due to the small sample sizes and other limitations of the analyses. Nevertheless, they serve to illustrate the potential utility of the model. CONCLUSION Seven well-validated instruments spanning the areas of attitudes, beliefs, skills, competencies, and technology integration proficiencies have been developed by the authors and their colleagues over the past decade and assembled within a framework for technology integration. Research related to the development of these instruments, and findings resulting from the instruments themselves, has lead to the conclusion that will (motives, positive attitudes), skill (ability to use software applications), and tools (access to hardware and software systems) are all essential ingredients for a teacher to effectively integrate information

Downloaded by [La Trobe University] at 18:29 16 January 2013

Instruments and Testing

21

Downloaded by [La Trobe University] at 18:29 16 January 2013

technology into his/her daily classroom practices. The authors conjecture that effective technology integration at the classroom level will then lead to a positive impact on student learning and achievement. Future research is planned to more fully explore parameters influencing level of technology integration, and to test the extent to which classroom technology integration influences student achievement. REFERENCES
Bailo, E. R., & Sivin-Kachla, J. (1995). Effectiveness of technology in schools, 1990-1994. Washington, DC: Software Publishers Association. CEO Forum. (1998). Year 2 STaR report: Professional development: A link to better learning. [Online]. Available: http://www.ceoforum.org Christensen, R. (1997). Effect of technology integration education on the attitudes of teachers and their students. [Online]. Unpublished doctoral dissertation, University of North Texas, Denton, Texas. Available: http://courseweb.tac.unt.edu/rhondac Christensen, R. (1999). Technology in Education Preservice Competency Survey. [Online]. Available: http://www.iittl.unt.edu/unt/students/tpsa2.htm Christensen, R., & Knezek, G. (1998). Parallel forms for measuring teachers’ attitudes toward computers. In S. McNeil, J. Price, S. Boger-Mehall, B. Robin, & J. Willis (Eds.), Technology and teacher education annual 1998 (Vol. 2, pp. 820-824). Charlottesville, VA: Association for the Advancement of Computing in Education. Christensen, R., & Knezek, G. (1999). Stages of adoption for technology in education. Computers in New Zealand Schools, 11(3), 25-29. Christensen, R., & Knezek, G. (2000a). Advancement of student technology integration skills through university preservice coursework. In D. Willis, J. Price, & J. Willis (Eds.), Technology and teacher education annual 2000 (Vol. 2, pp. 1505-1510). Charlottesville, VA: Association for the Advancement of Computing in Education. Christensen, R., & Knezek, G. (2000b). Strategies for integrating technology into the classroom. [Online]. Pre-conference symposium paper presented at the Society for Information Technology in Teacher Education Annual Conference, San Diego, CA. Available: http://www.iittl.unt.edu/IITTL/presentations Christensen, R., & Knezek, G. (2000c, April 25). Internal consistency reliability for the Technology in Education Competency Survey. Paper presented to the Preparing Tomorrow’s Teachers Evaluator’s Workshop, American Educational Research Association Annual Meeting, New Orleans, LA. Coley, R. J., Cradler, J., & Engal, P.K. (1998). Computers and classrooms: The status of technology in U.S. schools (Policy Information Report). Princeton, NJ: Policy Information Center, Educational Testing Service. Collis, B. A., Knezek, G. A., Lai, K. W., Miyashita, K. T., Pelgrum, W. J., Plomp, T., & Sakamoto, T. (1996). Children and computers in school. Mahwah, NJ: Erlbaum. Griffin, D., & Christensen, R. (1999). Concerns-Based Adoption Model (CBAM) Levels of Use of an Innovation (CBAM-LOU). Denton, TX: Institute for the Integration of Technology into Teaching and Learning.

22

Evaluation and Assessment in Educational Information Technology

Hall, G. E., & Rutherford, W. L. (1974). Concerns questionnaire. Procedures for Adopting Educational Innovations/CBAM Project, R&D Center for Teacher Education, University of Texas at Austin. Joreskog, K. G., & Sorbom, D. (1998). Lisrel 8: Structural equation modeling with the Simplis command language. Mahwah, NJ: Erlbaum. Klausmeir, H. J., & Goodwin, W. (1975). Learning and human abilities: Educational psychology (4th ed.). New York: Harper & Row. Knezek, G., Christensen, R., Hancock, R., & Shoho, A. (2000, February). Toward a structural model of technology integration. Paper presented to the Annual Hawaii Educational Research Association, Honolulu, Hawaii. Knezek, G., & Christensen, R. (2000). Attitudinal differences among integrated computing and traditional computer literacy students in the USA. In C. Morales, G. Knezek, R. Christensen, & P. Avila (Eds.), Impact of new technologies on teaching and learning (pp. 85-97). Mexico City, Mexico: Institute of the Educative Communication. Knezek, G., & Christensen, R. (1998). Internal consistency reliability for the Teachers’ Attitudes Toward Information Technology (TAT) questionnaire. In S. McNeil, J. Price, S. Boger-Mehall, B. Robin, & J. Willis (Eds.), Technology and teacher education annual 1998 (Vol. 2, pp. 831-832). Charlottesville, VA: Association for the Advancement of Computing in Education. Knezek, G., Christensen, R., Miyashita, K., & Ropp, M. (2000). Instruments for assessing educator progress in technology integration. [Online]. Available: http:// www.iittl.unt.edu Loucks, S. F., Newlove, B. W., & Hall, G. E. (1975). Measuring levels of use of the innovation: A manual for trainers, interviewers, and raters. Austin, TX: The University of Texas. Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999) West Virginia story: Achievement gains from a statewide comprehensive instructional technology program 1999: What impact does technology have on learning? [Online]. Available: http://www.mff.org/edtech Martin, C. D., Heller, R. S., & Mahmoud, E. (1992). American and Soviet children’s attitudes toward computers. Journal of Educational Computing Research, 8(2), 155-185. Norris, C., Box, K., & Soloway, E. (1999, June 20). Technology in the classroom: What do teachers do, believe, need? Snapshots from around the U.S.A. Paper presented at the International Society for Technology in Education Leadership Symposium, National Education Computer Conference, Atlantic City, NJ. Norris, C. A., Soloway, E., Knezek, G., Topp, N. W., Young, J., & Box, K. (2000). Snapshot survey: What do your administrators and teachers really need? Electronic School: The School Technology Authority, 187(6), 32-34. Pierce, D. (1998 October 5). ETS study shows how computers can help or hurt math achievement. eSchool News. [Online]. Available: http://eschoolnews.com. Rogers, E. M. (1983). Diffusion of innovations (3rd ed.) New York: The Free Press. Ropp, M. M. (1999). Exploring individual characteristics associated with learning to use computers in preservice teacher preparation. Journal of Research on Computing in Education, 31(4), 402-424.

Downloaded by [La Trobe University] at 18:29 16 January 2013

Instruments and Testing

23

Downloaded by [La Trobe University] at 18:29 16 January 2013

Russell, A. L. (1995). Stages in learning new technology: Naive adult email users. Computers in Education, 25(4), 173-178. Schumacker, R. E. (1996). A beginner’s guide to structural equation modeling. Mahwah, NJ: Erlbaum. U.S. Congress, Office of Technology Assessment. (1995, April). Teachers and technology: Making the connection (OTA-EHR-616). Washington, DC: U.S. Government Printing Office. Woodrow, J. E. (1992). The influence of programming training on the computer literacy and attitudes of preservice teachers. Journal of Research on Computing in Education, 25(2), 200-218.

APPENDIX A Stages of Adoption of Technology
Name: __________________________ Date: ____________

Instructions: Please read the descriptions of each of the six stages related to adoption of technology. Choose the stage that best describes where you are in the adoption of technology. Stage 1: Awareness I am aware that technology exists but have not used it–perhaps I'm even avoiding it. Stage 2: Learning the process I am currently trying to learn the basics. I am often frustrated using computers. I lack confidence when using computers. Stage 3: Understanding and application of the process I am beginning to understand the process of using technology and can think of specific tasks in which it might be useful. Stage 4: Familiarity and confidence I am gaining a sense of confidence in using the computer for specific tasks. I am starting to feel comfortable using the computer. Stage 5: Adaptation to other contexts I think about the computer as a tool to help me and am no longer concerned about it as technology. I can use it in many applications and as an instructional aid. Stage 6: Creative application to new contexts I can apply what I know about technology in the classroom. I am able to use it as an instructional tool and integrate it into the curriculum.
From: Christensen, R. (1997). Effect of technology integration education on the attitudes of teachers and their students. Doctoral dissertation, University of North Texas. Based on Russell, A. L. (1995). Stages in learning new technology. Computers and Education, 25(4), 173-178.

24

Evaluation and Assessment in Educational Information Technology APPENDIX B Concerns-Based Adoption Model (CBAM) Levels of Use of an Innovation

Downloaded by [La Trobe University] at 18:29 16 January 2013

Level 0 Non-use I have little or no knowledge of information technology in education, no involvement with it, and I am doing nothing toward becoming involved. Level 1 Orientation I am seeking or acquiring information about information technology in education. Level 2 Preparation I am preparing for the first use of information technology in education. Level 3 Mechanical Use I focus most effort on the short-term, day-to-day use of information technology with little time for reflection. My effort is primarily directed toward mastering tasks required to use the information technology. Level 4 A Routine I feel comfortable using information technology in education. However, I am putting forth little effort and thought to improve information technology in education or its consequences. Level 4 B Refinement I vary the use of information technology in education to increase the expected benefits within the classroom. I am working on using information technology to maximize the effects with my students. Level 5 Integration I am combining my own efforts with related activities of other teachers and colleagues to achieve impact in the classroom. Level 6 Renewal I reevaluate the quality of use of information technology in education, seek major modifications of, or alternatives to, present innovation to achieve increased impact, examine new developments in the field, and explore new goals for myself and my school or district. I best fit into Level ____________.

Instruments and Testing

25

APPENDIX C Technology in Education Competency Survey

Downloaded by [La Trobe University] at 18:29 16 January 2013

1 = Strongly Disagree (SD) 2 = Disagree (D) 3 = Undecided (U) 4 = Agree (A) 5 = Strongly Agree (SA)

SD 1. I feel competent using a word processor and graphics to develop lesson plans. 2. I feel competent using e-mail to communicate with colleagues. 3. I feel competent using the World Wide Web to find educational resources. 4. I feel competent using an electronic grade book. 5. I feel competent constructing and implementing project-based learning lessons in which students use a range of information technologies. 6. I feel competent to help students learn to solve problems, accomplish complex tasks, and use higher order thinking skills in an information technology environment. 7. I feel competent in recognizing when a student with special needs may benefit significantly by the use of adaptive technology. 8. I feel competent about teaching K-12 students age-appropriate information technology skills and knowledge. 9. I feel competent working with students in various IT environments (such as stand-alone and networked computers, on-computer classrooms, labs, etc.). 1

D 2

U 3

A 4

SA 5

1 1

2 2

3 3

4 4

5 5

1 1

2 2

3 3

4 4

5 5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

Adapted from Preparedness of graduates (pp. 41-43), International Society for Technology in Education (1999). Will new teachers be prepared to teach in a digital age? A national survey on information technology in teacher education. Santa Monica, CA: Milken Family Foundation, Milken Exchange on Education Technology.