IS Success Model in E-Learning Context Based on Students' Perceptions
Freeze, Ronald D; Alshare, Khaled A; Lane, Peggy L; Wen, H Joseph. Journal of Information Systems Education21.2 (2010): 173-184.
The title captures the objective of the study which is to evaluate the success of the E-learning based on the IS success model. Abstract
This study utilized the Information Systems Success (ISS) model in examining e-learning systems success. The study was built on the premise that system quality (SQ) and information quality (IQ) influence system use and user satisfaction, which in turn impact system success. A structural equation model (SEM), using LISREL, was used to test the measurement and structural models using a convenience sample of 674 students at a Midwestern university. The results revealed that both system quality and information quality had significant positive impact on user satisfaction and system use. Additionally, the results showed that user satisfaction, compared to system use, had a stronger impact on system success. Implications for educators and researchers are reported.
Keywords: IS Success, E-Learning, User satisfaction, System use, System quality, Information quality
1. INTRODUCTION
Both undergraduate and graduate courses are experiencing a migration away from the traditional classroom and toward a greater emphasis for electronic delivery of content (Allen and Seaman, 2008). This trend cuts across all departments and schools in the university system but is especially critical in business schools, since the preparation of students for successful business careers will rely on the students' abilities to accurately assess the quality of and rapidly adapt to the changing systems that reflect radical technological advances. The Information Systems Success (ISS) model focuses attention on the information and system quality of specific IT systems. The expanded use of electronic means of course delivery results in different IT systems in which students develop various views of the system quality and information quality that may affect their educational outcomes.
In a graduate online information management course, feedback provided in the e-learning environment affected student satisfaction, the typical outcome measure for the ISS model (Rossin, et al., 2008). Feedback, in the context of an e-learning environment, is a measure of the information quality provided by the instructor during course delivery. In addition, the perceived balance of challenge and skill necessary to be successful in the course also affected the satisfaction with the course. The balance of challenge and skill necessary for the online delivery of the e-learning experience is a measure of the system quality. Information quality can also be electronically delivered and assessed by individuals with an information system being absent from the process. In a business environment, the information needs of managers in different functional areas are critical aspects during the evaluation of information and subsequently its quality (Beard and Peterson, 2003). For students, information needs may vary from course to course as well as among various homework assignment styles (e.g. quizzes, short-answer questions, and case studies). The concluding goal of this study ends with a discussion of how an information system can facilitate the delivery of the required information.
While the ISS model is used in many instances, a basic assumption of the model is one of voluntary use by the user. This assumption is incorrect in the context of university elearning courses where usage of the system is required to complete the coursework. Usage of a non-voluntary system is not without its parallels in industry. The implementation of enterprise resource planning (ERP) systems for many companies requires the usage of these systems by employees. This industry need has translated these requirements into ERP system courses (Davis and Comeau, 2004). Since the usage of e-learning systems in academic settings is not voluntary, the application and possible changes to the ISS model to an online course environment is a necessary and critical extension of the study of information systems. This study applies the ISS model to study e-learning systems (ELS) in the context of individual impact for a student online environment. The remainder of this article presents the ISS model with its standard constructs. The methodology used to assess the study is reviewed. The data analysis and results are then addressed. Finally, a discussion of the conclusions along with limitations are presented.
2. LITERATURE REVIEW
The ISS model (DeLone and McLean, 1992) is among the most influential theories in predicting and explaining system use, user satisfaction, and IS success (Halawi, McCarthy, & Aronson, 2008; Guimaraes, Armstrong, & Jones, 2009). The ISS model can be used to assess ELS success due to the solid theoretical foundation and the numerous, successful empirical studies.
The base ISS model consists of six constructs or dimensions: (1) system quality, (2) information quality, (3) systems use, (4) user satisfaction, (5) individual impact and (6) organizational impact. DeLone and McLean (1992) suggested these six dimensions of success are interrelated rather than independent. System quality and information quality separately and jointly affect both use and user satisfaction. Additionally, the amount of use can affect the degree of user satisfaction - positively or negatively - and vice versa. Use and user satisfaction are direct antecedents of individual impact; and lastly, individual performance should eventually have some organizational impact.
DeLone and McLean (2003) proposed an updated ISS model and evaluated its usefulness in light of the dramatic changes in IS practice, especially the emergence and consequent explosive growth of web-based applications. Based on prior studies, the ISS model was updated by adding "service quality" measures as a new dimension and by grouping all the "impact" measures into a single impact or benefit construct called "net benefit" (DeLone and McLean, 2003). Thus, the updated model consists of six dimensions: (1) information quality, (2) system quality, (3) service quality, (4) use/intention to use, (5) user satisfaction, and (6) net benefits.
Within the e-learning context, learning activities are conducted through web-based applications. This makes an ELS both a communication and system phenomenon that lends itself to the updated ISS model. DeLone and McLean (2003) contend that web-based application processes fit well into their updated ISS model and the six success dimensions. We adopted DeLone and McLean's (2003) ISS model as part of the theoretical framework to develop an instrument for assessing the success of ELSs. ELS success will be maximized when learners perceive the systems are beneficial to their learning. However, since the ISS model is premised on a voluntary use assumption, research has often produced conflicting findings with respect to the relationships (Chen, Gillenson and Sherrell, 2002). One potential reason for this inconsistency might be the focus on a single theory that excludes consideration of other possible determinants. To evaluate this issue, we reviewed the information systems success literature and educational research and present that review in the order of dependent constructs and then the independent constructs.
2.1 System Use
System use is an important measure of system success (Chang and Cheung, 2001; DeLone and McLean, 1992; Lucas 1978; Van der Heijden, 2004). The system use construct has also been measured as a "possible to use" and an "intend to use" construct (DeSanctis, 1982). Delone and McLean (2003) suggest that the nature, quality, and appropriateness of system use are important outcomes, and a simple measure of time spent on the system is inadequate. System use is considered a necessary condition under which systems/technologies can affect individual (learning) performance. Such research highlights the importance of use for evaluating a system in terms of its success. System use, for this research, was defined as the extent and nature of using the ELS.
System use increases when the system is perceived as profitable and decreases if the system is perceived as not profitable (Ein-Dor, Segev and Steinfield, 1981). An ELS, in the context of course delivery, is mandatory in its use. From the student perspective, an ELS is not perceived as profitable or unprofitable. Students perceive system usage in terms of whether or not the ELS adds value to their learning experience. However, if students perceive the usage as adding value to their ability to improve performance in the course, the ELS will be perceived as successful. Thus, we hypothesize:
H1. Learners with a higher level of use are likely to agree that the ELS adds value to their learning experience.
2.2 User Satisfaction
User satisfaction is a measure of the successful interaction between an information system and its users. It is also defined as the extent to which learners believe the information system meets their needs (Ives, Olson and Baroudi, 1983). If a system meets the requirements of the users, their satisfaction with the information system will be enhanced (Bharati, 2003). Conversely, if the system does not provide the necessary information, they will become dissatisfied. Research findings (Lucas, 1978; Robey, 1979) provide evidence that heavily used systems are positively correlated to user satisfaction. In stark contrast, Schewe (1976) found no significant relationship between system use and user satisfaction; likewise, Lawrence and Low (1993) did not find this relationship to be significant. Similarly, Mawhinney (1990) found no relationship between user satisfaction and system use, and (Srinivasan, 1985) noted that the relationship is not always positive. For an ELS, usage and satisfaction with the ELS will not necessarily be related due to the focus and disparities that may be inherent in an online course environment.
Delone and Mclean (1992) studied articles that address the subject of user satisfaction in their research. They concluded that user satisfaction was widely used as a measure of IS success. However, while user satisfaction has been widely used as a surrogate for systems performance and IS success, critics have questioned its general applicability because of poor instruments that have been developed to measure satisfaction (Galletta and Lederer, 1989). As with ELS, when usage is not voluntary, measures of success should be based on educational outcomes (Gill, 2006). As a measure of educational outcomes, students can indicate the ELS success by the perceived value of their learning outcome. If students are satisfied with the system and its contribution to their learning, the ELS will be perceived as successful. Therefore, we hypothesize:
H2. Learners with a higher level of satisfaction are likely to agree that the ELS adds value to their learning experience.
2.3 System Quality
System quality is the individual perception of a system's performance. From an e-learning perspective, the system quality is measured in terms of both the hardware available to the user and the various software applications designed for their intended use and needs. While the user is not aware of the network requirements of an ELS, e-learning often requires network to network communication that necessitates Internet access. High quality ELSs demonstrate the following characteristics: availability, usability, realization of user expectations, ease of learning, and response time (Halawi, McCarthy and Aronson, 2008; Guimaraes, Armstrong and Jones, 2009).
In accordance with its focus on learning, a successful ELS is generally characterized as user friendly and effective in providing useful feedback to learners. Although some attractive features that apply to other systems, such as scalability, standardization, and security have been mentioned (Sakaguchi and Frolick, 1997), the success of an ELS is judged by learning effectiveness.
In terms of the relationship between system quality and system use, some studies (Seddon and Kiew, 1994; EtezadiAmoli and Farhoomand, 1996; Teo and Wong, 1998; Wixom and Watson, 2001) confirmed a direct relationship between system quality and the individual worker's decision-making performance, job effectiveness, and quality of work. Job effectiveness is difficult to measure for an ELS due to the potential remote nature of the participants. Diverse connection quality between participants may affect the individual's ability to use the ELS. This is especially true when participation is voluntary and usage or activity statistics would become important indicators of success (Gill, 2006). However, when participation is involuntary, educational outcomes or participant perceptions of the system's ability to promote their learning should be used as a measure of success.
In terms of the relationship between system quality and user satisfaction, researchers have long employed user satisfaction with their systems as a surrogate measure for success (Rai, Lang and Welker, 2002; Guimaraes, Staples and McKeen, 2003; Guimaraes, Armstrong, and O'Neal 2006). DeLone and McLean (2003) identified system quality as an important characteristic of the user perception to use information technology. This then leads to a direct positive impact on user satisfaction. With mandatory use of the ELS, user satisfaction is more critical and a larger hurdle to overcome for the system to be considered successful. Thus, the authors propose the following two hypotheses.
H3. The system quality will positively impact the use of ELS.
H4. The system quality will positively impact learner satisfaction.
2.4 Information Quality
Information quality traditionally refers to measures of system output, namely the quality of the information that the system produces primarily in the form of reports. The desired characteristics include accuracy, precision, currency, reliability, completeness, conciseness, relevance, understandbility, meaningfulness, timeliness, comparability, and format (Swaid and Wigand 2009). The main measures used in the information quality construct for ELSs are slightly different. The focus is more on information accuracy, completeness, relevance, content needs, and timeliness. These aspects are largely controlled by various entities that include IT departments and the learning organizations responsible for assembling the ELS requirements.
Information quality captures e-learning content issues. Providing students with learning information is the basic goal of a course web site (Bhatti, Bouch and Kuchinsky, 2000). Deciding what content to place on a web site is extremely important. Lin and Lu (2000) addressed the issue of how user acceptance is affected by features and accurate information. Huizingh (2000) distinguished content from design and operationalizes both concepts by using objective and subjective measures to capture features as well as perceptions. Perkowitz and Etzioni (1999) explored the importance of updated information with the notion of adaptive web sites. Student satisfaction is also affected by the feedback received in a course (Rossin, et al., 2008), and the feedback can be viewed as an element of information quality.
Course information quality is a crucial variable that affects the success of ELSs. According to Moore (1991), course information "expresses the rigidity or flexibility of the program's educational objectives, teaching strategies, and evaluation methods" and the course information describes, "the extent to which an education program can accommodate or be responsive to each learner's individual needs." Course information has two structural elements - course objectives and course infrastructure. Course objectives are specified in the course syllabus, including but not limited to: topics to be learned, workload in completing assignments, class participation expectations in the form of online conferencing systems, and group project assignments. Course infrastructure is concerned with the overall usability of the course website and organization of course material into logical and understandable components. These informational elements, needless to say, affect the satisfaction level, system use and learning outcomes (Eom, Ashill and Wen, 2006). We theorize that the quality of course information will strongly correlate to user satisfaction and system use. Thus, we hypothesize:
H5. Information quality will positively impact ELS use.
H6. Information quality will positively impact learner satisfaction.
The mandatory usage of the ELS prompted a revaluation of prior ISS model research in which system usage has affected user satisfaction and vice versa. An empirical result of increased system usage based on increased user satisfaction must be viewed as spurious since, regardless of how satisfied (or dissatisfied) the students are with the ELS, the increased usage may be mandated by the course content. Brown, et al. (2002) pointed out that in a mandatory setting, user attitude toward the system, not their usage, is a better representation of satisfaction with the system. Hypothesizing the reverse relationship, an increase in user satisfaction cannot result in increased system usage. This is again the result of the mandatory nature of the ELS in the context of course delivery. Students can be very dissatisfied (or satisfied) with the ELS and yet, due to course requirements, still be required to maintain a certain level of system usage. This révaluation, coupled with inconsistent findings in prior research (Baroudi, Olson and Ives, 1986; Cheney and Dickson, 1982; Srinivasan, 1985; Lawrence and Low, 1993) for these relations, prompted the removal of these relationships from consideration in our model.
Based on the above-mentioned literature review and hypotheses, we propose the following research model depicted in Figure 1.
3. METHODOLOGY
3.1 Instrument Development
To develop the survey instrument, the literature was reviewed for existing items that could be used. The items used to operationalize the constructs in Figure 1 were carefully adapted and reworded from past research to relate specifically to the context of e-learning. AU items used a 7point Likert scale ranging from 1 - "strongly disagree" to 7 - "strongly agree." The instrument items for information quality, system quality, usage, and user satisfaction were adapted from prior studies using the ISS Model (McGiIl, Hobbs and Klobas, 2003; Rai, Lang and Welker, 2002). The measures for ELS success were developed by the authors.
3.2 Samples and Data Collection
The developed ELS survey was administered over a three week period in the fall semester of 2007 at a Midwestern public university. The students involved took courses across multiple educational settings. In virtually all courses, elearning was not optional for the individuals involved, and all students selected were enrolled in at least one online course. Students received an invitation to take the survey when they logged into the course during the survey period. The entire population of 2,788 students was requested to participate in the study. Ofthat population, 674 surveys were returned resulting in a 24.17 percent response rate (See Appendix A). The students came from a variety of majors. The majors representing the two largest student population groups were education at 27 percent and business at 20 percent. The rest of the respondents come from other majors, such as nursing, engineering, and fine arts. The distributions of the students by their classifications (undergraduate and graduate) showed a similar pattern to the national distribution. For example, the majority of students were undergraduate (over 80 percent), while graduates consisted of about 17 percent (National Center for Education Statistics, 2008).
More than three quarters of the students indicated that they were taking only online classes that semester. With respect to the number of courses that students were taking, 38 percent of the respondents indicated that they had one course, 37 percent of the students reported that they had two courses, 13 percent of the students had three courses, and 12 percent had four or more courses.
The ELS used in this study is referred to as the Online Instructor Suite (OIS). OIS is a bundle of six locally developed applications that comprise a course management system for online course materials. Unlike other course software packages (e.g., Blackboard, WebCT), OIS does not manage content; instructors can develop online course content using any web content editing application (e.g. FrontPage, Dreamweaver). The OIS applications (Course Manager, GradeA, UTest, Forum, Calendar, DropBox, and Chat) are similar to BlackBoard.
There are four OIS applications that are designed to ensure system quality. Course Manager is the "heart" of the software package. It controls the user database and general properties used by all other modules. Course Manager can seamlessly import rosters from another server, database or text file, and allows instructors to divide students in sections of a same class and groups, among other features. GradeA is a spreadsheet-based gradebook that is easy to use yet powerful. GradeA provides instructors with a flexible way to create gradebooks that are securely accessible by students over the web. UTest lets the instructor administer tests over the Internet easily. It includes an array of features that provide flexibility and security to online tests. Tests can be taken using a standard browser or using the UTest Secure Browser, and grades can be automatically sent to GradeA for publishing in the gradebook. DropBox is an upload/download area where students can store files and submit assignments for grading. The instructor interface retrieves files from the server for viewing, changing and grading.
The rest of the OIS applications are designed to improve information quality. Forum is an asynchronous discussion space that can be used to increase interactivity among students and the instructor. A class forum is divided into discussions and topics, and instructors have full control over the entire area through an interface that allows them to read, create, reply to and grade students' posts. Calendar tracks important dates, announcements, test times, and other information that can be shared among all members of a group or class. Anything instructors enter in Calendar will appear in each student's personal calendar page. Students can add or remove items using a Web interface. Chat is a synchronous communication tool for OIS classes. This allows instructors and students to communicate with each other in real-time.
3.3 Statistical Procedure
A confirmatory factor analysis (CFA) approach to the data analysis was taken using the LISREL software package version 8.80. A two-step approach was taken for validating the research model. The initial step was construction of the measurement model in which the hypothesized scale items were loaded onto the independent constructs of System Quality, Information Quality, System Use, User Satisfaction and ELS Success. Factor loadings were checked against the guidelines provided by Comrey and Lee (1992). Modification indices were checked for cross loading and correlation of scale items. Four fit indices were used to assess the goodness of fit for the measurement model. The first three indices, Normed Fit Index (NFI), Comparative Fit Index (CFI), and Non-Normed Fit Index (NNFI), were expected to exceed 0.9 to indicate a good fit. The last index, the Root Mean Square Error of Approximation (RMSEA), should be less than .08 for a model of 'near fit'. An analysis of a more stringent standard presented by Hu and Bentler (1999) indicated that the NNFI and CFI should exceed a .95 criteria and the RMSEA should not exceed .06 to be considered a 'close fit'. Three of the four indices proposed by Hu and Bentler (1999) should meet these standards to indicate a close model fit. Additionally, SPSS computed the frequencies, means, standard deviation, reliability coefficients and Cronbach's Alphas for each construct (See Appendix B for means and standard deviations.)
A structural model was then developed from the resulting measurement model constructs. LISREL version 8.80 was again used to test the structural model and validate the proposed hypotheses. Model fit was assessed using the same criteria as the measurement model. Acceptance of the hypotheses was contingent upon achieving a .05 level of significance on the appropriate path coefficients. Details of the results are presented in the next section.
4. DATA ANALYSIS
4.1 Measure of Constructs Reliability and Validity
SPSS was initially used, with final validation conducted using LISREL, to implement the following steps for measuring the reliability and validity of the model. The itemtotal correlation was computed for each item using items belonging only to the same construct. The minimum acceptable value to keep a scale item with the latent construct is 0.5 (Hair, et al., 2006). More stringent reliability coefficients of 0.70 or higher have also been recommended (Nunnally 1978). In addition to the item-total correlations, Cronbach's alphas were computed for each construct. The result of the SPSS analysis was the identification, and subsequent removal, of two items (SU3 and USl) that did not load properly on their intended constructs. The two scale items were sequentially removed with noticeable improvements to Cronbach's Alpha and the corrected itemtotal correlations. The final model showed that all items demonstrated corrected item-total correlations above the 0.65 level with Cronbach's alpha above 0.80 for all constructs.
Twenty items were analyzed for construct validation and reliability as described above. A CFA using SEM was performed on the final measurement model. AU scale items were loaded on their indicated latent construct. AU coefficients were higher than the more stringent standard of .70 with the exception of SQl (0.67). The goodness-of-fit indices reviewed earlier were used to assess the validity of the measurement model. The NFI, CFI and NNFI all exceeded the .95 criteria for the model to be considered a close fit. The RMSEA was in the range of a near fit for the model. The results of the assessment of the reliability and validity of measures are reported in Table 1. The model fit statistics from the SEM measurement model analysis are reported in Table 2.
4.2 Structural Model
The structural model testing the research model and hypotheses met the more stringent standard of 0.95 for the NFI, NNFI, and CFI fit indices to indicate a model of close fit (Table 2). The RMSEA resulted in a value of 0.07, which indicated a model of near fit. With three of the four fit indices meeting the standard for close fit, the model is deemed an adequate test for the hypotheses.
RESULTS
All hypotheses presented in the research model, Hl through H6, achieved a significance level of 0.01. The standardized path coefficients and the hypotheses status are represented in Figure 2. The path coefficient for H2 (user satisfaction to system success) had the strongest effect on system success. It was the most significant factor with a tvalue of 31.25 and standardized path coefficient of 0.94. The explained variance for system success was excellent (R2 = 0.96).
Both system quality and information quality had a significant positive effect on user satisfaction and system use. While system quality had a slightly stronger effect on user satisfaction (with path coefficient of 0.48) compared to information quality (with path coefficient of 0.45), the nearly balanced path coefficients coupled with the high level of explained variance for user satisfaction (R2 = 0.83) indicate that both information and system quality were highly correlated to user satisfaction.
With respect to system usage, information quality (0.3 1) had a slightly stronger effect on system use than system quality (0.29). However, the explained variance of system usage (0.34) was much lower than that of user satisfaction. This indicated that there may be other factors or variables required in the explanation of system usage.
6. DISCUSSION AND CONCLUSION
This paper extended the ISS model to measure ELS success. Both system quality and information quality indirectly impact ELS success, predominantly through user satisfaction. To increase student satisfaction and ultimately affect ELS success, it is important for instructors to make available an ELS that provides students with needed, relevant, up-to-date information through a user friendly and interactive system. Given the high degree of explanation for user satisfaction (R2 = 0.83), the ISS model can be said to be an effective initial model for evaluating an ELS environment. Even though the surveyed population was using a mandated ELS, high system quality and information quality of an ELS are necessary for high levels of user satisfaction. This strongly relates to the ELS success.
While it is gratifying to recognize that user satisfaction strongly relates to ELS success, the relatively low explanation of system use (R^sup 2^ =0.34) coupled with the low path coefficient (0.07), although positive and significant, indicates that a further extension of the model may be necessary. There are some potentially confounding issues that may further explain some of the less significant findings of the research model in general and system usage specifically. With respect to the system quality impact on explaining system usage, the focus of the survey was on the ELS that all students were using for their online courses. While this provided a common point of reference for completing the surveys, multiple hardware, software and network issues outside the control of the ELS may also have influenced student usage. Specific issues influencing student use that would affect system quality but were not evaluated by the survey include, but are not limited to browser selection, network connection points (dial-up versus campus network versus cable), and operating system on the computer used. These system quality issues could create a wide range of time for students to wait before they received the same information necessary to be successful in the course and could therefore impact their perception of ELS success. In addition, given potential time constraints due to workload, students could potentially be using the system the same amount of time and yet receive different levels of quality due to specific issues that are outside the control of the ELS and the instructor.
The approach taken is an IS perspective, but the use of the Community of Inquiry (COI) model could also be useful in analyzing the results. The COI model (Garrison, Anderson and Archer, 2000) is a three component model that includes a cognitive, social and teaching presence that results in the final educational experience. In the ISS model perspective, the final educational experience can be viewed as user satisfaction or even system success. The measurement impact on the educational experience is purely characteristic of the system and information quality of the ELS. The COI model urges a more integrative role of both the student and teacher through a balance in each of the three presences of computer mediated communication. An increased understanding of the educational experience may occur through an understanding of how students construct meaning through sustained communication (Cognitive presence), project personal characteristics (Social presence), and realize personal meaning (Teaching presence). The COI model may help educators to understand the environment created by the ELS that facilitates the online learning experience.
In addition to the technology issues and COI model perspective, another potential aspect that may affect system usage is the skills the students bring to the e-learning environment. First-time students in an e-learning course may be substantially different in their system usage than those students who are more experienced and adept at minimizing their system usage while maximizing the learning from that usage. These skills may also extend to the self-efficacy of the students in their ability to organize their work and address issues with the course interface, namely their personal laptop, university computer or home desktop.
7. LIMITATIONS AND FUTURE RESEARCH
This study was exploratory in nature and thus had a few limitations that should be recognized. The use of selfreported scales to measure the study variables raises the possibility of common method variance. Student subjects are often viewed as a limitation but are ideal for this environment. The inclusion of both graduate and undergraduate students allows a possible extension for the comparison of a more experienced versus inexperienced study. This type of study could also be cross-referenced by a comparison of a subject group taking only online courses with a subject group taking only one online course. Additionally, the moderating factors of gender and age are other avenues of investigation for further insights into ELS success.
A goal of continuing research would be an exploration of how the ISS model would be supplemented in order to more accurately reflect the E-learning environment. Potential models for exploring ELS success would include a renaming of the ISS dimensions to better reflect the ELS along with the additions of human factors, technology issues, and the COI model perspectives that can moderate or mediate the current system-specific model. An instance of this modification would be changing the information quality dimension in the ISS model to course content quality. As indicated above, the information quality of the ELS is organized around two structural elements - course objectives and course infrastructure. Student expectations for course success are highly dependent on the instructor's ability to clearly and concisely communicate the course objectives. The course infrastructure depends on the options, layout and consistency of presentation provided by the ELS, in this case the OIS, and is not dependent on the instructor. The course content quality may be affected by both the course objectives, which are instructor controlled, and the course infrastructure, as presented by the OIS. The system quality would then need to be differentiated from the course infrastructure dimension. The system quality dimension may be relabeled as network quality with measurements that more fully explore how the users of the ELS modify their behavior to more efficiently access the ELS. In the case of our study, students may have the option to access the ELS from home (via dial-up) with a degraded network quality or from a campus network where access and efficiency are much improved. This separation could be explored since the OIS is a software application, and therefore functions consistently, but may appear to function differently due to the network used by the students to access the OIS.
Following the improvements suggested through the review of ISS model research (DeLone and McLean, 2003), two additional constructs of service quality and net benefits can be explored. Service quality in the context of an ELS model and in this particular study would encompass both the responsiveness of the instructor and the technology support provided by the university hosting the OIS. The inability of students to access the OIS from outside the university network may affect the perceived service quality due to their choice of connection and not the actual service quality of technical support or the responsiveness of the instructor. Finally, the net benefits of using the system can be further expounded upon by incorporating the different modules used in the OIS and measuring the quality of their contribution to the students' learning experience.
References
8. REFERENCES
Allen, I E, and J Seaman. Staving the Course: Online Eductiaon in the United States. 2008. Needham, MA: Sloan Consorium, 2008.
Baroudi, J J, M H Olson, and B Ives. "An empirical study of the impact of user involvement on system usage and information satisfaction." Communications of the ACM 29 (1986): 232-238.
Beard, Jon W, and Tim O Peterson. "Coming to Grips with the Management of Information: A Classroom Exercise." Journal of Information Systems Education 14, no. 1 (2003): 15-30.
Bharati, P. "People and information matter: Task support satisfaction from the other side." The Journal of Computer Information Systems 43. no. 2 (2003): 93-103.
Bhatti, N, A Bouch, and A Kuchinsky. "Integrating userperceived quality into web server design." Computer Networks 33. no. 1-6(2000): 1-16.
Brown, S A, A P Massey, M M Montoya- Weiss, and J R Burkman. "Do I really Have To? User Acceptance of Mandated Technology." European Journal of Information Systems 1 1, no. 4 (December 2002): 283.
Chang, M K, and W Cheung. "Determinants of the intention to use Internt/WWW at work: A confirmatory study." Information & Management 39. no. 1 (2001): 1-14.
Chen, L D, M L Gillenson, and D L Sherrell. "Enticing online consumers: An extended technology acceptance perspective." Information & Management 39, no. 8 (2002).
Cheney, P H, and W Dickson. "Organizational characteristics and information systems: an exploratory investigation." Academy of Management Journal 25, no. 1 (1982): 170-184."
Comrey, A L, and H B Lee. A First Course in Factor Analysis. 2nd. Hillsdale, NJ: Lawrence Erlbaum Associates, 1992.
Davis, Charles H, and Jana Comeau. "Enterprise Integration in Business Education: Design and Outcomes of a Capstone ERP-based Undergraduate e-Business Management Course." Journal of Information Systems Education 15, no. 3 (2004): 287-299.
DeLone, W H, and E R McLean. "Information system success: The quest for the dependent variable." Information System Research 3. no. 1 (1992): 60-90.
DeLone, W H, and E R McLean. "The DeLone and McLean model of information systems success: A ten-year update." Journal of Management Information Systems 19, no. 4 (2003): 9-30.
DeSanctis, G. "An examination of an expectancy theory model of decision support systems use." Proceedings of the Third International Conference of Information Systems. Ann Arbor, MI, 1982. 121-135.
Ein-Dor, P, E Segev, and A Steinfield. "Use of Mangement Information Systems: An Empirical Study." International Conference on Information Systems. 1981.39-42.
Eom, S, N Ashill, and J Wen. "The determinants of students' perceived learning outcomes and satisfaction in university online education: An empirical investigation." Decision Sciences Journal of Innovative Education 4, no. 2 (2006): 102-115.
Etezadi-Amoli, J, and A F Farhoomand. "A structural model of end user computing satisfaction and user performance." Information & Management 30, no. 2 (1996): 65-73.
Galletta, D F, and A L Lederer. "Some cautions on the measurement of user information satisfaction." Decision Sciences 20 (1989): 419-38.
Garrison, D R, T Anderson, and W Archer. "Critical inquiry in a text-based enviornment: Computer conferencing in higher education." The Internet and Higher Education 2, no. 2-3 (2000): 87-105.
Gill, Grandon. "Asynchronous Discussion Groups: A Usebased Taxonomy with Examples." Journal of Information Systems Education 17, no. 4 (2006): 373-383.
Guimaraes, T, C P Armstrong, and B M Jones. "A New Approach to Measuring Infromation Systems Quality." The Quality Management Journal 16, no. 1 (2009): 42-55.
Guimaraes, T, C P Armstrong, and Q O'Neal. "Empirically testing some important factors for expert systems quality." Quality Management Journal 13, no. 3 (2006): 28-38.
Guimaraes, T, S Staples, and J McKeen. Important human factors for systems development success: A user focus. Edited by Conrad Shayo. Vol. In Strategies for Managing IS Personnel. Hershey, PA: Idea Group Publishing, 2003.
Hair, J, W Black, B Babin, R Anderson, and R Tatham. Multivariate Data Analysis. New Jersey: Prentice Hall, 2006.
Halawi, L A, R V McCarthy, and J E Aronson. "An Empirical Investigation of Knowledge Management Systems." The Journal of Computer Information Systems 48, no. 2 (2008): 121-136.
Hu, L, and P M Bentler. "Cutroff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives." Structural Equation Modeling 6 (1999): 1-55.
Huizingh, E. "The content and design of web sites: An empirical study." Information & Management 37, no. 3 (2000): 123-134.
Ives, B, M Olson, and J Baroudi. "The Measurement of User Satisfaction." Communications of the ACM 26, no. 10 (1983): 785-793.
Lawrence, M, and G Low. "Exploring individual user satisfaction within user-led development." MIS Quarterly. June 1993: 195-208.
Lin, C J, and H Lu. "Towards an understanding of the behavioral intention to use a web site." International Journal of Information Management 20, no. 3 (2000): 19720$.
Lucas, H C. "Empirical model for a descriptive model of implementation." MIS Quarterly 2. no. 2 (1978): 27-41.
Mawhinney, C H. "A study of computer use b knowledge workers: user satisfaction versus system use." Proceedings Annual Meeting of Decisions Sciences Institue. San Diego, CA, 1990. 954-956.
McGill, T, V Hobbs, and J Klobas. "User-developed applications and information systems success: A test of DeLone and McLean's model." Information Resources Management Journal 16, no. 1 (2003): 24-45.
Moore, M G. "Editorial: Distance Education Theory." The American Journal of Distance Education 5, no. 3 (1991): 1-6.
Nunnally, J. Psychometric Theory. New York: McGrawHill, 1978.
Perkowitz, M, and O Etzioni. "Towards adaptive web sites: conceptual framework and case study." Computer Networks31.no. 11/16(1999): 1245-1258.
Rai, A, S Lang, and R Welker. "Assessing the validity of IS successmodels: An empirical test and theoretical analysis." Information Systems Research 13, no. 1 (2002): 50-69.
Robey, D. "User attitudes and management information system use." Academy of Management Journal 22, no. 3 (1979): 527-538.
Rossin, Don, Young K Ro, Barbara D Klein, and Yi Maggie Guo. "The Effects of Flow on Learning Outcomes in an Online Information Management Course." Journal of Information Systems Education 20. no. 1 (2008): 87-98.
Sakaguchi, T, and M Frolick. "A review of datawarehousing literature." Journal of Data Warehousing 2, no. 1 (1997): 34-54.
Schewe, C D. "The management information system user: an exploratory behavioral analysis." Academy of Management Journal 19. no. 4 (1976): 577-590.
Seddon, P B, and M Y Kiew. "A partial test and development of the DeLone and MclLean model of IS success." Proceedings of the International Conference on Information Systems. Atlanta: Association for Information Systems, 1994.99-110.
Srinivasan, A. "Alternative measure of systems effectiveness: Association and implications." MIS Quarterly 9, no. 3 (1985): 243-253.
Swaid, S I, and R T Wigand. "Measuring the quality of Eservice: Scale develoment and inital validation." Journal of Electronic Commerce Research 10, no. 1 (2009): 13-29.
Teo, TSH, and P K Wong. "An empirical study of the performance impact of computerization in the retail industry." Omega-The International Journal of Management Science 26. no. 5 (1998): 611-621.
Van der Heijden, H. "Hedonic information systems." MIS Quarterly 25, no. 4 (2004): 695-704.
Wixom, B H, and H J Watson. "An empirical investigation of the factors affecting data warehousing success." MIS Quarterly 25. no. 1 (2001): 17-41.
AuthorAffiliation
Ronald D. Freeze
Khaled A. Alshare
Peggy L. Lane
H. Joseph Wen
Accounting and Information Systems Department
Emporia State University
Emporia, Kansas 66801-5087 USA rfreeze@emporia.edu, kalshare@emporia.edu plane@emporia.edu, hwen@emporia.edu
AuthorAffiliation
AUTHOR BIOGRAPHIES
Ronald D. Freeze is an Assistant Professor of Information Systems at Emporia State University in Kansas. He received his Ph.D. from Arizona State University. His current research interests include Knowledge Management, Capability assessment, ERP Implementation and SEM. His publications have appeared in the Journal of Management Information Systems, Journal of Computer Information Systems, Journal of Knowledge Management and International Journal of Knowledge Management.
Khaled A. Alshare is a Professor of Information Systems at Emporia State University. He received his Ph.D. from the University of Texas at Arlington. His current research interests include technology acceptance, information security, cross-cultural studies in information systems, distance education, and data envelopment analysis (DEA). His publications have appeared in the JCIS, Journal of Internet Commerce, International Journal of Quality & Reliability Management, and other journals.
Peggy L. Lane is an Associate Professor of Information Systems at Emporia State University in Kansas. She received her Ph.D. from the University of Arkansas. Her current research interests include ERP implementation, e-learning, group decision making, and ethics. Her publications have appeared in the International Journal of Education Research, International Journal of Management and Enterprise, Communications of the Association for Information Systems, The Journal of Computer Information Systems, Journal of Information Systems Education, Journal of End User Computing, and Communications of the ACM.
H. Joseph Wen is Dean of the School of Business and Jones Distinguished Professor at Emporia State University. He holds a PhD from Virginia Commonwealth University. He has published over 100 articles in academic refereed journals, book chapters, encyclopedias, and national conference proceedings. He has received over six million dollars in research grants from various state and federal funding sources. His areas of expertise are Internet research, electronic commerce, transportation information systems, and software development. He has also worked as a senior developer and project manager for various software development contracts since 1988.
A COMPARISON OF ERP-SUCCESS MEASUREMENT APPROACHES
Kronbichler, Stephan A; Ostermann, Herwig; Staudinger, Roland. Journal of Information Systems and Technology Management : JISTEM7.2 (2010): 281-310. Abstract ERP projects are complex purposes which influence main internal and external operations of companies. There are different research approaches which try to develop models for IS / ERP success measurement or IT-success measurement in general. Each model has its own area of application and sometimes a specific measurement approach based, for instance, on different systems or different stakeholders involved. This research paper shows some of the most important models developed in the literature and an overview of the different approaches of the models. An analysis which shows the strengths, weaknesses and the cases in which the specific model could be used is made. [PUBLICATION ABSTRACT]
Full Text * Translate Full text * Turn on search term navigation
Headnote
ABSTRACT
ERP projects are complex purposes which influence main internal and external operations of companies. There are different research approaches which try to develop models for IS / ERP success measurement or IT-success measurement in general. Each model has its own area of application and sometimes a specific measurement approach based, for instance, on different systems or different stakeholders involved. This research paper shows some of the most important models developed in the literature and an overview of the different approaches of the models. An analysis which shows the strengths, weaknesses and the cases in which the specific model could be used is made.
Keywords: ERP, success, measurement, information system, review
1.INTRODUCTION
An ERP system is an integrated, configurable and customizable information system which plans and manages all the resources in the enterprise, streamlines and incorporates the business processes within and across the functional or technical departments in the organization (She and Thuraisingham, 2007). ERP systems consist of different modules which represent different functional areas and they offer integration across the entire business, including Human Resources, Accounting, Manufacturing, Materials Management, Sales and Distribution and all other areas which are required in different branches (Davenport, 1998). ERP supports a process-oriented view of the enterprise and provides standardised business processes and real-time financial and production information for the management (Nah and Delgado 2006; May, 2003).
There are not only benefits that can be achieved from an ERP system implementation; there is already evidence of failure in projects related to ERP implementations which are found in the literature (Davenport, 1998). Competitively and technically, implementing ERP is a must do, but economically there are costs which are difficult to justify and it is difficult to implement a long lasting business advantage (Willcocks and Sykes, 2006). An investment in ERP represents a significant commitment of resources and it has a dramatic effect on all operational aspects of a business (Nicolau, 2004).
Business needs are changing rapidly and new requirements are often influencing business processes. Because of the new business needs which are coming up the company which wants to hold up or achieve competitive advantage has to react immediately and the quality of the adopted or implemented solution is often poor (Kronbichler et al., 2009). According to different studies, a lot of ERP projects do not reach the expected results or lead to the failure of the project. The study of Cooke et al. (2001), for example, listed 117 companies which implemented ERP and had the following results: 25 percent of all the projects were out of budget, 20 percent of the projects were abruptly discontinued for various reasons, and 40 percent of the remaining 55 percent stated that they did not reach the defined goals within one year after the official project ended. Although some of these problems arise from technical aspects, the majority of these problems result from management, social and organisational issues within the companies. For a successful ERP implementation, these issues must be considered because there are a lot of challenges for organisations during ERP projects. Businesses are expected to change their business processes to fit the standardised business processes from the ERP-solution selected and, as a result, to fully benefit from the ERP (Nah et al., 2003). Project management often has a technical focus and nontechnical issues are ignored. The project management only monitors if the project is in time, in scope and in budget.
A lot of research in critical success factors (CSF) in ERP-implementations or ERPprojects has been done (Kronbichler et al., 2009, Esteves-Sousa and Pastor-Collado, 2000, Holland and Light, 1999, Nah and Lau, 2001, Sumner, 1999) but there are a few publications which show the different approaches of ERP-success measurement and the advantages and disadvantages of these investigations. A review of different success measurement approaches is necessary to oppose the CSF and the success of the running system for further research. This ongoing research paper analyzes the different aspects and possibilities of ERP and information system success measurement and it concentrates on the post implementation phase, summarizes the different approaches and shows how relevant these approaches are for the measurement of ERP-success. It can be used as a basis for decision support during the selection of a suitable success measurement model for a specific research question in the field of ERP-success measurement.
The purpose of this research is to identify possibilities of success measurement and to show which of these possibilities turned out to be of importance in order to provide indispensable information for further ERP-research.
The main steps for the research study are:
* Literature review related to success measurement in ERP / Information system projects
* Investigate the relevance for the measurement of ERP success and list the main models identified
* Point out the strategic importance of success measurement
* Build an overview which assists practitioners and researchers in selecting a success measurement model
Through an extensive review of the existing publications in the field of success measurement different models were identified. After finishing the literature review, the models were examined. In a third step, the relevance of the models for different use cases is shown.
2.ERP SUCCESS MEASUREMENT
Research Methodology
The search term for appreciable publications was "success" and "measure" in combination with "information system" or "ERP". The second search term for important publications for this research paper was "enterprise systems success" which is often used in the literature. Every result was analysed through a review of the abstract. The findings of the first step of the literature review were analysed and further publications in the subject of the measurement of success within information systems and ERP-systems were found because of the references of the analysed publications.
Later, the findings of the publications were analysed and the current state of the field was built through an investigation of the different success measurement models / possibilities which were found in the literature. At the end of the paper different approaches were made which ensure a support for the selection of an appropriate success measurement model.
Success and Quality
Success is a dependent variable of the reached quality level. If the quality of the ERP-system running is poor, the success will be also poor in most cases.
According to the ISO 9000 2005 standard, the quality of something can be determined by comparing a set of inherent characteristics with a set of requirements. If those inherent characteristics meet all the defined requirements, high or excellent quality is achieved. If those characteristics do not meet all defined requirements, a low or poor level of quality is achieved. By linking quality to requirements, ISO 9000 argues that the quality of something cannot be established in a vacuum and quality is always relative to a set of requirements (Praxiom Research Group Limited, 1997). The success or failure of information systems is relative too and must be measured in relation to the expectations of the organisation that implements the system (Curlee and Tonn, 1987).
Although success is complex and difficult to measure, researchers are making efforts in doing so. Most of the practical measurements focus on delivering a functional IS product within certain economic and temporal constraints. A system must first be accepted to be used and that should increase the probability of system success (Behrens et al., 2005). A lot of research has been focused on defining factors and measures that should capture the characteristics of an information system but such factors might not capture the intangible or indirect value generated by the according system (Ding and Straub, 2007). It is reasonably easy to evaluate tangible implementation costs, e.g. software license, hardware, consultancy, and training, but other intangible costs are much more difficult to measure and evaluate (e.g. productivity dip) (Hedman and Borell, 2005).
3. SUCCESS MEASUREMENT APPROACHES
In the following section of this paper some models for success measurement are listed and explained. It is an overview of the existing approaches without an extensive explanation of each framework. The different measurements which are used at the most detailed level of the success measurement of the models are not enumerated, because this is not necessary for the understanding and the purpose of the models themselves.
The DeLone McLean I/S Success Model
The most cited model for success measurement in the field of information systems is the DeLone and McLean (DeLone and McLean 1992, DeLone and McLean, 2002, DeLone and McLean, 2003) model which moved to a user centred approach when trying to judge overall IS success. The DeLone and McLean model consists of six interdependent measurements of success. System quality, information quality, use, user satisfaction, individual impact and organisational impact are the main measurement dimensions.
DeLone and McLean had different purposes when they were writing their publication in 1992. They wanted to organize and summarize management information system research related to defining the dependent variable, to measuring progress on defining the dependent variable and to improving the information systems research practice (DeLone and McLean, 1992). The methodology they were using to build the model consisted of the following main steps:
* Literature review
* Collection of IS Articles from 1981 to 1988
* Build a framework / model for organizing success measures
* Definition of empirical measures which are grouped into six success categories
This model provided a comprehensive taxonomy on IS success and identified over 100 IS success measures during the analysis of the collected articles (Elpez and Fink, 2006).
In 2002 DeLone and McLean published a reformulated IS-success model which offered the addition of service quality and the collapsing of individual impact and organizational impact on net benefits (DeLone and McLean, 2002, Ding and Straub, 2007). The change of the model was based on alterations in the role and management of information systems and on research contributions since publishing their original paper. The "use" was replaced by "Intention to use", which is an attitude, whereas "use" is behaviour; this new part of the model may resolve some of the process versus causal concerns that Seddon (1997) raised . But attitudes, and their links with behaviour are difficult to measure and many researchers may choose to keep "use" but with a more extensive understanding of it. The new model shows that "use" must precede "user satisfaction" in a process sense, but positive experience with "use" will lead to greater "user satisfaction" in a causal sense. That's the reason why increased "user satisfaction" will lead to increased "intention to use," and, thus, "use." As a result, "net benefits" will occur. The lack of positive benefits can lead to decreased use and possible discontinuance of the system or of the whole IS department itself (e.g. outsourcing) (DeLone and McLean, 2003). The new construct "Net benefits" is the collapsing of Individual and Organisational Impact which were mentioned in the original model of 1992. This was necessary to broaden the impact of the information system also depending on the context in which the model was used (DeLone and McLean, 2003, Wu and Wang, 2006).
The arrows between the 6 Dimensions of the DeLone & McLean model show the relations and interdependencies between the dimensions. System quality, for example, influences the Intention to use, Intention to use influences the user satisfaction and, as a result, the net benefits occur. If the system quality is poor, the net benefits are poor too. The 6 dimensions are the dimensions DeLone and McLean identified during their research when they were investigating the dependencies of information systems success.
The Gable et al. model
Gable et al. (2003) made an exploratory inventory survey which was used for model building. They built a model which was used for enterprise system success measurement approaches - the "A Priori Model". The "A Priori Model" was using five constructs and forty-two sub-constructs. The aim of the test of the "A Priori model" originally showed that the ERP success depends on the size of the organisation (Myers et al., 1997).
The Delone and McLean constructs and measures were used as the basis of the starting ES success model and were synthesized with the associated measures from Sedera et al. (2003). The constructs/ measures of the Delone and McLean model provided a holistic view across the different roles within the organization and provided a detailed categorization of success dimensions. A main difference to the DeLone & Mc Lean model is that the construct use was omitted from the a priori model. The mapping exercise of the 2 different measures facilitated identification and inclusion of other, new measures related to ES. Therefore some measures were considered unsuitable and were omitted from the a priori model. The build model was tested for its validity and the validity of four model dimensions and their convergence in a single higher-order phenomenon, enterprise system success, were shown. The revised model is the result of Gable et al. research. It has four quadrants, individual impact, organisational impact, information quality and system quality which are related dimensions of the multidimensional phenomenon: enterprise system success (Gable et al., 2003).
The main differences to the DeLone and McLean model are:
* it is a measurement model and does not purport a causal/process model of success
* the construct use is omitted
* satisfaction is treated as an overall measure of success (no explicit dimension for user satisfaction)
* new / additional measures were added to reflect the contemporary IS context and organizational characteristics (Gable et al., 2003)
Individual impact means the impact of the system on the individual working with the system, e.g., decision effectiveness or individual productivity. Organisational impact measures the impact of the system on the organisation, e.g,. organisational costs or staff requirements. System quality consists of measures like ease of use, flexibility or data accuracy. Information quality on the other hand consists, e.g., of timeliness, relevance or importance of the information worked up.
The Gable et al. model can be used for measures at a certain point of time, a snapshot of the organisation's experience. The impact dimensions are an assessment of benefits which are caused by the system (in a negative or positive way). The quality dimensions of the model show the future potential. Together, the four dimensions reflect a complete view of the enterprise system and the success reached (Gable et al. 2003).
The extended ERP Systems Success measurement model
(Ifinedo, 2006) extended the dimensions of success proposed by Gable et al. (2003) because of the growing body of knowledge in this research field. The author found through literature review and interviews that ERP systems success measurement models might be limited because 2 important dimensions may not be considered. One new dimension which was added to the model was the Vendor/Consultant Quality because the result of empirical evidence revealed that firms tend to associate the role and quality of the providers of their software with its overall success of the organization (Ifinedo, 2005, Ifinedo and Nahar, 2006). ERP-projects are very complex and take a lot of time, that's why competent partners are needed. A know-how transfer and mixture between internal and external staff is necessary to manage it. Vendor / consultant quality measures the influence of external quality on the ERP-systems success. Vendor and consultant are grouped together because they represent an external source in the model. Ifinedo (2006) argued that the client will be in a better position to use the acquired software efficiently and effectively in achieving organizational goals when an arrangement between externals and the implementing firm exists. When this is the case, success with the software increases. Typical measures for this dimension are technical support provided, relationship with the organisation or credibility and trustworthiness.
The author considered the research of Myers et al. (1996) who argued that any IS success model should incorporate workgroup impact. Workgroup impact, the second added dimension, in the notion of Ifinedo (2006) means that "workgroup" encompasses sub-units and/or functional departments of an organisation. According to Ifinedo (2006) workgroups like teams or groups can contribute a lot to the success of an ERP-project. The author refers to CSF research, which showed that workgroup impact is one of the most important success factors. Typical measures for this dimension are improvement of interdepartmental communication or organizational-wide communication.
Laterm Ifinedo (2006) made an attempt at replicating, validating and extending the model. An additional finding was that System Quality and Organizational Impact were found to be perhaps the two most important dimensions for ERP systems success.
The main differences to the Gable et al. model are the 2 additional dimensions vendor / consultant quality and workgroup impact. The Ifinedo (2006) model has nearly the same area of application as the Gable et al. (2003) model, but it provides a framework that allows to collect more comprehensive data influencing the ERP-systems success.
Markus & Tanis
Markus and Tanis (2000) tried to define success based on their observations of enterprise systems. According to the authors there are different phases characterized by key players, typical activities, characteristic problems, appropriate performance metrics and a range of possible outcomes. Each experience made with ERP is unique, and experiences may differ from company to company and from the specific point of view (Markus and Tanis, 2000). Markus and Tanis (2000) defined an enterprise system experience cycle with different phases and for each phase the publication includes a description, key actors, typical activities, common errors or problems, typical performance metrics and possible outcomes. Figure 4: Adopted Enterprise System Experience Cycle of Markus & Tanis (2000) shows how the success measurement model of Markus and Tanis works.
The success measurement model of Markus & Tanis (2000) can be used for multiple success measurement approaches at different stages of an ERP-project. It provides the possibility to make plans and take actions if a result is not as good as expected and to get better results in the next phase because every outcome of a phase is influencing the next phase. At the end of the research of Markus and Tanis (2000) there is a table which shows "A Process Theory of Enterprise System Success" with the phases name, the successful outcome, necessary conditions, probabilistic processes and a recipe for success. The difference to other models is that this model provides a theoretical framework for analyzing retrospectively and prospectively, the business value of enterprise systems.
Ex-ante evaluation of ERP software
The main focus of the research is the ex-ante evaluation and the selection process of ERP systems (Stefanou, 2001). The difference to the other models which are part of this research is that all of the models (except the Markus and Tanis (2000) model) concentrate on an ex-post evaluation which concentrates on an evaluation of an existing system. According to Stefanou (2001) an ex-ante evaluation is necessary because of the fact that selecting an ERP is a long time commitment which is very costly too. The model of Stefanou (2001) is divided into 4 main phases which are demonstrated in Figure 5: Major phases of ERP-lifecycle (Stefanou, 2001).
The first phase (Clarification of the business vision) considers the business vision as a starting point for ERP initiation/acquisition. Investments in ERP are strategic actions which have consequences for the company. For the selection of an appropriate system, a clear business vision is necessary because it has to be clear which aims the implementation of the new system should achieve and if the evaluated system enables it. The first part of the second phase (Comparing needs vs. capabilities) consists of the detailed examination and definition of business needs and of the company's capabilities and various constraints according to the ERPs functionality. That means that the decision, if the ERP can support the business processes of the company or if an adoption would be necessary, has to be made. Therefore, a list of the required technological changes for a successful implementation must be made. The constraints which are limiting the possibilities are classified into 5 categories: Technical, organisational, human, financial and time constraints. The second part of the second phase considers the selection of needed ERP modules and additional software which is necessary to handle the daily business. Additionally, an ERP product, vendor and support services evaluation should be made in this phase too. The implementing company has to decide if an all-in-one solution is a better choice than best-of-breed solutions. In the third phase of the Stefanou's (2001) ex-ante model costs and benefits arising from the ERP implementation project are estimated. The costs of consulting fees and the user training are only examples for some points the evaluators shouldn't ignore in this phase. The last phase of the suggested model is "operation, maintenance and evolution" which means that changes in the market and new business channels cause in updates or new releases of the implemented software. That means that after finishing the implementation project there is a continuous check necessary if the solution fulfils the needs of the business. This phase includes estimation of the costs and benefits which will arise in the future from operating, maintaining and extending the ERP system with additional functionality (Stefanou, 2001). The proposed framework of Stefanou (2001) shows how companies can evaluate a planned ERP-implementation project ex-ante. That means that it provides instruments to evaluate the future outcome and helps the management to decide the best way for the company. The framework guides the evaluator through all the important stages which must be considered when evaluating ERP systems because a simple evaluation based on ease of use, usefulness and involvement of end users, as it has been suggested by Montazemi et al. (1996) is not longer valid for complex systems like ERP.
Balanced Scorecard Approaches
The management of ERP Software consists of two main tasks-the implementation and the use of this comprehensive software afterwards (Rosemann and Wiese, 1999). The intention of the Balanced Scorecard is the supplementation of traditional financial measures with three additional perspectives - the customer perspective, the internal business process perspective and the learning and growth perspective (Kaplan and Norton, 1997). The Balanced Scorecard (BSC) can be used for evaluation of these tasks and afterwards for the strategic planning of the future development of the system based on the evaluation results (Rosemann and Wiese, 1999, Martinsons et al., 1999). There are two publications which investigate the usage of a BSC-approach for IS evaluation, one of Rosemann and Wiese (1999) and another one of Martinsons et al. (1999). In this research the BSC approach of Rosemann and Wiese (1999) is demonstrated because the BSC approaches are very similar and they should demonstrate how a BSC can be used for ERP success measurement. Martinson's et al. (1999) BSC is not ERP specific; it's an IS BSC in general.
Rosemann and Wiese (1999) provided two different BSC approaches in their research. The first BSC approach is measuring the project performance and in addition to the classical perspectives (financial/cost, customer, internal process and innovation and learning), a fifth perspective, which represents the typical project management tasks, the project perspective was added to this BSC. The second BSC approach of Rosemann and Wiese, the operational BSC, which is more relevant for this research, is measuring the business performance and can be used for (continuous) controlling of the ERP software.
The operational BSC is shown in Figure 6: The ERP operation Balanced Scorecard (Rosemann and Wiese, 1999)
For the purpose of using the Balanced Scorecard to control the running of ERP software, the four standard perspectives of the original model have to be adjusted to the specific object of an ERP system.
Financial Perspective
An ERP-system represents a capital investment which causes expenses as well as revenues. These expenses and revenues are not quantifiable in an easy way. But a financial follow-up is nevertheless required and can be usefully take on the form of a gap analysis concentrating on the actual expenses versus those expenses budgeted. According to Rosemann and Wiese (1999) the results of the financial perspective can help to identify poor performance. Negative deviations of actual training costs versus budgeted costs may indicate that the system's functions are not efficiently used by staff members. A continuous increase in external consulting expenses may point to deficiencies in the internal training staff's competence.
Customer Perspective
Rosemann and Wiese (1999) differentiate between employees directly dealing with the system and external business partners like suppliers, subcontractors and customers which are indirectly working with the system. For the purpose of measuring business performance, concentrating on internal users seems more adequate, since the system's effects on external partners are rather remote and indirect. There are 2 aspects which should be differentiated:
* The share of types of business processes covered by the system. An example for this is the retailing sector with business process types like "classical" retailing, third party orders, settlement, promotion and customer service.
* The share of total transaction volume handled by the system versus transactions performed outside of it needs to be considered.
Because of the bottom-up approach, which is followed by Rosemann and Wiese (1999), measures should be designed so as to allow easy identification of bottlenecks connected with the system.
Internal Process Perspective
The internal process perspective focuses on the internal conditions for satisfying the customer expectations. These conditions can be grouped into processes needed for operating the system (Figure 9: The internal process perspective - operational view (Rosemann and Wiese, 1999)) on the one hand and those for improving and enhancing the system (Figure 10: The internal process perspective - development view (Rosemann and Wiese, 1999)). Essential measures for evaluating its internal processes are the number and type of trends in user complaints. Analysis of these measures should lead to a ranking of system defects by disutility to users. Further important are the bottlenecks which should be identified when measuring response time, transaction volume, and their respective evolution over time. These measures are early indicators of the need for capacity augmentation.
The Internal Process Perspective can help to eliminate defects as well as to improve the system's present capabilities and introducing new functions. In order to evaluate the effectiveness of the enhancement process, standardised indices should be defined. For example the actual time needed for development compared to schedule. Or an index to measure the quality of the developed software.
The innovation and learning perspective is dedicated to an examination of the company's ability to effectively make use of the ERP system's functions as well as to enhancement and improvement of the system. This ability depends on the know-how of personnel and entails including employee-centred measures covering both users and IT staff. A useful indicator for measuring this dimension is the level of training courses, measured by the amount of time or expenses spent. For system developers, there are specific measures like their type of formal qualification which can additionally be surveyed. Another important measure is dependence on external consultants which are often necessary for the implementation of an ERP system and ERP projects. However, the company desires a quick know-how to its internal staff in order to reduce its need and dependency for highly paid consultants.
Task-Technology Fit (TTF) construct as an indicator of ERP success
The Task-technology fit (TTF) theory has the main clear statement that IT is more likely to have a positive impact on individual performance and can be used if the capabilities of the IT match the tasks that the user must perform. It measures the acceptance with the 3 main influence factors: task, ERP (technology) and user. These 3 factors are influencing the acceptance of the system. ERP is viewed as a tool used by individuals carrying out their tasks. Tasks are the actions carried out to transform inputs into outputs. That means, for example, input is an order of a customer and output is the delivery of the specific article. Users use the technology to support them in performing of their tasks. Task-technology fit measures the degree to which a technology supports an individual in performing his or her portfolio of tasks (Goodhue, 1995). Smyth (2001) adopted the original model of Goodhue and Thompson (1995) and added 2 other accepted success indicators, Perceived usefulness, what Ives and Olson (1984) call "aggregate organizational benefit" and "user satisfaction" what DeLone and McLean (1992) reported as a further important indicator of IS implementation success. The framework describes the match between the functionality provided by the ERP package, the tasks undertaken by the users of that package, and the skills and attitudes of the individual users. In the TTF ERP Success Model of Smyth (2001) TTF, perceived usefulness and user satisfaction are shown as the three constructs that are the most important indicators of ERP success.
In this model poor TTF would contribute to a low level of User Satisfaction, while poor TTF and low user satisfaction each would contribute to the lack of success of the ERP package. Perceived usefulness is influenced by organisational factors and that's influencing the user satisfaction in a direct and an indirect way. To use this model in practical use it is necessary to go through the publications which are the basis for this new framework (mentioned above).
Other Success Measurement Models
There are models which are very similar to existing models such as the research of Seddon (1997) or very specific for measuring only one aspect of IS success, like the research of Sedera et al. (2003) who studied the relation of the size of the organisation and the success achieved or Wu and Wang (2007) who investigated the key user's viewpoint in success measurement approaches. The focus of this paper is to show the most important approaches of success measurement for IS / ERP systems, that's why only a selection of the probably most interesting approaches for future research in this field is shown. A lot of models arise from different research approaches but only a few have really new or alternative basic approaches which need to be considered in this research. One interesting aspect is the difference of some models identified. Some arising from the DeLone McLean model, others like the BSC models showed a new attempt when the researches were published. The TTF is interesting because of the alternative point of view which is concentrating on the fit of the system to the needs of the users / involved parties. That shows that every model discussed has a right to exist and a ranking according to the functionality / gained currency doesn't make sense.
4.COMPARISON AND AREA OF APPLICATION OF THE MODELS
In this section of the paper, the success measurement models which are mentioned above are investigated regarding the different use cases and the different dimensions of success measurement approaches in the field of ERP systems. That means that every model has specific strengths and weaknesses and for every practical success measurement intention different needs occur, which can result in different models used. The possible outcomes of success measurement differ on the intention the company has when using a success measurement framework in practice. Table 1 shows an overview of the different features of the investigated models which should help in selecting an appropriate model. Some of the models are ERP specific, others are concentrating on information systems in general and may need adoption when used for ERP success measurement. The dimensions were defined by the authors of this research. Therefore, some criteria which are interesting for the use of a model were investigated through the literature review. The first dimension on the y-axis is the "No of different perspectives" which means the number of the ranges in which the success metrics were defined. As mentioned above the DeLone & McLean model has the dimension "Service Quality" with some metrics which make the dimension measurable. "Suggested Measures" means that the authors who build the model in their research already defined measurement metrics for the dimensions of their model. The authors of some models, such as the IS Effectiveness Matrix, are only listing a few metrics which should help as an assistance for the defining of appropriate measures. The third dimension "Tested in practical use" shows if the model was already used for the evaluation of a IS / ERP and not only a theoretical construct. "Process model / Causal model" shows which type the model is. Process models often represent a networked sequence of activities or dimensions. Such models can be used to develop more precise and formalized descriptions of success measurement approaches. Causal models represent the causal interdependencies between the dimensions of the models. As mentioned above in the DeLone and McLean model "User satisfaction" is influencing "Use".
In the section following table 2 considers the stakeholders' interests and other dimensions of interest when selecting a model in daily business practice.
4 of the models mentioned above were tested in practical use with different outcomes and different evaluation purposes. Some of the authors tested the models while building the framework. The model of DeLone and McLean (1992; 2003) was tested in different use cases like in 2007 by Chien and Tsaur who found out that system quality, service quality, and information quality seem to be the most important successful factors when they were investigating the success of ERP-systems in Taiwanese high-tech industries. Another finding of their investigation was that the results indicated that technological newness was the most important factor in determining the quality of the system. System quality, such as performance, flexibility of changes, response time, and ease of use, is a technical issue. The result of Chien and Tsaur (2007) confirmed conventional wisdom that the pursuit of state-of-the art technology is a risky proposition.
The Gable et al. model was tested in practical use by Gable et al. (2003) when the authors were validating their findings. But the authors were only interested in model building and not in the results of the 310 valid responses; that's why the results of the survey were not directly published. In 2008 the paper of Sedera studies the proposition that the size of an organization (i.e. small, medium, and large) may have contributed to the differences in receiving benefits from Enterprise Systems. For this research the author used the Gable et al. (2003) model and the study included 66 respondents representing small organizations, 198 respondents from medium and 66 respondents from large organizations, from a total of 27 organizations that had implemented a market leading Enterprise System in the second half of 1990. The author demonstrated that their research provides counter evidence to the popular belief that Enterprise Systems are unsuitable for Small organizations, demonstrating similar benefits and impacts on their larger counterparts.
The model of Ifinedo (2006) was tested by the author himself in the publication of Ifinedo and Nahar (2006) when the authors did a study with companies in the Nordic Baltic region. The authors believe that firms that have no formal methods of evaluating the success of their ERP software could use their revised ERP system success framework for such an exercise as reported in their case studies. Ifinedo and Nahar (2006) found out that system quality and information quality are considered the two important dimensions in the assessment of ERP success for their sampled firms.
The framework of Stefanou (2001) was tested by the author because he wanted to validate his research results. Stefanou (2001) made personal semi-structured interviews and structured interviews conducted through e-mail with nine ERP consultants and project implementation leaders. But there was no test with practical results in the meaning of a selection process done, only the testing of the model previousl described.
5.CRITICAL CONSIDERATION OF EVALUATION IN GENERAL
Drucker (2004) once said: "What you measure is what you get." "Ensure that every measure of performance is pertinent to the achievement of a goal or value of your organization. Otherwise, you risk misdirecting your organization."
IT executives know that the right investments in technology can deliver competitive advantage. But in today's business environments the role of the IT is often like electricity, to be managed at minimal cost. By investments in IT innovation, companies have the opportunity to gain a competitive advantage or to change the rules in their industries. The information technologies that support businesses should be adopted and measured with the same decision-making process used for investments in general (Craig and Tinaikar, 2006). Only by measurement of IT success or success metrics the gap between the optimum and the current state can be identified and a strategy for the future development can be made. This shows the importance of success measurement approaches in the IT field for strategic thinking and planning. If the measurement result says that the ERPs performance is poor, that the users are not satisfied with the implemented solution and the transaction time is too high, this outcome is worthless without any action to change this.
Evaluation is often based on standardized questionnaires which were made by evaluator without considering the stakeholder's opinion. Guba, Lincoln (1989) who created the "Fourth generation Evaluation" criticized that common, quantitative evaluation methods are not able to support companies with their complex and dynamic business in a sufficient way. According to Guba and Lincoln (1989) the quantitative evaluation is not appropriate to measure complex interdependences between internal and external influences. Another weak point is that there are a lot of measures which can be of importance which are not considered in a common measurement model and, therefore, a clear statement about the strength and weaknesses of the system can't be made. The "Fourth generation Evaluation" is based on an intensive collaboration and the consideration of the concerns of the stakeholders. That means that an open-minded approach is used and no preconceptions should influence the concept.
The models analysed in this research often include predefined measures as shown in Table 1: Comparison of the different success measurement models which can be used for evaluation of the implemented IS or ERP-system. To measure other key figures, which were not considered in the selected model could be an additional challenge. The intensive collaboration between the stakeholders and the team which is doing the success measurement is necessary because the measurements are not limited to those which are currently included. Selecting appropriate measurements can influence the outcome of the evaluation. If a person who is not directly involved and impartial is doing the success measurement, the risk to get sophisticated results is not as high. But if a person, who wants to direct the result of the evaluation to a particular result, is doing the evaluation the risk of a falsification is higher. The selected success measurement model can influence the outcome too because every model has a specific focus.
The stakeholders' participation, which is claimed by Guba and Lincoln (1989,) depends on the model selected. Therefore, the next section of the paper investigates how different stakeholders are considered in each model and which model could be appropriate when doing a practical evaluation for specific stakeholder groups.
6. SUCCESS MEASUREMENT MODELS AND STAKEHOLDER INVOLVEMENT
From the stakeholders' perspective, each stakeholder has a different view on the project outcome. For the achievement of a complete perspective on ERP-success, these views have to be considered when doing the success measurement.
This section shows which models are useful for specific measurement approaches. An investigation regarding dimensions which are interesting for companies or researchers when selecting a success measurement model was made. Each stakeholder has a specific expectation of the outcome of the success measurement. Therefore, the view of the stakeholder must be considered in the used model or else the opinion is not considered in the success measurement result. Due to the fact that the evaluation result is used for different purposes, the selection of the model should be done considering the interests of the group for which the evaluation is made. If e.g. the IT department evaluates the system, the outcome and afterwards the actions taken do not improve the system in a way the users want to . User expectations and IT purposes are often widely different. The top management has other interests as users for example. The top management is interested in cost reduction or in an IT strategy plan; users often want to improve the usability and simplify their daily work. In table 2 the models are investigated in respect of different stakeholders and categorized into 3 groups. X means that the evaluation is integrating or affecting the stakeholder or the evaluation fulfils the dimension defined in the matrix.
Table 2 shows that most models consider the user's point of view. That's clear because the users are working with the system when doing their daily business and are influenced by the (poor) performance directly. For the investigation, the success metrics of the different models were analysed.
The different stakeholders defined are the users, the top management, the IT and the externals. The dimension "Process improvement" which is shown in the matrix means that success measurement leads to a clear identification of the processes which are not optimized and possibly need to be changed. Some models focus on the processes, like the TTF model which tries to show the gaps between the daily tasks and the fit of the processes the according to the tasks.
"Future needs" means that the model investigates if the future needs of the company can be fulfilled by the investigated system or if any changes should be made or if new implementations which may be needed. "Future needs" signifies the strategic planning of the ERP and is a middle to long term dimension. The BSC approaches especially concentrate on the future needs because of the usage of the BSC as a strategic planning instrument in the business environments nowadays.
The dimension "Financials" shows if the model considers financial aspects - like for example external cost or support costs and provides a cost planning / evaluation. This could lead to potential cost reductions and a clear cost structure and is interesting for the (IT-) management.
The "net benefits" in the DeLone and McLean model measure, for example, the cost savings or the additional markets, that's why financials are rated with an X. An evaluation of with this model can lead to a process improvement, but processes are not influenced directly. "IT" is rated with an X because 3 dimensions are affecting the IT department.
Table 2 additionally shows that the interests of the vendor / externals are not considered in most of the models. Only (Ifinedo, 2006) added an external perspective to his model. But this is only used to evaluate the performance of the externals and not to consider their opinion. It seems that this view is not important for success measurement in practice. An external view would be of interest if the externals would evaluate different customer implementations, the result could be used to compare systems or else the consideration of an external view makes no real sense.
The model of DeLone and McLean (1992; 2003) is primary supporting the users and the IT because it tries to evaluate the net benefits the users are getting from the system. The system and the service quality are directly influenced by the performance of the IT department. The model can lead to a process improvement because of the service quality dimension, which can reveal processual faults. The financials are considered by the defined measures of DeLone and McLean, like cost savings or additional sales.
The Gable et al. (2003) model has two similar dimensions to the DeLone and McLean (1992; 2003) model (system quality and information quality) and 2 different dimensions. The organisational impact has measures which are measuring possible business process changes and financial changes. The other dimension which is different to the DeLone and McLean (1992; 2003) model is individual impact. Individual impact consists of measures which are measuring the progress of the user when the user is working with the system; that's why users play an important role in the Gable et al. (2003) framework.
Ifinedo (2006) added the external perspective to the Gable et al. (2003) model that's the reason why the externals play a role in that model. The other line-ups of table 2 are the same as in the Gable et al. (2003) model.
The model of Markus and Tanis (2000) is considerable for the whole enterprise. The authors defined key actors for every phase of their enterprise system experience cycle and in every phase there are different stakeholders involved. Beyond all phases, all the stakeholders,shown in table 2, are involved and that's why all the stakeholders are rated with an X. Because of the widespread activities, shown in the different phases of the model, they can lead to a process improvement, they consider the future needs of the company (because of the step by step phases) and they also can be used to control the budget (financial metrics).
Stefanou (2001) is considering the all the stakeholders mentioned below. For the exante evaluation of an ERP-system, it's important to involve all the stakeholders in the evaluation and when coming to a final decision. The organisational constraints of the ex-ante evaluation are considering the business processes of the company, and, because of that, the model can lead to an improvement or a change of business processes (based on the selected system). The future needs of the company are considered, otherwise an ex-ante evaluation wouldn't make sense. The aim of the ex-ante evaluation is the selection of an appropriate ERP-system which covers the future needs of the company. The financials are considered in the financial and time constraints and in the ERP product, vendor and support services evaluation which has to be made in the second part of the second phase of the model.
Rosemann and Wiese (1999) presented a model which considers the users and provides measures which are interesting for the management. The BSC provided by the authors leads to a process improvement (process view), it considers the financials (financial perspective) and the future needs (innovation and learning perspective). Because of this, the related fields in the table below are rated with an X.
Smyth et al. (2001) were concentrating on the tasks and the fit of the task to the technology which is the basis for the fulfilment of those tasks. In the model the users play the most important role because the users are working with the system and they have to manage their daily work with the processes provided by the system. The outcome of the investigation (task-technology-fit) is affecting the business processes and can lead to a process improvement or a business process reengineering. The IT department is only indirectly involved if a change in the system (customizing or modification) is necessary.
7.CONCLUSION
The objective of this research is a review of different models which could be used for ERP success measurement We found through literature review that ERP systems success measurement models might be limited in scope and do not suit for every practical case. In particular, this research attempts to build a long needed theoretical base for success measurement studies.
The DeLone & McLean IS Success Model seems to remain the most popular, comprehensive framework for IS success measurement. But there are other models which show interesting alternative approaches to success measurement. Some of the models have a specific approach (e.g. especially designed for the measurement of success for ERP systems) which can simplify success measurement for companies because of the defined, validated metrics. A recommendation which model should be used or which one is the best is not possible.
This paper offers the reader a critical overview with the specific properties and an alignment of the models discussed and allows them to get to know which success measurement approaches exist in the literature and which one would be applicable for the research or practical success measurement case. Some of the success measurement models identified were not discussed in this research due to the fact that the models
* were very similar to other models
* did not contain a suitable approach for ERP success measurement
* had a specific approach in the field of success measurement (e.g. to measure only the management perspective of success in IS)
This study has implications for practice as well. As noted, this study is partly motivated by the need to present practitioners a basis for the selection of a success measurement model. These practitioners need guidelines for assessing the success of their ERP software. The two tables in this research show which models could be of interest for practitioners and researchers. Therefore, the authors defined different dimensions which are differentiating the models from each other and should be used as a basis for the selection of a model. As stated in a section above, success measurement models and stakeholder involvement, the findings are limited to the criteria investigated in this paper. That means that there are different possible criteria which can be used to differentiate one model from another model and the criteria defined in this research are only a possible subset.
Evaluation of success is a difficult approach and it only makes sense if the result of an evaluation is used as a basis for actions which can result in an improvement of the systems performance. Possible outcomes of improvements can be measured through a new evaluation of the systems performance and a comparison of the results of each evaluation. For future research it would be interesting to investigate which actions can be set if an evaluation result of a system is poor in a dimension (information quality, for instance) and which improvements should be made.
Another difficulty in evaluation approaches is that the results are often manipulated by the department which is making the evaluation. The IT department, which is usually the department doing the success measurement, for example, would be interested in a positive evaluation of the system quality. Therefore, the measures, which are part of the evaluation, can be defined in a way which leads to a positive result. If a model prepares those most important measures, it should lead to a convincing result.
References
REFERENCES
Behrens S., Jamieson, K., Jones, D. & Cranston, M. (2005). Predicting System Success using the Technology Acceptance Model: A Case Study. 16th Australasian Conference on Information Systems, Sydney
Cameron, K. S., & Whetten, D. A. (1983). Some conclusions about organizational effectiveness. In K.S. Cameron & D.A Whetten (ed.) Organizational Effectiveness: A Comparison of Multiple Models, New York: Academic Press.
Chien, S-W. & Tsaur, S. M. (2007). Investigating the success of ERP systems: Case studies in three Taiwanese high-tech industries', Computers in Industry, 58(8-9): 783 - 793.
Cooke, D., Gelman, L., & Peterson, W. J. (2001). ERP Trends. The Conference Board, pp. 1-28.
Craig, D., & Tinaikar, R. (2006). Divide and conquer: Rethinking IT strategy, (Online), McKinsey & Company, http://www.mckinsey.de/downloads/publikation/mck_on_bt/2006/MoIT9_ITStrategy_VF.pdf
Curlee, T. R., & Tonn, B. T. (1987). The Success or Failure of Management Information Systems: A Theoretical Approach', OAK RIDGE NATIONAL LABORATORY for the U.S. Department of Energy under Contract no. DE-AC05- 840R21400, pp. 1 - 23.
Darshana, S., Gable, G., & Rosemann M. (2001). A Balanced Scorecard Approach to Enterprise Systems Performance Measurement. Conference Proceedings, Twelfth Australasian Conference on Information Systems, Coffs Harbour.
Davenport, T. H. (1998, July-August). Putting the enterprise into the enterprise system. Harvard Business Review, 76(4): 121 - 131.
DeLone, W. H., & McLean E. R. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, 19(4):9-30.
DeLone, W. H. and McLean, E. R. (1992). Information Systems Success: The Quest for the Dependent Variable', Information Systems Research, 3(1):60-95.
DeLone, W. H., & McLean, E. R. (2002). Information Systems Success Revisited. Conference Proceedings, 35th Hawaii International Conference on System Sciences, Los Alamitos.
Ding, Y., & Straub, D. (2007). Using Marketing Exchange Theory to conceptualize IS Quality and Re-conceptualize the IS Success Model. Twenty Eighth International Conference on Information Systems, Montréal
Drucker, P. F. (2004). The Daily Drucker. New York: Harpercollins.
Elpez, I., & Fink, D. (2006). Information Systems Success in the Public Sector: Stakeholders' Perspectives and Emerging Alignment Model. Issues in Informing Science and Information Technology,3: 219-231.
Esteves-Sousa, J., & Pastor-Collado, J. (2000). 'Towards the unification of critical success factors for ERP-implementations. 10th Annual Business Information Technology Conference (BIT), Manchester
Gable, G., Sedera, D., & Chan, T. (2003). Enterprise Systems Success: A Measurement Model', Twenty Fourth International Conference on Information Systems, Seattle.
Goodhue, D. L., & Thompson, R. L. (1995). Task-Technology Fit and Individual Performance. MIS Quarterly, 19(2): 213 - 235
Guba, E. G., & Lincoln, Y. S. (1989). Fourth Generation Evaluation. SAGE.
Hedman, J., & Borell A. (2005). Broadening Information Systems Evaluation Through Narratives', The Electronic Journal of Information Systems Evaluation, 8(2): 115-122.
Holland, C. P., & Light, B (1999). A critical success factors model for ERP implementation', IEEE Computer Society, 16(3): 30 - 35.
Ifinedo, P. (2005). Do Organisational Technological Contingency Factors Influence the Perception of ERP Systems Success? An Exploratory Study in the Baltic-Nordic Region of Europe. Conference Proceedings, The 4th International Business Information Management Association (IBIMA) Conference, Lisbon, pp. 427 - 437.
Ifinedo, P. (2006). Extending the Gable et al. Enterprise Systems Success measurement model: a preliminary study. Journal of Information Technology Management, 17(1): 14 -33.
Ifinedo, P., & Nahar, N. (2006). Quality, Impact and Success of ERP-Systems: A Study Involving Some Firms in the Nordic-Baltic Region. Journal of Information Technology Impact, 6(1): 1-28.
Kaplan, R. S., & Norton, D. P. (1997). Using the Balanced Scorecard as a Strategic Management System. Harvard Business Review OnPoint, 1-13.
Kronbichler, S. A., Ostermann H., & Staudinger R. (2009). A Review of Critical Success Factors for ERP-Projects. The Open Information Systems Journal, 3:14-25.
Markus, M. L., & Tanis, C. (2000). The Enterprise System Experience-From Adoption to Success. In Zmud, R.W (ed.). Framing the Domains of IT Management: Projecting the Future Through the Past, Pinnaflex Educational Resources, Inc., Cincinnatti, (Chapter 10, pp.173-207).
Martinsons, M., Davison, R. , & Tse, D. (1999). The balanced scorecard: a foundation for the strategic management of information systems. Journal of Decision Support Systems, 25: 71 - 88.
May, M. (2003). Business Process Management - Integration in a web-enabled environment. Pearson Education: Great Britain.
Montazemi, A., Cameron, D., & Gupta, K. (1996). An empirical study of factors affecting software package selection', Journal of Management Information Systems, 13:89-105.
Myers, B. L.., Kappelman, L. A., & Prybutok, V. R. (1996). A Care for Including Work Group Productivity Measures in a Comprehensive IS Assessment Model. Conference Proceeding. The 27th Annual Meeting of the Decision Science Institute, Orlando.
Myers, B. L.., Kappelman, L. A., & Prybutok V. R. (1997). A Comprehensive Model for Assessing the Quality and Productivity of the Information Systems Function: Toward a Theory for Information Systems Assessment. Idea Group Publishing, Hershey.
Nah, F. F.-H., & Delgado S. (2006). Critical Success factors for Enterprise Resource Planning Implementation and Upgrade. Journal of Computer Information Systems, 99 - 113.
Nah, F. F.-H., & Lau, J. L.-S. (2001). Critical factors for successful implementation of enterprise systems. Business Process Management Journal, 7(3): 285 - 296.
Nah, F. F.-H., Zuckweiler, K. M., & Lau, J. L.-S. (2003). ERP Implementation: Chief Information Officers Perceptions of Critical Success Factors. International Journal of Human-Computer Interaction, 16(1):5 - 22.
Nicolau, A. I. (2004). Firm Performance Effects in Relation to the Implementation and Use of Enterprise Resource Planning Systems. Journal of Information Systems, 18(2): 79-105.
Praxiom Research Group Limited (1997) 'ISO 9000 2005 - Management Dictionary', (Online), http://www.praxiom.com/iso-definition.htm#Quality, Exeter: Author.
Redenius, J. O. (2005) 'Im Mittelpunkt steht die Wertschöpfung', (Online), IT Management, 3(1-5), http://www.inditango.com/downloads/IT_Management-032005.pdf
Rosemann, M., & Wiese, J. (1999). Measuring the Performance of ERP Software - a Balanced Scorecard Approach. Conference Proceeding, The 10th Australasian Conference on Information Systems, 8(4).
Seddon, P. (1997). A Respecification and Extension of the DeLone and McLean Model of IS Success. Information Systems Research, 8(3): 240 - 253.
Sedera, D., Gable, G., & Chan, T. (2003). ERP success: Does organisation Size Matter?. 7th Pacific Asia Conference on Information Systems,Adelaide:1075 -1088.
Sedera, D. (2008). Does Size Matter? Enterprise System Performance in Small, Medium and Large Organizations', Research presentation at the 3gERP Workshop (Online), http://www.diku.dk/hjemmesider/ansatte/henglein/3gERP-workshop-2008/papers/sedera.pdf
She W., & Thuraisingham B. (2007). Security for Enterprise Resource Planning Systems. Information Systems Security, 16:152-163.
Smyth, R. W. (2001). CHALLENGES TO SUCCESSFUL ERP USE. The 9th European Conference on Information Systems, Bled.
Stefanou, C. J. (2001). A framework for the ex-ante evaluation of ERP software. European Journal of Information Systems, 10:204-215.
Sumner, M. (1999). Critical success factors in enterprise wide information systems projects. Conference Proceedings; The American Conference on Information Systems, New Orleans, pp. 232 - 234.
Willcocks, L. P., & Sykes R. (2000). The Role of the CIO and IT Function in ERP.Communications of the ACM, 43(4):32 - 38.
Wu, J. H., & Wang Y. M. (2006). Measuring KMS success: A respecification of the DeLone and McLean's model', Information and Management, 43:728 - 739.
Wu, J. H., & Wang, Y. M. (2007). Measuring ERP success: The key-users' viewpoint of the ERP to produce a viable IS in the organization. Computers in Human Behavior, 23:1582-1596
AuthorAffiliation
Stephan A. Kronbichler
University of Health Sciences, Austria
Herwig Ostermann
University of Health Sciences, Austria
Roland Staudinger
University of Health Sciences, Austria
Recebido em/Manuscript first received: 22/11/2009 Aprovado em/Manuscript accepted: 08/04/2010
Endereço para correspondência/ Address for correspondence
Stephan A. Kronbichler, Junior Researcher, University of Health Sciences, Medical Informatics and Technology, Oberndorf 50d, 6341 Ebbs. Tel.: 0043/664/8159748. E-mail: Stephan.Kronbichler@umit.at
Herwig Ostermann, University of Health Sciences, Medical Informatics and Technology, Eduard Wallnöfer Zentrum 1, 6060 Hall in Tirol, Austria, Department for Human and Economic Sciences. Fax. No.: +43 (0)50 8648 67 3880.E-mail: Herwig.Ostermann@umit.at
Roland Staudinger, University of Health Sciences, Medical Informatics and Technology, E-mail: Roland.Staudinger@umit.at

Effective ERP adoption processes: the role of project activators and resource investments
Bernroider, Edward W N. European Journal of Information Systems, suppl. Special Issue: Information Technology Funding and System22.2 (2013): 235-250. Introduction
Effectively assessing and implementing enterprise-wide Information Systems (IS) such as Enterprise Resource Planning (ERP) in organizations remains to be difficult (Gunasekaran et al , 2008; Aloini et al , 2012). The adoption of ERP systems in an organization is a time and cost-intensive venture with far-reaching consequences for the way the entity is structured and conducts its business. Because of the complex acquisition and implementation procedures in companies (Bernroider & Koch, 2001; Umble et al , 2003; Uwizeyemungu & Raymond, 2009), projects are far too often perceived as only partial successes or are even abandoned prior to completion. Managers find it difficult to assess the performance of ERP projects against the backdrop of changing stakeholder perceptions (Markus et al , 2000a; Besson & Rowe, 2001) and demanding resource requirements (Bernroider & Koch, 2001; Sharma et al , 2008).
IS research has only begun to recognize the importance of activators in framing and setting the direction for an ERP adoption project (Boonstra, 2006; McLaren & Jariri, 2012). Being able to appreciate the source of any IT innovation project is essential to understanding its requirements and how the project team materializes. In previous research, early team formation was emphasized as a central aspect of ERP adoptions (Bernroider & Koch, 2001). Project teams may be participative, balanced or biased towards different internal or external stakeholder groups. Effective IS planning should involve extensive participation (Peffers et al , 2003) and a meaningful dialogue to avoid conflict escalation during implementation (Besson & Rowe, 2001).
Organizations continue to struggle with the high levels of resources needed for successful ERP adoption. ERP systems are cross-functional platform solutions associated with great socio-technical complexity, and therefore demand resource-intensive justification and funding stages (Stefanou, 2001) and implementation procedures (Sharma et al , 2008). Business executives are facing a lot of uncertainty about when to invest in which resource to effectively adopt ERP in their organizations.
Against this backdrop, I investigate whether effective ERP adoption projects can be associated with resource investment decisions at different project stages, and analyze the respective roles of stakeholders in ERP project design. Our results offer managerial insights on the timing of effective staffing and resource investment decisions for ERP projects. The three central points of this empirical study include the important role of stakeholders in ERP project initiation, a two-staged view of expended resources for ERP adoption, and the question how these resource investment levels impact the overall performance of the ERP project. The methodology is a quantitative empirical survey of Austrian ERP adopters. Our stratified random sample comprises 88 mid-sized and large organizations. I used non-parametric statistical methods (independence tests and correlation analysis) and principal component analysis (PCA) to test five research hypotheses.
This study makes a number of contributions to the IS adoption literature:
The paper offers new descriptive insights in terms of ERP activators, distributions of various resource metrics over ERP project stages, and ERP project effectiveness levels. This information aims at giving managers an understanding of some common design practices in ERP adoptions.
It offers a better understanding of relationships between the roles of ERP activators in team formation and resource investment decisions with regard to different project stages.
It distinguishes between resources expended for different project stages and shows that levels of expended resources are related between stages.
It demonstrates that a broad definition and multiple measures of project effectiveness can describe the failures and successes of ERP projects. By linking these measures with levels of expended resources , the study shows that, for example, heavy-weight ERP projects are less effective.
Literature review and research motivation
This section very briefly summarizes results of previous research about the nature and scope of ERP projects emphasizing resources expended and the role of stakeholders.
Nature and scope of ERP projects
ERP projects continue to experience schedule delays, cost escalations, and reduced quality when the system is finally operational. Only 13% of organizations think their ERP adoption projects are meeting expectations with regard to improvements in business processes or business value delivery. More than 50% of companies rate their ERP adoption as unsatisfactory (Panorama Consulting Group, 2009). The main reason for this situation is the complexity of the underlying radical organizational change, which is of strategic nature and software-intensive (Besson & Rowe, 2001). An ERP adoption requires high levels of resource investments and has far-reaching organizational implications (Gunasekaran et al , 2008; Stockdale et al , 2008).
Adding to the problem of underachieving ERP projects are changing perceived expectations of stakeholders and their actions to influence the course of the project. Stakeholders in ERP projects usually try to influence the course of the project (Boonstra, 2006). However, a specific success or failure at one point in time may only be loosely related to the perceived situation at another point in time (Markus et al , 2000a). It seems important to develop some governance and control over stakeholders (Johnstone et al , 2006). The early stage of ERP project initiation may already determine the influence of stakeholders and levels of implementation conflicts (Besson & Rowe, 2001).
The ERP adoption project consists of multiple stages (Bernroider & Koch, 2001). Project management theory suggests two main stages in any IT project: a design and a delivery stage (Maylor, 2010). In the context of ERP, prior literature used the terms 'selection', 'chartering', and, more recently, 'justification and funding' to describe the design stage. Typical design tasks include evaluating requirements, risks, alternatives and implementation options, and framing the project including the funding strategy (Bernroider & Stix, 2006; Aloini et al , 2012). Project delivery, also termed 'implementation' or 'project phase', relates to adapting organizational routines and introducing the information system to different organizational units. An ERP project may also include a 'shakedown' or 'early-use' stage, referring to the period after implementation until a routine service is established (Markus & Tanis, 2000). Prior research has shown that ERP projects exhibit different characteristics in these stages, leading to the notion of 'ERP dynamics' (Besson & Rowe, 2001). During the project, perceptions of involved stakeholders change from technological to organizational imperative positions. The latter dominate in the implementation stage, when integration/differentiation choices and diverse stakeholder conflicts need to be overcome.
It is hard to measure the failure or success of any dynamic and software-intensive project. This particularly applies to ERP projects. In project management, the meaning and choice of performance metrics remain an active area of research. No clear-cut definition of successful and failed projects is available (Agarwal & Rathod, 2006). Process metrics of IT projects usually comprise the implemented scope of original requirements, plan effectiveness, and early operational impacts (Mabert et al , 2006; Maylor, 2010). Early operational impacts refer to the time between going operational and achieving a 'routine use' of the system (Markus & Tanis, 2000). This phase was considered in prior studies as a time period of several months causing organizational performance dips, which may be (McAfee, 2002; Jones et al , 2011) or may not be (Markus et al , 2000b) recovered. Performance dips were reported in regard to, for example, process cycle times, inventory levels, and operating labor costs (Boonstra, 2006). Outcome quality, either system or information related, is the main independent success variable in the D&M IS success model (DeLone & McLean, 1992). Leading indicators of quality problems are conflicts arising during system implementation. Literature distinguishes between conflicts over strategy (Lee & Myers, 2004), relationship and social conflicts (Jehn & Bendersky, 2003), and task conflicts stemming from disagreements about the nature and fit of tasks and functions, in particular between IS users and developers (Liu et al , 2011b). Consequently, ERP projects should be evaluated against multiple goals to understand their overall levels of effectiveness (Markus et al , 2000a).
Research problem and objectives
The above discussion has shown that failure to account for changing stakeholder perceptions has repeatedly been identified as a major problem in troubled ERP projects. Furthermore, considerable amounts of a company's resources are invested in multi-staged ERP projects. Finally, it was found that despite the significance of ERP projects, far too many result in only partial success or even abandonment prior to completion. While the mentioned studies have increased our understanding of these three aspects in isolation, little empirical work has been conducted to establish the important associations between these dimensions. To investigate these links further, I now define three research objectives to guide the paper.
Firstly, it is crucial to find out whether early activators impact team formation and influence the resource investment decisions made with regard to different stages of the project. It is possible that despite the literature's consensus on the stakeholders' general importance, the critical issues of staffing a project team and assigning resources to the project are not related to the stakeholders dominating ERP activation.
Secondly, I set out to investigate expended resources not only for the ERP project implementation stage but also for the ERP project justification and funding stage. It is essential to understand whether levels of expenditures are related between these two stages, and whether they are influenced by the ERP activators.
Thirdly, in order to contribute to a better understanding of the failures and successes of ERP projects, we need to investigate whether the levels of resources expended in a project stage are associated with particular levels of project effectiveness.
Research design
Conceptual model
Drawing upon the literature review and the three research objectives, I developed a three-tier conceptual model as shown in Figure 1 - See PDF,. The dependent variable in our model is ERP project effectiveness. The middle dimension reflects the expended resources, and the independent variable captures the roles of stakeholders in ERP activation. The next section will develop the hypotheses.
Research hypotheses
Our review of the literature on ERP adoption in the previous section suggests that successful adoption requires both high levels of resource investments and that decisions about the structure and levels of expended resources may be dependent on stakeholder involvement. Next, we will explore each of these anticipated relationships more specifically, and summarize the assumptions as research hypotheses to be tested with the empirical data.
The role of ERP activators
The activators of the ERP project can promote certain team structures, possibly to the advantage of their stakeholder groups. Participative and balanced designs, equally reflecting the values and ideas of many, support effective decision-making and increase acceptance rates (Davenport, 1993; Sarker & Lee, 2003; Ke & Wei, 2008). The early screening process can be managed to further a more widespread inclusion of stakeholders in the project team (Hsu et al , 2011). We assume that the source of the initial need has implications for staffing the project. A project initiated by the IT department may be configured as a technology-driven project rather than an organizational change project (Kumar et al , 2002). ERP systems triggered by senior management may, in turn, be perceived as threats to the internal IT department (Besson & Rowe, 2001).
Stakeholders influence the design and course of the ERP project, and potentially to the advantage of their own interest groups (Boonstra, 2006). This should equally apply to stakeholders activating the project with regard to team design, and the levels and structures of expended resources. A technology-driven project may require a more extensive justification and funding stage, while a strategy-led project or a re-organization project may require more overall resources as they reflect more radical shifts in the organization's culture. Early participation of senior management should provide more leadership in the strategic formulation process (Sarker & Lee, 2003), which may require less support from outside strategy consultants. Therefore, I derive the following hypotheses from prior research.
H1a:The role of the ERP activators is associated with the functional balance of the ERP project team.
H1b:The role of the ERP activators is associated with ERP resource investment decisions at different stages of the project.
The role of expended resources
ERP projects are in general resource-intensive (Bernroider & Koch, 2001). However, we have little information on how resource expenditures for the justification and planning stage are related to expenditures in the implementation stage. Most available studies focus on one or the other stage. It is known that complex projects require more careful planning (Maylor, 2010). The higher the complexity, the more eventualities need to be considered, which likely increases the efforts for all stages of the project. We assume that the complexity of the ERP project will be equally reflected in both stages when it comes to expended efforts.
The so-called Iron Triangle of projects predicts that by accepting higher levels of expended resources (costs and time), higher levels of quality can be realized. These trade-offs can be done deliberately based on goal preferences to achieve an effective project outcome (Barney et al , 2012). Contemporary research suggests more variables or holistic views (e.g. Jha & Iyer, 2007; Marques et al , 2011), but the mentioned principle can still be considered valid.
Related research has provided contradictory results as to whether and which types of conflicts are helpful in IS projects. Empirical research seems to predominately confirm a negative view. For example, it was reported that task conflicts related to requirement diversity correlate negatively with final project performance (Liu et al , 2011a) and that interpersonal conflicts are major barriers to IT project success (Johnstone et al , 2006). However, it was also established that avoiding conflicts altogether is not beneficial for an organizational change (Besson & Rowe, 2001; Meissonier & Houze', 2010), in which conflict is a natural and necessary aspect of any innovation. The complexity of the subject is further increased by fluctuating conflict characteristics in different IS project phases (Yang & Tang, 2004).
Hence, these considerations lead us to the following hypotheses.
H2a:The more resources are expended in the justification and funding stage, the more resources are expended in the ERP implementation stage.
H2b:The more resources are expended for the project, the more effective the ERP project becomes.
H2c:Encountered implementation conflicts are associated with lower ERP system quality and project performance.
Instrument development and pre-testing
The instrument confirms in large parts with related ERP studies (Bernroider & Koch, 2001; Baki & Çakar, 2005). A panel of ERP experts from two universities in Austria and the U.K. examined the survey instrument for content validity (Dillman, 1978). In particular, a clear separation of stages from the initial ERP adoption decision to system use was established to structure the instrument and account for the process-oriented view of the study. According to their suggestions, the questionnaire was revised and used in pre-tests conducted in the U.K. and Austria.
Variable selection and operationalization
The following list describes how the various variables were conceptualized and measured. Table A3 specifies the instrument with the respective questions, and the IDs and scales of the variables.
Profile of respondents
Respondents were asked to categorize their organization in terms of the numbers of customers (SC) and suppliers (SS). I also asked for the implemented modules of the ERP system (SM). The European Amadeus Database (Bureau-van-Dijk, 2003) provided more firm-level information such as the numbers of employees and subsidiaries, legal forms, and the industry sector.
Activation stakeholders and project teams
I considered five different ERP project initiator types (EI) and four different team structures (PT) in differently balanced formations based on prior research on ERP selection (Bernroider & Koch, 2001).
Expended resources
This section distinguished between expended resources for the justification and funding, and implementation project stages. Resources were conceptualized by capturing durations (RT), expended labor time and external support (RL), and monetary costs excluding licensing (RC). I later complemented the analysis with an indirect assessment of personnel costs following suggestions from prior research (Buxmann & König, 1997). This allowed the construction of more estimation variables (RL3-4, RC2). Estimations were based on the given person months (RL1) and the proportion of external support (RL2). The estimated costs, derived from salary estimates (Grohs, 2003), for an external person month were 23,100 EUR (used to calculate RL4), and for an internal person month 6000 EUR (used for RL3).
Implementation conflicts
I conceptually linked conflicts to perceived related implementation problems in a very broad view. The options (IS) covered social, task, technical and resource-related aspects (Jehn & Bendersky, 2003; Johnstone et al , 2006; Liu et al , 2011b).
System level effectiveness
I considered a number of criteria (SQ) to account for the quality of the implemented ERP system (DeLone & McLean, 1992).
Project performance
I included three dimensions of project performance in the survey. Scope achievements (PP1) reflect the implemented functionalities of the ERP software against the original requirements. Plan performance (PP2) shows the expended efforts against the plan (Mabert et al , 2006; Maylor, 2010). Performance dips after going operational (PP3) reflect early-use performance (Markus et al , 2000b; Boonstra, 2006).
General outcome
Finally, this study considered three ERP outcome variables (OU), which I used for separately assessing reliability of responses and non-response bias.
Data collection procedures
The data were collected in Austria through a nationwide empirical survey based on a stratified and disproportional random sample comprising 1000 Austrian companies. The sample was randomly drawn from the European Amadeus database (Bureau-van-Dijk, 2003). The large sample size was necessary to ensure a satisfactory representation of ERP adopters from a population of 24,081 organizations. The target population for this study, however, is smaller and can be defined as all registered medium and large enterprises in Austria having at least started to implement ERP. A stratified and disproportional random sample with subgroups according to company size was necessary to avoid under-representing large enterprises. The hardcopy questionnaire was distributed to the business-management of each of the 1000 companies with a link to an electronic version of the questionnaire. We used follow-up calls and reminder/thank-you emails to explain the study and increase participation. Incentives included the survey report and possible collaboration in case study research.
Sample characteristics and evaluation of non-response bias
The multi-staged data collection of the empirical survey resulted in 209 valid returns and a 22% initial response rate considering neutral dropouts (49 companies). These dropouts refer to companies with wrong addresses or to companies that no longer existed. In accordance with our target group, I excluded small enterprises, where ERP is not a common IT strategy, to allow for a more homogenous sample and more reliable statistical analyses. This procedure reduced the initial sample to 152 medium and large enterprises, but increased the response rate to 24%. Additionally, 64 non-adopters and early-stage ERP adopters still evaluating systems were also excluded. This exclusion did not reduce the response rate as the target population was narrowed down by an equal proportion. Consequently, this study used 88 medium and large organizations in data analysis. All of these organizations have progressed at least to the ERP implementation stage according to the four-phase framework of Markus & Tanis (2000). The screening for possible aberrant response behavior, such as random responding (Thompson, 1975; Berry et al , 1992), triggered no further exclusions of data sets.
Table 1 (See PDF) shows sample characteristics derived from primary and secondary data (Bureau-van-Dijk, 2003). Most organizations have high numbers of customers and suppliers. Medium enterprises employ between 50 and 249 persons with an annual turnover not exceeding EUR 50 million (EC, 2003).
Potential non-response bias was assessed following two different approaches. The first approach compared respondents and non-respondents. The analysis based on variables from the Amadeus database revealed no significantly different characteristics between these groups in terms of legal form (e.g., limited or public companies), number of employees, and number of subsidiaries as measured by chi-square ( χ2 ) and two-sample unpaired t tests (see Table A1). As this approach can only be calculated for characteristics known for both subgroups, I also compared early vs late respondents. This wave analysis approach regularly used in IS and management accounting surveys (Van der Stede et al , 2006; Wu & Wang, 2006) is based on the assumption that late respondents more likely resemble non-respondents than early respondents. In this case, the following ERP outcome variables were considered: changes in the workforce structure and quantity connected with ERP (O1), competitive edge through ERP (O2), and the availability of IT/IS services after ERP implementation (O3). The detailed chi-square ( χ2 ) test results were also included in the Appendix (Table A2). The comparison revealed no statistically significant differences for either variable between waves, thus providing no evidence of non-response bias.
Common method bias
Common method bias or common method variance (CMV) is generally considered in empirical organizational research (Podsakoff & Organ, 1986; Malhotra et al , 2006). This paper is based on a mono-method research design and a self-report instrument, which may cause a certain amount of covariance shared among all indicators. This research applied the Harman's single-factor test as a diagnostic technique to test for CMV. It involves entering all constructs into a principal components factor analysis to see if either a single factor or a general factor emerges that may account for the majority of covariance among measures (Podsakoff et al , 2003). Seven factors emerged. The first accounted for 34% of the variance. The other six (with eigenvalues greater than one) contributed to the remaining 67% of the variance explained by the set, each accounting for 5-19%. This suggests that while there is likely to be some CMV, the effect is relatively small.
Statistical methods
For data analysis, I used SPSS with activated sampling weights to account for the disproportional, stratified random sample (Purdon & Pickering, 2001). The research hypotheses were mainly tested with non-parametric statistical tests including the Mann-Whitney test and the Spearman rank correlation coefficient (Sprent & Smeeton, 2007). Both tests work well with the ordinal responses in our data set and are more robust than their parametric equivalents. I used PCA to achieve an orthogonal transformation to convert possibly correlated variables into a smaller set of linearly uncorrelated factors. Varimax rotation was used to see how groupings of variables measure the same concept (Hair et al , 1984). The factor scores for the composite variables representing the factors were calculated with the regression approach (Thurstone, 1934). PCA was also applied to test for common method bias to understand the systematic error variance shared among variables due to the measurement method (Podsakoff et al , 2003).
Research results
Descriptive analysis
The aim of this section is to present a descriptive summary with regard to levels of ERP project effectiveness, structures and levels of expended resources, and the distribution of ERP activators over business functions. The following three subsections directly relate to the three specific research objectives proposed earlier. I ran independency tests (Mann-Whitney test) to understand the roles of our control variables (organizational size, implementation scope) and only mention statistically significant findings in this respect. Implementation scope is conceptualized with the number of implemented ERP modules (mean=2.48 modules).
ERP project effectiveness
I developed three dimensions - conflicts, system quality, and project performance - to gain a richer understanding of project effectiveness.
The occurrences of implementation conflicts are shown in Table 2 (See PDF) . In the mean, 2.56 conflicts are observed in an ERP adoption project and almost 90% of all cases experience at least one conflict. Most problematic are resource escalations, system-related incompatibilities, and user resistance. LEs regularly experience more conflicts in ERP implementation (Mann-Whitney test, P <0.05).
In the mean, organizations implement effective ERP systems (see Table 3 (See PDF) ). All system-level aspects were evaluated on or above the middle threshold of three, which accounts for a neutral assessment. Respondents were most satisfied with the reliability of their ERP systems and least impressed by the ERP system as an enabling technology for follow-on investments.
Effective ERP systems are delivered by slightly underperforming projects with regard to classic project management metrics (see Table 4 (See PDF) ). ERP projects do not achieve the full scope of original requirements with a reported mean gap of almost 15%. Plan performance is also lower than expected. Furthermore, more than every other project experienced organizational performance dips after switching the system to operational use.
Structure and levels of expended resources
ERP projects are resource-intensive and show high levels of variation in terms of expended resources. Table 5 (See PDF) provides all considered metrics divided into the justification and funding, and the implementation stages. The average time to complete an ERP project is 18.9 months with a standard deviation of 12.9 months. This finding compares well with a survey of Swedish ERP projects reporting a mean ERP project time of 17 months with high levels of variability between projects (Olhager & Selldin, 2003). In LEs, overall ERP project costs amount to EUR 1.2 million (RC1) plus EUR 265,000 worth of internal time consumption for the project (RL3). The Swedish study reported average total project costs of USD 1.68 million.
Only the resource metrics related to the implementation stage are significantly dependent on the size of the organizations (Mann-Whitney test, P <0.05), the metrics for the justification and funding stage are not. This is a surprising result. MEs show a different balance of resource investments over project stages. They need to invest proportionally more in the justification and funding stage than their larger counterparts. The ratio of implementation costs to overall costs is significantly lower in MEs (67.5%) than LEs (93.3%) (Mann-Whitney test, P <0.01).
ERP activators and teams
Senior managers are by far the most common ERP activators; the IT department plays a much less prominent role in this aspect (see Table 6 (See PDF) ). The other stakeholders known to potentially influence the launch of ERP projects do not make themselves heard. External stakeholders such as vendors and consultants surprisingly seem to have hardly any direct relevance in this early pre-appraisal stage of the project. A notable influence of other non-conventional initiators was especially identified in MEs.
The most common project-team type is biased and dominated by the IT department. However, participative forms are the most common project teams among LEs. The equal participation of major actors was also found to be essential for the success of IS projects (Besson & Rowe, 2001; Peffers et al , 2003).
Factor analysis
I applied exploratory PCA to the variables measuring conflicts and system-related performance criteria to reduce dimensionality and transform the correlated items into a reduced set of factors. This allowed for a more meaningful testing of the research hypotheses in the following section.
A sensible four-factor solution comprising change, resource, task, and integration conflicts emerged from a Varimax-rotated component matrix (see Table 7 (See PDF) ). Factors are marked by high loadings of their respective items, which are shaded in each column of the Table. This solution fits the discussed theory on conflicts, in which similar categories can be found, well (e.g. Besson & Rowe, 2001; Liu et al , 2011b). Each of the nine indicators loaded above the 0.5 threshold on its respective factor, with cross-loadings below this threshold (Hair et al , 2008) for all but one marginal case (IS6). The four factors accounted for 67% of the variance. Factor scores were used as composite variables in subsequent analyses providing information about the project's placement on the respective factors (Mîndrila, 2009).
The same procedure produced a sensible two-factor solution for system-related achievements representing a joint variance of 60% once I dropped one item (see Table 8 (See PDF) ). Again, factors are marked by high loadings of their respective items, which are shaded. I then distinguished between system quality and integration/information quality. Theoretically, this aligns well with the same two-tier quality dimension from the original D&M IS success model (DeLone & McLean, 1992). Again, regression scores for these two factors were used as consolidated measures in subsequent analyses.
Testing of research hypotheses
The role of ERP activators
ERP activators impact the functional balance of the project team in ERP adoptions (supporting Hypothesis 1a). Table 9 (See PDF) shows that ERP activators from each of the three business functions have the power to influence team composition in favor of their own functional home. For example, an ERP project initiation by stakeholders from the IT department leads to a project team that is dominated by the IT department. Participative teams, widely considered as the preferred team design for large-scale IS changes (Sarker & Lee, 2003; Ke & Wei, 2008), are most likely to develop if an internal organization department activates the ERP adoption project.
Next, we turn our attention to levels of expended resources, which in many cases depend on the role of the ERP activator (supporting Hypothesis 1b). Senior management initiates shorter projects, which are less resource-intensive in justification and funding and involve less external labor. IT departments trigger projects with lower overall costs. The organization department activates projects that are labor-intensive with regard to individual stages and both stages taken together. These projects also imply longer durations particularly during implementation (Table 10 (See PDF) ).
The role of expended resources
The levels of expended resources in ERP implementation depend on the levels of resources expended in the prior stage (supporting Hypothesis 2a). The findings from the Spearman rank correlation analyses show that every measure from one stage positively correlates with the same measure of the other stage. In addition, several positive and no negative correlations can be seen between one measure from one stage and another measure from the other stage. In other words, high levels of expended resources in the justification and funding stage are associated with high levels of expended resources in the implementation stage in all dimensions, namely in terms of time, internal and external labor use, and overall project costs (Table 11 (See PDF) ).
Contrary to Hypothesis 2b, high levels of expended resources are associated with lower levels of project effectiveness. This finding is supported by measures of project effectiveness from all three domains: Conflicts, System Quality, and Project Performance. Table 12 (See PDF) presents the results in more detail. The data revealed a pattern showing that the more resources a company assigns to a project, the more conflicts arise during implementation and the worse the project's results in terms of quality and project performance become. Conflicts and adverse project performance most distinctively occur when high levels of resources are expended in the ERP implementation stage.
Finally, I expected implementation conflicts to lead to lower project quality and performance (Hypothesis 2c). This assumption was supported by the data with the exception of integration conflicts. All other types of implementation conflicts led to adverse project outcomes in some aspect. In general, resource conflicts, which are characterized by time and cost escalations, are most distinctively related to less successful ERP projects. Change and task conflicts are associated with lower levels of operational performance in the early-use stage. The clear positive correlation between resource conflicts and plan performance validates the used research instrument. Resource conflicts during implementation explain that expended project efforts were higher than originally planned (Table 13) (See PDF) .
Discussion: the roles of ERP activators and expended resources
In this section, I discuss the major findings and make inferences according to the findings on the research hypotheses depicted in Table 14 (See PDF) , followed by elaborations on the limitations.
The role of ERP activators
In accordance with our expectations, the ERP activators generally impact the design of the project team (H1a) to the disadvantage of the organization. It is known that ERP projects benefit from balanced and participative team designs and need to account early for organizational resistance to change (e.g. Sarker & Lee, 2003; Ke & Wei, 2008). However, ERP activators regularly establish non-participative teams biased towards their own interest groups. The IT department most successfully establishes the most common form, IT-biased project teams. Organizations should establish more control over this team-building process. Methods from literature, such as the Critical Success Chains approach (Peffers et al , 2003), are available to foster cost-effective widespread inclusion of stakeholders in ERP projects.
ERP activators also influence the levels and balance of resource investments over the ERP justification and funding stages, and the implementation stages (H1b). Strategy-led projects triggered by senior management are indeed shorter and involve lower costs for external support. These projects tend to require less external knowledge through more explicit leadership in the project process (Sarker & Lee, 2003). The internal organization unit triggers many heavy-weight projects, which are labor-intensive and take longer to implement. Organizations need to be aware of the substantially higher resource and complexity implications mainly occurring during system implementation when pursuing original re-organization ideas.
The role of expended resources
Resource-intensive projects or so-called heavy-weight projects are characterized by resource-intensive justification and funding stages followed by resource-intensive implementation stages (H2a). This in particular applies to internal and external labor time investments. The use of external knowledge (most commonly through consultants) during the justification and funding stage increases the levels of external support needed for the implementation stage. Organizations may inadvertently develop a dependency on consultants by using their help during the appraisal of ERP. Theoretically, this situation applies to organizations with less developed dynamic capabilities to undergo an effective organizational change (Teece et al , 1997). Innovation theory argues that the successful adoption of IT depends on the organizational ability in fully assessing IT (Rogers, 2003). In this context, history matters. In other words, organizations that have not invested in analytical capacities in a quickly moving environment may have greater difficulties to assimilate ERP on their own.
Contrary to expectations, heavy-weight ERP projects are not more effective (H2b). They are related to higher levels of change and resource conflicts, lower levels of achieved system and integration quality, and lower early-use performance. Heavy-weight projects suffer from cost and time escalations and a shortage of skilled people. A trade-off between cost or time with quality within the Iron Triangle of project management does not work with ERP projects. Brooks' law, which predicts that incremental person-power added to a software-intensive project makes it longer, not shorter, seems to apply to ERP projects (Brooks, 1995).
Finally, resource, task and change conflicts during implementation are associated with lower project effectiveness in various dimensions (H2c). Resource conflicts are related to lower system quality, plan and scope performance. Change and task conflicts are related with early ERP use problems, which can be reduced by training measures (Jones et al , 2011). However, this is a very late measure to make ERP projects more effective. Control at an early stage can be achieved by the mentioned Critical Success Chains approach (Peffers et al , 2003), which widely includes participants at the project start without gravely increasing resource expenditures. Owing to the important role of conflicts, more research is needed to investigate when and how conflicts should be tackled and resolved (Meissonier & Houzé, 2010).
Limitations
Finally, it is important to note some limitations. The study is based on self-reported measurements known as the most common form of data collection in the social sciences (Malhotra et al , 2006). The effectiveness of the method is dependent on the respondents' willingness to pay attention and to answer as instructed. Lack of attention or rationality can interfere with inferences drawn from the data (Pokorny et al , 2001). I tried to mitigate this risk with preventive controls (offering incentives and inviting target persons), and with detective controls (screening for possible aberrant response behavior and analyzing CMV). To a certain extent, however, fluctuating and careless responding cannot be avoided in survey-based approaches. This equally applies to this study, in which the respondents needed to assess their ERP projects retrospectively.
On a different note, the mono-method research design did not accommodate data source triangulation by using multiple sources of data or different data gathering methods to ensure validity of the estimates given (Denzin, 1984; Yin, 2003). Consequently, the acquired responses are likely to be biased towards an internal manager's perception of ERP projects, which to some degree may have inflated reported ERP project success levels. However, the two-factor response bias analysis, which was in particular based on an ERP success metric, did not indicate any bias related to non-responses due to ERP failures. The survey instrument itself was validated by panel and expert discussions and the wording of questions (face validity) and appropriate scales were pre-tested. Related research reported no significant statistical differences between the views of different managerial ERP stakeholder groups (Ifinedo & Nahar, 2006), which supports a coherent internal management perspective on ERP projects presented in this paper.
Concluding remarks
This paper presents significant new evidence about the importance of ERP activators and expended resources, based on a large survey of senior managers conducted in Austria. This is one of the few studies that distinguish between investments for different ERP project stages. It was confirmed that early activators impact team formation and influence resource investment decisions made for the different stages of the project. Moreover, it was shown that these resource investments are interrelated and critical. Heavy-weight projects are less effective and troubled by numerous problems, in particular resource and change conflicts. These, in turn, are related to lower overall project effectiveness. Such insights are of particular importance now that organizations are pressured by stagnant markets and scarce resources while becoming increasingly dependent on changing IT requirements. Yet, they still experience significant issues with respect to larger-scale IT adoptions. While the findings will be of most significance to the organizations operating in the European Union, it is likely that they apply to other regions as well, as ERP projects are seen as global phenomena.
AuthorAffiliation
[1] Institute for Information Management and Control, Vienna University of Economics and Business, Vienna, Austria
Correspondence : Edward W.N. Bernroider, Institute for Information Management and Control, Vienna University of Economics and Business, Augasse 2-6, Vienna 1090, Austria. Tel: +43 (1) 31336 4466; Fax: +43 (1) 31336 746
References
Agarwal N and Rathod U (2006) Defining 'success' for software projects: an exploratory revelation. International Journal of Project Management 24 (4), 358-370.
Aloini D, Dulmin R and Mininno V (2012) Modelling and assessing ERP project risks: a petri net approach. European Journal of Operational Research 220 (2), 484-495.
Baki B and Çakar K (2005) Determining the ERP package-selecting criteria: the case of Turkish manufacturing companies. Business Process Management Journal 11 (1), 75-86.
Barney S, Petersen K, Svahnberg M, Aurum A and Barney H (2012) Software quality trade-offs: a systematic map. Information and Software Technology 54 (7), 651-662.
Bernroider EWN and Koch S (2001) ERP selection process in midsize and large organizations. Business Process Management Journal 7 (3), 251-257.
Bernroider EWN and STIX V (2006) On the applicability of data envelopment analysis for multiple attribute decision making in the context of information system appraisals. Communications of the International Information Management Association (CIIMA) 6 (2), 105-116.
Berry DTR, Wetter MW, Baer RA, Larsen L, Clark C and Monroe K (1992) MMPI-2 random responding indices: validation using a self-report methodology. Psychological Assessment 4 (3), 340-345.
Besson P and Rowe F (2001) ERP project dynamics and enacted dialogue: perceived understanding, perceived leeway, and the nature of task-related conflicts. The DATA BASE for Advances in Information Systems 32 (4), 47-66.
Boonstra A (2006) Interpreting an ERP-implementation project from a stakeholder perspective. International Journal of Project Management 24 (1), 38-52.
Brooks FP Jr. (1995) The Mythical Man-Month: Essays on Software Engineering . Addison-Wesley, Reading, MA, 322 pages.
Bureau-van-Dijk EP (2003) Amadeus. [Online]. Bureau-van-Dijk. (accessed January).
Buxmann P and König W (1997) Empirische Ergebnisse zum Einsatz der betrieblichen Standardsoftware SAP R/3. WIRTSCHAFTSINFORMATIK 39 (4), 331-338.
Davenport TH (1993) Process Innovation: Reengineering Work through Information Technology . Harvard Business School Press, Boston, MA.
Delone WH and Mclean ER (1992) Information systems success: the quest for the dependent variable. Information Systems Research 3 (1), 60-95.
Denzin N (1984) The Research Act . Prentice-Hall, Englewood Cliffs, NJ.
Dillman DA (1978) Mail and Telephone Surveys: The Total Design Method . John Wiley and Sons, New York.
EC (2003) Commission recommendation of 6 May 2003 concerning the definition of micro, small and medium-sized enterprises. Official Journal of the European Union 46 (L 124), 36-41.
Grohs E (2003) Standards und Trends in der Unternehmensberatung auf dem Gebiet des Controlling in Österreich , Fachhochschule Wien, Vienna, Austria.
Gunasekaran A, Ngai EWT and Mcgaughey RE (2008) Information technology and systems justification. In Evaluating Information Systems (Irani Z and Love P, Eds), chapter 1, pp 1-34, Butterworth-Heinemann, Oxford.
Hair JF, Anderson RE, Tatham RL and Black WC (1984) Multivariate Data Analysis . Prentice-Hall, New Jersey.
Hair JF, Black WC, Babin BJ, Anderson RR and Tatham RL (2008) Multivariate Data Analysis . Pearson Education, London.
Hsu JS, Liang TP, Wu SPJ, Klein G and Jiang JJ (2011) Promoting the integration of users and developers to achieve a collective mind through the screening of information system projects. International Journal of Project Management 29 (5), 514-524.
Ifinedo P and Nahar N (2006) Prioritization of enterprise resource planning (ERP) systems success measures: viewpoints of two organizational stakeholder groups. In Proceedings of the 2006 ACM Symposium on Applied Computing (Haddad HM, Chbeir R, Ossowski S, Wainwright SL, Liebrock LM, Palakal MJ, Yetongnon K and Nicolle C, Eds), pp1554-1560, ACM, Dijon, France.
Jehn KA and Bendersky C (2003) Intragroup conflict in organizations: a contingency perspective on the conflict-outcome relationship. Research in Organizational Behavior 25 , 187-242.
Jha KN and Iyer KC (2007) Commitment, coordination, competence and the iron triangle. International Journal of Project Management 25 (5), 527-540.
Johnstone D, Huff S and Hope B (2006) IT projects: conflict, governance, and systems thinking. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences HICSS '06 , p 197b, Vol. 8Kauai, HI, USA.
Jones DC, Kalmi P and Kauhanen A (2011) Firm and employee effects of an enterprise information system: micro-econometric evidence. International Journal of Production Economics 130 (2), 159-168.
Ke W and Wei KK (2008) Organizational culture and leadership in ERP implementation. Decision Support Systems 45 (2), 208-218.
Kumar V, Maheshwari B and Kumar U (2002) ERP systems implementation: best practices in Canadian government organizations. Government Information Quarterly 19 (2), 147-172.
Lee JC and Myers MD (2004) Dominant actors, political agendas, and strategic shifts over time: a critical ethnography of an enterprise systems implementation. The Journal of Strategic Information Systems 13 (4), 355-374.
Liu JY-C, Chen H-G, Chen CC and Sheu TS (2011a) Relationships among interpersonal conflict, requirements uncertainty, and software project performance. International Journal of Project Management 29 (5), 547-556.
Liu JYC, Chiang JC, Yang M-H and Klein G (2011b) Partnering effects on user-developer conflict and role ambiguity in information system projects. Information and Software Technology 53 (7), 722-729.
Mabert VA, Soni A and Venkataramanan MA (2006) Model based interpretation of survey data: a case study of enterprise resource planning implementations. Mathematical and Computer Modelling 44 (1-2), 16-29.
Malhotra NK, Kim SS and Patil A (2006) Common method variance in IS research: a comparison of alternative approaches and a reanalysis of past research. Management Science 52 (12), 1865-1883.
Markus ML, Axline S, Petrie D and Tanis C (2000a) Learning from adopters' experiences with ERP: problems encountered and success achieved. Journal of Information Technology 15 (4), 245-265.
Markus ML and Tanis C (2000) The enterprise systems experience: From adoption to success. In Framing the Domains of IT Research: Glimpsing the Future Through the Past (Zmud RW, Ed), pp.173-207, Pinnaflex Educational Resources Inc., Cincinnati.
Markus ML, Tanis C and Van Fenema PC (2000b) Multisite ERP implementations. Communications of the ACM 43 (4), 42-46.
Marques G, Gourc D and Lauras M (2011) Multi-criteria performance analysis for decision making in project management. International Journal of Project Management 29 (8), 1057-1069.
Maylor H (2010) Project Management . Prentice-Hall, London.
Mcafee A (2002) The impact of enterprise information technology adoption on operational performance: an empirical investigation. Production & Operations Management 11 (1), 33-53.
Mclaren T and Jariri I (2012) Stakeholder assessment and management for enterprise systems implementation projects. In Proceedings of the International Conference of Information Resources Management (Conf-IRM) (Brandtweiner R and Lech J, Eds), paper 68, pp 1-8, Vienna, Austria.
Meissonier R and Houzé E (2010) Toward an 'IT conflict-resistance theory': action research during IT pre-implementation. European Journal of Information Systems 19 (5), 540-561.
Mîndrila D (2009) Understanding and using factor scores: considerations for the applied researcher. Practical Assessment, Research & Evaluation 14 (20), 1-11.
Olhager J and Selldin E (2003) Enterprise resource planning survey of Swedish manufacturing firms. European Journal of Operational Research 146 (2), 365-373.
Panorama Consulting Group (2009) 2008 ERP Report . Panorama Consulting Group, Denver.
Peffers K, Gengler CE and Tuunanen T (2003) Extending critical success factors methodology to facilitate broadly participative information systems planning. Journal of Management Information Systems 20 (1), 51-85.
Podsakoff PM, Mackenzie SB, Lee J-Y and Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology 88 (5), 879-903.
Podsakoff PM and Organ DW (1986) Self-reports in organizational research: problems and prospects. Journal of Management 12 (4), 69-82.
Pokorny SB, Jason LA, Schoeny M, Curie CJ and Townsend SM (2001) Eliminating invalid self-report survey data. Psychological Reports 89 (1), 166-168.
Purdon S and Pickering K (2001) The use of sampling weights in the analysis of the 1998 workplace employee relations survey. London, National Centre for Social Research.
Rogers EM (2003) Diffusion of Innovations . Free Press, New York.
Sarker S and Lee AS (2003) Using a case study to test the role of three key social enablers in ERP implementation. Information & Management 40 (8), 813-829.
Sharma R, Yetton PW and Zmud RW (2008) Implementation costs of IS-enabled organizational change. Information and Organization 18 (2), 73-100.
Sprent P and Smeeton NC (2007) Applied Nonparametric Statistical Methods . Chapman and Hall/CRC, Boca Raton, FL.
Stefanou CJ (2001) A framework for the ex-ante evaluation of ERP software. European Journal of Information Systems 10 (4), 204-215.
Stockdale R, Standing C, Love PED and Irani Z (2008) Revisiting the content, context and process of IS evaluation. In Evaluating Information Systems (Irani Z and Love P, Eds), chapter 2, pp 35-48. Butterworth-Heinemann, Oxford.
Teece DJ, Pisano G and Shuen A (1997) Dynamic capabilities and strategic management. Strategic Management Journal 18 (7), 509-533.
Thompson AH (1975) Random responding and the questionnaire measurement of psychoticism. Social Behavior and Personality 3 (2), 111-115.
Thurstone LL (1934) The vectors of mind. Psychological Review 41 (1), 1-32.
Umble EJ, Haft RR and Umble MM (2003) Enterprise resource planning: implementation procedures and critical success factors. European Journal of Operational Research 146 (2), 241-257.
Uwizeyemungu S and Raymond L (2009) Exploring an alternative method of evaluating the effects of ERP: a multiple case study. Journal of Information Technology 24 (3), 251-268.
Van Der Stede WA, Mark Young S and Xiaoling Chen C (2006) Doing management accounting survey research. In Handbooks of Management Accounting Research (Christopher S, Chapman AGH and Michael DS, Eds), vol. 1, pp 445-478, Elsevier, Amsterdam.
Wu J-H and Wang Y-M (2006) Measuring KMS success: a respecification of the DeLone and McLean's model. Information & Management 43 (6), 728-739.
Yang H-L and Tang J-H (2004) Team structure and team performance in IS development: a social network perspective. Information & Management 41 (3), 335-349.
Yin RK (2003) Case Study Research - Design and Methods . SAGE Publications, London.
Appendix
Appendix
Tables A1 (See PDF) , A2 (See PDF) , A3 (See PDF)
(Table Omitted: See Article Image)
About the author
Edward W.N. Bernroider is a University Professor in Information Management and Control at the Vienna University of Economics and Business in Austria. His research focuses on Accounting Information Systems, and IT Governance, Risk and Compliance. He has engaged in a variety of educational programs, international consultancies, and advisory activities for commercial and nonprofit enterprises, and made numerous presentations at national and international conferences. His publications have appeared in journals such as Decision Support Systems , Information and Management , European Journal of Information Systems , European Journal of Operational Research , Computers and OR , and the Business Process Management Journal. He currently serves on the editorial board of three major IS journals and is a regular member of academic and professional bodies in the field of IS.