e-Education Systems Implementation Success Model

Saadiah Yahya and Noor Habibah Arshad
Universiti Teknologi MARA (UiTM), Selangor, Malaysia

Nor Zaila Wahab
Ministery of Education, Malaysia

ABSTRACT

Originally developed for delivering distance learning programs, e-education systems in their current forms are powerful, web-based information systems with capabilities that match Enterprise Resource Planning (ERP) systems. By carrying out their deployments on an enterprise-level basis, many Higher Education Institutions (HEI) are able to leverage on their powerful functionalities and ERP-like integrative capabilities to put together academic and administrative systems (Twigg 2002). 

This study is proposing that it may no longer relevant to still evaluate e-education systems as only educational technologies in view of the current implementation environment where new forms of e-education systems are now information systems of strategic relevance in many HEI (Moul 2002). E-education systems should therefore be studied as: mainstream information systems, i.e. within the paradigms of information systems evaluation research and no longer as educational technology or innovation. This study looks into information systems success model that can be developed to explain and predict the success of e-education systems implementation. It identifies constructs to measure e-education systems implementation success; investigates the appropriate dependent variable as a proxy for e-education systems implementation success and finally develops and validates the model. The findings from this research, showed that the ‘organizational IS continuance’ construct is a viable proxy for information systems implementation success and the model developed can be equally applied and generalizable across different stakeholder groups.

Keywords: e-Education System Success Model; Perceived Organizational Benefits; Perceived Usefulness; Expectation Confirmation; Organizational IS Continuance

BACKGROUND TO THE RESEARCH

Education organizations unlike business organizations are very slow to adopt new information technologies (IT). Such changes may also face resistance and rejection from faculty members (Jones 2000). Hence,while e-education systems initiatives are actively underway in many universities, the high risks, uncertainties and many examples of failed implementations (Paulsen 2002) associated with these systems, appropriate evaluation of these systems then remains an important agenda for higher education management if they wish to see a good return for their costly investments.

RESEARCH QUESTIONS AND OBJECTIVES

By tradition e-education systems are regarded as educational technologies and their evaluation are more on their efficacies as teaching and learning tools rather than as organizational information systems. This study believes that research should inform on new ways to evaluate e-education systems based on their latest roles as important organizational information systems. Since e-education systems have improved the operational management of teaching and learning processes; institutional performance and in many instances as key institutional transformation drivers (Wang, et al. 2007; Zemsky & Massey 2004; Twigg  2002; Oblinger, et al.  2001), it is apparent that a new evaluation perspective is needed.

Objectives of this study are to:

• identify constructs to measure e-education systems implementation success;
• identify most appropriate dependent variable as a proxy for e-education systems implementation success; and
• develop a model of e-education systems implementation success.

E-EDUCATION SYSTEMS IMPLEMENTATION SUCCESS AND RELATED EFFORTS

This study represents a confluence of two disciplines, educational technology research and information systems research. Within the educational technology literature, web-based education systems are studied as education innovations based on the diffusion of innovation perspective (Rogers 1995; Surry & Brennan 1998). As innovations, their adoptions are usually investigated at the individual level since adoption of an innovation is concerned with an individual’s behavior during the innovation diffusion process (Rogers 1983). Also, their efficacy and effectiveness are usually measured along the lines of pedagogical value for learners (Pulkinnen 2003; Lund & Volet 1998).

In parallel, adoption of information systems has often been studied under technology acceptance (Davis 1989; Bhattacherjee 2001) within a social context or environment such as the institution. In these studies, the adoption or acceptance of these technologies by individual users is used as an indicator for their implementation success within the institution. Implementation success is synonymous to user acceptance and their decision to proceed with systems use or continuance. Continuance of information systems or sustained use is analogous to post-adoption use after the initial adoption decision. Post-adoption of information systems has been identified as more salient for effective organizational performance and their survivability than first-time adoption (Bhattacherjee 2001). The importance of continuance as opposed to acceptance or initial adoption decision is evident in many e-commerce firms where the cost of acquiring new customers may cost as much as five times more than retaining existing ones (Parthasarathy and Bhattacherjee 1998). In addition, Kwon and Zmud (1987) suggested that organizations would benefit more from their information systems implementation when the technology has reached the diffusion level of infusion. At this level of diffusion, the information systems use is routinized and sustained in the organization. Since Rogers’ (1983) diffusion of innovation (DOI) theories are widely used in both educational technology and information systems implementation, it is the main referent theory for this study. 

In addition to DOI theories, this study is also framed within the information systems (IS) success literature in which DeLone and McLean (1992, 2003) and Seddon’s (1997) reiterated IS success model are the main referent theories. The IS success literature provides understanding of how information systems can be evaluated, the criteria that guide evaluation and the process of how information systems lead to more individual and organizational effectiveness. On the other hand, the IS evaluation research literature provided the understanding with regards to information systems benefits and their realization. ISs benefits are unique to the particular technology and benefits realization is dependent on other factors such as organizational support and successful management change (Ward & Griffiths 1996; Ballantine & Stray 1998; Remenyi, et al., 1997; Lederer & Mirani 1998). Review of the educational technology literature and institutional studies on higher education, specific benefits of e-education systems were identified and classified. This study’s objectives culminate in the development of an e-education systems implementation success model. In conclusion, the theoretical framework which guided the development of the research model is largely based on the IS evaluation literature.

THE PROPOSED E-EDUCATION SYSTEM SUCCESS MODEL

The research model proposed and validated in this study is an extension of Bhattacherjee (2001) Expectation-Confirmation model of IS Continuance. The study showed that information systems continuance intention result of perceived usefulness, expectation confirmation and user satisfaction. However, a system can be considered successful or valuable only when it can yield benefits or positive impacts to the organization and its stakeholders. Therefore, the proposed research model of e-education systems implementation success utilized both previously used constructs perceived usefulness (Davis, 1989), as well as newly developed constructs which are adapted from previous work. The adapted and extended constructs are perceived organizational benefits (DeLone & McLean 1992; Shang & Seddon 2003; Seddon 1997); organizational information systems continuance (Bhattacherjee 2001); and expectations confirmation (Bhattacherjee 2001). The proposed model is given in Figure 1.

Figure 1: Proposed Causal Model - e-Education Systems Success

Since the theories behind the research model are well established and supported in the literature, the confirmatory factor analysis approach using structural equation modeling (SEM) techniques was seen as appropriate to be used in the data analysis to assess the psychometric properties of research instruments and validation of the research model. It was also chosen because it provides the required statistical rigor for theory testing studies (Boudreau et al. 2001). Teo et al. (2003) describes SEM techniques as a useful approach in model validation due to its emphasis on overall model fitting, which is well-suited for testing theory as a whole.

The site selected for the survey is a public Malaysian higher education institution which had implemented an education enterprise portal, which is an example of an e-education system, for the last four years. The length of implementation period is an important criterion for site selection because previous IS studies have shown that for large complex systems organizational impacts are realized only after a period of at least three years after implementation (Shang & Seddon 2001). The selected site fulfilled the two criteria required.

KEY ASSUMPTIONS AND RESEARCH METHODOLOGY

Since the area of e-education systems evaluation is an understudied area, the current study suffers from several theoretical and methodological limitations. Two major ones are:

• This study is exploratory because it was a first attempt to test a causal model through a cross-sectional survey of e-education systems users. This method may not be appropriate because cross-sectional field surveys by their nature tend to limit the generalizability of their results when used to investigate changes in behaviour over time (Pinsonneault & Kraemer 1992). In contrast, longitudinal survey which collects data for at least two points in time is better suited for these purposes. Thus, longitudinal design provides greater confidence for causal inferences than cross-sectional design as used in this study. Future research in this area should consider longitudinal design if the survey method is chosen.
• Most previous studies on IS implementation studies in organizations were based on simple stand-alone voluntary information systems (e.g. TAM, Davis, 1989). As such, instruments adapted from these studies such as those used to measure the perceived usefulness construct may not have been appropriate for large complex, mandatory systems such as e-education systems.

The methodology used to develop the e-education systems benefits classification framework followed Shang and Seddon (2002) methods for identifying ERP benefits. Methods for assessing psychometric soundness of the proposed e-education systems benefits groupings followed those proposed by Mirani and Lederer (1998). The development and validation of e-education systems implementation success model followed the methodology used by Churchill (1979) criteria for construct measurement. Assessment of psychometric properties of the study’s instruments and in validating the hypothesized relationships between variables in the research model followed a confirmatory factor approach using structural equation modeling techniques. These analyses had followed closely Geffen’s et al. (1995), and Shih’s (1994) criteria for using the CFA approach employing structural equation modeling techniques.

PILOT AND PROPER SURVEY

Before the survey was carried out, an application to conduct the survey was made to the Malaysian Ministry of Higher Education. Once permission from the Ministry was obtained, a pilot study was conducted. The pilot study was carried out to test the validity of the questionnaire. It was conducted through a web survey to test the feasibility of conducting the survey proper over the Internet. The participants for the pilot survey were from three selected universities i.e. one from private university and two from public universities. However, the overall response for the pilot survey was very poor. For the private university, the response rate was 14.3%. For public university A, it was 36%, while for public university B, it was 8%. The results from the pilot indicated that a web survey would probably fail to gather enough responses for the study. Based on the response rate and comments from some participants who were contacted after the survey, it was decided that the survey would not be conducted over the Internet but would be a combination of paper-based survey distributed manually and in electronic format (MS Word) sent through emails. Another result of the pilot showed that public university A, which had a response rate of 36% would most probably have enough willing participants for the study as compared to the other two universities. In addition, the response of the private university to the researcher’s request to conduct the study in the particular institution was late thus prompting the decision to choose public university A as the appropriate site for the survey.

A questionnaire described as a survey of end-users perceptions on e-education systems implementation was distributed to two groups of end-users comprising lecturers and students from three faculties, the engineering; economics and management sciences; and information and communication technology faculties respectively. For the student group, the questionnaires were distributed to students in the three faculties who were approached at random by the researcher during a three day period. In total, 115 questionnaires were completed out of 250, representing a response rate of 58%. For the lecturers’ group, 100 questionnaires were distributed manually with the help of a lecturer from the economics and management sciences faculty, while the electronic form of the questionnaire was distributed through emails to 150 lecturers from the three faculties selected (engineering, economics and management sciences; ICT). The survey that was distributed through emails was conducted over two rounds where those who did not respond to the first email, were sent a reminder. A total of 147 responses were received from the lecturers group indicating a response rate of 59.0%.

DATA ANALYSIS FOR VALIDATING E-EDUCATION SYSTEMS SUCCESS MODEL

A rule of thumb found in the literature on the appropriate sample size is that it should be at least eight times the number of variables in the model (Jaccard & Wan  1990). Stevens (1996) suggests that each measured variable should have at least 15 cases while Bentler and Chou (1987) recommend at least five cases per parameter estimate (including error terms as well as path coefficients). The number of samples for this study (262 cases) was within the required minimum of samples as suggested by Jaccard and Wan (1996), Stevens (1996) and Bentler and Chou (1987).

While this study adopted validated and reliable instruments used in earlier research where possible, the demanding requirements of psychometric validity under the guidelines suggested in the literature (Churchill 1979; Nunally 1978; Cronbach 1951) were followed closely.

This study’s final finding showed the applicability of the model across different groups. The information systems success literature has shown that information systems success is a subjective variable where different users will give different views based on their personal preference and values (DeLone & McLean 1997; Seddon 1997). Hence it was suggested that objectives measures of IT/IS success may be impossible to arrive at and is best assessed by the use of behavioral proxies such as IT/IS satisfaction and intentions to use (Davis, et al., 1989). Table 1 summarizes the results of path analysis and support for hypotheses.

Table 1: Results of Path Analysis and Support for Hypotheses

RESULTS AND FINDINGS

The results of this part of the analysis revealed that all assumptions of the model were confirmed by the data. ‘Perceived organizational benefits’ and ‘perceived usefulness’ explained 59 percent of the variance in ‘expectations-confirmation’, while all three constructs together explained about 84 percent of the variance in the dependent variable, ‘organizational IS continuance’. The high percentage of variance indicated that the latent constructs of ‘perceived organizational benefits’, ‘perceived usefulness’ and ‘expectations-confirmation’ have significantly explained data variations for ‘organizational IS continuance’ and its underlying dimensions. Support for the power of the model in explaining the causality or direction of the relationships was further expounded with the examination of the goodness of fit indices of the model. Examination of the selected fit indices demonstrated that the model fitted the data well, indicating that the causal direction and relationships between the exogenous (‘perceived organizational benefits’, ‘perceived usefulness’) and the endogenous (‘expectations-confirmation’, ‘organizational IS continuance’) latent variables as proposed in the research model had statistical validity.

While the variance estimates provided statistical evidence to the predictive powers of the research model, path analysis was conducted to test the hypothesized relationship of the variables within the causal structure of the model. All path coefficients were found to be in the direction specified and were significantly large. This showed that the data provided support for hypotheses one to six. For the last hypothesis, which sought to validate that the research model can be applied or is generalizable across all groups of stakeholders (students and lecturers), multi-group analysis involving a Chi-square difference test was carried out. The results of this analysis revealed that there was no statistical evidence to accept the null hypothesis that the model is not applicable to different groups of stakeholders. 

In conclusion, results of the analysis carried out indicated that (1) the model has significant explanatory powers; (2) the hypothesized relationships and its direction are found to have empirical support; (3) there is empirical evidence to support hypothesis one to six; and (4) the model is generalizable across all groups of stakeholders.

Sustained usage is more important than first time use after users’ initial adoption decision. This is because users may change their intention of using the system in the future after initially accepting it. Sustained usage is an indication of routinization as well as a factor leading to routinization. Furthermore, if a technology has been routinized in the organization, it is in effect the final outcome of a successful information system implementation. Routinization was described by Zmud and Apple (1989) as the stage that must be reached for the information system to be judged a success. This study likewise had focused on sustained usage of e-education systems usage behavior or its routinization as a proxy for implementation success.

SUMMARY OF FINDINGS

The preceding paragraphs have explicated several major findings with regards to the overall theoretical framework of the research model. These are summarized as follows: 

1. The overall model fitness and significant paths between the variables showed that model can be an appropriate model to explain e-education systems implementation success
   
2. The overall model fitness and significant paths between the variables showed that the model can be an appropriate model to explain sustained information systems use
   
3. This study had successfully extended Bhattacherjee’s (2001) expectation confirmation model of IS continuance to include organizational-level and individual-level constructs encompassing a technology innovation process view of e-education systems implementation within a quasi-mandatory and full mandatory information systems usage.
4. The research model had been shown to be useful in explaining the relationship or chain of events that leads to the final decision or users’ intention based on the significant paths between ‘perceived organizational benefits’ and ‘perceived usefulness’ to ‘expectations confirmation’.
   
5. Within the research model, it was found that, both exogenous variables, ‘perceived usefulness’ and ‘perceived organizational benefits’ are significant predictors for organizational e-education systems continuance hence its implementation success. Between these two, however ‘perceived usefulness’ is the stronger predictor based on its more significant path coefficient (R2 = 0.38) as compared to ‘perceived organizational benefits’ (R2 = 0.13).
6. This study found empirical support for the assumptions that ‘perceived usefulness’ and ‘perceived organizational benefits’ appropriately represent individual-level and organizational-level constructs to predict e-education systems implementation success.
7. The study was able to show that ‘perceived organizational benefits’ is an appropriate construct to represent the organizational views of benefits or IT value of e-education systems.
8. This study had shown that the significant relationship between ‘perceived organizational benefits’ to the dependent variable, which represents sustained system use, had provided support for the IT/IS use- IT/IS business value linkage proposed by Cronk and Fitzgerald (1999).
9. This study’s e-education benefits classification typology and its subsequent validation provided support to Orlikowski (1999), Davenport et al. (2000), Brown (2005) and similar others findings that IT benefits are specific to the technology and its context of use where identifying them are salient for implementation success.
10. The difference in paths strengths between ‘perceived organizational benefits’ and ‘perceived usefulness’ and the dependent variable provided support for Ciborra’s (2000) technological drift theory.
    
11. The study’s results shows that there is significant empirical evidence to support the assumption that the ‘organizational IS continuance’ construct is a viable proxy for information systems implementation success (Variance explained, R2 = 0.84).
12. The study’s results show that there is significant empirical support that the model can be equally applied and generalizable across different stakeholder groups.

DISCUSSIONS AND RECOMMENDATIONS 

This research has put forth arguments that e-education systems, when implemented on an enterprise level, has the power to transform the education organization and make good on its role in providing education that is relevant to the digital and knowledge economy. Thus, it has become imperative for education organizations that have invested so much in e-education systems to be able to gauge whether their e-education systems are paying off in terms of benefits not only to the teaching and learning aspects but also towards overall organizational performance. In view of these developments, this study has presented arguments for e-education systems research to be carried out based on the perspectives of mainstream information systems research. The practice of basing research of e-education systems through the perspectives of educational technology does not reflect the realities of e-education systems implementation in universities today.

The main contribution of this study is the e-education systems implementation success model. Results of the data analysis indicated that the proposed benefits dimensions and relationships between variables in the model have empirical support. The study had successfully applied behavioural and diffusion of innovation theories to a system (e-education systems) implementation area and provided an appropriate theoretical framework for future research. The research model is comprehensive since it has behavioural variables at two levels of analysis, individual- and organizational-levels.

There are three key limitations in making the results of this study more significant in its empirical validity. They are (1) the cross-sectional nature of data collected; (2) stakeholders group not representative enough of the university’s organizational members; (3) non-collection of data related to specific organizational attributes and the education industry.

The results of this study have four implications for research. They are: (1) e-education systems research should be examined through the perspectives of mainstream information systems research; (2) the new variables in this study need to be further examined using behavioural theories as in technology acceptance and information systems implementation research; (3) future research on e-education systems to focus on technical characteristics to better understand its diffusion process; and (4) to inform future information systems researchers on the importance of understanding information systems benefits.

REFERENCES

Ballantine, J., and Stray, S. 1998. “Financial appraisal and the IS/IT investment decision-making process”. Journal of Information Technology, 13, pp. 3-14.

Bentler, P., & Chou, C. 1987. “Practical issues in structural modeling”. Sociological Methods and Research, 16, pp. 78-117.

Bhattacherjee. 2001. “Understanding information systems continuance: An expectation-confirmation model”. MIS Quarterly, 25 (3), pp. 351-370.

Churchill, G. 1979. “A paradogm for developing better measures for marketing constructs”. Journal of Marketing Research, 10, pp. 64-73.

Davis, F. 1989. “Perceived usefulness, perceived ease of use, and user acceptance of information technologies”. MIS Quarterly, 13 (3), pp. 319-340.

Davis, F., Bagozzi, R. and Warshaw P. 1989. “User acceptance of computer technology: a comparison of two theoretical models”. Management Science, 35 (8), pp. 982-1003

DeLone, W. and McLean, E, 1992. “The DeLone and McLean Model of Information Systems Success: The quest for the dependent variable”. Information Systems Research, 3 (1), pp. 60-95

DeLone, W. and McLean, E. 2003. “The DeLone and McLean model of information systems success: a ten-year update”. Journal of Management Information Systems, 19 (4), pp. 9-30.

Farquhar, J., and Surry, D. 1999. “Adoption analysis: An additional tool for instructional developers”. Education and Training Technology International, 31 (1), pp. 19-25.

Geffen, D., Straub, D., and Boudreau, M. 2000. “Structural equation modeling techniques and regression: Guidelines for research practice”. Communications of AIS, 4.

Jaccard, J., Turrisi, R., and Wan, C. 1990. Interaction effects in multiple regression. Newbury Park, CA: Sage Publications.

Jones, D. 2000, “Emergent Development and the Virtual University”, Proceedings of Learning 2000: Reassessing the Virtual University Conference, September 28--October 1, 2000, Virginia Tech,  Roanoke, VA.  

Kwon, T., and Zmud, R. 1987. “Unifying the fragmented models of information systems implementation”. In R. B. Hirschheim (Ed.), Critical Issues in Information Systems Research, Chichester,  Wiley (pp. 88-97).

Lund, C., and Volet, S. 1998. “Barriers to study online for the first time: Students' perceptions”. In McBeath, & Atkinson (Ed.), The Biennial Conference of the Australian Society for Educational Technology, July 5-8, 1998. Perth, Western Australia.

Mirani, R., and Lederer, A. 1998. An instrument for assessing organizational benefits of information systems projects. Decision Sciences, 29 (4), pp. 803-838.

Moul, B. 2002, Administrative and Academic Information Systems: A shared vision. Campus Technology, viewed 12 May 2003,  <http://www.campustechnology.com.article>

Nunally, J. 1978. Psychometric Theory. New York: McGraw-Hill.

Parthasarathy, S., and Bhattacherjee, A. 1998. “Understanding post-adoption behavior in the context of online services”. Information Systems Research, 9 (4), pp. 362-379.

Paulsen, F. M., 2002, NKI Distance Education,  viewed 21Jun 2005, <http://home.nettskolen.com/~morten>.

Pinsonneault, A. and Kraemer, K. 1992. “Survey research methodology in management information systems: An assessment”. Working paper for Queen's HEC workshop, Ecole des Hautes Etudes Commerciales de Montreal and the University of California at Irvine. Montreal, Canada.

Pulkkinen, J. 2003. The paradigms of e-education, an analysis of the communication structures.  Research Dissertation on information and communication technology integration in education in the years 2000-2001. University of Oulu, Finland, Faculty of Educational Sciences and Teacher Education.

Remenyi, D., Sherwood-Smith, M., and White, T. 1997.  Achieving Maximum Benefits from your Information Systems. Chichester: Wiley and Sons.

Rogers, E. 1995. The diffusion of innovations (3rd ed.). New York: Free Press.

Seddon, P. 1997. “A respecification and extension of the DeLone and McLean model of information systems success”. Information Systems Research, 8 (3), pp. 240-253.

Shang, S., and Seddon, P. 2002. “Assessing and managing the benefits of enterprise systems: The business manager's perspective”. Information Systems Journal, 12,  pp. 271-299.

Shih, H. 1994. “Extended technology acceptance model of internet utilization behavior”. Information & Management, 41 (6),  pp. 719-729.

Stevens, J. 1996. Applied Multivariate Statistics for the Social Sciences.

Surry, D., & Brennan, J. 1998.  Diffusion of instructional innovations: Five important unexplored question.

Teo, H., Wei, K., & Benbasat, I. 2003. ”Predicting intention to adopt interorganizational linkages: An institutional perspective”. MIS Quarterly, 27 (1), pp. 19-49.

Twigg, C. 2002. “The impact of changing economy on four-year institutions of higher education: The importance of the Internet”. In P. Graham, & N. Stacey (Eds.), The Knowledge Economy and Post-Secondary Education: Report of a Workshop (pp. 77-104). Washington, DC: National Academy Press.

Wang et al. 2007. “Measuring e-learning system success in an organizational context: Scale development and validation”. Computers in Human Behavior, vol. 23, Iss. 4, pp 1792-1808.

Ward, J., & Griffiths, P. 1996. Strategic planning for information systems. Chichester, UK: John Wiley & Sons.

Zemsky, W., & Massy, W. 2004. Thwarted innovation: What happened to e-learning and why. Report, University of Pennsylvania, The Weatherstation Project of the Learning Alliance.

Copyright for articles published in this journal is retained by the authors, with first publication rights granted to the journal.
By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings. Original article at: http://ijedict.dec.uwi.edu//viewarticle.php?id=601&layout=html