Students' perceptions of prehospital web-based examinations

Brett Williams
Monash University , Victoria , Australia

ABSTRACT

This study examined the attitudes of prehospital undergraduate students undertaking a web-based examination (WBE) as an adjunct to the traditional paper-based examination (PBE). Following the completion of the WBE second year Bachelor of Emergency Health (BEH) undergraduate students at Monash University (n = 94) were asked to complete a questionnaire which was designed to obtain information about students' attitudes of WBE. Quantitative results produced high student satisfaction and acceptance of WBE as an appropriate teaching and assessment resource in the BEH degree. Generally, students found the WBE experience to be very positive and preferred WBE to PBE.

Keywords: Prehospital education; higher education; web-based examinations; paper-based examinations

INTRODUCTION

With financial constraints now being placed on higher education sector, this has forced staff and administrators to develop more cost-effective programs, which can be delivered in a more flexible environment without reducing the sophistication of educational pedagogical approaches. Other areas that have emerged have been programs that involve collegial merging of partnerships and more efficient ways of conducting non-teaching activities such as written examinations.

One teaching and learning solution postulated has been the online teaching and learning paradigm. Many institutions now offer many or part thereof in both undergraduate and postgraduate studies. This paradigmatic shift however, has not been without pedagogical obstacles or resistance by teaching staff, with terms such as 'Digital Hemlock' and 'Sophisticated Obstacle' being associated with online teaching, learning and assessment.

Aims

This study was undertaken in two phases comparing attitudes of second year BEH prehospital undergraduate students. The BEH degree is pre-employment or pre-registration professionally oriented undergraduate degree, offered full-time on campus by Monash University . The goal of the study was to assess BEH students' attitudes and perceptions of WBE and comparing WBE to PBE.

Literature Review

A literature review was undertaken using several databases: MEDLINE, EMBASE and Cochrane Library. The following MeSH terms were used in the search: web-based examination, web-based testing, computer-assisted instruction, computer testing, online testing, internet testing, EMS, paramedic, prehospital and out-of-hospital.

No papers were obtained under designated prehospital filters in MEDLINE, EMBASE or ERIC. A second tiered search response was undertaken with an improved outcome using an un-filtered search approach. WBE literature does exist in other health science disciplines, however this is the first study of its kind relating to prehospital undergraduate students

BEH DEGREE

The BEH degree is a full-time on-campus degree completed over two years (accelerated entry) or three years (standard entry). Students receive education relating to prehospital practice, clinical epidemiology, public and population health and professionalism issues with a strong emphasis towards a more general community-based health approach.

The BEH program is a newly formed thematic degree offered at Monash University Department of Community Emergency Health and Paramedic Practice (DCEH&PP). The degree offers students the opportunity to develop skills, knowledge and an understanding of the attributes required of a prehospital practitioner. Many students undertake this degree in anticipation of becoming an ambulance paramedic; however the unique nature of this degree allows students to consider other community-based occupations with the program offering five themes as highlighted below.

•  Theme 1. Science, knowledge and evidence

•  Theme 2. Population health and illness in society

•  Theme 3. Foundations of the paramedic clinician

•  Theme 4. Community-based Emergency health in integrated health and emergency systems

•  Theme 5. Personal and professional development

These themes 'intersect' both vertically and horizontally throughout the curricula and shape the teaching and learning under the founding paradigm of community-based emergency health. DCEH&PP staff has responded to Monash University 's Graduate Attributes which include development of information technology (IT) skills and thus have included not only online teaching and learning but also WBE to improve their skills with computers and health informatics.

Traditionally, Australian prehospital education with vocational or on-the-job training historical origins have generally measured students' theoretical competency by way of the traditional PBE. Often this traditional PBE would consist of multiple choice questions (MCQ), short answers, case studies, long answers and word matching. With the transition of prehospital education moving to the higher education sector, it is important that further exploration of other pedagogical and assessment options are evaluated. One such option is the move from the traditional PBE to one of innovative, flexible and state-of-the-art way of assessing students' competency and cognitive ability.

WBE PROCESS

The unit used in the study utilises the educational paradigm of case-based learning (CBL) via a blending mode of face-to-face (F2F) and online teaching and learning. The learning management system used during the study was WebCT (CE) TM / WebCT ( Vista ) TM and required significant student participation in the online classroom forum. All online testing facilities were undertaken in WebCTTM.

The initial phase of this project was to formally assess the students in an online capacity during the semester 1, 2005 and 2006 in a prehospital clinical unit of study. Students were advised on the first day of class and also in their unit booklets that such a testing procedure was taking place during week 5. Advice was provided to students who were not "IT-savvy" to seek student assistance on campus. The test was worth 15% of the overall grade.

The testing procedure was conducted in designated computer laboratory; each laboratory contained 20 personal computers with full network capabilities. The allocated test time was 1 hour and consisted of 55 questions. Multiple attempts were not allowed. The computer laboratory was structured in a designated space, thus no computers or chairs were altered to increase proximity or distance during the examination process. Questions were generated from the first 4 weeks of lectures and consisted of MCQ's, diagram MCQ's, true or false questions and word matching.

Authentication was ensured by standard login and passwords via the normal WebCTTM URL. Given the blended teaching mode lecture; notes and material were available in WebCTTM, nevertheless security was maintained by selectively blocking students to their lecture notes during the 1-hour examination. This type of security is discussed by (Peterson et al. 2004) who also discusses the capacity for students to view electronic text books and resources via the Internet during the exam. No notes or paper material were allowed on desktops and security was maintained with 2 invigilators present during the examination. These general security measures are repeatedly identified in the literature (Gilmer et al. 2003) .

Figure 1: This is a representative screen from the WebCTTM examination questioning area highlighting the multiple-choice questioning layout. Please note the timer in the top right hand corner.

A timing clock was clearly highlighted in the testing screen advising the students of the remaining time (Figure 1). Once the 1 hour time interval had lapsed the exam automatically closed and any unanswered questions could not be attempted. Printing of the exam was not permitted and students could change their answers (if required) during the exam. This point is important as other studies have shown increased student distress when some uncertainty exists with clicking the wrong answer (DeAngelis 2000) .

Figure 2: This is a representative screen from the WebCTTM examination questioning area highlighting the capacity to provide feedback with correct answers.

Method

The convenience sample consisted of (n = 94) BEH undergraduate students. Second year students enrolled in the BEH degree in semester 1, in 2005 and 2006 were selected in this project. The study population consisted of 100 students enrolled in one of the clinical units from the degree, 94 (94%) students participated in the study.

There were 37 second year [2005] and 57 second year [2006] students at the time of the study. A cross-sectional survey design using a paper-based questionnaire was adopted in this study. The questionnaire contained 10 multiple-choice questions. The survey attempted to elicit students' attitudes towards WBE's and its key features and attitudinal comparison between WBE's and PBE's. Students were asked to report their attitudes and opinions on a 4-point Likert Scale (1 = strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree).

Data Analysis

Analytical statistics were used to report the students' perceptions and their attitudes towards WBE using a 2-tailed unpaired t test. The results are considered statistically significant if the P value is < 0.05. The Likert scale responses were analysed using SPSS (Statistical Package for the Social Sciences, Version 14.0, SPSS Inc., Chicago , Illinois , U.S.A. ).

Ethics

Ethics approval for the study was granted by the Monash University Standing Committee on Ethical Research in Humans.

RESULTS

The following data are a concise description of the student's responses to each question of survey collected and calculated from both second year cohorts.

Table 1: Combined data second year students 2005-2006 (n=94)

WBE a learning experience

The vast majority of students (94%) strongly agreed or agreed that the WBE was a positive experience. In contrast, only (5%) of respondents disagreed this statement. In terms of the WBE itself, (97%) of students either strongly agreed or agreed that the WBE was not confusing and (100%) students stated they either strongly agreed or agreed that the WBE was efficient. Interestingly, almost 100% (97%) of respondents felt the WBE was not tedious.

WBE testing environment

Just over (90%) of students felt the WBE was undertaken in a conducive testing environment, (6%) of respondents disagreed and only (1%) of students strongly disagreed with this statement. Almost 85% (83%) of students believed the WBE ensured fairness and equity. One in ten (13%) students disagreed with this and only (2%) of respondents strongly disagreed with this statement.

WBE versus PBE

Just over 80% (84%) of students either strongly agreed or agreed that the WBE was better than PBE's. Less than one in ten (9%) disagreed and only (5%) of respondents strongly agreed with this statement. The far majority of students (80%) strongly agreed or agreed that they would prefer WBE's compared to PBE's. Less than 15% (14.9%) disagreed and less than one in twenty (4.3%) strongly disagreed with this. Almost one in nine (87%) strongly agreed or agreed that they would prefer to receive their test results automatically using WBE's. Just over 5% (6.4%) and (5.3%) disagreed and strongly disagreed with this respectively.

Statistical Analysis of Group Differences

The statistical analysis aimed to identify whether any statistical significance existed with the second year students and the efficacy of WBE as an assessment tool in undergraduate prehospital education. Analysis of the Likert responses showed that the second year cohort had an overall mean satisfaction of 1.74 (SD 0.58) [95% CI 1.52-1.77], P = <0.0685.

Overall analysis of the two groups combined shows that the students generally liked WBE as an appropriate assessment tool. A t-test revealed that there was no significant difference (t=-1.84 P = 0.0685) in students' perceptions and attitudes towards WBE.

DISCUSSION

Whilst internet-based education has increased dramatically, the actual testing and assessment via the World Wide Web is not often reported in the health care literature. Several articles have examined students' perceptions or attitudes toward WBE. The results highlight evidence that prehospital students generally prefer and enjoy the opportunity to participate in WBE's. Notably, similar satisfaction levels have been identified in the literature by (Butzin et al. 1984; Legler & Realini 1994; Bloom & Trice 1997; Bocij & Greasley 1999; Ogilvie et al. 1999; Cotugna & Vickery 2001; Hong et al. 2002; Gilmer et al. 2003) .

Over 80% of students stated they strongly agreed or agree that they would prefer WBE over PBE. Other studies (Griffiths 1994; Zandvliet & Farragher 1997; Hong et al. 2002; Schultze-Mosgau et al. 2004) also found similar results. These findings are significant in the context of prehospital education and shows great promise in general prehospital higher education. Some of the advantages of WBE's are found below. One benefit WBE's have over PBE's not directly identified in the literature is the capacity for faculty members to support individual or group learning problems that may have not been possible in the past given the time constraints of traditional PBE's. A high proportion of students would prefer undertaking a WBE compared to the traditional PBE. These results are substantially higher than other reports in the literature (Butzin et al. 1984; Bocij & Greasley 1999; DeAngelis 2000) . This raises the question whether WBE's are just another sophisticated piece of technology in reducing the teachers making time or do WBE's produce better exam scores?

•  Timely feedback

•  Flexible scheduling

•  Cost effective

•  Reliable

•  Student/teaching flexibility of time and location

•  Reduction in teacher marking time

•  Instant scoring

•  Enhanced security

•  Inclusion of multimedia

•  Immediate student grade storage

•  Allows greater student progress to be monitored

•  Unbiased marking

•  Easy identification of 'common' question errors

•  Standardisation of examination environment

•  (Griffiths 1994; Stephens 1994; Bloom & Trice 1997; Ogilvie et al. 1999; Bartlett et al. 2000; DeAngelis 2000; Lynch et al. 2000; Cotugna & Vickery 2001; Hong et al. 2002; Gilmer et al. 2003; Doty & Lucchesi 2004; Peterson et al. 2004; Schultze-Mosgau et al. 2004; Vrabel 2004)

High student satisfaction results were gained in instructional design, presentation and efficiency with almost 100% of the participants claiming the WBE was not confusing and all students claiming the WBE was efficient. Similar results were found in the studies by (Butzin et al. 1984; Ogilvie et al. 1999; Bartlett et al. 2000; DeAngelis 2000) . Comparable student satisfaction was also highlighted with almost 100% of students stating that the WBE was not tedious. Bocij & Greasley (1999) and DeAngelis (2000) also state parallel conclusions in their studies.

Whilst the majority of students felt the test was undertaken in a conducive environment several aspects should be taken into consideration for WBE's. Consideration should be sought for computer-to-computer proximity, although question randomisation would remedy this. Also, 'quiet signs' should be posted on doors and hallways to reduce external noise and potential student traffic (Gilmer et al. 2003) . Faculty staff should pay particular attention to reducing external noise and the room temperature. In this particular study, given the multiple computers being used simultaneously this increased the ambient temperature dramatically. Network printers should also be considered if they are placed in computer laboratories. In Gilmer et al. (2003) study a network printer was identified as a distracter that unfortunately had to be subsequently removed during the exam.

Over 80% of students felt adequate fairness and equity was maintained for the WBE. These data are substantially higher than other studies found in the literature (Bartlett et al. 2000; Gilmer et al. 2003) . This could be attributed to that the fact that students were given the opportunity to undertake a 'mock' test in an informal environment allowing them to familiarise themselves with the testing environment and type of questions. These 'mock' tests during each study module has been identified by several authors to provide students with a non-threatening and fair means of undertaking a WBE, perhaps for the first time (Danielsen et al. 1998; Bocij & Greasley 1999) .

Only one in ten of participants felt threatened by the WBE, further analysis is required to determine the cause of this feeling. Was it due to the proximity of computers, first exposure at WBE or simply the subject matter being assessed? This is reinforced by other studies who also found similar findings (Bocij & Greasley 1999; Gilmer et al. 2003) . Other potential or perceived disadvantages (see below) that may affect student fairness and equity might include generational learners, experience with WBEs, level of expertise with computers and associated anxieties. Notably, several studies have shown that computer anxieties have little effect on student's performance or academic results when undertaking WBE's (Powers & O'Neill 1992; Lynch et al. 2000; Cotugna & Vickery 2001) .

•  Increased cost

•  Student apprehension

•  Need for new test-taking skills

•  Non-conducive testing environment

•  Potentially reliant on information-savvy students

•  (Cotugna & Vickery 2001; Peterson et al. 2004)

The literature highlights that for exams to provide the most useful learning experience then feedback should be timely or automatic (Peterson et al. 2004). Almost 90% of students preferred their test results automatically using the web-based system. So what are the advantages of this? The WebCTTM online exam function allows not only automatic scoring results but also the ability of the instructor to place descriptors in correct and incorrect answers. The capacity for faculty to include descriptors in the answer section has been identified as a positive teaching method by several authors in their studies (Bloom & Trice 1997; Ogilvie et al. 1999; Gilmer et al. 2003). This provides the student with timely feedback but also feedback that is meaningful and assists in clearing misconceptions with particular questions. In DeAngelis' (2000) randomisation study she identified several benefits of automatic and descriptive scoring. Firstly, this led to a reduction of actual testing time, and secondly, further reduction in marking time with no need to review test results.

Further research

Further research into students' capacity and equity with computer use could be undertaken via an experimental control group study. One aspect could be to compare and contrast similar educational content with other universities - WBE's could be used simultaneously across the country or globe (Schultze-Mosgau et al. 2004) . Additional analysis into w hether WBE's can be determined to be more valid, reliable and cost effective in terms of financial costs but also in terms of students assessment requirements than traditional examination methods. Also, comparison of different learner generations should be undertaken, for example, Generation Y versus X Generation and how this may impact on user-friendliness of WBE's.

Limitations of study

Several elements were not identified in the survey including gender, English language proficiency and participant's previous experience with WBE's; these specific backgrounds and exposure may have affected some of the results. Moreover, since no control groups were used who did not use the WBE; the author cannot be sure the testing system itself was responsible for academic performance.

CONCLUSION

Prehospital education in many facets is undergoing significant change in pedagogical approaches, principles and assessment methods. As identified a small number of articles have examined students' perceptions or attitudes toward WBE. In this study undergraduate prehospital students found the WBE experience to be very positive, preferred WBE to PBE and preferred to receive their test results automatically. Providing students with more exposure with WBE's and other contemporary assessment technology should become an integral component of prehospital higher education.

Acknowledgements

Many thanks to Mr Mal Boyle for his assistance with the statistical analysis.

Also, many thanks to the reviewers for their time and constructive comments during the review process.

REFERENCES

Bartlett, J., M. Alexander, et al. (2000). A comparison of online and traditional testing methods in an undergraduate business information technology course. Delta Epsilon National Conference Book of Readings.

Bloom, K. C. and L. B. Trice (1997). "The efficacy of individualized computerized testing in nursing education." Computers in Nursing, vol. 15, no. 2, pp. 82-88.

Bocij, P. and A. Greasley (1999). "Can computer-based testing achieve quality and efficiency in assessment?" International Journal of Educational Technology, vol. 1, no. 1, pp. 1-21.

Butzin, D. W., C. P. Friedman, et al. (1984). "A pilot study of microcomputer testing in paediatrics." Medical Education, vol. 18, no. 5, pp. 339-42.

Cotugna, N. and C. E. Vickery (2001). "Perceptions and evaluation of the computerized registration examination for dietitians." Journal of the American Dietetic Association , vol. 101, no. 12, pp. 1453-5.

Danielsen, R., R. Davis, et al. (1998). "Computerized testing: uses and pitfalls." Perspectives on Physician Assistant Education, vol. 9, no. 3, pp. 152-155.

DeAngelis, S. (2000). "Equivalency of computer-based and paper-and-pencil testing." Journal of Allied Health, vol. 29, no. 3, pp. 161-4..

Doty, C. I. and M. Lucchesi (2004). "The value of a web-based testing system to identify residents who need early remediation: what were we waiting for?" Academic Emergency Medicine. vol. 11, no. 3, pp. 324.

Gilmer, M. J., J. Murley, et al. (2003). "Web-based testing procedure for nursing students." Journal of Nursing Education , vol. 42, no. 8, pp. 377-80.

Griffiths, F. (1994). "Freedom and flexibility in the higher education examination process." Active Learning, vol. 1, no. 1, pp. 1-3.

Hong, C. H., D. McLean, et al. (2002). "Using the internet to assess and teach medical students in dermatology." Journal of Cutaneous Medicine & Surgery, vol. 6, no. 4, pp. 315-9..

Legler, J. D. and J. P. Realini (1994). "Computerized student testing as a learning tool in a family practice clerkship." Family Medicine , vol. 26, no. 1, pp. 14-17.

Lynch, D. C., T. W. Whitley, et al. (2000). "Variables that may enhance medical students' perceived preparedness for computer-based testing." Journal of the American Medical Informatics Association, vol. 7, no. 5, pp. 469-74.

Ogilvie, R. W., T. C. Trusk, et al. (1999). "Students' attitudes towards computer testing in a basic science course." Medical Education, vol. 33, no. 11, pp. 828-31.

Peterson, M. W., J. Gordon, et al. (2004). "Computer-based testing: initial report of extensive use in a medical school curriculum." Teaching & Learning in Medicine, vol. 16, no. 1, pp. 51-59.

Powers, D. and K. O'Neill (1992). Inexperienced and anxious computer users: coping with a computer-administered test of academic skills. New Jersey, Princeton.

Schultze-Mosgau, S., T. Zielinski, et al. (2004). "Web-based, virtual course units as a didactic concept for medical teaching." Medical Teacher , vol. 26, no. 4, pp. 336-42.

Stephens, D. (1994). "Using computer assisted assessment: time saver or sophisticated distraction?" Active Learning, vol. 1, no. 1, pp. 1-5.

Vrabel, M. (2004). "Computerized versus paper-and-pencil testing methods for a nursing certification examination: a review of the literature. Computers, Informatics, Nursing, vol. 22, no. 2, pp. 94-8.

Zandvliet, D. and P. Farragher (1997). "A comparison of computer-administered and written tests." Journal of Research on Computing in Education, vol. 29, no. 4, pp. 423-438.

Copyright for articles published in this journal is retained by the authors, with first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

Original article at: http://ijedict.dec.uwi.edu//viewarticle.php?id=292&layout=html