RAT Online. The Wheel Spanner: An online self-defence course

Steven Yates
Derivco, South Africa

ABSTRACT

The task of design, delivery and evaluation of online learning environments is a complex one. There is a multitude of learning approaches, software design tools and evaluation methods from which to choose. This paper focuses on the design, delivery and evaluation of an online course in Rough and Tumble (RAT), a South African martial art (wheel spanner for self-defence). The researcher uses the 'eclectic-mixed methods-pragmatic paradigm' as a theoretical framework, which allows researchers to link in other relevant theories. Development research forms the core research methodology, allowing the researcher to carry out both formative and effectiveness evaluation. Learner output, as well as questionnaires, expert reviews, and course interactions indicate that learners gained favourably in knowledge, skills and attitudes. Positive and less positive feedback forms a useful part of the evaluation guiding the researcher in further decision making for other online courses of this nature.

Keywords: Multimedia, martial arts, knowledge representation, constructivism, development research

INTRODUCTION

In this paper, the researcher evaluates a Rough and Tumble (RAT) online course on how to use a wheel spanner for self-defence. This paper forms part of a larger project known as the RAT Online project, which consists of a series of online courses and a multimedia resource CD-ROM known as the RAT CD-ROM. This project was initiated mainly because of an increasingly geographically dispersed learning community.

RAT is a martial art of South African origin. Practitioners may use principles from various martial arts from around the world and from a wide variety of martial arts disciplines. RAT is a martial art developed to answer the practical needs of specific real-life applications. The researcher created, developed and expanded RAT in the early 1990s to deal with a greater number of self-defence situations than conventional martial arts can offer.

The wheel spanner used for this course is the four-sided variety used for changing car wheels. Because of the incidence of violent crime on the roads in South Africa, the wheel spanner course may be beneficial to motorists who find themselves stuck on the roadside. RAT practitioners generally do not carry weapons, but they are expected to be able to use implements found in their environment as weapons should they be required to do so. This mindset is similar to some old martial arts where practitioners adapted farming implements as weapons.

RAT and other martial arts learning situations do not only involve learning physical skills. Learners need to develop a high degree of creativity as well as appropriate and ethical attitudes. In the RAT Online courses, the researcher aims to develop creative and evaluative cognitive abilities in learners through well-planned learning activities and collaborative tasks with tools such as discussion forums and simulation tools. The tasks encourage the development of appropriate attitudes, and the final video grading exam tests that physical skills have been developed. These exams have to be unedited video recordings showing a continuous flow of activities and skills. This enables course examiners to observe whether participants have taken the time to practise the skills, as the exam covers a broad range of activities. It would be difficult to record the grading exam in an unedited version without first having practised the skills.

The next section of this paper states the main aims of the course and then describes the theoretical framework, followed by the methodology section. There is a discussion of the course and the evaluation, and, in conclusion, some closing remarks.

AIMS OF THE COURSE

The aims of this course and evaluation are reflected in the main research questions of the RAT Online project. This study set out to determine the following:

Can RAT martial arts knowledge, skills and attitudes (KSAs) be facilitated in computer-supported learning environments?

What design would constitute effective martial arts computer-supported learning environments?

What kinds of learning activities and software tools are effective in martial arts computer-supported learning environments?

These questions help guide a formative process of RAT Online course design, delivery and evaluation based on a theoretical framework and methodology.

THEORETICAL FRAMEWORK

The researcher makes use of an evaluation framework, called the 'eclectic-mixed methods-pragmatic paradigm' (EMMPP), which is a theoretical framework that allows for a flexible approach to the design, delivery and evaluation of interactive learning systems (Reeves & Hedberg, 2003). The EMMPP allows researchers to adopt different perspectives and theoretical approaches depending on the practical needs of a project. In this project, there are three main theoretical sections: EMMPP, RAT, and learning. EMMPP is the theory that binds the study together. The RAT section includes the underlying principles of RAT, Chomsky's universal grammar (UG) (Chomsky, 1965; Cook, 1988; Cook & Newson, 1996), and the prototype theory of semantics (Taylor, 1989). The learning section includes social constructivism and Vygotsky's Zone of Proximal Development (ZPD) (1978), cognitive flexibility theory (CFT) as discussed by Spiro, Feltovich, Jacobson & Coulson (1991), and Bloom's Taxonomy (cognitive, affective and psychomotor – KAS/KSA) (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956; Krathwohl, Bloom, & Masia, 1964; Singer, 1982).

Outside martial arts practitioners consider RAT an eclectic martial art, as RAT practitioners use principles and techniques that are found in a variety of martial arts. RAT practitioners, on the other hand, think of RAT as a complete martial art that allows learners to incorporate any self-defence principle.

Chomsky's UG provides an analogical framework, as Chomsky (Cook, 1988) tries to provide a framework of principles that apply to all languages,  in which individual languages, such as English, are instances of the universal set of language rules. RAT practitioners encourage an open-minded approach to combat where all principles of combat are important and individual martial arts are instances of the universal set of martial arts principles. The course facilitator encourages this approach in all courses.

Prototype theory (Taylor, 1989) is another analogical linkage from linguistics. Prototype theorists recognise that different people understand concepts in different ways and have different mental images of concepts, such as 'shoe' for example. Females and males may have different mental images of these words and yet individuals somehow know what is meant when speaking. In this approach, it is possible for people to change attributes of concepts to 'create' different mental images. For example, if we change one attribute of a female's high heel shoe, such as the heel, to a flat heel, it would be a different shoe. One could create a 'meaning chain' by continuously changing the attributes of the shoe until it included all examples of shoes and also peripheral examples of the word shoe, such as 'boot', or 'slipper'. RAT practitioners use this to develop new techniques by applying the idea of meaning chains to self-defence techniques. RAT practitioners call this multiple contextual training (MCT). In all courses and training, facilitators encourage learners to (1) change the characteristics of individual techniques, such as a straight punch to a hook punch along such meaning chains to create new techniques; and (2) to change the attributes of self-defence situations using a single technique. For example, a particular defence to a front strangle might be used in multiple situations, such as a punch attack, or a hair grab attack.

Vygotsky (1978) speaks of a Zone of Proximal Development where learners can solve problems with the help of more knowledgeable peers. Learning occurs in socially mediated environments. The researcher applies a social constructivist approach in the wheel spanner course. Learners are expected to construct new techniques from self-defence principles through a process of dialogue and collaboration. This is far more valuable in self-defence situations compared to merely mimicking the techniques of others, as you can develop and improve the techniques and principles to deal with new self-defence problems.

The researcher structured the course around Spiro et al's (1991) Cognitive Flexibility Theory, which says that learning can be structured at less advanced levels, but at advanced levels, learning is far more ill-structured and does not  necessarily follow a linear path. The wheel spanner course began with tasks to introduce learners to topics that they may want to think about and techniques that they could experiment with, but later tasks were far more complex, requiring learners to create a syllabus based on their own understanding, experience and what they gained from others. The objectives and assessment criteria of such tasks were clear, but the tasks were open-ended.

Bloom (1956) classified educational objectives into three broad objectives: knowledge, attitudes and psychomotor, which is known as Bloom's Taxonomy. All the tasks in the wheel spanner course were constructed with objectives that addressed one or a combination of these objectives. Learners had easy access to the assessment criteria that were derived from the objectives. Learners had to physically demonstrate their knowledge and skills; create and evaluate knowledge, and this was done through a collaborative approach.

The researcher uses a methodological approach that logically conforms to the EMMPP: development research; this is discussed next.

METHODOLOGY

Development research allows the researcher to make design decisions based on a theoretical framework and evaluate the design according to the framework (Reeves & Hedberg, 2003, p 274). This is a formative process where learning system designers can use theory in a practical way. The researcher uses formative and effectiveness evaluation in the RAT Online project. The researcher will use the results of this study to improve the next online course, the Belt course. This course and evaluation is an 'experimental course' to move closer to better course design and evaluation principles.

Methods, Instruments and Analyses

The researcher uses Reeves & Hedberg's (2003, p144) evaluation criteria to evaluate the wheel spanner course: functionality, usability, appeal, and effectiveness, which are incorporated across several research instruments. Such a multi-dimensional study requires several methods of data collection and several data collection instruments. The researcher adopts a mixed-methods approach to data collection and analysis in this study. The research methods are grouped into the following three categories in this study: expert evaluation (user interface rating form and teaching evaluation form), learner feedback about the course (online questionnaire, post-questionnaire interview, and course assessment) and records (participant observation and email communication).

Due to the small number of participants in this study (see The Wheel Spanner Course: Participants), the researcher makes minimal use of quantitative data. The researcher uses SPSS (a statistical software package) to generate basic frequency statistics to analyse the online questionnaire. Most of the data analysis is qualitative in nature and the researcher uses QSR NVivo (a qualitative software analysis package) to analyse the data. In cases where the textual data is minimal, the researcher uses a manual reading approach to analyse the data.

Some of the instruments allow researchers to gain a visual sense of the data collected, such as the expert review instruments. The researcher uses the instruments as a convenient method of analysing data and to make practical design decisions.

Expert review

User interface rating form

For the expert evaluation, the researcher administered a user interface rating form and a teaching evaluation form. The user interface rating form (Figure 1) helps the researcher make decisions and comparisons between evaluators about the functionality, usability, and appeal of the learning environment (Reeves, 1997). This evaluation tool is a blend of criteria used by Nielsen (1994b), Reeves & Hedberg (2003) and Tognazzini (2003) and has been adapted accordingly.  The researcher constructed the instrument with the aim of incorporating as inclusive a range of evaluation criteria, while at the same time ending up with a tool that would not be too demanding on evaluators' time. The power of the instrument lies in its visual characteristics, and therefore the instrument would need to fit onto a medium (e.g. a single A4 page) that would allow easy viewing. Evaluators were given a user interface rating form that contained a full description of each criterion.

Figure 1: User Interface Rating Form

In the next paragraph, the teaching evaluation form is briefly outlined.

Teaching evaluation form

The teaching evaluation form (see Figure 2), which is adapted from Reeves & Hedberg (2003, p191) helps guide the researcher in the teaching effectiveness of the learning environment. The researcher constructed this tool by combining the characteristics of effective learning environments and important characteristics of martial arts learning environments. These scale diagrams provide a quick and rich visual summary of the evaluation dimensions. Researchers can quickly see where they can improve certain evaluation dimensions and make further design and delivery decisions by making comparisons with expected evaluation outcomes. Each criterion is based on the underlying theories of the theoretical framework. As with the user interface rating form, the teaching evaluation form contains a full description of each criterion.

Figure 2: Teaching Evaluation Form

The learner feedback about the course is crucial in a study of this nature where the researcher is attempting to establish the effectiveness of the learning environment and process.

Learner feedback about the course

Online questionnaire

An online questionnaire administered to the course participants covered questions relating to functionality, usability, appeal and effectiveness of the course and course environment. The learners had to rate each criterion according to a five-point rating scale (1=Strongly Disagree, 2=Disagree, 3=Neither Agree nor Disagree, 4=Agree, 5=Strongly Agree). The data was then analysed using simple frequency statistics. Some learners added additional feedback in the comments sections.

Post questionnaire interview

The online questionnaire was followed by some small-scale semi-structured interviews with a random selection of available participants.

Course Assessment

The assessment of the learners' outputs during the course helps make informed judgements about the effectiveness of the course. The assessments relate to the three domains of Bloom's Taxonomy: cognitive (knowledge), affective (attitudes), and psychomotor (skills), or KASs (KSAs). Each task in the course was assessed according to explicit marking criteria. The assessment consists of analysis of written activities (discussions and chats), assessment of learner output (mind maps), and video recorded performances. The researcher used QSR NVivo to search for and code the results according to the marking criteria of the course.

Records

Participant observation

The researcher acted as participant observer and documented an implementation log, then made appropriate changes during the delivery of the course.

Email communication

Finally, the researcher recorded email communication between the researcher and learners during the course. This last stage was used as a means of bringing any other issues to the fore that were not covered by the other instruments.

Next, the researcher documents the course development, delivery and findings.

THE WHEEL SPANNER COURSE

Description

The wheel spanner course was scheduled to run for one month (23 February - 31 March 2004), but continued for two weeks longer, as learners required more time to complete tasks.

The course was based on a previous smaller scale RAT Online course that ran for a period of two weeks (Bear Hug course). Bear Hug course participants requested that future RAT Online courses include more opportunities to engage with multimedia (i.e. more images and videos) and requested that the course run for a longer period. Learners recognised the potential of learning martial arts online and thus agreed to take part in future courses of this nature.

While the Bear Hug course served as a pilot to this study, the Wheel Spanner course would serve as part of a practical formative process to develop a high-level design template for future RAT Online courses and to satisfy the aims of the study described above. The researcher set out to design a more in-depth course.

Participants

There were two participant groups in this study, the course participant group and the expert reviewer group.

Course participants

The course participant group consisted of 12 registered participants. Eight participants actually took part in some of the learning tasks, while only six participants completed most of the tasks. Two out of the six participants did not complete the final task.

Due to the high quality of the learner output, the course facilitator awarded all six of the course participants their course completion certificates.

The all-male course participant group was homogenous in terms of interest, as all participants had participated in martial arts before (9 months – 27 years experience). The age group category was quite broad (14 years of age – 50 years of age), with the majority of participants falling in the 20 to 36 years age group. Only three out of eight participants had taken part in a previous online martial arts course. The participants had an interesting mix of backgrounds and experiences. Between them, there were 12 different martial arts types.

The course participants were located in various places. Out of the eight active participants on the course, five were from Durban (South Africa), one was from Johannesburg (South Africa), one was from London (United Kingdom), and one was from Melbourne (Australia).

Expert reviewers

The expert reviewer group (two participants) took part in the study after the course was complete. The expert reviewers took part in two areas of post-course evaluation, User Interface Evaluation and Teaching Evaluation. Both expert reviewers in this study were located in Durban (South Africa).

Experts were chosen for their education and teaching experience. Both have a Master's degree in a field associated with online learning (one is working towards a PhD) and both teach using online technologies in a higher education institution. Each evaluator also has experience creating, developing and evaluating online learning environments.

The researcher addressed issues of consent by setting in place an informed consent process. The researcher then began the development process.

Development

The Bear Hug course was limited in scope to a single self-defence situation. The Wheel Spanner course would have to cover a much broader range of situations, as it would need to serve as a course that learners could apply in any self-defence situation. The course needed to cover principles of self-defence, weapons (the wheel spanner in particular), strategy, self-defence attitudes and performance (skills). The online course design and environment therefore required tasks and tools that would enable learners to acquire and create knowledge, skills and attitudes in all these areas. In addition, learners needed to develop attitudes that demonstrated that they could collaborate on tasks, as the success of the course relied on social interaction.

The researcher designed and delivered a course with eight tasks using the theoretical framework described above. The course was structured enough to guide learners, but was also open-ended enough to conform to Spiro et al's (1991) theories around cognitive flexibility. The eight tasks were structured so that learners could engage with theory, practice and develop attitudes appropriate for self-defence and for collaboration (i.e. Bloom's Taxonomy). To encourage reflection and evaluation the participants were required to mark each other's work. The eight tasks included:

Task 1: Reflect, research and experiment (a non mark-bearing task designed to encourage reflection and experimentation with the wheel spanner);

Task 2: Watch a movie (a movie clip showing how the wheel spanner can be used in a self-defence situation);

Task 3: Comment and practise (a mark-bearing discussion eliciting knowledge and shared experiences by participants);

Task 4: Video chat (a mark-bearing live text-based chat with web cams where learners can demonstrate techniques) (see Figure 3);

Task 5: Experiment and discuss (a mark-bearing focused discussion around specific techniques – learners could upload example media such as images, animations and video clips);

Task 6: Represent your knowledge (a mark-bearing two-part task – first a live chat with an online whiteboard [a shared space where participants can draw and write] and then an asynchronous group activity [discussion forum] to develop a mind map of wheel spanner techniques, principles and self-defence strategies) (see Figure 4 and Figure 5). See Buzan & Buzan (1993) for the benefits of mind-mapping;

Task 7: Virtual self-defence (a mark-bearing discussion and simulation) (four chat rooms with movable images of people in different settings where learners can discuss strategy) (see Figure 6); and

Task 8: Practise and record your knowledge (a mark-bearing unedited video recording of a learner constructed syllabus and demonstration of development of skills).

Figure 3: Video chat tool

Figure 4: Whiteboard chat

Figure 5: Example: Free-mind mind map

Figure 6: Virtual self-defence room 2

The researcher intended to extend the learning experience by enabling communication between learners, creating learning situations that are difficult to simulate in real life (virtual self-defence), and to give learners the opportunity to represent their knowledge in different ways with the course tools. This would be an example of using computers as 'mind tools', as espoused by Jonassen, Carr & Yueh (1998).

The site also contained a Resources page that had links to useful example images (such as an example mind map), the movie, the old wheel spanner course, a mark sheet, certificate and software.

The researcher created a website with Dreamweaver MX using Active Server Pages (ASP), HTML, Macromedia Flash MX and Macromedia Flash Communication Server 1.5. The discussion forum was a free web-based discussion forum called Web Wiz Forums. A video camera captured the movie clip and a digital camera was used to capture the images. Adobe Photoshop 6.0 and Macromedia Fireworks MX were used to prepare the graphics to appropriate Web usable file formats.

Next, the results, analyses and findings are presented.

RESULTS, ANALYSES AND FINDINGS

Expert review

User interface rating form

Figure 7 contains a visual summary of the user interface rating form results, analysis and findings. The squares represent the researcher's development goals, the circles represent the expected ratings based on the course design and implementation, and the dashed and solid lines represent ratings by evaluator 1 and evaluator 2 respectively.

Figure 7: Evaluation results of user interface evaluation form

The ratings of both evaluators do not stray too far from the development goals (Figure 7). It would be ideal if all the ratings could form a straight line through the squares. Despite there being room to improve, especially in the areas of ease of use, navigation and media integration, the ratings are satisfactory and do not reveal much of a problem with the user interface. The researcher expected rating also reveals that more could be done to improve cognitive load, mapping, screen design, information presentation, and overall functionality of the site. Although the expected rating for use of metaphors is not as high as it should be, it is not a concern, as a standard Web page metaphor was used with drop-down menus to each relevant task, course tool, or resource. Evaluators seemed happy with the result. One evaluator commented: "Interactive reference library metaphor?" which is satisfactory. It is therefore not necessary to change the metaphor in the next RAT Online course.

While the user-interface evaluation results seem straightforward, the teaching evaluation results visually represent more of an irregular pattern.

Teaching evaluation form

Figure 8 is a visual summary of the teaching evaluation form results, analysis and findings. Each symbol denotes the same meaning as in the user-interface rating form.

Figure 8: Evaluation results of teaching evaluation form

The results reflect that the evaluation closely matches the development goals. The biggest areas of concern are the 'source of motivation' and 'cultural sensitivity' dimensions. The development goal for 'source of motivation' is intrinsic, but this is not easy to achieve. As a participant observer, the researcher observed that he had to issue many statements of encouragement and reminders about incomplete tasks. These statements of encouragement could have been based on a biased perception and might have been unnecessary.

For the 'cultural sensitivity' dimension, one of the evaluators commented that: "I think all the 'shoulds' in the language might alienate those unused to disciplined fields of endeavour? Can these be rephrased on the course info?" This observation is useful and indicates that the language on the course website needs to change.

The teaching evaluation instrument needs to change in future iterations of the evaluation process. One evaluator did not complete the two dimensions: 'martial application' and 'martial theory'. These two dimensions might be better placed on a Content Expert Evaluation Form. One evaluator indicated the difficulty of evaluating the teaching dimensions on the rating scale. She suggested a quadrant graph format, but the rating scale would have lost its visual effect. In later courses, the researcher included a third dimension, 'integrated' along the rating scale.

While the user-interface rating form and the teaching evaluation form help developers make informed decisions based on expert evaluations, the learner feedback about the course contributes much to the effectiveness evaluation.

Learner Feedback About The Course

Online questionnaire

There are too few respondents to make any significant statistical measures, but based on the frequencies of the eight participants who did complete the questionnaire combined with learner comments, the results of the expert reviews, course assessment, and records, the course was functional (but with several technical difficulties), usable, appealing, and effective. Rather than using only mean values for all decision-making, the researcher considered each response on a case-by-case basis and adopted the approach that qualitative feedback would be valuable and support decisions.

The questionnaire elicited data about specific tasks and tools in the course, as well as more general issues, such as the appeal of the environment, and facilitator behaviour. The results suggested a need for further analysis and corroboration in the following areas: the usefulness and usability of the 'represent your knowledge' task and tools, the whiteboard tool, peer marking, the video chat, and the 'watch a movie' task.

Learner comments indicate that the 'represent your knowledge' task was a useful task.

Participant 1: "Represent your knowledge (mind map) – Encouraged a free flow of interesting ideas and forced one to really think, test and formulate what they had learnt."

Participant 2: "The mind map...this was a more do it yourself task than any other task, it was like you [were] presenting your "whole" self to be marked, commented on, criticised...best learning experience and also I learned a lot from looking at other people's mind maps."

Participant 3: "Mind map. By creating a mind map, one gets a better understanding of the entire system. Once you actually look at a mind map, you can apply it to almost every weapon based system that fit[s] similar parameters. Also there is no such thing as a complete mind map."

The 'represent your knowledge' task was divided into two parts: a synchronous activity where everyone got together in the whiteboard room, and the asynchronous activity where they created individual or group mind maps. The conflicting questionnaire results point to an ineffective task, but the learner feedback about the task says something more positive about the task. The feedback suggests that learners leverage on Bloom's taxonomy higher cognitive levels, such as application, analysis, synthesis, and evaluation (Krathwohl, Bloom, & Masia, 1964, pp 191-193). The feedback also suggests that learners found value in organising their own knowledge from this open-ended task, which points to evidence of both the affective domain of Bloom's taxonomy (Krathwohl, Bloom & Masia, 1964, pp 180-183) and CFT. The learners also seemed to enjoy the social nature of the task, which is support of social constructivist learning environments. The comments around the 'least useful tool' show that it was not the 'represent your knowledge' task that was ineffective but the tool used for the synchronous part of the task.

Learner feedback suggests that the whiteboard tool, used for the synchronous part of the task, was not easy to use.

Participant 1: "Don't know what to call it but when we got together in groups with the whiteboard to discuss and formulate the mind maps. The mind maps were great but I found the disjointed process of trying to communicate without talking over each other frustrating."

Participant 2: "Whiteboard – too technical or too simple – I don't know. It just didn't do it for me. The mind maps consolidated everything."

Participant 3: "Wasn't enough control – too chaotic, but the idea is possibly useful."

Most participants agreed that the peer marking aspect of the course was useful, but most did not take part completely. One participant's comment reflects more accurately how participants might have really felt: "It took too much time, was difficult to remember who had done what and felt counter-productive."

The questionnaire results reveal that one person thought that the video chat was the 'least useful learning task', but three people thought the video chat tool was the least useful tool. The effectiveness of the chat task requires further testing in future courses. One participant's comment provides a reason to investigate the chat task further: "Video chat – it was like entering the others people's minds and see[ing] what and how they think; in a way also opening yourself also, increasing your awareness."

For the most part, participants agree that the 'watch a movie' task was useful, but one participant's thoughts on this task are interesting support of learning environments that foster the generation of new martial arts knowledge: "'watch a movie: to be honest, one does not need to actually see the wheel spanner being used. It also can work negatively whereby we assume that the techniques used in a video may be the only techniques that are possible. It would be better to start with a blank mind and see what we can develop, then watch the video." The researcher used this feedback in later courses to develop example learning material and better placement in the sequence of tasks.

The discussion forum and virtual self-defence rooms came out strongest in the 'most useful learning tool' rating, which both support the social constructivist approach used in this course, as the success of these tools relied on collaboration and discussion tasks between course participants. Once again, participant feedback provides evidence of the effectiveness of these two tools.

Participant 1, commenting on the discussion forum: "Discussion forum – Able to communicate your ideas, receive feedback and encouragement. Able to review at own convenience and revisit to refresh oneself if one had been away for a while."

The discussion forum seemed to be the main 'meeting place' for any course activity and, as the comment above shows, the 'asynchronicity' of the tool corresponds to the expert ratings of an open course delivery.

Participant 1, commenting on the virtual self-defence rooms: "Virtual self-defence room...you can visually represent your thoughts and actions and (...) the technology works well and it's easy to use."

Participant 2, commenting on the virtual self-defence rooms: "I liked the virtual self-defence tool [that allowed me to] move the people around and comment on the positions. Good for tactical and visualising situations. Strangely enough, the chat attached to it seemed to work for me, even though I did not like the 'plain' chat. Perhaps this is because it focuses the conversation and only [one] situation at a time can be displayed."

Even though the virtual self-defence chats failed for technical reasons at times, the value of learning self-defence strategy in such a simulated environment has potential, given the positive learner feedback above about the task and the rich discussion observed by the researcher during these tasks. The researcher therefore included a discussion forum and investigated other means of delivering the virtual self-defence task in the next course.

Three (3) people felt that there was no 'least useful task'. Perhaps they feel that all the tasks contribute holistically to the learning experience, which is the intention of the course designer.

Seven (7) out of the eight (8) participants feel that after taking part in the RAT Online Wheel Spanner course that you can learn martial arts online. Respondent 8 neither agreed nor disagreed that you could learn a martial art online, as he felt that people might think that it is too easy to get a certificate, because much more practice and reinforcement would be required to acquire skill and this is a valid observation. As in any kind of class, it would be up to individuals to be motivated to continue their practice. The video grading demonstrated to the researcher that participants had acquired skills to complete the physical part of the course successfully.

Three comments made by different participants in the 'further comments' section offer concise and rich support of the theoretical framework, expert evaluations and participant observation, with concepts such as application, physical skills, discussion, exploring, social learning, active learning, development of prior knowledge, creation, and motivation coming to the fore.

Participant 1: "In the beginning I was a bit apprehensive [about whether] learning online would work. But discussing and exploring opened my mind to things I would have never thought of. I learnt most when I tried to apply all the theoretical skills to practice. When I made the video I can definitely say I learnt the most."

Participant 2: "This course I would say challenges your knowledge of martial arts and also expands it too, you learn new things all the time so you just got to have a go."

Participant 3: "Overall I thought the course was excellent and I have developed as a martial artist. The design of the site is very professional and the content of the course kept you interested. Thanks to (...) and all who participated for their hard work."

After conducting the questionnaire, the researcher did a small-scale interview with some participants.

Post-questionnaire interview

The post-questionnaire interview was a short interview with four of the participants, but it helped expose areas of weakness in the questionnaire by providing further information on questionnaire ratings. One example is where one participant rated the question item "It was easy to find my way around the site" with a 3 (neither agree nor disagree). The interview revealed that it was the not the site that was difficult to use, but that the site address was difficult to remember. The learner also commented that: "It took me a long time to get orientated [referring to the emails with the site addresses]. When I'm in the site, it's fine. When I got used to it, it was fine."

These findings demonstrated the need for more in-depth post-questionnaire interviews in the next RAT Online course.

Course Assessment

Most of the course tasks were assessment tasks. Although not everyone took part in each task, participants were able to demonstrate new knowledge, attitudes and skills. Participants' participation and artefacts generated in the course tasks were measured against the learning objectives and explicit marking criteria. The researcher coded the interactions for each participant in NVivo and then awarded each participant a mark for each task. The researcher recorded each participant's marks for each task and calculated the total in a Microsoft Excel spreadsheet.

The various discussion forum tasks reveal that online course participants can use existing knowledge and create new martial arts knowledge. During the discussion activities, they showed that they could work collaboratively to construct knowledge. In the 'represent your knowledge' task, some participant groups submitted text documents; one submitted a scanned hand-written mind map and others created digital mind maps. These submissions were all high in quality (see Figure 5 for an example), showing that they could analyse and synthesise knowledge covered in the course.

The live chat tasks also generated some lively debate and ideas and showed that learners gained in collaborative attitudes. However, the video chat server crashed before the chat session was complete and some participants could not take part due to problems with access from work. The whiteboard chat was not fully successful, but it was a useful way to get the 'represent your knowledge' task going. The virtual self-defence chat worked well. Participants discussed a wide range of topics on self-defence strategy, including ethics, law, body language, techniques, escape, the environment, multiple attackers, weapons, body structure of the opponent, and psychology. The researcher believes that the virtual self-defence task contributes to an effective martial arts course and favourably affects the final video grading by giving learners insight into a broad range of self-defence problems that they could apply in the video grading.

Participants had to make a video recording of their constructed syllabus in the final task: Practise and Record Knowledge. Only four participants took part in the video grading. The performances in this task demonstrated that learners could gain in skills in online martial arts courses.

Records

Participant observation

The implementation log reveals that there were several technical difficulties; participants arriving late for chats, or participants not arriving at all. The researcher had to contact participants by telephone on several occasions to remind them to take part. These issues contribute to an ineffective course experience and one that is less than perfectly functional.

In the questionnaire, some participants rated as 'least useful learning task' the synchronous part of the 'represent your knowledge' task (whiteboard tool), marking each other's work, the video chat, and the movie clip in 'watch a movie'. As a participant observer, the researcher observed that participants did not seem willing to mark each other's work. The problems with the nature of the whiteboard tool caused some difficulty with the synchronous part of the 'represent your knowledge' task, as it was difficult to control with everyone trying to have their say at the same time. However, it was a useful task to kick-start the process of the 'represent your knowledge' task. The implementation log reveals that the video chat crashed and the movie clip was not of a high quality, both reasons for these two components of the course not to be fully effective.

E-mail communication

The course generated a substantial amount of e-mails, mostly from the course facilitator/researcher (80 e-mails received, 109 sent by the facilitator). This represents a substantial amount of work in addition to the time required by the facilitator in the course activities. The e-mails mostly covered topics about technical issues and negotiation around task deadlines. The facilitator observes that it is necessary to maintain communication in order to promote motivation. Course participants can sometimes 'get lost' from the course environment and e-mail seems to be an effective way to 'bring them back'.

The findings above provide positive evidence to pursue this research project further, although much investigation is still required.

CONCLUSION

This paper began with an introduction and background to the RAT Online Wheel Spanner course. Next, the researcher provided the aims of the course, followed by a theoretical framework to the study. The binding theory for the study is the eclectic-mixed methods-pragmatic paradigm, which has several linked theories. The other theories include RAT, Chomsky's universal grammar, prototype theory, social constructivism, cognitive flexibility, and Bloom's Taxonomy. Following the theoretical framework, the researcher introduced the methodology, development research, and the various instruments and analysis methods. The study is a mixed-methods one incorporating minimal quantitative data from the online questionnaire, and qualitative data in the form of learner feedback about the course, visual rating scales and evaluator comments, and coded texts derived from course tasks and records. The researcher then described the course development, results, analyses and findings.

Potential martial arts teachers wanting to develop and deliver online learning courses in martial arts need to be aware that these courses demand substantial investments in time. There is also some frustration resulting from learner dropouts, setbacks when certain learners do not complete tasks on time, user problems, motivation issues and technical difficulties. The design, development, implementation, and evaluation of the wheel spanner course reflected many of these frustrations.

Despite the negative aspects of the course, this study shows that martial arts (in this case RAT: the Wheel Spanner) can be learned online with the chosen theoretical framework, which answers the first question in the aims of the study. The researcher used universal grammar and prototype theory to guide him as course facilitator, as this approach kept negotiation around techniques and principles open and diverse. These theories form part of the foundations of RAT, the effectiveness of which has been tested in numerous situations, competitions and demonstrations.

Expert evaluation, learner feedback about the course, and participant observation, provide evidence of social negotiation of knowledge and that this was effective. The social nature of the course provides observable evidence of desirable attitudes for the creation and sharing of knowledge. This supports the social constructivist part of the theoretical framework. The final tasks, such as 'represent your knowledge', 'virtual self-defence', and 'video grading' suggest that cognitive flexibility theory might be a useful theory to ground the high level design of such courses. Learner feedback about the course and participant observation about the final tasks show that the open-ended nature of the final tasks where learners could draw on new and shared knowledge from earlier more structured tasks was a useful learning experience. Furthermore, the learner assessment results and participant observation show that learners can gain in all of Bloom's Taxonomy domains: cognitive, affective and psychomotor.

The learners demonstrated their construction of knowledge and development of attitudes in the collaborative tasks of the course, such as the discussion forums, the chats, virtual self-defence chats and 'represent your knowledge' tasks. They also showed that they could create new self-defence techniques and demonstrate their physical skills in these techniques. The chosen theoretical framework therefore contributes to a useful design in computer supported learning environments, which helps toward answering question two in the aims of this study. The successes and failures of the learning tasks and software tools suggest that minor changes to the sequence of some tasks is necessary, and some new tools are necessary. This answers the third question in the aims of this course and helps in sequencing tasks and choosing tools in subsequent RAT Online courses.

The EMMPP forms a useful theoretical framework and development research is a useful methodology allowing a mixed-methods approach. The expert reviews, online questionnaires, assessments and records help to make effective design and delivery decisions in the areas of functionality, usability, appeal and effectiveness of online learning environments. The researcher recognises such evaluation as a complex task in which much work is still needed in this formative process.

REFERENCES

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (1956) Taxonomy of educational objectives: Book 1: Cognitive domain. New York: David McKay Company, Inc.

Buzan, T. & Buzan, B. (1993) The mind map book. London: BBC Books.

Chomsky, N. (1965)  Aspects of the theory of syntax. Cambridge, Mass.: MIT Press.

Cook, V. J. (1988) Chomsky's universal grammar: an introduction. Oxford: Blackwell.

Cook, V. J. & Newson, M. (1996) Chomsky's universal grammar: an introduction (2nd updated ed.). Oxford, OX, UK; Cambridge, Mass. Blackwell Publishers.

Jonassen, D. H., Carr, C. & Yueh, H.-P. (1998) Computers as mindtools for engaging learners in critical thinking. TechTrends, 43(2), 24-32.

Krathwohl, D. R., Bloom, B. S. & Masia, B. B. (1964) Taxonomy of educational objectives. The classification of educational goals. Handbook II: Affective domain. New York: David McKay Company, Inc.

Nielsen, J. (1994b) Ten usability heuristics.  Retrieved 9 July, 2004, from http://www.useit.com/papers/heuristic/heuristic_list.html

Reeves, T. (1997, 21 November 1997) Evaluating what really matters in computer-based education.  Retrieved 25 May, 2005, from http://www.educationau.edu.au/archives/cp/reeves.htm

Reeves, T. & Hedberg, J. (2003) Interactive learning systems evaluation. New Jersey: Educational Technology Publications.

Singer, R. N. (1982) The learning of motor skills. New York: Macmillan Publishing Co., Inc.

Spiro, R. J. Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1991) Cognitive flexibility, constructivism, and hypertext: random access instruction for advanced knowledge acquisition in ill-structured domains.   Retrieved 2 April, 2004, from http://phoenix.sce.fct.unl.pt/simposio/Rand_Spiro.htm

Taylor, J. R. (1989) Linguistic categorization: Prototypes in linguistic theory. New York: Oxford University Press.

Tognazzini, B. (2003) First principles in interaction design.  Retrieved 9 July, 2004, from http://www.asktog.com/basics/firstPrinciples.html

Vygotsky, L. S. (1978) Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press.

Copyright for articles published in this journal is retained by the authors, with first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

Original article at: http://ijedict.dec.uwi.edu//viewarticle.php?id=427&layout=html