This is G o o g l e's cache of http://www.educationau.edu.au/archives/cp/reeves.htm as retrieved on Jun 13, 2005 14:05:36 GMT.
G o o g l e's cache is the snapshot that we took of the page as we crawled the web.
The page may have changed since that time. Click here for the current page without highlighting.
This cached page may reference images which are no longer available. Click here for the cached text only.
To link to or bookmark this page, use the following url: http://www.google.com/search?q=cache:xziu1MUJEeQJ:www.educationau.edu.au/archives/cp/reeves.htm+reeves+pedagogical+dimensions&hl=en


Google is not affiliated with the authors of this page nor responsible for its content.
These search terms have been highlighted: reeves pedagogical dimensions 

Reeves: Evaluating What Really Matters in Computer-Based Education

Learning with software: pedagogies and practices

Evaluating What Really Matters in Computer-Based Education

Dr Tom Reeves
University of Georgia

 


Introduction

This paper describes fourteen pedagogical dimensions of computer-based education (CBE), each based on some aspect of learning theory or learning concept, that can be used as criteria for evaluating different forms of CBE. The pedagogical dimensions described in this paper are:

  1. epistemology;
  2. pedagogical philosophy;
  3. underlying psychology;
  4. goal orientation;
  5. experiential value;
  6. teacher role;
  7. program flexibility;
  8. value of errors;
  9. motivation;
  10. accommodation of individual differences;
  11. learner control;
  12. user activity;
  13. cooperative learning;
  14. cultural sensitivity.

An initial attempt to apply these criteria (Application of the dimensions in CBE evaluation) to two different examples of CBE is included.

Systematic evaluation of computer-based education (CBE) in all its various forms (including integrated learning systems, interactive multimedia, interactive learning environments, and microworlds) often lags behind development efforts (Flagg, 1990). There are several reasons for this lack of evaluation. First, consumers of technological innovations for education seem to assume that because these innovations are advertised as effective, they are effective. The fallacy of this assumption should be clear to anyone familiar with the generally poor success of CBE in most educational contexts (cf., Cuban, 1990; Siegel, 1994, Shlechter, 1991). Nonetheless, the dominant strategy of the business interests that underwrite the development of CBE has been and continues to be investing much more money in marketing CBE than in evaluating it.

Second, evaluation of CBE has often been reduced to a numbers game wherein the value of CBE is represented by 1) the amount of money spent on hardware and software, 2) the ratio of students to computers, or 3) the amount of time students have access to CBE within a school day, week, month, or year (Becker, 1992). The utility of such indicators in evaluating the ultimate effectiveness and worth of CBE is extremely limited, but their pervasiveness is obvious in the reports produced by national, state, and local education agencies around the world (National Center for Educational Statistics, 1993). This type of quantitative data is relatively easy to collect, analyze, and report. Further, judgments concerning progress within a specific educational entity (school, district, state, or nation) as well as comparisons among different entities can be rendered with a "certainty" that is untainted by the complexity of more ambiguous indicators such as measures of implementation, motivation, and learning.

A third reason for the lack of the evaluation of CBE is the inadequate utility of the evaluations that have been previously conducted. Evaluation reports are usually presented in the format of social science research reports, a format that "is almost useless for most clients and audiences" (Scriven, 1993, p. 77). Further, evaluations of CBE are rarely carried out in a manner timely enough to have sufficient impact on the decisions that must be made in the midst of significant development or implementation efforts. The inadequate utility problem will not be resolved unless educators create evaluation systems that are as integral to educational practice as student assessment systems are today (Reeves, 1992a). In addition, the results of evaluations must be communicated in formats that are accessible to as wide an audience as possible.

A fourth factor in the paucity of useful evaluations of CBE may be that evaluators often rely upon traditional empirical evaluation methods that compare an instructional innovation with another approach. Frequently the results of these studies have been disappointing (Clark, 1992). A major weakness in traditional empirical approaches to evaluation is that the treatments being compared (e.g., interactive multimedia versus classroom instruction) are often assumed to be cohesive, holistic entities with meaningful differences. Berman and McLaughlin (1978) and other implementation researchers (Cooley and Lohnes, 1976) have illustrated the fallacy of assuming that meaningful differences exist between two programs just because they have different names. It is imperative to open up the "black boxes" of instructional alternatives and reveal the relevant pedagogical dimensions they express if evaluations are to be meaningful and have utility. Pedagogical dimensions are the keys to unlocking the black boxes of various forms of CBE.

Pedagogical dimensions can be used to compare one form of CBE with another or to compare different implementations of the same form of CBE. Scriven (1993) maintains that there is an "almost universal necessity to do comparative evaluations" (p. 58), despite the tendency of some evaluation theorists to deny the utility of such comparisons (Cronbach, 1980). The "universal necessity" to conduct comparative evaluations is evidenced by the strong desire of most clients and audiences for such comparisons. Therefore, it is imperative that criteria for evaluating various forms of CBE be developed that will result in more valid and useful evaluations. That is the intent of this paper.

Purpose

The purpose of this paper is to describe fourteen pedagogical dimensions of CBE that have the potential to provide improved criteria for understanding, describing, and evaluating CBE. In physics, dimensions are used to describe a physical quantity or phenomenon in terms of certain fundamental properties such as mass, length, time, or some combination. For example, velocity has the dimensions of length divided by time as in "the car has a maximum speed of 120 miles per hour." Similarly, the phenomena that are forms of CBE can be described in terms of pedagogical dimensions. Pedagogy is defined as the art, science, or profession of teaching. Pedagogical dimensions are concerned with those aspects of the design and implementation of CBE that directly affect learning.

Pedagogical dimensions

Pedagogical dimensions refer to the capabilities of CBE to initiate powerful instructional interactions, monitor learner progress, empower effective teachers, accommodate individual differences, or promote cooperative learning. My first attempt to describe these dimensions was made at the 1992 Information Technology for Training and Education Conference, in Queensland, Australia (Reeves, 1992b). Since then, the dimensions have been revised based upon feedback from colleagues in Australia and the USA. This current set of dimensions is by no means final and further modifications are inevitable.

Pedagogical dimension 1 - Epistemology

Epistemology is concerned with theories about the nature of knowledge. A dimension of CBE important to users of these systems is the theory of knowledge or reality held by the designers. Figure 1 illustrates a dimension of CBE ranging from an objectivist theory of knowledge to a constructivist one. Tobin and Dawson (1992) describe these two theories in relation to interactive learning environments.

Figure 1. Epistemological dimension of CBE

Objectivist epistemology (Thorndike, 1913) encompasses the following facets:

If the designers and users of CBE lean toward an objectivist epistemology, they will be primarily concerned with assuring that the content of the CBE they create and implement is comprehensive and accurate with respect to ultimate "truth" as they know it. They will seek to establish the definitive structure of knowledge for a given domain based upon the advice of the most widely accepted experts in a field. For example, in science education, they will seek to transmit to students the "immutable laws" of any given field.

Advocates of constructivist epistemology, on the other hand, are much more concerned with assuring that the content in CBE reflects the complete spectrum of views of a given domain, ranging from the traditional academic perspectives to the views of the most radical "fringe." Constructivist epistemology calls for a multiplicity of perspectives so that learners have a full range of options from which to construct their own knowledge. In science education, constructivists might provide students with opportunities to rediscover the currently accepted theories of a given science as well as rival theories that may eventually replace the current positions. They might provide coaching or scaffolding to assist students in their discovery, but they would not overly direct the learning process. Constructivist pedagogy is increasingly popular in educational literature today, but few examples exist of its adoption in schools (Nix and Spiro, 1990).

Within education today, there is a tension between those who promote objectivist epistemologies and those who espouse constructivism. Within the context of CBE, the objectivist perspective is perhaps best represented by those who promote integrated learning systems (ILS), (Levinson, 1994), whereas the constructivist perspective may be best represented by those who promote electronic "mindtools" (Jonassen, in press). Major corporations have developed ILS for various sections of the primary, middle, and secondary school curricula. ILS are large scale, networked systems that integrate instruction, assessment, and management functions. In the USA, examples include the Integrated Learning System from Jostens Learning, and SuccessMaker developed by the Computer Curriculum Corporation, a subsidiary of Paramount Communications. These ILS have been produced to take over large portions of the established school curriculum and relegate teachers to the roles of "facilitators." Efficiency in attaining prespecified educational objectives is a frequently touted value of ILS.

In contrast, other educators are leading a movement away from a predominantly "instructivist" pedagogical culture to one that is "constructivist" in nature (Jonassen, in press; Papert, 1993). Instead of regarding knowledge as something that exists outside students which they must passively ingest, knowledge is recognized as being socially and individually constructed on the basis of experience. A recognition is growing that there is no "absolute" knowledge and that there is more than one viable perspective on knowledge in many areas, including mathematics and science. Electronic "mindtools" such as hypertext and multimedia provide opportunities for teachers and students to collaborate in the construction of unique knowledge representations. "HyperCard" from Apple Computer as well as spreadsheets and database programs are examples of "mindtools."

Pedagogical dimension 2 - Pedagogical Philosophy

Rieber (1992) and others (Duffy and Jonassen, 1992; Papert, 1993) make a clear distinction between instructivist and constructivist approaches to teaching and learning. Another way of thinking about these orientations is in terms of pedagogical philosophies. Figure 2 illustrates a dimension of CBE ranging from a strict instructivist philosophy to a radical constructivist one.

Figure 2. Pedagogical philosophy dimension of CBE

Instructivists stress the importance of goals and objectives that exist apart from the learner. These goals and objectives are drawn from a domain of knowledge, e.g., algebra, or extracted from observations of the behaviors of experts within a given domain, e.g., surgeons. Once goals and objectives are delineated, they are sequenced into learning hierarchies, generally representing a progression from lower to higher order learning. Then, direct instruction is designed to address each of the objectives in the hierarchy, often employing instructional strategies derived from behavioral psychology (Rieber, 1992). Relatively little emphasis is put on the learner per se who is usually viewed as a passive recipient of instruction. CBE based on instructivist pedagogy generally treats learners as empty vessels to be filled with learning. Direct instruction demands that content be sharply defined and that instructional strategies focus as directly on prespecified content as possible.

Alternatively, constructivists emphasize the primacy of the learner's intentions, experience, and metacognitive strategies. Rieber (1992) describes the constructivist view of learning as involving "individual constructions of knowledge" (p. 94). In this view, learners attain a state of cognitive equilibrium through reconstruction of concepts, schema, mental models, and other cognitive structures in the face of new information and experience that may conflict with earlier constructions. A major goal in constructivist pedagogy is to ensure that the learning environment is as rich as possible. Emphasis is placed on identifying the unique interests, styles, motivations, and capabilities of individual learners so that learning environments can be tailored to them. Instead of an empty vessel, the learner is regarded as an individual replete with pre-existing knowledge, aptitudes, motivations, and other characteristics that are difficult to assess, much less accommodate. Constructivists often argue for replacing direct instruction with self-directed exploration and discovery learning.

Different forms of CBE are based upon different pedagogical philosophies. Traditional computer-based tutorials, drill-and-practice programs, and contemporary ILS mesh well with instructivist pedagogies. Alternatively, interactive learning environments (Hannafin, 1992), microworlds (Rieber, 1992), and "mindtools" (Jonassen, in press) are forms of CBE that enable the implementation of constructivist pedagogy. It must be noted that the degree to which educators, parents, and community leaders emphasize one pedagogical philosophy over another appears to be strongly influenced by religious and political beliefs.

Pedagogical dimension 3 - Underlying Psychology

At the risk of ignoring a number of other important theoretical perspectives (e.g., developmental psychology), a dimension related to the basic psychology underlying CBE is proposed. Figure 3 illustrates this dimension with behavioral psychology at one end of the continuum and cognitive psychology at the other.

Figure 3 Underlying psychology dimension of CBE

Debunking behavioral psychology has become quite fashionable, despite a few staunch defenders (Gilbert and Gilbert, 1991). Therefore, it seems ironic that behavioral psychology continues to be the underlying psychology for many forms of CBE. According to classical behavioral psychology (Skinner, 1968), the important factors in learning are not internal states that may or may not exist, but behavior that can be directly observed. Instruction consists primarily of the shaping of desirable behaviors through the scientific arrangement of stimuli, responses, feedback, reinforcement, and other contingencies. First, a stimulus is provided, often in the form of a short presentation of content. Second, a response is demanded, often in the form of a question. Third, feedback is given as to the accuracy of the response. Fourth, positive reinforcement is given for accurate responses. Fifth, inaccurate responses result in either a repetition of the original stimulus or a somewhat modified (often simpler) version of it, and the cycle begins again.

Cognitive psychology, on the other hand, has captured the attention of many educators today, and virtually all self-respecting instructional design theorists now claim to be cognitivists (Gagné and Glaser, 1987). Without ignoring behavior, cognitive psychology places much more emphasis on internal mental states than behavioral psychology. Kyllonen and Shute (1989) have proposed a taxonomy that represents the spectrum of internal states with which cognitive psychologists are concerned. Their taxonomy begins with simple propositions (e.g., stating that Japan sells more electronic products than any other nation), proceeding through schema, rules, general rules, skills, general skills, automatic skills, and finally, mental models (e.g., analyzing the potential of a trade war between Japan and the United States based on an analysis of balance of trade trends). The latter type of knowledge seems particularly important because mental models are the basis for generalizable problem-solving abilities (Halford, 1993).

Cognitive psychologists recognize that a wide variety of learning strategies may have to be employed in any given instructional setting depending upon the type of knowledge to be constructed. Learning strategies include memorization, direct instruction, drill-and-practice, deduction, and induction (Schank and Jona, 1990). Different forms of CBE vary in their capacity to implement these different learning strategies. Whereas an ILS may provide adequate opportunities for direct instruction and drill-and-practice, some sort of mindtool or microworld may be required to support deductive and inductive learning strategies.

Pedagogical dimension 4 - Goal Orientation

The goals and objectives of CBE can range from sharply focused ones (e.g., following strict protocols for handling medical emergency situations) to more or less unfocused ones (e.g., learning to appreciate modern art). Figure 4 illustrates a dimension of CBE related to the degree of focus represented in the goals of an interactive program.

Figure 4 Goal orientation dimension of CBE.

Cole (1992) clarifies the relevance of different types of goals to the design of CBE. She maintains that some knowledge "has undergone extensive social negotiation of meaning and which might most efficiently and effectively be presented more directly to the learner" (p. 29). In such cases, direct instruction, perhaps in the form of a computer-based tutorial, may suffice for learning. Other knowledge is so tenuous, creative, or of a higher level (e.g., mental models) that direct instruction is inappropriate. In the latter cases, CBE programs that promote inductive learning such as microworlds (Rieber, 1992), virtual reality simulations (Henderson, 1991), and learning environments (Hannafin, 1992) are much more appropriate.

Although, there are many advocates of discovery-based environments for the learning of social studies, science, and even mathematics in schools, most of these people would probably prefer their brain surgeons to be trained via direct instruction. However, there are examples of alternative approaches to learning being applied even in medical schools. Bransford, Sherwood, Hasselbring, Kinzer, and Williams (1990) describe the sequencing problem in the context of medical education. Most medical schools follow a sequence whereby students memorize great quantities of factual information during their first two years of training and then spend the next two years in various clinical settings where they may or may not have opportunities to use the memorized knowledge. A few enlightened medical schools have begun to place students in clinical settings from day one while providing them with the pedagogical support to learn basic knowledge and skills as needed. Perelman (1992) describes medical schools in Canada and The Netherlands that successfully employ this innovative approach.

Although it might be tempting to delegate the teaching of sharply-focused goals and objectives to commercial ILS, tutorials, and drill-and-practice programs, insufficient research has been conducted on which to base this decision. In addition, the infusion of mindtools, learning environments, and microworlds within traditional school curricula has been so limited that their effects on various types of learning goals and objectives are unclear.

Pedagogical dimension 5 - Experiential Validity

The earliest type of systematic learning activity probably involved some sort of apprenticeship whereby a novice worked side by side with a master. Apprenticeships have high, i.e., concrete, experiential value. More abstract learning activities, e.g., classroom lectures, were developed much later in history. A major criticism of much of our current dominant pedagogical schemes is that they are too abstract, removed as they are from "real world" experience (Brown, Collins, and Duguid, 1989). Figure 5 illustrates an experiential value continuum ranging from abstract to concrete.

Figure 5 Experiential value dimension of CBE

An important concern for educators and trainers alike is the degree to which classroom learning transfers to external situations in which the application of knowledge, skills, and attitudes is appropriate. The cognitive theories of Newell and Simon (1972), Anderson (1983), Brown (1985), and others support the fundamental principle that the way in which knowledge, skills, and attitudes are initially learned plays an important role in the degree to which these abilities can be used in other contexts. To put it simply, if knowledge, skills, and attitudes are learned in a context of use, they will be used in that and similar contexts. This principle is especially important in vocational education.

In traditional instruction, information is presented in encapsulated formats, often via abstract lectures and texts, and it is largely left up to the student to generate any possible connections between conditions (such as a problem) and actions (such as the use of knowledge as a tool to solve the problem). There is ample evidence that students who are quite adept at "regurgitating" memorized information rarely retrieve that same information when confronted with novel conditions that warrant its application (Bransford et al., 1990; Perelman, 1992).

CBE can be designed to present a focal event or problem situation that will serve as an "anchor" or focus for collaborative efforts among instructors and students to retrieve and construct knowledge (Brown et al., 1989). Cognitive psychologists at the Cognition and Technology Group at Vanderbilt University (CTGV) call this type of instruction "anchored instruction" (Bransford et al, 1990; CTGV, 1992) because the process of constructing new knowledge is situated or anchored in meaningful and relevant contexts. They maintain that events and problems presented in CBE should be designed to be intrinsically interesting, problem-oriented, and challenging. They have evidence that in response to these types of events and problems, students construct useful as opposed to inert knowledge (Bransford et al., 1990; CTGV, 1992).

Pedagogical dimension 6 - Teacher Role

CBE can be designed to support different pedagogical roles for teachers. Some CBE are designed to place teachers in the role of a "facilitator." Other programs are designed to support the more traditional didactic role of an instructor as "the teacher." Figure 6 represents a continuum of teacher roles ranging from didactic to facilitative.

Figure 6 Teacher role dimension of CBE

The didactic roles of teachers are well-established. A quarter century ago, Carroll (1968) told us that "By far the largest amount of teaching activity in educational settings involves telling things to students..." (p. 4). More recent analyses of teaching indicate that little has changed since then (cf., Goodlad, 1984; Kidder, 1989; Perelman, 1992). Where teacher exposition is an appropriate instructional strategy, CBE can be designed to support, reinforce, and extend teacher presentations.

It has become commonplace today in education circles to talk about changing the teacher's role from a traditional didactic one to that of a facilitator. The Cognition and Technology Group at Vanderbilt (CTGV, 1992) describe a shift in the teacher's role "from authoritarian provider of knowledge to a resource who at times is consulted by students and at other times can become the student whom others teach" (p. 73). In addition to the constructivist learning environments such as the Jasper Woodbury Problem-Solving Series (CTGV, 1992), producers of large scale integrated learning systems (ILS) claim to assign teachers roles as facilitators. However, there may be important differences between the facilitator tasks of a teacher using Jasper and those carried out by a teacher implementing a commercial ILS. For instance, there is a danger that teachers using ILS may be so occupied with making sure the ILS are functioning properly and troubleshooting any problems, that they might not be able to conduct the one-to-one and small group teaching that the systems were supposed to allow teachers to conduct.

Pedagogical Dimension 7 - Flexibility

A hidden agenda of some forms of CBE seems to be making them "teacher-proof," perhaps because of a belief that earlier instructional innovations have failed as a result of teacher interference (Winn, 1989). Alternatively, other forms of CBE exist in which teachers have considerable leeway to modify program activities. Figure 7 represents a continuum of program flexibility ranging from "teacher-proof," i.e., unchangeable, to "easily modifiable."

Figure 7 Program flexibility dimension of CBE.

Teacher-proof approaches have fervent contemporary advocates. Some forecast the replacement of teachers with increasingly humanlike CBE. For example, Winn (1989) wrote "Educational technology can only become a viable discipline and profession if it concentrates on developing alternatives to the teacher-based model of public education rather than trying to alter it, improve it, or even just serve it" (p. 36). A new video created by ATandT as its vision of the future portrays students sitting in front of individual terminals interacting with computer-generated teachers while an adult stands nearby, seemingly in the role of monitoring their behavior (ATandT, 1993).

On the other hand, proponents of program flexibility must deal with the history of inadequate implementation that has hindered decades of educational innovations (Berman and McLaughlin, 1978). Modifying an innovative program has often resulted in insufficient fidelity in implementing a program's effective dimensions. Nonetheless, prohibiting local adaptation will lessen opportunities for creative modifications that may actually enhance effectiveness. The issue of program flexibility is a complex one that must be addressed by efforts to assess implementation very carefully during any evaluation. Forms of CBE must be designed to walk the fine line between being so "teacher-proof" that they do not allow local adaptation (and may even encourage sabotage) and being so open or unstructured that they do not provide sufficient guidance and support for valid implementation.

Pedagogical dimension 8 - Value of Errors

The old maxim that "experience is the best teacher" reflects a belief that we learn much in life through trial and error (CTGV, 1992). Although this approach is inefficient and even dangerous in some contexts, experiential learning is highly valued simply because it provides opportunities for us to "learn from our mistakes." On the other hand, some educational theorists, especially proponents of programmed instruction, have maintained that ideal learning involves no errors. These developers attempt to arrange the contingencies of instruction in such a way that learners can only make correct responses. Figure 8 presents a continuum of perspectives concerning the value of errors ranging from errorless learning to learning from "trial and error" experience.

Figure 8 Value of errors dimension of CBE

An example of a CBE program that prohibits errors is the Principles of the Alphabet Learning System (PALS) designed for the IBM Corporation by Dr. John Henry Martin (1986). PALS uses interactive videodisc technology to teach basic literacy skills to adolescents and adults. At specific intervals, learners are required to type in letters to form the words that on-screen characters say. However, only those keys that match an acceptable spelling of the words are enabled. Pressing the wrong keys puts nothing on the screen except more and more refined directions as to the desired response. This "errorless" approach is also an element of IBM's Writing To Read program (Freyd and Lytle, 1990).

Such an errorless approach contrasts sharply with forms of CBE that employ high fidelity simulation as an instructional strategy. In "The Case of Dax Cowart," an interactive videodisc simulation created at the Center for the Design of Educational Computing at Carnegie Mellon University (Covey and Cavalier, 1989), college students are placed in the roles of members of a hospital ethics panel that must decide whether a horribly burned patient can be allowed to die as he has requested or must undergo months of excruciatingly painful treatments. Regardless of a student's decision, he or she is confronted with the negative outcomes of that decision. In this simulation, each choice is treated as an "error" from which valuable lessons can be learned.

Pedagogical dimension 9 - Origin of Motivation

Motivation is a primary factor in instructional models (Carroll, 1963). Rieber (1992) describes five design principles for CBE derived from constructivism. The first is to "provide a meaningful learning context that supports intrinsically motivating and self-regulated learning" (p. 98). Intrinsic motivation has been held forth as the "Holy Grail" to which all CBE programs should aspire (Malone, 1984). Figure 9 illustrates a motivation dimension that ranges from extrinsic (i.e., outside the learning environment) to intrinsic (i.e., integral to the learning environment).

Figure 9 Motivation dimension of CBE.

Intrinsically motivating instruction is very elusive regardless of the delivery system, but virtually every new approach to come along promises to be more motivating than any that have come before. Interactive multimedia is the latest type of interactive learning system that is supposed to motivate learners automatically, simply because of the integration of music, voice, still pictures, text, animation, motion video, and a friendly interface on a computer screen. In practice, as Keller (1987) has specified, motivation aspects must be consciously designed into multimedia just as rigorously as any other pedagogical dimensions. An assumption underlying many commercial multimedia packages seems to be that students will be intrinsically motivated to explore these systems in search of new knowledge. Very little research exists that examines this assumption, but Harmon (1992) found that students using these programs were more likely to seek out confirmation of things they already knew than to seek new knowledge. It seems that the current state-of-the-art of CBE is such that extrinsic motivation will remain a critical factor in many educational contexts.

Pedagogical dimension 10 - Accommodation of Individual Differences

Although it might be assumed that the main reason for employing CBE would be accommodating individual differences among learners, this is not always the case. Some CBE programs have very little, if any, provision for individual differences whereas others are designed to accommodate a wide range of individual differences including personalistic, affective, and physiological factors (Ackerman, Sternberg, and Glaser, 1989). Figure 10 illustrates a continuum of accommodations of individual differences that ranges from non-existent to multi-faceted.

Figure 10 Accommodation of individual differences dimension of CBE

The impact of individual differences is a major factor in the effectiveness of CBE. Learning is a function of the learner, the content to be learned, and the features of the instruction (Sternberg, 1985). Many theoretical models of learning treat individual differences among learners as the major predictor of differential learning outcomes (cf., Carroll, 1963). In most educational contexts, we cannot be guaranteed that learners will be homogeneous in terms of aptitudes, prerequisite knowledge, motivation, experience, learning styles, eye-hand coordination, etc. Therefore, we must provide scaffolding, cognitive bootstrapping, and other types of metacognitive support to promote learning (Resnick, 1989). Examples of CBE that provide comprehensive metacognitive support are difficult to identify (Cates, 1992).

Pedagogical dimension 11 - Learner Control

Learner control has been one of the most heavily researched dimensions of CBE in recent years (Steinberg, 1989). Figure 11 illustrates a dimension of CBE that can range from complete program control to unrestricted learner control.

Figure 11 Learner control dimension of CBE

Learner control refers to the options in CBE that allow learners to make decisions about what sections to study and/or what paths to follow through interactive material. The popular wisdom is that learner control makes CBE more effective by individualizing the instruction and making it more motivating, but all too often experimental studies have led to no significant results in terms of the predicted main effects (Williams, 1993). Reeves (1993) describes critical theoretical and methodological flaws in learner control studies. Ross and Morrison (1989) concluded that "research findings regarding the effects of learner control as an adaptive strategy have been inconsistent, but more frequently negative than positive" (p. 28). Better research is needed before questions about the learner control issue can be answered (Reeves, 1993).

Pedagogical dimension 12 - User Activity

Hannafin (1992) identified another important dimension of CBE, especially those forms of CBE that he and others characterize as "learning environments." He maintains that some learning environments are primarily intended to enable learners to "access various representations of content" (p. 59). He labels these "mathemagenic" environments. Other learning environments, called "generative" by Hannafin, engage learners in the process of creating, elaborating or representing knowledge. Figure 12 illustrates this continuum of user activity.

Figure 12 User activity dimension of CBE

Generative learning environments are aligned most closely with constructivist pedagogy whereas mathemagenic environments are often based upon instructivist pedagogy, but this is not necessarily always obvious. Contemporary CBE programs such as the ABC News Interactive series (ABC News Interactive, 1991) and the IBM Ultimedia programs (IBM Corporation, 1991) include generative capabilities nested within otherwise mathemagenic presentations of content. On the other hand, mindtools such as HyperCard have great potential for enabling generative learning (Jonassen, in press).

Pedagogical dimension 13 - Cooperative Learning

Support for the value of cooperative learning is growing throughout education circles (Slavin, 1992). CBE can be designed to thwart or promote cooperative learning. In fact, some CBE programs require cooperative learning (IBM Corporation, 1986) whereas others make no provision for its support. Figure 13 illustrates a cooperative learning dimension ranging from a complete lack of support for cooperative learning to the inclusion of cooperative learning as an integral part of CBE.

Figure 13 Cooperative learning dimension of CBE

Cooperative learning refers to instructional methods in which learners work together in pairs or small groups to accomplish shared goals (Slavin, 1992). Johnson and Johnson (1987) and Slavin (1990) present evidence that when CBE (and other instructional delivery systems) are structured to allow cooperative learning, learners benefit both instructionally and socially. Some commercial ILS have been designed to be used by two or more learners working cooperatively. In addition, multimedia construction programs (such as Authorware Professional and Macromind Director) are so complex that they usually require team-based usage in school contexts.

Pedagogical dimension 14 - Cultural Sensitivity

Henderson (1994) provided a valuable critique of an earlier version of these pedagogical dimensions (Reeves, 1992b). Henderson maintains that the assumptions underlying a specific point on any of these dimensions has a cultural element that should not be ignored. For example, whereas a constructivist pedagogy advocates, indeed demands, persistent questioning on the part of learners, questions, especially "why?" questions, are inappropriate in cultures such as the Torres Strait Islanders of Australia. Although CBE may not be able to adapt to every cultural norm, they should be designed to be as culturally sensitive as possible (Powell, 1993). Figure 14 illustrates a cultural sensitivity dimension ranging from non-existent to integral.

Figure 14 Cultural sensitivity dimension of CBE

Powell (1993) revealed that few instructional design courses include cultural diversity as an important factor in designing effective instructional programs. Therefore, it should not be surprising that few CBE programs have been developed in which cultural sensitivity is integral to their design. To be sure, a few instances of CBE include what Henderson (1994) labels "tokenistic gestures" by allowing an occasional minority role for an actor or perhaps by including culturally diverse, albeit safe, references in terms of music, location, or other cultural aspects. It is difficult to describe what comprehensively culturally sensitive CBE would be like, but at the very least such programs would accommodate diverse ethnic and cultural backgrounds among learners.

Application of the dimensions in CBE evaluation

To illustrate the potential utility of the pedagogical dimensions of CBE described above, the last part of this paper presents an analysis of two examples of CBE employing these dimensions. Experienced developers and users of CBE possess the necessary background and objectivity to provide a reliable and valid assessments of the pedagogical dimensions of these systems. Although the final ratings reported below are mine, they have been influenced by colleagues in Australia and the USA with whom I have discussed these programs. There is an inevitable degree of subjectivity in this analysis and additional applications of these dimensions involving experienced personnel in other education contexts are invited.

The two examples of CBE used in this application are the Writing To Read program designed by Dr. John Henry Martin and widely disseminated by the IBM Corporation (IBM Corporation, 1985) and the Jasper Woodbury Problem Solving Series developed by the Cognition and Technology Group at Vanderbilt University (CTGV, 1992). The Writing To Read (WTR) program is intended to improve the reading and writing performance of students in kindergarten and first grade. During WTR periods lasting an hour per day, children rotate among five workstations, two of which involved CBE. The primary computerized workstation in WTR provides students opportunities to learn and practise phonics skills. Computer guided activities include keying in sounds, words, and eventually sentences. The program emphasizes the learning of 42 phonemes, letter-sound combinations that can be used to "spell" any words in the English language. Often, the computer requires verbal as well as keyed in responses, and occasionally students are prompted to clap or stomp their feet in time with computer presentations.

Few examples of CBE have been more extensively evaluated than WTR and few programs are more controversial. Slavin (1990b) concluded that the results of WTR are disappointing. On the other hand, Chira (1990) described the enthusiastic reception of WTR as a statewide program in Mississippi. Estimates are that more than ten percent of the kindergarten and first grade students in the USA used WTR in the 1992-93 academic year, making it one of the largest implementations of CBE in any setting.

The Jasper Woodbury Problem Solving Series (CTGV, 1992) was created in an academic environment within the context of a long term research and development program. Its use until recently has been confined largely to a few dozen schools in the southeast section of the USA, but it is now commercially available. The Jasper Series represents an attempt to implement constructivist learning principles. These programs (which are provided in both interactive videodisc and linear video versions) provide students with opportunities to develop advanced mathematical problem-solving skills within the context of a series of high-interest video adventures. Students discover the need to develop mathematical skills within the context of flying planes and operating motor boats to solve simulated dilemmas. Numerous studies have been and are being conducted using the Jasper series of programs (Bransford et al., 1990).

The Jasper series is an example of what Hannafin (1992) calls a "generative" learning environment, i.e., a program that requires students to construct or generate their own knowledge as opposed to one that requires them to select knowledge from prepackaged options. Knowledge constructed in generative environments is more likely to generalize than the inert knowledge acquired in traditional passive learning environments (CTGV, 1992).

Figure 15 presents a profile of the Writing To Read and Jasper programs using fourteen pedagogical dimensions. My ratings of these programs are based on limited observations in schools, demonstrations of the programs at professional conferences, and reading several extensive reports about them, but not first-hand experience in implementing the programs myself. My analysis reveals that WTR is based on objectivist, instructivist, and behavioral foundations. It is a highly structured program. One of its most notable features is its provision for errorless learning, e.g., students are not allowed to key in incorrect responses to questions. Alternatively, the Jasper programs are grounded in constructivist and cognitivist foundations. Teachers are integral facilitators in implementing Jasper, and they are encouraged to modify it according to their local needs. Collaborative learning is strongly supported in this program. It appears to be an advanced example of a generative learning environment.

Figure 15. Pedagogical dimensions of the Writing To Read and Jasper programs

Conclusions

The preceding analysis is an admittedly preliminary investigation into the value of these pedagogical dimensions. Hence, the following recommendations are made for improving their utility. First, the dimensions should be subjected to rigorous expert review by leaders in the design and application of CBE. Second, once there is evidence for the qualitative validity of the dimensions, quantitative scales should be integrated into each dimension, e.g., a ten point rating system. Quantitative values have not been added to the dimensions up to now for fear that reviewers might get too distracted by the numerical values to concentrate on the qualitative aspects of the dimensions themselves. However, there is certainly merit and utility in eventually grounding the ratings in quantitative values. Third, the validated dimensions should be applied to many different forms of CBE in a wide variety of educational contexts to provide evidence for their utility. Fourth, research should be initiated into the relationships among ratings of the pedagogical dimensions of CBE and actual data regarding the instructional effectiveness and impact of these same programs.

The fourteen pedagogical dimensions described above are by no means the final answer to improving evaluations of CBE in education. A comprehensive approach to evaluating CBE requires multiple levels of design, data collection and interpretation. We must explore many alternatives. Each month sees the introduction of new commercial CBE packages advertised as effective instructional systems. Yet systematic evaluation of the implementation and efficacy of these systems is sadly lacking. In addition, many evaluators continue to employ outmoded experimental designs. Papert (1993) sums up the inadequacy of these traditional evaluation designs: "The method of controlled experimentation that evaluates an idea by implementing it, taking care to keep everything else the same, and measuring the result, may be an appropriate way to evaluate the effects of a small modification. However, it can tell us nothing about ideas that might lead to deep change" (p. 27).

In education today, we need "deep change," and therefore improving evaluation of CBE has never been more important. Technological advancements are increasing at an ever faster pace especially with respect to telecommunications and multimedia. At the same time, few teachers feel confident and competent with respect to the goals and functions of CBE in their classrooms (Becker, 1992; Siegel, 1994). Despite some efforts to introduce pre-service teachers to computer education in their teacher preparation programs, Becker (1992) found that over half of the future teachers he surveyed never used a computer in any of their college courses. At least part of the problem may stem from a restricted vision of CBE as simply an alternative delivery system for traditional pedagogy rather than as a tool for implementing alternative pedagogical dimensions. Evaluation approaches based upon clearer delineation of the pedagogical dimensions within different types of CBE will surely be a step forward.

References

ABC News Interactive Corporation. (1991). In the holy land: A history of hatred in the Middle East [Interactive multimedia program]. New York: ABC News Interactive.

Ackerman, P. L., Sternberg, R. J., and Glaser, R. (Eds.). (1989). Learning and individual differences: Advances in theory and research. New York, W. H. Freeman.

Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.

ATandT. (1993). Connections: ATandT's vision of the future [Videotape]. Newark, NJ: ATandT.

Becker, H. J. (1992). Computer education. In M. C. Alkin (Eds.), Encyclopedia of educational research (pp. 232-235). New York: Macmillan.

Berman, P., and McLaughlin, M. (1978). Federal programs supporting education change. A model of education change, Vol. VIII: Implementing and sustaining innovations. Santa Monica, CA: Rand.

Bransford, J. D., Sherwood, R. D., Hasselbring, T. S., Kinzer, C. K., and Williams, S. M. (1990). Anchored instruction: Why we need it and how technology can help. In D. Nix and R. Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology (pp. 115-141). Hillsdale, NJ: Lawrence Erlbaum.

Brown, J. S. (1985). Process versus product: A perspective on tools for communal and informal electronic learning. Journal of Educational Computing Research, 1, 179-201.

Brown, J. S., Collins, A., and Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-41.

Carroll, J. B. (1968). On learning from being told. Educational Psychologist, 5, 4-10.

Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64, 723-733.

Cates, W. M. (1992, April). Considerations in evaluating metacognition in interactive hypermedia/multimedia instruction. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco.

Chira, S. (1990, January 24). Electronic teacher: A Mississippi experiment. New York Times, A1, B5.

Clark, R. E. (1992). Media use in education. In M. C. Alkin (Ed.), Encyclopedia of educational research (pp. 805-814). New York: Macmillan.

Cognition and Technology Group at Vanderbilt. (1992). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 65-80.

Cole, P. (1992). Constructivism revisited: A search for common ground. Educational Technology, 32(2), 27-34.

Cooley, W. W., and Lohnes, P. R. (1976). Evaluation research in education. New York: Irvington.

Covey, P., and Cavalier, R. (1989). The Case of Dax Cowart [Interactive videodisc program]. Pittsburgh, PA: Center for the Design of Educational Computing, Carnegie Mellon University.

Cronbach, L. J., Ambron, S. E., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F., Weiner, S. S. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.

Cuban, L. (1990). Reforming again, again, and again. Educational Researcher, 19(1), 3-13.

Duffy, T., and Jonassen, D. H. (Ed.). (1992). Constructivism and the technology of instruction: A conversation. Hillsdale. NJ: Lawrence Erlbaum.

Flagg, B. N. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum.

Freyd, P., and Lytle, J. H. (1990). A corporate approach to the 2 R's: A critique of IBM's Writing To Read program. Educational Leadership, 47(6), 83-88.

Gagné, R. M., and Glaser, R. (1987). Foundations in learning research. In R. M. Gagné (Ed.). Instructional technology foundations (pp. 49-83). Hillsdale, NJ: Lawrence Erlbaum.

Gilbert, M. B., and Gilbert, T. F. (1991). What Skinner gave us. Training, 23(9), 42-48.

Goodlad, J. I. (1984). A place called school. New York: McGraw-Hill.

Halford, G. S. (1993). Children's understanding: The development of mental models. Hillsdale, NJ: Lawrence Erlbaum.

Hannafin, M. J. (1992). Emerging technologies, ISD, and learning environments: Critical perspectives. Educational Technology Research and Development, 40(1), 49-63.

Harmon, S. W. (1992). On the nature of exploratory behavior in hypermedia environments: Considerations of learner use patterns of hypermedia environments for the design of hypermedia instructional systems. Unpublished doctoral dissertation, The University of Georgia.

Henderson, J. (1991). Designing realities: Interactive media, virtual realities, and cyberspace. Multimedia Review, 1(2), 47-51.

Henderson, L. (1994, January). Reeves' pedagogic model of interactive learning systems and cultural contextuality. Paper presented at the Second International Interactive Multimedia Symposium, Perth, Western Australia.

IBM Corporation. (1991). Illuminated books and manuscripts [Interactive multimedia program]. Atlanta, GA: IBM Corporation.

IBM Corporation. (1986). Principles of the alphabet learning system (PALS) [Interactive videodisc program]. Atlanta, GA: IBM Corporation.

IBM Corporation. (1985). Writing to read [Interactive learning system program]. Atlanta, GA: IBM Corporation.

Johnson, D. W., and Johnson, R. T. (1987). Learning together and alone (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.

Jonassen, D. H. (in press). Mindtools in schools. New York: Macmillan Publishing.

Keller, J. M. (1987). Strategies for stimulating the motivation to learn. Performance and Instruction, 26(8), 1-7.

Kidder, T. (1989). Among schoolchildren. Boston: Houghton Mifflin.

Kyllonen, P. C., and Shute, V. J. (1989). A taxonomy of learning skills. In P. L. Ackerman, R. H. Sternberg, and R. Glaser (Eds.), Learning and individual differences: Advances in theory and research (pp. 117-163). New York: W. H. Freeman.

Levinson, E. (1994). The word on buying an ILS. Electronic Learning, 13(5), 12-13.

Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In D. F. Walker and R. D. Hess (Eds.), Instructional software: Principles and perspectives for design and use. Belmont, CA: Wadsworth Publishing.

National Center for Educational Statistics. (1993). The digest of educational statistics 1993 (No. 065-000-00606-5). Pittsburgh, PA: U. S. Government Printing Office.

Newell, A., and Simon, H. (1972). Human problem-solving. Englewood Cliffs, NJ: Prentice-Hall.

Nix, D., and Spiro, R. (Eds.). (1990). Cognition, education, and multimedia: Exploring ideas in high technology. Hillsdale, NJ: Lawrence Erlbaum.

Papert, S. (1993). The children's machine: Rethinking school in the age of the computer. New York: Basic Books.

Perelman, L. J. (1992). School's out: Hyperlearning, the new technology, and the end of education. New York: William Morrow.

Powell, G. C. (1993). Incorporating learner cultural diversity into instructional systems design: An investigation of faculty awareness and teaching practices. Unpublished doctoral dissertation, The University of Georgia.

Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39-46.

Reeves, T. C. (1992a). Evaluating schools infused with technology. Education and Urban Society Journal, 24(4), 519-534.

Reeves, T. C. (1992b, September). Effective dimensions of interactive learning systems. Invited keynote paper presented at the Information Technology for Training and Education (ITTE `92) Conference, Queensland, Australia.

Resnick, L. B. (1989). Introduction. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum.

Rieber, L. P. (1992). Computer-based microworlds: A bridge between constructivism and direct instruction. Educational Technology Research and Development, 40(1), 93-106.

Ross, S. M., and Morrison, G. R. (1989). In search of a happy medium in instructional technology research: Issues concerning external validity, media replications, and learner control. Educational Technology Research and Development, 37(1) 19-33.

Schank, R. C., and Jona, M. Y. (1990). Empowering the student: New Perspectives on the design of teaching systems (Tech. Rep. No. 4). Evanston, IL: The Institute for the Learning Sciences, Northwestern University.

Scriven, M. (1993). Hard-won lessons in program evaluation. San Francisco, CA: Jossey-Bass.

Shlechter, T. M. (Ed.). (1991). Problems and promises of computer-based training. Norwood, NJ: Ablex Publishing.

Siegel, J. (1994). No computer know how. Electronic Learning, 13(5), 58.

Skinner, B. F. (1968). Technology of teaching. New York: Meredith Publishing.

Slavin, R. E. (1992). Cooperative learning. In M. C. Alkin (Ed.), Encyclopedia of educational research (pp. 235-238). New York: Macmillan.

Slavin, R. E. (1990a). Cooperative learning: Theory, research, and practice. Englewood Cliffs, NJ: Prentice Hall.

Slavin, R. E. (1990b). IBM's writing to read: Is it right for reading? Phi Delta Kappan, 72, 214-216.

Steinberg, E. R. (1989). Cognition and learner control: A literature review, 1977-88. Journal of Computer-Based Instruction, 16(4), 117-121.

Sternberg, R. J. (1985). Beyond I.Q.: A triarchic theory of human intelligence. Cambridge, MA: Cambridge University Press.

Thorndike, E. L. (1913). Educational psychology: The psychology of learning (Vol. 2). New York: Teachers College Press.

Tobin, K., and Dawson, G. (1992). Constraints to curriculum reform: Teachers and the myths of schooling. Educational Technology Research and Development, 40(1), 81-92.

von Glasersfeld, E. (1989). Cognition, construction of knowledge, and teaching. Syntheses, 80(1), 121-140.

Williams, M. D. (1993). A comprehensive review of learner-control: The role of learner characteristics. In M. R. Simonson (Ed.), Proceedings of the Annual Conference of the Association for Educational Communications and Technology (pp. 1083-1114). New Orleans, LA: Association for Educational Communications and Technology.

Winn, W. (1989). Toward a rationale and theoretical basis for educational technology. Educational Technology Research and Development, 37(1), 35-46.

 


Contact inform@educationau.edu.au with questions or problems.

Last modified: 21 November 1997