how to evaluate a CBT course

When reviewing a CBT or Web-based training program, it is easy to be dazzled by way-cool graphics, video and animations--with the result that the practicality of the course, given general bandwidth constraints, is not weighted and the content and instructional value of the program aren't carefully appraised. We see that tendency in reviews of the several CD-ROM based encyclopedias; reviewers focus on the number of video segments and the ease of navigation, for example, but rarely compare the content of several articles for thoroughness, accuracy, balance and appropriateness for the audience's purpose, which might be quick checking of a fact, getting an overview, or as a jumping off place for further research. The result is that a supermarket-based encyclopedia (Funk & Wagnalls, a.k.a. Encarta) is ranked more highly than those derived from encyclopedias that the American Library Association rates much better.
      Our Avoiding Sexual Harassment course contains limited graphics because we've found that photos and drawings of harassing behavior are a distraction that actually interferes with the objectives. That's even more true of the program on Avoiding Racial and Sexual Harassment we've recently created for UPS. Similarly, our Avoiding Wrongful Discharge program, which is really an interactive checklist or virtual consultant to be used when one is deciding whether to fire an employee, has no graphics at all, but users have found it invaluable as a means of limiting their liability from certain employment practices.

Kirkpatrick's criteria
On the assumption that you care about achieving Kirkpatrick's four criteria for training effectiveness, here are some suggestions for evaluating CBT programs. Most important:

• What are the aims (instructional objectives, outcomes) of the program?
• Are they significant to the organization and the individual?
• How will you know if they are achieved?
• Are the behaviors or competencies observable?
• Are they quantifiable?

Is there an examination that provides a high level of assurance that the student who completes the course and passes the exam has indeed achieved the objectives? Many exams are patty-cake; the trainee could pass them without completing the course. Others include passable, but irrelevant questions. The questions need to measure the achievement of the instructional objectives. If you could get all the answers correct and still not perform the specified outcome of the program, the test is not acceptable.

multiple tracks
If trainees are likely to vary widely in their education, experience, responsibilities, etc., are there separate paths through the material for each segment of the audience? For example, since the law places substantial responsibility on supervisors to prevent and deal with sexual harassment incidents, it was appropriate to provide separate tracks for supervisors and employees in the program on Avoiding Sexual Harassment. People have different learning styles, and some competencies are easier to learn from, say, a case study or project model than from a presentation model. Does the program adapt the instructional strategy to learning sytles and the nature of the material?

appropriate content
Is the content and style appropriate to the audience and the objectives of the program? Some subjects are trivialized by trying to be cute or to show how creative the graphics can be. Wrongful discharge is a very serious issue and anything that distracts from that seriousness ought to be reconsidered.
      Is the program adaptable to a variety of distribution channels? Increasingly, these kinds of courses will be available on networks and on the Web, but the initial efforts are still not much more than electronic page-turners (little interaction, no practice, no contingency branching, no proficiency testing). Video is another issue--it's amazing how a little video can bring a network to its knees, so the trade-offs between high bandwidth requirements and availability need to be considered.

data aggregation & reporting
Can the exam scores and other measures of trainee achievement be gathered and analyzed automatically? Is the data format compatible with standard databases and spreadsheets?
      How significant (cost and time) is the task of customizing a proprietary course or revising the content of a custom program?

If you'd like to discuss these issues with people who've been in the business since 1983, please give us a call (908) 627-1234). We'll give you straight answers, not a sales pitch.
 
         Copyright (c) 1999-2002 Guided Learning Systems LLC