to evaluate a CBT course
When reviewing a CBT or Web-based training program, it is easy to be
dazzled by way-cool graphics, video and animations--with the result that
the practicality of the course, given general bandwidth constraints, is
not weighted and the content and instructional value of the program aren't
carefully appraised. We see that tendency in reviews of the several CD-ROM
based encyclopedias; reviewers focus on the number of video segments and
the ease of navigation, for example, but rarely compare the content of
several articles for thoroughness, accuracy, balance and appropriateness
for the audience's purpose, which might be quick checking of a fact, getting
an overview, or as a jumping off place for further research. The result
is that a supermarket-based encyclopedia (Funk & Wagnalls,
a.k.a. Encarta) is ranked more highly than those derived from encyclopedias
that the American Library Association rates much better.
Our Avoiding Sexual Harassment course
contains limited graphics because we've found that photos and drawings
of harassing behavior are a distraction that actually interferes with
the objectives. That's even more true of the program on Avoiding Racial
and Sexual Harassment we've recently created for UPS. Similarly, our
Avoiding Wrongful Discharge
program, which is really an interactive checklist or virtual consultant
to be used when one is deciding whether to fire an employee, has no graphics
at all, but users have found it invaluable as a means of limiting their
liability from certain employment practices.
On the assumption that you care about achieving Kirkpatrick's
four criteria for training effectiveness, here are some suggestions for
evaluating CBT programs. Most important:
- What are the aims (instructional objectives, outcomes) of the program?
- Are they significant to the organization and the individual?
- How will you know if they are achieved?
- Are the behaviors or competencies observable?
- Are they quantifiable?
Is there an examination that provides a high level of assurance that
the student who completes the course and passes the exam has indeed achieved
the objectives? Many exams are patty-cake; the trainee could pass them
without completing the course. Others include passable, but irrelevant
questions. The questions need to measure the achievement of the instructional
objectives. If you could get all the answers correct and still not perform
the specified outcome of the program, the test is not acceptable.
If trainees are likely to vary widely in their education, experience,
responsibilities, etc., are there separate paths through the material
for each segment of the audience? For example, since the law places substantial
responsibility on supervisors to prevent and deal with sexual harassment
incidents, it was appropriate to provide separate tracks for supervisors
and employees in the program on Avoiding
Sexual Harassment. People have different learning styles, and
some competencies are easier to learn from, say, a case study or project
model than from a presentation model. Does the program adapt the instructional
strategy to learning sytles and the nature of the material?
Is the content and style appropriate to the audience and the
objectives of the program? Some subjects are trivialized by trying to
be cute or to show how creative the graphics can be. Wrongful discharge
is a very serious issue and anything that distracts from that seriousness
ought to be reconsidered.
Is the program adaptable to a variety of
distribution channels? Increasingly, these kinds of courses will be available
on networks and on the Web, but the initial efforts are still not much
more than electronic page-turners (little interaction, no practice, no
contingency branching, no proficiency testing). Video is another issue--it's
amazing how a little video can bring a network to its knees, so the trade-offs
between high bandwidth requirements and availability need to be considered.
aggregation & reporting
Can the exam scores and other measures of trainee achievement
be gathered and analyzed automatically? Is the data format compatible
with standard databases and spreadsheets?
How significant (cost and time) is the
task of customizing a proprietary course or revising the content of a
If you'd like to discuss these issues with people who've been in the
business since 1983, please give us a call (908) 627-1234). We'll give
you straight answers, not a sales pitch.
(c) 1999-2002 Guided Learning Systems LLC