TEST SCORES FOOLISHNESS

October 6, 2014

Imagine my surprise when I read a long-established training vendor’s recent press release. Omitting the organization’s name, I’m going to quote from it:

“(Company X) conducted a survey of its clients to determine the effectiveness of online training. Comparing over 100,000 pre- and post-test scores, (Company X) found that clients using (Company X’s) courses significantly increased their trainees’ technical and safety knowledge.

After reviewing the tests results from a large population of trainees, (Company X) can demonstrate a 30% increase in technical skills knowledge when comparing pre-test to post-test scores.”

What an empty claim! Ridiculous! Nonsensical propaganda!

And, in the 1980s, an assertion made by many training vendors regarding the videotapes they then hawked — before the customers figured out the vacuousness of such claims.

Think about the meaninglessness of these results for a minute.

The only true measure of the courseware you provide your trainees will be tied directly to an increase in the trainees’ on-the-job performance — which translates into long term retention and increased skills.

Short-term improvement, as measured by “immediate testing” (pre- and post-test score differential) means next to nothing.

As the Pittsburgh Science of Learning Center has discovered: “Long-term retention involves the recall and use of knowledge after a relatively long period of time has passed since instruction on that knowledge. . . . In PSLC studies, a post-test given immediately after the instruction is not a long-term retention test.”

Just remember your own experience in high school and college classrooms. You studied late at night to pass a test and then, within days or weeks, you forgot much of what you had temporarily learned. You experienced short-term retention but impacted your long-term retention knowledge very little.

The truth is that every vendor of training can make the same claim as Company X. All of them can demonstrate a marked increase in test scores between the pre-test and the post-test. No vendor has an advantage in that measurement tool.

But, you should be looking at long-term retention which can only be measured if you re-administer a post-test six months, or more, after the initial training. Combined with the evaluation given by the trainee’s supervisor that is the only way you will be able to measure the long term effectiveness of your training initiatives.

What’s more, when it comes to correctly designed online learning, a student should be able to demonstrate 100% short term mastery. Well designed online learning allows the trainee to repeat all of the practice exercises and problems as many times as he wishes — in order to thoroughly master the material before taking the final exam.

So, forget comparing pre- and post-test scores. They matter little. It’s finding a long-term retention measurement that will prove the worthiness of your training.