ROI and the Levels of Evaluation

: Posted By Super User; 14 April 2014; Return on Investment

The complete story of a training program success or failure

Patricia and Jack Phillips, from ROI Institute, included return on investment (ROI) as the fifth level of the traditional Kirkpatrick evaluation levels. According to this new ROI methodology for training and education practitioners, the new evaluation framework includes five levels of data:

Level 1 Reaction, Satisfaction, and Planned Action—Data representing participants' reactions to the program and their planned actions is collected and analyzed. Reactions may include participants' views of the course content, facilitation, and learning environment. This category of data also includes data often used to predict application of acquired knowledge and skills, including relevance, importance, amount of new information, and participants' willingness to recommend the program to others.

Level 2 Learning—Data representing the extent to which participants acquired new knowledge and skills is collected and analyzed. This category of data also includes the level of confidence participants have in their ability to apply what they have learned.
Level 3 Application and Implementation—Data is collected and analyzed to determine the extent to which participants effectively apply their newly acquired knowledge and skills. This category also includes data that describes the barriers that prevent application and any supporting elements (enablers) in the knowledge transfer process.
Level 4 Business Impact—Data is collected and analyzed to determine the extent to which participants' applications of acquired knowledge and skills positively influenced key measures that were intended to improve as a result of the program. When reporting data at Level 4, a step to isolate the program's effects on these measures from other influences is always taken.
Level 5 Return on Investment—Impact measures are converted to monetary values and compared with the fully loaded program costs. You can have improvement in productivity, for example, but you must determine the monetary value of that improvement and what that improvement cost you in order to calculate ROI. If the monetary value of productivity's improvement exceeds the cost, your calculation results in a positive ROI.

Each level of evaluation answers basic questions regarding the program success.

Level of Evaluation	Key Questions
Level 1: Reaction, Satisfaction, and Planned Action	Was the program relevant to participants’ jobs and mission? Was the program important to participants’ jobs and mission success? Did the program provide new information? Do participants intend to use what they learned? Would participants recommend the program to others? Is there room for improvement with facilitation, materials, and the learning environment?
Level 2: Learning	Did participants acquire the knowledge and skills presented in the program? Do participants know how to apply what they learned? Are participants confident to apply what they learned?
Level 3: Application and Implementation	How effective are participants at applying what they learned? How frequently are participants applying what they learned? If participants are applying what they learned, what is supporting them? If participants are not applying what they learned, why not?
Level 4: Business Impact	So what if the application is successful? To what extent did application of learning improve the measures the program was intended to improve? How did the program affect output, quality, cost, time, customer satisfaction, employee satisfaction, and other measures? How do you know it was the program that improved these measures?
Level 5: ROI	Do the monetary benefits of the improvement in business impact measures outweigh the cost of the program?

The reason for referring to evaluation data as levels is that it facilitates managing and reporting the data. More important, however, these five levels present data in a way that makes it easy for the audience to understand the results reported. Each level of evaluation provides important, stand-alone data. Reported together, the five-level ROI framework represents data that tells the complete story of program success or failure.

Reference: Return of Investment Basics, Patricia Pulliam Phillips, Jack J. Phillips, ASTD Press, 2005 ROI Institute

Rating:

( 0 Rating )

INSOURCES BLOG