By Jack Phillips, PhD, and Patti Phillips, PhD
Before training evaluation begins, program objectives must be developed. Program objectives are linked to the needs assessment. Let's look at how objectives can be developed within the context of the five-level ROI framework. Setting clear objectives at each level is important because success with training objectives helps answer basic questions regarding program effectiveness.
Level 1 Objectives (Reaction, Satisfaction, and Planned Action)
These describe issues that are important to the success of the program, including facilitation, relevance and importance of content, logistics, and intended use of knowledge and skills. The best Level 1 objectives should:
- Identify issues that are important and measurable
- Be attitude-based, clearly worded, and specific
- Specify how participants have changed their thinking or perceptions as a result of the program
For example, participants' perceptions of program relevance is a strong predictor of how well they will apply learned knowledge and skills. So, a good Level 1 objective is:
- At the end of the course, participants will perceive program content as relevant to their jobs.
Reaction data can be collected with feedback questions such as, "Was this program useful, necessary, motivational, and important to your success?" Measures of success with Level 1 Objectives include:
- 80 percent of participants view the knowledge and skills as relevant to their daily work activity as indicated by rating this measure 4.5 out of 5 on a Likert scale.
Level 2 Objectives (Learning)
Level 2 objectives communicate expected outcomes from instruction. The best learning objectives:
- Describe outcome-based behaviors that are observable and measurable
- Describe competent performance that should occur as the result of learning
- Spell out what the participant must be able to do as a result of learning
As with Level 1 objectives, Level 2 objectives should be clearly worded and specific. A typical learning objective may be:
- At the end of the program, participants will be able to implement Microsoft Word.
Sounds reasonable. But, how will you know you have achieved success? You need a measure, such as:
- Within a 10-minute time period, participants will be able to demonstrate to the facilitator the following applications of Microsoft Word with zero errors:
- File, Save as, Save as Web Page.
- Format, including font, paragraph, background, and themes.
Now, you can evaluate the success of learning.
Level 3 Objectives (Application and Implementation)
Where learning objectives and their success measures tell you what participants can do, Level 3 objectives tell you what participants are expected to do when they leave the learning environment. The best Level 3 objectives are observable and measurable, outcome-based, clearly worded, specific, and:
- Emphasize applying what was learned
- Describe the expected outputs of the training program
- Provide the basis for evaluating on-the-job performance changes
A typical application objective might be:
- Participants will use effective meeting behaviors
Again, you need specifics in order to evaluate success. What are effective meeting behaviors and to what degree should participants use those skills? With Level 3 evaluation, you can also follow-up to assess success with learning transfer. Here you look for barriers to application as well as enablers that support learning. Gathering data around these issues allows you to take corrective action when evidence of a problem exists.
Level 4 Objectives (Business Impact)
Level 4 objectives provide the basis for the questions that you ask during the overall evaluation process. They measure the consequences of participant¡¦s applied skills and knowledge and place emphasis on achieving bottom-line results. The best Level 4 objectives:
- Are results-based, clearly worded, and specific
- Spell out what the participant has accomplished in the business unit as a result of the program.
A sample Level 4 Objective might be:
- Increase market share of young professionals by 10% within nine months of new ad launch.
Success with Level 4 objectives is critical when you want to achieve a positive ROI.
Level 5 Objectives (ROI)
Level 5 objectives target the specific economic return anticipated when an investment is made in a program. Some organizations are satisfied with a 0% ROI break-even. This says that the organization got the investment back. Remember, not all programs are suitable for ROI -- typically only 5-10% of all learning programs are evaluated at this level.
Summary
The value of training occurs when participants react positively to an event; acquire new knowledge, information or skills; apply those on the job after a program; and, as a result of these applied actions, positively influence targeted business measures. Defining specific success measures across multiple levels of evaluation will help you move beyond simple tracking of reaction data and make it easy for your audience to understand the results reported. Each level of evaluation provides important, standalone data. Reported together, the five-level ROI framework gives data that tells the complete story of program success or failure.
Measuring ROI in Training and Development - Workshop
Connect training objectives to industry results. Use evaluation data for continuous improvement.
Measuring the effectiveness of training to ensure it meets workforce performance needs and linking training program objectives to business results are essential parts of the training cycle. They are also critical compliance requirements linked to continuous improvement and industry engagement.
This two-day workshop emphasises the Phillips ROI Methodology and participants will develop ways to apply the return-on-investment techniques to learning and performance solutions.
At the end of the program, participants will be able to:
- Link training program learning objectives to industry needs
- Apply ROI techniques to training evaluation
- Develop an evaluation plan that meets organisational and compliance requirements
- Use evaluation data for continuous improvement purposes
