How much to evaluate?

/

#6 in our training evaluation blog post series:

I read an article the other day called Measuring the Effectiveness of Training, by Teresa Hiatt, Director of Sales Education, Ricoh Americas Corporation. She stated some interesting statistics in her article that I’ve also run across in other research articles and best industry reports:

  • Research from the American Society for Training and Development (ASTD) states that top companies will spend more than $1,500 per employee per year for education opportunities.       
  • Fortune Magazine's 100 Best Companies to Work For lists development opportunities as an important indicator to attract and retain top employees.
  • The latest estimate of the size of the Training Industry is about $135 billion.

She concludes that all this research confirms that leading organizations and key management think that training employees is an important part of business today. Yet despite the time, effort and money companies spend for training and development, training is rarely held accountable for results.

Many organizations (and even T&D professionals) are under a misconception that measuring levels 3 and 4 (and even 5) are complicated, require too many resources and cost too much money so they just don't do it. They also believe that the levels of evaluation should be applied equally to all courses they offer. But not every training program requires evaluation at all levels. Business needs, strategic positioning, performance discrepancies, stakeholder support and the ability/willingness of the organization to observe, document, evaluate and support evaluation within an organization are all important factors when determining what training programs are chosen for evaluation. 

Best practice organizations use the following recommended evaluation targets when conducting evaluation of training programs:

Level 1 = Conduct evaluation for 100% of training programs

Level 2 = Conduct evaluation for 60% of training programs

Level 3 = Conduct evaluation for 30% of training programs

Level 4 = Conduct evaluation for 10% of training programs

Level 5 = Conduct evaluation for 5% of training programs

What percent of your training programs are you evaluating and at what level(s)? I’d be interested to hear.

Be sure to check out our other evaluation blog posts in this series:

Selling the importance of evaluation

/

#5 in our training evaluation blog post series:

In the not so distant past, evaluation of learning was an isolated activity relegated to the training team who’s responsibility didn’t extend much past gathering level 1 and level 2 evaluation. Today the emphasis is on the bottom line and how organizations can get the best value for their money and efforts. New evaluation tools, processes and strategies are available to help companies become more strategic; the evaluation of learning has become less of an isolated activity and more of a culture/philosophy. Learning teams are now becoming drivers of change, helping to support evaluation efforts within their organizations. But what if your stakeholders and senior management don’t see or understand the importance of evaluation?

Implementing levels 3, 4 and even 5 can be a challenging and daunting task even if the training team is fully involved and committed, because commitment and participation is also required from employees, managers, supervisors, business partners, stakeholders and senior management. An organization’s executives need to be on board as top-down messaging is critical to success – you’ll be swimming upstream trying to get managers to participate in evaluation if they don’t feel that their own bosses are behind it.

Many senior leaders already acknowledge that employee education is a critical success factor for future growth and prosperity. Use this as your ‘hook’ to sell them on the importance and value of a solid system of evaluation. Here are some tips to help you along the way:

  • Show how evaluation contributes to success: Be able to show a direct correlation between the organization’s strategic needs and goals, business unit operational needs, individual development needs, the training that is designed to address these and the evaluation techniques that will be used to quantify improvements.
  • Share a roadmap to implementation: Create an evaluation strategy that will systematically guide the organization from the present situation to the desired amount of evaluation. Be prepared to provide costs in terms of time and manpower.
  • Give confidence with examples: Gather relevant case studies of best practice organizations who have implemented evaluation within their organization with positive results. Use this information to support your position. There are a number of websites with best practice research. Check out our favourites in a previous blog post.
  • Start small to prove your case: Run a pilot and communicate/share results. Work with a key stakeholder/business partner to address a business need through training. Use this training as the “test case” for your evaluation plan. Apply each level of evaluation gathering testimonials and data, and tracking trends along the way. Share results and testimonials. Use stories and case studies based on the training results to capture attention and highlight the positives. Prove that measuring the value of learning can have a positive effect on your organization.
  • Communicate, communicate, communicate! Communication at all levels of the organization is critical to success. People need to understand the What, Where, Why, When and How of your evaluation strategy and what their role and commitment will be. Be patient. This will take time, but your efforts will be worthwhile!

Be sure to check out our other evaluation blog post in this series:

 

 

8 tips for creating a successful evaluation strategy

/

#4 in our training evaluation blog post series:

In this tight economy, organizations want to know that they’re getting their money’s worth wherever it’s spent – and training is no exception. But when it comes to training, many organizations are unable to clearly identify what they are getting in return for their dollars. Research shows Levels 4 (Results) and 5 (ROI) are the two training evaluation levels least integrated into organizations, and yet they can provide the greatest value, as they measure the impact that learning has on the business and determine the return on investment.

So how do you get your company headed in the right direction and where do you start the evaluation process? A good first step is to create an overall evaluation strategy for training. Use it as a roadmap to help you stay focused on the big picture while implementing the details. When creating a strategy for your organization, consider the following tips:

  1. Flip the Kirkpatrick evaluation model upside down and start with Level 4 evaluation (Results). Identify key corporate goals and strategies and determine how training aligns with them. Identify stakeholder expectations. What are they looking to achieve as a result of training? This information will drive your evaluation strategy.
  2. Get buy-in from the top and commitment from appropriate stakeholders.
    • What strategies can you put into place to "sell" senior leaders and stakeholders on the benefits that evaluation will have to the organization?
    • What senior leaders and stakeholders could be strong champions providing support and commitment?
  3. Determine who owns the evaluation process.
  4. Identify what training course(s)/program(s) will be evaluated and at what levels. For each course or program, answer the question, "Is there a strong business need for this training?"
  5. Identify how each level of evaluation will be measured. What tools will be used to gather the information needed for your evaluation strategy?
  6. Determine what resources will be required for each level of evaluation. Levels 1 and 2 can be directly controlled and managed by a training team while participation, time and resources will be required from employees, managers, senior leaders, stakeholders and business partners for levels 3, 4 and 5. Are they willing and able to provide it?
  7. Identify challenges and risks for each evaluation level.
  8. Talk to other departments to determine what information is currently being tracked in your organization that can be used for levels 4 and 5 evaluation.  Don't reinvent the wheel. Use what's available then figure out if you need to fill gaps with further information.

Evaluation doesn’t need to be complicated. There are a number of fairly simple ways to implement evaluation in your organization that can provide a lot of value. Start by creating your evaluation strategy then stay tuned here for future tips and ideas that will help you implement evaluation successfully in your organization.

Be sure to check out our other evaluation blog post in this series:

 

Refreshing the Kirkpatrick Four Levels evaluation model - Kirkpatrick Then and Now

/

#3 in our training evaluation blog post series:

As I’ve written in past evaluation blog posts, more and more organizations are now focusing on how training impacts business results, wanting to know if they are getting the most for their return on training investment. If you follow evaluation blogs and read articles and research on evaluation, you’ll notice that Training and Development professionals are now looking to improve their learning process by “beginning at the end”. They are first defining business outcomes and determining the desired Level 4 results then working down the model rather than starting with traditional level 1 evaluation and working upwards.

Don Kirkpatrick’s concept of the 4 levels of evaluation - Reaction, Learning, Behavior and Results - was first introduced in the 1950’s during and after his Ph.D. dissertation.  But it wasn’t until after numerous articles and speaking engagements at national conferences over the years that his landmark book, Evaluating Training Programs: The Four Levels was published in 1994. It quickly became a cornerstone for T&D professionals providing a logical structure and process for measuring learning. One interesting note I learned:  Don never called his evaluation concept the “Four Levels”.  Someone else did and it caught on!

I recently picked up the Kirkpatrick Then and Now book by Jim Kirkpatrick and Wendy Kayser Kirkpatrick (2009). Taking the original 4 levels concepts developed by Don, Jim (Don’s son) and Wendy (Don’s daughter-in-law) have created an updated and fresh version of Don’s original model.  The book is an Interesting read, covering the “Then” – a description of Don’s 4 levels as well as his first-hand account of how he developed his model, testimonials from his colleagues and a photo gallery. The “Now” part of the book defines the new five Kirkpatrick Foundation Principles and provides practical advice on how to implement the updated Kirkpatrick model successfully starting from the top and working on down.

So grab a copy of the book, have a read and enjoy learning how to turn the Kirkpatrick model on its head!

Be sure to check out our other evaluation blog posts in this series:

Measuring the value of learning

/

#1 in our training evaluation blog post series:

Good training evaluation techniques identify and measure what learning has occurred during and after learning, whether job performance improvements have been realized, and most importantly, what the cost-benefits are to the organization. When it comes to training evaluation techniques, Kirkpatrick's four level evaluation model (including level 1-reaction, level 2-learning, level -behaviour and level 4-results) is the most widely used and respected model for evaluation and measurement of business value. Level 5 evaluation, Return on Investment (ROI), is an additional step to training evaluation that has been developed by respected author and founder of the ROI Institute, Dr. Jack Phillips.

I found an interesting statistic in a 2009 ASTD study on The Value of Evaluation: Making Training Evaluations more Effective. 92% of respondents said they measure at least Level 1 of the Kirkpatrick/Phillips module of learning evaluation, but the use of the model drops off dramatically with each subsequent level. The study suggests that organizations may not fully grasp how evaluation should be used. I have to agree with this insight.  Many people I’ve talked to and worked with during my career don’t realize that good evaluation tools and techniques are available that can be used to make their organization stronger. There are ways to implement evaluation that can have a positive impact without being overly complicated or time-consuming. During our recent Kirkpatrick certification, I came across many T&D and HR practitioners who were looking for tools that could successfully work in their organization and processes to make evaluation “easier” to implement.

We, at Limestone, are passionate about the value of training evaluation and would like to share some of our thoughts, strategies, and ideas with you through a series of “evaluation” blog posts.  So stay tuned.