Research shows that executives want to see what the behavior change influences in the organization as a result of leadership development programs.

Few would question the importance of leadership and leadership development. Great leaders are needed now more than ever, and for the most part, they must be developed internally. Executives stress this topic in many surveys and reports. The Conference Board routinely reports that leadership and leadership development are often included among the top five issues facing executives. It is one of the most important talent development initiatives that we can implement.

Most professionals involved in implementing leadership development programs understand that there are five levels of outcomes possible for any program. These are (1) reaction to the program, (2) learning the content, (3) applying the content, (4) the impact on key measures, and (5) the financial ROI (is it worth it?). However, not everyone is measuring at the impact and ROI levels.

A recent article in MIT Sloan Management Review highlighted the problem. In the article “Leadership Development Is Failing Us. Here’s How to Fix It,” the authors argue that leadership development providers do not deliver impact and offer a prescription for the consumers, the L&D and HR managers who purchase the program.¹ In reality, there are four stakeholder groups involved in the situation. They are the leadership development providers, the consumers (L&D/HR) who purchased the program, the participants, and the executives who fund it. They are all important stakeholders, but perhaps the most important opportunity is to ensure that the funders are pleased with the results.

Previous research shows that executives want to see impact, not just behavior change. They want to see what the behavior change influences in the organization. Even some would like to see the financial ROI. Classic research has shown that impact and ROI are the number one and two measures for this kind of program, with 96% and 74%, respectively, indicating their desired level of evaluation.² If someone is not convinced of that, we suggest they ask the executives, who fund the programs, if they would like to see the program’s impact or the financial ROI. We predict that the answer to both questions will be yes.

Additionally, in our evaluations, participants are usually excited when they can see the impact of their new or improved behavior. They would like to see results at the impact level. When the leadership program is connected to impact measures in the beginning, they have more ownership because now they are solving a problem or addressing an opportunity under their control—not just learning some new behavior.

These days, more consumers are asking for impact and ROI, primarily driven by the influence of the funders, the executives. The consumers of leadership development, HR and L&D teams, are now requiring more measurement at the impact and ROI levels. This is often driven by the executives. We see this from two perspectives. When we are asked to conduct ROI studies in an organization in the L&D function, it often involves evaluating a leadership development program. Those requests come from the top executives. When the practitioners attend our ROI Certification sessions, they are asked to select a program that needs to be evaluated all the way to ROI. One of the criteria is that executives may be expecting an evaluation at this level. In the last ten years, the number one topic for ROI Certification projects has been leadership development. In our work, we are encouraging our practitioners to be proactive and not wait for a request from their executives.

In many situations, the leadership development providers are reluctant to evaluate at these levels. The concern is that the program may not deliver the value. The reality is that if the program doesn’t deliver a positive impact, it is probably not the content that is the issue. More than likely, it was not implemented or supported very well in the organization. As one vice president of a large technology company told us, “Any model will work. It’s just a matter of how much we are willing to support it to deliver the desired results.” An ROI study helps the provider stand out from the rest of the field.

The challenge to leadership development providers is to measure at the impact and ROI levels now. Being proactive places you in a much better position. If you wait until the request, you and your client will have a short time frame to deliver, and you’re defending a challenge. You want to be on the offense, not defense. ROI is on someone else’s agenda. You want to keep ROI on your agenda so you can drive this with your clients.

The good news is that many organizations do this systematically now. Our recent book, Proving the Value of Leadership Development: Case Studies from Top Leadership Development Programs, presents 12 case studies of how organizations have addressed this issue. This is our fourth book on the value of leadership development. Let us know if you are interested in learning more about these other books.

We would invite all of those who offer leadership development programs to consider pushing the evaluation to the impact level, and maybe ROI.

References:

1. Leroy, Hannes, Moran Anisman-Razin, and Jim Detert. “Leadership Development Is Failing Us. Here’s How to Fix It,” MIT Sloan Management Review, Winter 2024.

2. Phillips, Jack J., and Patti P. Phillips. Measuring for Success: What CEOs Really Want from Their Learning Investments. Alexandria VA: ASTD Press 2010. Pg. 261

Written By: 

Patti P. Phillips, Ph.D., and Jack J. Phillips, Ph.D.

This article was originally published on May 9, 2024 at trainingmag.com.

Need for the standard

Practitioners continue to ask what should be measured, how it should be measured, and what should be done with the metrics once they are calculated. For some metrics, we don’t even agree on what to call them or how to define and calculate them. This is not surprising since there has not been a comprehensive framework, common language or agreed-upon definitions for measurement and reporting.

We have the Katzell/Kirkpatrick/Phillips five levels for program evaluation, but this accounts for only five to 10 of the 100-plus metrics we have in learning. How about the rest, including metrics for use at the department level (for example, percentage of on-time completions and reach) and for organized informal learning (for example, performance support), which is becoming increasingly popular?

Contrast the state of L&D measurement in particular and our profession in general to that of more mature professions such as accounting, chemistry, or biology. These mature professions have agreed-upon frameworks and a common language that facilitates understanding and communication among practitioners, as well as the development of standardized curricula at universities and conferences. For example, accounting has four types of measures (income, expense, assets, and liabilities) and three basic statements (income, balance sheet, and cash flow). Accounting terms are well-defined, and accountants agree on how to calculate and use their measures. A comprehensive framework and common language are the foundation for any profession.

Standard names and definitions are also important if a profession is to mature. Imagine if accountants did not agree on how to calculate the values of the measures in the financial statements. Imagine if scientists did not agree on the properties of the elements in the periodic table. It is hard to advance as a profession when there is no agreement on terminology.

Standard names and definitions are also required for accurate benchmarking. Without agreement on how to define and calculate metrics, organizations submit data for “their version” of a metric, which means that the resulting aggregated results are not as accurate as they could be if all reporting organizations calculated their data uniformly.

In short, there has been an unmet need in our profession for an agreed-upon framework and common language.

A deeper look at the standard

Developed by a global team of ISO industry experts from 2021 to 2023 (who I had the opportunity to lead), this 42-page standard provides a framework and guidance for measurement and reporting. The standard was created to provide clear, practical advice to practitioners on how to create a measurement and reporting strategy as well as how to define and calculate the metrics. It begins by describing the five broad types of data users and the four broad reasons to measure. Next, the standard shares three categories of metrics and four types of reports.

The standard recommends 19 metrics for smaller organizations and 52 metrics for larger organizations, including metrics for informal learning (knowledge sharing through communities of practice, performance support, and non-course-related content available through an organization’s learning portal). Recommendations are made for each type of user, which makes it easy for the practitioner to select the most appropriate metrics for each user. For example, eight L&D metrics are recommended for the CEO of a smaller organization, 13 for the head of learning, and seven for a program manager. The standard also includes recommended names and definitions for each metric and examples of their calculation and use in scorecards.

The framework, based in part on TDRP (Talent Development Reporting Principles, created between 2010 and 2012 by a group of industry thought leaders and practitioners), is designed to be easy to understand, easy to use, and easy to remember, much like the Katzell/Kirkpatrick/Phillips framework which employs just five levels. A framework with more than four or five categories may be more accurate and discerning but is likely to be harder to remember and use.

With usability in mind, the ISO standard includes just five broad categories of users: Senior organization leader (for example, the CEO), group leader (for example, head of a business unit), head of learning (the CLO or person in charge of learning), program manager (responsible for developing and managing the program) and individual learner. Guidance is to begin by knowing your users since different users will be interested in different metrics.

Likewise, while there are many reasons to measure, the standard collapses them into four broad categories, which also dictate the type of report to use when sharing the data. The most common reason to measure is to inform (answer questions and identify trends). The second reason to measure is to monitor (ensure the reported value of a metric is at least as good as it has been historically). The third reason to measure is to evaluate a program (determine its effectiveness and impact). Finally, the fourth reason to measure is to manage a program to deliver planned results (requires specific, measurable goals and special reports generated monthly to determine if additional actions are required for success).

Three categories of metrics are offered (efficiency, effectiveness, and outcome). Efficiency metrics comprise the bulk of L&D metrics and include activity, utilization, and cost metrics). Common examples include the number of participants, completion rates, and cost. Effectiveness metrics include participant reaction, goal owner satisfaction, learning, application, and ROI (return on investment). Following TDRP and a number of thought leaders, the standard separates outcome metrics Level 4) from the rest of the effectiveness metrics (Levels 1-3, 5) since the isolated impact of learning (Level 4) is the metric most CEOs want to see but are least likely to receive from their L&D department.

The ISO framework concludes with reports. Scorecards and dashboards are the most common. Scorecards typically are Excel spreadsheets with metrics as rows and time periods as columns. While scorecards are excellent for presenting detailed data, dashboards are designed to be more visually appealing with charts and graphs. Dashboards typically show less data, but it is more aggregated (for example, year-to-date). Program evaluation reports are designed to brief the results of a program upon completion. Management reports are the fourth type. These are specially designed reports to use monthly when managing a program to a successful conclusion.

With a common framework and language established, the standard recommends the following:

• Know your users and learn why they want the data. In other words, what is their reason to measure? Employ the four-reason framework as your mental model.
• Recommend appropriate metrics from the standard to meet their need. Once the type of user is known, simply refer to the table in the standard for the list of recommended metrics. There should always be several efficiency and several effectiveness metrics. Include an outcome metric if the program is designed to help achieve a specific, measurable organizational goal (for example, increase sales by 10 percent). Use the standard name of the metric.
• Calculate the value of the selected metrics using the definitions and formulas in the standard.
• Share the data with the user in the recommended report, which is determined by the reason to measure:
  
  • Inform: Scorecard or dashboard.
  • Monitor: Scorecard or dashboard with a threshold or legend based on an historical best.
  • Evaluate: Program evaluation report.
  • Manage: Management report.

This approach applies to creating measurement and reporting strategies for both individual programs and for the department as a whole.

A framework for the future

The new ISO standard is designed to address the need described above for a framework and common language as well as for standard names and definitions. It also provides detailed, practical guidance for selecting metrics for both smaller and larger organizations, including metrics for both individual programs and for the department as a whole as well as for informal and formal learning.
This standard should help our profession become better organized and more mature, which in turn will allow us to have greater impact and become better business partners, but only if it is widely adopted and used. So, please consider purchasing a copy and implementing in your L&D group. Even if you do not buy the standard, you can begin to use the framework for types of users, reasons to measure, types of metrics and types of reports.

This article was originally published on August 9, 2024 at ChiefLearningOfficer.com

About the Author

Dave Vance is the author of The Business of Learning: How to Manage Corporate Training to Improve Your Bottom Line, Second Edition and co-author of Measurement Demystified: Creating Your Measurement, Analytics, and Reporting Strategy and Measurement Demystified Field Guide.