OPM Workshop Evaluating Leadership Development Programs: Easing into Levels 3 & 4 Presenters: Cheryl Ndunguru & Yadira Guerrero Senior Executive Resources and Performance Management, Work-Life and Leadership and Executive Development UNITED STATES OFFICE OF PERSONNEL MANAGEMENT Workshop Purpose and Objectives
PurposeTo empower participants to competently execute results-focused evaluations for their agency leadership development program. ObjectivesParticipants will: Articulate the importance of training evaluation Effectively address barriers to conducting level 3 & 4 Create a logic model that focuses on training effectiveness
Introduction to Evaluation Definitions EvaluationThe making of a judgment about the value of something Subjective Data Beliefs Attitudes Perceptions
Objective Data Observation Measurement Definitions cont. InputsResources ActivityWhat you do/Target audience OutputWhat you produce (immediate result of the activity)
# of participants who completed the course # courses offered # of training hours % participant satisfaction with the training 5 Definitions Cont.
OutcomeThe difference/impact made by what you produced (result of the output) Everyone loved the training, so what? The class was full, so what? The instructor was outstanding, so what? Everyone learned something, so what?
MeasurableSpecific, observable, and quantifiable characteristics Timeliness Quality Quantity
Cost-effectiveness Evaluation in Government Executive Branch Office of Management and Budget Operating Agencies OPM MSPB Legislative Branch Congress
GAO CBO Civil Society Advocacy and Funding for Government Oversight Private Sector Researchers and Evaluators Evaluation Guidance and Tools Government Performance Results Act (GPRA) Government Performance Results Modernization Act (GPRA-MA)
Program Assessment Rating Tool (PART) Performance Improvement Council www.Performance.gov 5 CFR 410.201(d)(4) Heads of agencies are required to assess periodically, but not less often than annually, the overall agency talent management program to identify training needs within the agency
Human Capital Framework (HCF) Leadership Development Programs Training Coaching Mentoring Rotations Workforce Planning Recruitment &
Outreach Employee Development Leadership Development Retention Knowledge Management Strategies/ Programs
Goals Human Capital Framework Talent Management Performance Management
GAO: Federal Training Investments (2012) Evaluate the benefits achieved through training and development programs, including improvements in individual and agency performance: Has a formal process for evaluating employee satisfaction with training. (Levels 1 &2) Has a formal process for evaluating improvement in employee performance after training. (Level 3) Has a formal process for evaluating the impact of training on the agencys performance goals and mission. (Level 4)
Reactive vs. Strategic: Where are You? ATD Best Awards Video Program Evaluation vs. Training Evaluation 410.202 Responsibilities for Evaluating Training Agencies must evaluate their training programs annually to determine how well such plans and programs contribute to mission accomplishment
and meet organizational performance goals. Program Evaluation Program evaluations are individual systematic studies conducted periodicallyto assess how well a program is working. They are often conducted by experts external to the program, , as well as by program managers. A program evaluation typically examines achievement of program objectives in the context of other aspects of program performance to learn the
benefits of a program or how to improve it. (GAO) SES Candidate Development Program Program Goal: Create a pool of effective and diverse leaders for sustained organizational success Recruitment Process Selection
Process Training &Developmen t Process (5 CFR 412) Certification Process
Program Outcomes: 1)QRB Certified candidates 2) Increased leadership diversity Program Evaluation Questions A Program evaluation would assess (thru questions, interviews, etc.) the effectiveness of each process in the program in helping to accomplish the long term goal. Was a need for the program identified? Was program funding adequate? Did recruitment efforts attract a diverse pool of
applicants? Did senior leaders fulfill their roles in the selection process? Was the training evaluated? To what extent did external factors impact the program? Were the program goals met? Training Evaluation Training evaluation is an objective summary of quantitative and qualitative data gathered about the effectiveness of training. The
primary purpose of evaluation is to make good decisions about use of organizational resources. Training evaluation data helps the organization to determine whether training and subsequent reinforcement is accomplishing its goals and contributing to the agency mission. (Training Evaluation Field Guide, 2011) SES Candidate Development Program Create effective and diverse leaders for sustained organizational
success Recruitment Process Selection Process Training & Developme nt
Process Certification Process Outcomes: 1)QRB Certified candidates 2) Increased leadership diversity What is a Logic Model? What is a Logic Model?
21 A picture of your program. Graphic and text that illustrates the causal relationship between your programs activities and its intended results. To To produce produce these
these outputs outputs Weuse use We these these resources resources
Forthese these For activities activities Sothat that So participants participants
changetheir their change behaviorsinin behaviors thefollowing following the ways ways
Leadingtoto Leading this this program program result! result! 5 USC 4101Definition of Training training means the process of providing for and
making available to an employee, and placing or enrolling the employee in, a planned, prepared, and coordinated program, course, curriculum, subject, system, or routine of instruction or education, in scientific, professional, technical, mechanical, trade, clerical, fiscal, administrative, or other fields which will improve individual and organizational performance and assist in achieving the agencys mission and performance goals;
Levels of Evaluation Did it matt er? Did they use it? Did they learn it?
Did they like it? Level 1 Did they like it? Training Reactions Learning Behavior Results Know how the trainees felt about the training event. Point out content areas that trainees felt were missing from the training event. Tell how engaged the trainees felt by the training event.
Formative evaluation Importance of Level 1 Positive attitudes toward the training can be quite beneficial to ensuring positive level 2 and level 3 outcomes Evaluation of specific aspects of the training provides import information about what can be improved (instructor, topics, presentation style, schedule, audio visuals, etc.)
Level 2 Did they learn it? Training Reactions Learning Behavior Results Demonstrates participant learning (Pre and Post test) Formative evaluation Importance of Level 2 Helps promote the development program. Positive level two evaluation can help in interpreting the results of level three evaluation
(e.g., if level three results do not occur, it may due to work place factors and not because of any flaw in the training). Can provide formative evaluation information that can be used to improve the training (e.g., you may find certain learning objectives that are not being met). True or False? If participants are happy or satisfied at the end of
a training course, it usually means they will use the skills that theyve learned. False Research indicates there is no significant relationship between: - perceptions of enjoyment of a training and performance - perceptions of the instructors effectiveness and performance - perceptions of the amount learned and
performance Levels 3 and 4 Level 3Did they use it Level 4Did it matter To Be Continued Training Effectiveness Level 4 Overview: Kirkpatrick Business
Partnership Model National Museums Agency (NMA) Leadership Development Program NMA Strategic Goals Build and maintain a strong agency leadership pipeline and talent pool for leadership continuity and viability Develop future leaders who are ready to step into higher positions
Enhance and grow and strong pan-institutional leadership team Situation a front-page expose of funds misuse by one museum director, reduced donations and lack of a consistent succession plan across the organization. Finally, there was an apparent lack of pan-institutional cooperation among the museums. Competition between museums had reached a level that surpassed friendly
competition. Does anyone want to share the situation that was/is the catalyst for your LDP? Level 4 Did it matter? Training Reactions Learning Behavior Results Level four outcomes tend to fall far down outcome lines, which means that many intervening factors must take place in order for the level four outcomes to take place.
Connect the training program to a larger organizational strategic program that is designed to produce level four changes. NMA: Level 4 Business Need Business Need Maximize and demonstrate impact from donations. Create leadership pipeline for sustained institutional success. Build a pan-institutional culture where
decisions are made with the betterment of the entire NMA in mind. Level 4: Pitfalls to Avoid Pitfalls to Avoid Creating a training program without first identifying stakeholders that will judge its success Trying to please everyone instead of identifying the few, most critical group of stakeholders that need to be satisfied
Assuming that business/organizational leaders have expectations and targeted results in mind when they make a training request Level 4: How to Avoid the Pitfalls Get Involved Obtain leadership support Get Involved: ADDIE Model
What is ADDIE? A systematic approach (model) for developing effective instruction. One of the most popular models in instructional design. Outcome of each step feeds into the subsequent step. Evaluation is ongoing throughout each layer of design. Get Involved: Effective Learning
Interventions for Developing ECQs OPM Leadership Development Matrix Adapted from the Draft OPM document Effective Learning Interventions for Developing ECQs A Quick reference guide that highlights the most effective and targeted approach for developing each competency within the ECQs Example: Leading Change
Obtain Leadership Support: Get Stakeholders Involved in the Training Stakeholder Engagement Benefits for the Training Department Benefits for the Stakeholder Streamlined policy and program
development processes Greater opportunities to contribute directly to development of training Increased efficiency in and effectiveness of training delivery More open and transparent lines of communication increasing the accountability of Government and
driving innovation Improved risk management practices allowing risks to be identified and considered earlier, thereby reducing future costs Improved access to decision-making processes, resulting in the delivery of more efficient and responsive training
Enhanced organizational confidence in the training department More effective training department Activity: Stakeholder Map Draw and label the Stakeholders who are invested in the accomplishment of your Level 4 results. Draw lines with arrows connecting stakeholders.
Write a label on the line to describe relationships. How can you inform these stakeholders of the strategic role your leadership development program plays in accomplishment of that goal? Activities: Training Effectiveness Level 4 Planning: Identify the program results and measures Level 3 Planning: Identify critical behaviors and leading indicators
NMA Results and Measures Level 4 Result To sustain the ability of the NMA to share knowledge with the world. Level 4 measurement (observable, measurable) The sustainment of the NMA would be measured in two ways: 1. Donation levels 2. Cross-organizational agreement on funding usage
Level 4 Activity: Identify the Program Outcomes and Measures Input (Resources) Activity (What you do) Output (Level 1 & 2)
Behaviors (Level 3) Outcomes (Level 4) Sample Succession Planning Results To increase the organizations ability to fill key jobs with internal candidates To sustain diversity in promotions
To increase positive performance evaluations To maintain leadership effectiveness To increase high potential retention & attrition How will you collect data to verify that youve accomplished these results? Data Collection Methods (pg. 31 & 32) NMA Data Collection Methods Sample Level 4 Method
Level 3 Overview Did they use it? Program Reactions Learning Behavior Results Measures actual behavior on the job, rather than only measuring or demonstrating positive reaction, learning or intent to apply the learning. Level three outcomes are required for level four outcomes Sometimes, evidence of level 1 outcomes, level 2 outcomes, and level 3 outcomes will be sufficient
evidence of the merit and usefulness of a training program. The Fun Theory https://www.youtube.com/watch?v=2lXh2n0aPyw Training Transfer After During Before
GROUP 2 12 supervisors report a 15% decrease in the amount of time they spend making unnecessary edits to reports written by those who attended the course;. supervisors attribute half of this improvement to training. GROUP 1 Saved 22 man-hours (valued at $10,437). participants report a 40% decrease in the number of final drafts
returned to them, by supervisors for additional edits. 30 employees completed the course 70% of supervisors report more positive feedback from 4.5 of 5.0 satisfaction end users. 95% said they will use what they learned back on the job Required Drivers
Level 3: Determine Critical Behaviors The degree with which critical behaviors are performed on the job determines the degree to which desired results are obtained. Purpose Define clearly exactly what needs to be done in measurable, observable, quantifiable terms Identify the few, critical behaviors that will have the greatest impact on the desired goal and agency mission
Level 3: Identify Leading Indicators Purpose Provide early validation that the correct critical behaviors were selected Inform and reassure stakeholders, training professionals and initiative participants that long term targeted results are on track for success Level 3 Activity Identifying Critical Behaviors & Leading
(Level 3) Outcomes (Level 4) Activity: Identifying Critical Behaviors & Leading Indicators NMA: Critical Behaviors and Leading Indicators
Data Collection Methods (pg. 31 & 32) Quick Tip: Writing Good Evaluation Questions Belief Behavior Evaluation Action Planning Activity Now that youve created measurable Level 3 and Level 4 outcomes and measurements how
will you proceed to effectively evaluate your program at these levels? Stakeholder support Get involved in the process Create relevant questions Ensure drivers are in place
Cette différence de réaction sur le sable ne peut donc être expliquée que par la variation d'une grandeur liant le poids (la force) à la surface de contact, On peut en déduire que La pression est l'application d'une force sur...
Includes Language Acquisition each year. Units revolve around Statements of Inquiry. Enriches lessons by investigating global contexts. Requires vertical and horizontal collaboration. Will finish with the moderated Personal Project. Academy of Richmond County is a candidate school* for the Middle...
Overview. This paper offers an instructional class exercise of the capital structure decision-making process including a hands-on application of how to use four gain to leverage (G. L) equations including a recent equation that is used when managing growth firms.....
How Does a Solution Form Energy Changes in Solution Energy Changes in Solution Why Do Endothermic Processes Occur? Why Do Endothermic Processes Occur? Enthalpy Is Only Part of the Picture Enthalpy Is Only Part of the Picture Student, Beware! Student,...
Food shopping tips. Responsible weight-loss planning. Tips for long-term weight management. Web Tools for Tracking Your Health Goals. Learn about free online tools that can help you keep track of your goals. Stress Management Tips. Relaxation techniques.