Evaluation is a systematic process of inquiry that provides information about something; in this case, a program, like that of an apprenticeship, to determine its efficiency and effectiveness.
How vital is evaluation in program management? On a scale of one to ten, it is definitely a 10. Whether or not an organization chooses to hire an outside evaluator, conduct an internal evaluation, or even a combination of both, the importance factor remains. Did participants learn what they were supposed to? Do changes need to be made within the program itself? Other measurement factors may also include ROI (return on investment), employee retention and engagement, and, of course, organizational outcomes.
Program evaluation is a collaborative effort. An apprenticeship program comprises on-the-job training (OJT) provided by the employer via a mentor and related technical instruction (RTI) often provided by a trainer or an educational institution. When an employer evaluates an apprenticeship program, the evaluation should include the learning components or competencies – RTI – and vice versa. When an educational institution or trainer evaluates a learning program, the evaluation should incorporate the on-the-job training (OJT) activities. It is imperative that the OJT and the RTI complement each other.
By working together through collaboration and evaluation efforts, the employer and the RTI provider can identify gaps in knowledge and skills and ways to enhance learning and development, training, and apprenticeship programs to ensure they are more efficient and effective.
As Donald Kirkpatrick outlined in his 2006 evaluation framework, training programs should be evaluated for three key reasons:
Although there is no short supply of program evaluation modes or types, they generally fall into three common categories: formative, summative, and process evaluation.
Formative evaluations and assessments measure what program participants or apprentices are learning during the program; think of this as a quiz or exam in a traditional classroom in the middle of a semester or term. A tool trainers, educators, and mentors can use to monitor progress – an assessment FOR learning that occurs DURING learning.
Summative evaluations and assessments measure what program participants or apprentices have learned after the program is complete; think of this as a post-test as opposed to a pre-test. A tool used to evaluate whether learning has occurred – an assessment OF learning that occurs AFTER learning.
On the other hand, process evaluations measure the program activities and how programs are implemented. A process evaluation answers questions like, are program activities being completed or going according to plan; is the program reaching the intended audience; are all learning materials (RTI) and activities (OJT) relevant or appropriate, etc.
Various frameworks exist for assessing program effectiveness, with the Kirkpatrick Model and the Rainbow Framework being two prominent approaches. The Kirkpatrick Model structures program evaluation across four sequential levels, each building upon the previous one. Level 1 measures participant reaction to the program, Level 2 assesses learning outcomes, Level 3 examines behavioral changes, and Level 4 evaluates overall results. Because each level influences the next, all four must be addressed systematically without skipping any stage.
Level 1
The first level measures the program participants' reactions to the training program. It is necessary to obtain feedback from the learners or apprentices in an apprenticeship program. Who better to acquire program information from than the individuals who participate in the learning aspect of the program itself. Do the OJT and the RTI complement each other? Does the sequence of learning make sense? Are reactions optimistic or negative? If they could do it all over again, would they? Would they recommend the program to their friends? Are the content, competencies, and environment conducive to learning? Ultimately, are participants' reactions positive?
Level 2
Level two of the Kirkpatrick Model – learning – involves “the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program” (p. 22) . What is the goal of an apprenticeship program? Simply speaking, to improve knowledge through RTI and to increase skills through OJT. Program goals and objectives must be clear and intentional for the learning evaluation to be valuable.
Level 3
In Kirkpatrick’s framework, level three focuses on measuring changes in the behaviors of program participants. In other words, has a change occurred?
Level 4
Finally, level four focuses on the final program results. What are the objectives of the learning and development, training, or apprenticeship program, and did the program outcomes meet these objectives? The Kirkpatrick Model is a best-in-class framework for program evaluation.
Of course, other evaluation frameworks are as beneficial as the Kirkpatrick Model. BetterEvaluation, part of the Global Evaluation Initiative, encompasses a global community that works to improve evaluation theory and practice. BetterEvaluation also provides an extensive knowledge platform where you can find more information on evaluation approaches and methods, as well as evaluation tools and resources, including The Step-by-Step Guide to Evaluation, and my favorite, the Rainbow Framework.
Regardless of the type of evaluation or framework you use, the important thing is that program evaluation actually occurs and that some form of each of these factors takes place:
ROI cannot be measured without conducting a proper program evaluation, analyzing business impact, and calculating ROI:
ROI = (Net Benefits of Program / Total Program Costs) x 100
Calculating the return on investment for training or apprenticeship programs is rarely straightforward. While comparing program costs to monetary benefits provides one measure of value, ROI cannot always be quantified financially. As evaluation expert Rosalie Torres notes in Evaluation Strategies for Communicating and Reporting: Enhancing Learning in Organizations, "The objective of all evaluative work is to promote insight." This perspective reminds us that the true value of program evaluation often lies in the understanding it generates rather than the dollar figures it produces.