Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance
Introduction to Evaluation
Authors and practitioners alike seem to agree that to evaluate is to place a value on or judge the worth of a person, place, thing, or event. In the context of PT, evaluation is a way to "compare results with intentions and delve into the usefulness of methods and resources so that we may move toward the required results." [1]
The International Society for Performance Improvement (ISPI) has long supported "the integral role of evaluation in PT and in the ongoing functioning of any organization." [2] According to ISPI's HPT Model shown below, evaluation is "one of the basic components of PT," [3] along with performance analysis, cause analysis, intervention selection and design, implementation, and change management.
The authors of Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance have enhanced the HPT Model by adding the three types of evaluation ” formative , summative , and confirmative ”and the meta-evaluation process to the evaluation component. Enhancing the evaluation component (see Table 7-1) broadens the scope of the component and increases the value of the entire HPT Model.
| |
| |
| |
Definition and Scope
There are three types of evaluation ”formative, summative, and confirmative ”plus a process called "meta evaluation," which literally evaluates the evaluation. Taken collectively, formative, summative, confirmative, and meta evaluation:
-
Cover the life cycle of the performance intervention from analysis through implementation and evaluation.
-
Guide the decision-making processes involved in why, what, when, and how to evaluate.
Table 7-1 illustrates the scope of evaluation within the HPT Model.
Formative Evaluation
Formative evaluation is diagnostic and is "used to shape or mold an ongoing process...to provide information for improvement...." [4] Formative evaluation is developmental and continuous; it begins during the analysis stage and continues through the selection and design of interventions and sometimes into early implementation.
Summative Evaluation
Summative evaluation focuses on the effectiveness of a performance intervention after it is implemented. Effectiveness is measured by immediate reaction and personal acquisition of knowledge and skills. Summative evaluation takes place during implementation and change management.
Confirmative Evaluation
Confirmative evaluation identifies and explains long- term or enduring effects. [5] It is "the process of collecting, examining and interpreting data and information in order to determine the continuing competence of learners or the continuing effectiveness of instructional materials." [6] Confirmative evaluation builds on and goes beyond formative and summative to place a value on knowledge or skills transfer to the job, organizational impact, and return on investment.
Meta Evaluation
Meta evaluation is a process for "assuring and checking the quality of evaluations." [7] Through meta evaluation the PT practitioner or evaluator validates the formative, summative, and confirmative evaluations and acquires valuable insight into evaluation processes and products. The best way to distinguish among the three types of evaluation and the meta evaluation process is to look at why and when each type of evaluation and meta evaluation should occur during the life cycle of the HPT Model. Table 7-2 provides an overview.
| Type/Process of Evaluation | Why? (Purpose) | When? (Timing) |
|---|---|---|
| Formative | Improve performance intervention package | During performance analysis, cause analysis, and selection or design of interventions |
| Summative | Determine immediate competence of user and effectiveness of package | During implementation and change management |
| Confirmative | Determine continuing competence of user and effectiveness of package | 6 “12 months after implementation |
| Meta | Evaluate formative, summative, and confirmative processes to provide insight to evaluator | After confirmative evaluation |
Purpose
In broad terms, "the purpose of evaluation is to illuminate and improve the organization," [8] More specifically , the purpose of evaluation is to generate information that will accomplish two outcomes :
-
Help the organization to value or judge the results of a performance
-
Trigger or support a decision regarding the performance, the performer, or, ultimately, the organization itself.
Traditionally, quantitative and qualitative evaluation is used to prove or disprove the value of a performance intervention. When the HPT Model was first introduced in 1992, "Evaluation of Results" was the last step in the model. Marc Rosenberg, one of the authors of the HPT Model, summarized the evaluation component of the Model as follows : "After applying or implementing the PT solution, it's important to monitor it to determine its effect on performance improvement and on the organization." [9]
On the other hand, "Most evaluators would agree that the purpose of evaluation is to affect decision making." [10] Feedback from evaluation can determine whether to begin, maintain, change, or end the performance ”or the performer. To provide feedback that will trigger or support decisionmaking, the PT practitioner or evaluator may focus the evaluation on one or more of the following goals:
-
Confirm that a particular procedure or treatment is being carried out as prescribed.
-
Indicate the effectiveness of some treatment or intervention in order to decide whether to continue, expand, or eliminate the treatment.
-
Audit the current state of affairs.
-
Determine whether the cost of an activity is justified by its effects. [11]
Earlier, this chapter indicated that the type of evaluation helps to determine the purpose of the evaluation. However, there is also a whole matrix of decisions (see Figure 7-2) that determines the purpose of evaluation, and the purpose of evaluation in turn determines when, what, and how to evaluate. [12]
When the purpose of the evaluation is explicit (clear, specific, and detailed), true (real and undistorted), and determined in advance, the resulting when, what, and how will unfold more smoothly.
When to Evaluate
PT practitioners should not think of evaluation as an afterthought or a one-time event. Ideally, "evaluation may occur at any time and with any frequency. It depends on the purpose of the evaluation." [13] Below are two supporting viewpoints from the literature.
-
"During analysis one often carries out the activities earlier defined as evaluative: clarifying decisions to be made, collecting information, and feeding it into the decision process. A good part of the needs assessment ” of front-end analysis in general ” is evaluation." [14]
-
"Establish a framework for accountability during the selection and implementation phases and ensure the personal involvement of key decision makers in choosing bottom-line measures to demonstrate the value of their efforts." [15]
Role of the PT Practitioner/Evaluator
Before completing this introduction to the evaluation component of the HPT Model, it is necessary to take a brief look at the role of the PT practitioner or evaluator. The PT practitioner or evaluator is responsible for performing or monitoring five major tasks , [16] whether the evaluation is formative, summative, confirmative, or meta. The tasks include:
-
Setting the boundaries (or focus) by establishing the purpose, goals, objectives, and scope of the evaluation.
-
Selecting the appropriate method (s) based on the purpose and the context (political environment, available resources, time, and cost).
-
Collecting all the information that is feasible , available, and relevant.
-
Analyzing the data using sound qualitative or quantitative methods.
-
Reporting the findings after first determining who needs to know, what they need to know, how best to inform them, and how often to keep them informed.
The remaining sections in this chapter will help the PT practitioner or evaluator to plan and conduct a formative, summative, confirmative, and meta evaluation within the framework of the HPT Model.
Job Aid 7-1: TYPES OF EVALUATION: ADDRESSING THE ISSUES
| |
Directions: The columns are labeled with the four types of evaluation; the rows are labeled to describe issues that must be addressed when planning an effective evaluation. Fill in each cell before the evaluation begins and revise as the evaluation progresses.
| Formative | Summative | Confirmative | Meta | |
|---|---|---|---|---|
| Primary Audience | ||||
| Primary Emphasis in Data Collection | ||||
| Primary Role of PT Practitioner | ||||
| Primary Role of Evaluator | ||||
| Typical Methodology | ||||
| Frequency of Data Collection | ||||
| Primary Reporting Mechanisms | ||||
| Reporting Frequency | ||||
| Emphasis in Reporting | ||||
| Requirements for Credibility |
ISPI 2000 Permission granted for unlimited duplication for noncommercial use.
| |
[1] Kaufman, Keller, and Watkins, 1997, p. 9
[2] Geis and Smith, 1992, p. 130
[3] Geis and Smith, 1992, p. 131
[4] Geis and Smith, 1992, p. 134
[5] Hanson and Siegel, 1995, p. 27
[6] Hellebrandt and Russell, 1993
[7] Madaus, Scriven, and Stufflebeam, 1987, p. 16
[8] Geis and Smith, 1992, p. 133
[9] Rosenberg, 1996b, p. 9
[10] Geis and Smith, 1992, p. 133
[11] Geis and Smith, 1992, pp. 133 “134
[12] Geis and Smith, 1992, p. 134
[13] Geis and Smith, 1992, p. 138
[14] Geis and Smith, 1992, p. 139
[15] Rothwell, 1996b, p. 283
[16] Herman, Scriven, and Stufflebeam, 1987