CHAPTER

12

Measuring Training Effectiveness

A large number of organisations are setting up their own training facilities. They are also inviting outside faculty to conduct in-company courses for executives at all levels. This trend will continue because new orientation in managerial and technical areas is necessary for effectively responding to the changing environment of business. Training is expensive if it does not serve the purpose for which it is given. The training should be able to improve the capabilities of individuals and, collectively that of the organisation. If the programme of training does not contribute to the building of organisational capabilities, and, reflect, over a period of time, in improved performance, the loss to the organisation is far greater than the money spent on training. The organisation could lose its competitive edge and its market position. Hence training has to serve identified purposes for the organisation based on a systematic analysis of its own capabilities and the demands upon it that the future scenario may make. The organisation has to identify what skills, attitudes, behaviour, knowledge, conceptual and operating capabilities are needed and which of these can be developed through training and which of these require reorientation in policy and management practices. In our experience very few organisations spend enough time to analyse the situation with care before launching on training programmes. We have seen that even some of the best managed companies set out vague goals such as ‘training for leadership’ or ‘team building’, etc. without seriously questioning what they expect from ‘leaders’ or ‘team effort’. In many organisations some management practices have to be changed in order that leader or team-building exercises in training can give the desired results. Without clarity of purpose training effectiveness is difficult to measure.
Note : Reproduced by permission from Indian Journal of Industrial Relations, Vol. - 36, No. - 3, 2001

98

Education & Training

Another critical area for deriving expected benefits from training is the measurement of training effectiveness–how well the training inputs are serving the intended purpose. This aspect is often neglected by organisations, saying that measurement is difficult. We believe that the techniques in social sciences have developed sufficiently to measure most important aspects of training that contribute to its effectiveness. Here we will discuss the techniques that we have used in our work, and found satisfactory to improve upon training inputs. There are three kinds of training outputs that organisations need to measure. They are: – Relating to course planning, relevance, comprehension and whatever goes on in the teaching programme and the environment. – The utilisation of what is learnt on the job, i.e. transfering the classroom learning to the job in terms of skills, competencies, decision making, problem-solving abilities and relationships and the like. – The changes in the mind set such as work related attitudes, values, interpersonal competencies and personal attributes.

Measures
All measures are devised to assess the difference between pre and posttraining. There are no absolute measures. The measures compare the effect of training interventions with some other comparable data. Hence, the organisation must obtain pre-training data or some benchmark on the same dimensions as expected from training interventions. The second consideration is that the trainer has to be clear about two things, (i) that the objectives of training are specific, i.e. the outcome expected and (ii) that the training material and the methodology are capable of achieving the stated results. – Measures of the first training output have been used for many years. They consist of two kinds of instruments: (a) Detailed questionnaire that solicit responses of participants on various dimensions of the programme, relevance, evaluation of each session on a 3, 5 or 6 point scale, instructor effectiveness, overall evaluation of the programme, etc. There are many variations in the items and the scales. (b) Identifying the concepts or the ideas that the participants have found useful and how they would employ them at work. The responses give an overall assessment of the content and delivery

Measuring Training Effectiveness

99

of the programme and the usefulness of the programme for the participants. Both kinds of instruments are often used by training institutions. – The second concern has to do with how has the participant applied the concepts or the ideas that he may have learnt in the training programmes. Some measures are developed in the class-room situation in respect to certain identifiable areas. More often the measures have to seek data from post-training work situation. These data are obtained from the participant concerned, his peers and supervisors. The type of data ranges from questionnaires to ‘incidents’ relating to the participant. At times, open ended data are obtained from the peers and supervisors. There are many variations of this data. In-company programmes are able to obtain such data; open programmes have difficulty in obtaining reliable data in post-training work situations. – Data about mind set, attitudes, values and other person-related factors have to use indirect methods to assess training. The nature of changes in the mind set take place ‘within’ the individual and the processes that induce changes are not visible. It is often mentioned that any changes that might occur within the individual will show up in the post training behaviour of the participant. The difficulty is that such changes are slow and, in many cases, observable over a period of time. This being so, it is difficult to distinguish the contribution of training and that of other factors that may have contributed to the behavioural change. And without the assessment of the contribution that training may have made to the change process, where applicable, it is not possible to assess the effectiveness of the training. Hence two kinds of measurements are necessary to evaluate training that aim at modification in person-related behaviour: (a) Measuring the changes within the individual that the training may have initiated. These processes are not manifest or observable. (b) Actual behaviour changes at work.

(i)

Measures of Changes within the Individual

The measurement of internal (or intra-person) processes is possible by the insights gained in the theory of psycho-dynamics i.e. the causal relationships between behaviour and the inner processes that cause them. It is argued that the choices that an individual makes to deal with a situation are guided by his personal orientation (set of beliefs, values, perceptions, etc.). Changes in the personal orientation are necessary if the individuals’ choices leading to decision making have to change.

100

Education & Training

The instrument to measure the impact of training would have to identify the factors that the individual uses to deal with a live problem before and after training. If the training has an impact on the participant, he would employ different factors to analyse the situation compared to factors he may have used before training. For this purpose we have used what is in literature mentioned as Field Force Diagram postulated by Kurt Lewin. The logic is simple: an event at a given time represents a balance of positive forces pushing in one direction and a set of negative forces pushing the particular event in the opposite direction. The event at that time occurs because the positive and the negative forces are equal. In case the balance has to be altered, the equilibrium of the two sets of forces must be disturbed. The participants in the programme are required to identify the forces that they would manipulate to alter the situation. They have to give reasons for the choice. The before and after training exercise would indicate if any change in the participants’ approach has occurred. We have also used certain ‘projective tests’ to understand the “inner” dynamics of the person. We used three pictures, for example, for the participants to construct a story (a variant of the Thematic Apperception Test) and analysed these stories to identify if any ‘process” changes have occurred. Some other methods used for this purpose are the following: – The evaluator poses a problem and requires the individual to solve it and then identify the factors that he/she took into consideration in arriving at the decision. – Analysing a series of incidents in which the individual is involved at work. – Observation over a period of time and analysing the behaviour of the individual. – Analysing the level of learning by in-depth interviews by a consultant with counselling background or a clinical psychologist. The measures that indicate the changes ‘within’ the individual due to training have been used by many trainers. All the measures used for this purpose have to be ‘projective’, i.e. provide an index of the process within the individual. From the perspective of a trainer, the results on these tests are invaluable for knowing the impact of training.

Measuring Changes in Behaviour
Post training changes in behaviour of a participant can be assessed only at the place of his work. Many organisations identify certain critical dimensions

Measuring Training Effectiveness

101

of change so that specific data on these aspects are available. Life Insurance Corporation, for example, identified four areas—decision making, problem solving, supervisory skills and relations with others. The rationale for choosing these areas was that these tasks require assimilation of data from varied sources and employs the individual’s overall competence in dealing with the situation. The trainer has to obtain data in these areas from people with whom the participant interacts the most. In this exercise pre-training data are not necessary because the respondent provides these data in the interview. The problem in this measurement is that it is difficult to reconcile the data from respondents in case of discrepancies in their reporting. The methods of quantifying the responses are also crude. However, the trainer can get the trends and the direction and, in most cases, these could serve his/her purpose adequately. The interviewer can seek concrete examples of the changes. The incidents reported by respondents can be subject to content analysis. The behavioural changes on account of training inputs have to be assessed within a short time after training. If the exercise is delayed the respondents may find it difficult to recall post and pre-training behaviour of the trainee. Consequently the data obtained in the interviews could be impure. Organisations use questionnaires to be filled by the trainee and his colleagues, including the supervisor. We have, however, found it necessary to obtain data from personal interviews at least from a sub-sample of the population.