Problems arising in self-assessment
In a nutshell, the greatest disadvantage of self-assessment as a method of obtaining data is the greater chance of measurement error. In a meta-analysis of 44 self-assessment studies in higher education, Falchikov and Boud (1989) reported correlations between self-assessed and external measures of performance ranging from -0.05 to 0.82, with a mean correlation of 0.39. In a similar review of 18 self-assessment studies in the health professions, Gordon (1991) reported correlations ranging from 0.02 and 0.65. Although Ward et al. (2002) have cast doubt as to how much credence should be given to these correlations - most were between self-assessments and expert ratings, and the latter may themselves be flawed - it is clear that even in the most favourable case selfassessments paint a less than perfect picture. What are the main sources of error?
In principle, errors can be divided into those resulting from a more or less ‘intentional’ manipulation of answers by respondents, and unintentional discrepancies between the real and reported values.
Unintentional measurement errors arise when the answers given by respondents in good faith do not correspond to the ‘real’ value on the variable in question. There are various reasons why unintentional measurement errors may occur. First of all, the content of the question may be unclear or ambiguous.
This problem is likely to give rise to discrepancies between the concept as intended by the researchers and the concept as understood by respondents, as well as to discrepancies between the understanding of the concept by different groups of respondents (Ward et al. 2002). Dykema &
Schaeffer (2000) have shown that complexity and clarity are strong predictors of measurement errors.
Such errors seem particularly likely in the case of characteristics such as skills and competencies, which are by nature complex, abstract and difficult to delineate.
A second factor that can give rise to unintentional measurement errors is that of limitations to respondents’ comprehension or memory. Even if the formulation of a question as such is completely clear, respondents can only report on what they understand, and what they can readily retrieve from memory. If confronted with questions that fall outside these limitations, respondents will be forced to
12
choose between skipping the question altogether or making a guess as to the answer. If the limitations are themselves differentially distributed between different groups of respondents, item non-response will be selective and constitute a form of measurement error. If graduates fail to comprehend the question fully but still offer an answer, the validity of the data will be compromised. An important point is that limitations to comprehension also apply to instructions or explanations given to help respondents understand the question. Very detailed or subtle instructions are themselves likely to be poorly understood. Problems with comprehension may be of particular importance when considering self-assessment of skills. It takes a certain level of (meta-)cognitive skills to be able to reflect about one’s job, the requirements that are imposed and the possessed level. This may imply that selfassessment may be more difficult to administer among lower educated groups.
A third source of unintentional measurement error stems from the so-called anchor problem. This refers to ambiguity or lack of clarity of the measurement scale used. In contrast to variables such as working hours or income, there is no natural numerical scale on which to measure skills. This places a burden on researchers to provide a scale that is clearly understood in a uniform way by all categories of respondents. Ideally, all respondents should share the same understanding of what the extreme values and midpoint - the anchors in the scale - represent. Self-assessments of skills often use very general terms to indicate extreme values, such as ‘very low’ and ‘very high’. Such scale values are not explicitly related to any objective characteristics in the real world. As a result of the ambiguous nature of the anchor points, different groups of respondents are likely to use their own frames of reference when answering the questions, so that the answers will not be comparable between groups (Ward et al., 2002). This can lead to systematic overestimation or underestimation of skills by different groups, whose reference groups have respectively a lower or higher level than the population at large.
Implicitly, the extremes and midpoint on the scale might be assumed to correspond to the extremes and midpoint of the distribution of skills in the population as a whole. However, most respondents are unlikely to have a comprehensive overview of the total distribution of a particular skill in the population, particularly if that population is very broadly defined. In the absence of clear clues as to what ‘very low’ or ‘very high’ means, respondents will tend to use their own frame of reference of what is considered
‘normal’ or ‘average’. In the case of skills, this is likely to be strongly biased by the respondent’s own educational background or occupational affiliation. This implies that differences between occupational groups or fields of study are probably biased towards the mean, making it difficult if not impossible to assess the overall skill level or to compare different groups. Lack of clarity in the scale used may also give rise to the so-called halo-effect (Gray, 1996). This refers to the tendency of certain respondents to use only a small range of the scale (say 3 or 4 on a 5-point scale) for all questions.
In addition to these sources of unintentional errors by respondents, there are also various reasons why respondents might intentionally alter their true responses (Richter & Johnson, 2001). Many of these reasons fall under the general heading of what Orne (1962) calls "demand characteristics". This refers to any aspect of the research environment or the research instrument that communicates a "demand" for the respondent to behave in a particular way. One of the most commonly reported reasons is that of social desirability (Victorin, Haag-Gronlund, & Skerfving, 1998): respondents may alter their responses in order to appear more ‘normal’. In the case of skills, some respondents may find it embarrassing to report very low or very high levels, for fear of appearing like ‘dunces’ or ‘geeks’.
Alternatively, respondents may have reasons to report more extreme values than apply to them in reality, for example out of boastfulness or modesty, or to deliberately mislead researchers. They may wish to appear consistent, unusual (Berg, 1967) or extreme. Despite reassurances about the confidentiality of the data, some respondents may fear that the information could be used against them. It is important to note that graduates may not always be fully aware of the fact that the answers they are giving are less than truthful. Even when they believe that they are answering honestly, individuals are often ignorant of their own motivations and internal states (Nisbett & Wilson, 1977). As a result, in practice it may at times be difficult to distinguish between intentional and unintentional measurement errors. This could imply that some remedies applied to reduce unintentional errors can also help reduce ‘intentional’ alteration.
Some solutions
There are several strategies that can be deployed to help researchers come to grips with measurement errors resulting from the use of self-assessments. The strategies can be divided into two broad categories. First of all, one can look for ways of improving the research instrument so as to
13
reduce or eliminate avoidable errors. Secondly, since some error is almost certainly unavoidable, an attempt can be made to gain an indication of the validity of the results, and thereby of the applicability of the data for various kinds of analyses. In some cases, the process of validation may provide a means of (partially) correcting for measurement errors, for example by recalibrating the data or developing appropriate control variables.
Addressing unintentional measurement error and increasing overall comprehension
There are a number of strategies that can be adopted to increase the overall reliability by reducing unintentional measurement error and increasing comprehension. Dykema & Schaeffer (2000) argue that complexity, clarity, and affective intensity are important determinants of measurement error.
Although their approach cannot be directly applied to measurement of skills (their own research is about important events in respondents’ lives rather than internal states such as skills), many of their arguments are relevant. The process of retrieval of information from long-term memory is affected by the nature of the stimuli used to trigger it. Retrieval is expected to be less accurate when the information is complex, indistinct from other information, and emotionally neutral. This suggests that measurement errors can be reduced by formulating items that are clear and unambiguous, that are clearly distinguishable from other items, and that elicit an emotional response from graduates. In the case of skills, the challenge is to formulate items that have a clear and uniform meaning to all graduates, to avoid items that are composites of several underlying dimensions, to choose items that are conceptually distinct from other skills, and to formulate the items in such a way as to tap into the feelings graduates have about their own (lack of) abilities. It is doubtful to what extent the latter suggestion can be implemented, but a minimum requirement is probably a formulation that is as active and - within the restrictions imposed by a general list - as concrete as possible. It seems advisable in any case to exercise a certain degree of caution and restraint in aiming for an emotional response, since a too emotionally charged formulation may constitute an unwanted “demand characteristic”, and induce a deliberately altered response from some graduates.
Addressing the anchor problem
There are several ways to address the anchoring problem. Ex ante expert anchoring is probably the most widespread technique. It involves the a priori development by experts of an answer scale that has a clear and uniform meaning for all respondents. This comes down to providing explicit anchors for the evaluation criteria (Ward et al., 2002; Martin et al., 1998). The values assigned to the different levels of a rating scale convey information to the respondent regarding what is expected (Richter &
Johnson (2001). Respondents will use such anchors as frames of reference for estimating their own responses (Schwarz, 1999). Ideally, the extreme points on the scale, as well as the mid-point, should correspond to something that all respondents know and assign the same meaning or interpretation to.
Two different forms of ex ante expert anchoring can be distinguished. One method uses occupational titles as anchors in the scale.9 10 Algera (1981) applied this method to 24 different task characteristics and skill requirements (see Figure 1 for an example). On the basis of expert ratings, characteristic examples of occupations are located at appropriate points over the full range of the scale.
Respondents are requested to position their own skill level with respect to the listed occupations.
Although this method is attractive in theory, it is based on a number of assumptions which may not necessarily hold. First of all, the anchor occupations are assumed to be clear to all respondents.
Finding occupations that fit this requirement may be easier said than done. For some skills it might prove difficult to find good anchors. Second, some inter-expert consistency has to be established before applying this method in a survey. This involves the usual set of methods to assess consistency between the rates of the different experts (e.g. inter-rating reliability tests). Third, the anchors must be clearly transitive: starting with the lowest level, each subsequent anchor in the scale must correspond to a more difficult level. Fourth, if the anchor coincides with the respondent’s own occupation, he/she might fill in that anchor point even if their own level is very different. Fifth, a general assumption which
9 This method could also be applied using fields of education or other clearly recognisable social categories.
10 If occupations are used to provide the main anchors, one might also think of using experts to rescale the occupations afterwards: ex post expert anchoring. The assumption is that respondents will bias their group mean towards the over-all mean, leading to a decrease of the between-group variation (e.g. between occupations), compared to the within-group variation. Under the condition that the bias is only partial (i.e. there is still between-group variation left), the differences between groups can be rescaled using the rating of experts of some typical occupations at both ends of the distribution. Note however that the variation within groups may also be biased and that this way of anchoring does not change that problem.
14
to some extent underlies all of the other assumptions is that an occupation can be regarded as a good proxy of given level on a particular skill. The actual assessment involves two steps: respondents are required to form an image of the skill level associated with each occupation, and then they are required to score their own level on that skill relative to these occupations. The use of this method may be less problematic if only three job titles are used, representing the extremes and the midpoint of the
scale.