Psychological Assessment
PYC 4807

Assignment 1
Unique number: 553283

IRMA VISAGIE
Student number: 42056829

Open Rubric

2
Table of Content

Pages

1. Introduction

p4

Section A
2. Steps in developing a measure

p4

2.1 The planning phase

p4

2.1.1 Specify the aim of the measure
2.1.2 Define the content of the measure
2.1.3 Develop the test plan
2.2 Item writing

p7

2.2.1 Write the items
2.2.2 Review the items
2.3 Assemble and pre-test the experimental version of the measure

p8

2.3.1 Arrange the items
2.3.2 Finalise the length
2.3.3 Answer protocols
2.3.4 Develop administration instruction
2.3.5 Pre-test the experimental version of the measure
2.4 The item-analysis phase

p9

2.4.1 Classical test-theory item analysis: Determine item difficulty (p)
2.4.2 Classical test-theory item analysis: Determine discriminating power
2.4.3 Item response theory (IRT)
2.4.4 Identify items for final pool
2.5 Revise and standardise the final version of the measure

p 11

2.5.1 Revise the items and test
2.5.2 Select items for the final version
2.5.3 Refine administration instructions and scoring procedures
2.5.4 Administer the final version
2.6 Technical evaluation and establishing norms
2.6.1 Establish validity and reliability
2.6.1.1 Reliability
2.6.1.2 Validity
2.6.2 Establish norms, set performance standards or cut-scores

p 11

3
2.7 Publish and refine continuously

p 15

2.7.1 Compile the test manual
2.7.2 Submit the measure for classification
2.7.3 Publish and market the measure
2.7.4 Revise and refine continuously
3. Evaluate a measure

p 16

Section B
4. Steps in the adaptation of an assessment measure for cross-cultural application p 16
5. Conclusion

p 18

6. References

p 18

4
1. Introduction
In the following essay in section A we are going to consider the process involved in developing a psychological assessment measure. We are going to discuss the steps that you would take in this process, including how you would choose items for your test, evaluate reliability and validity and the issue of establishing norms. Furthermore in section B, we are going to list and briefly discuss the steps that should be followed in the adaptation of an assessment measure for cross-cultural application.
2. Steps in developing a measure
A psychological measure needs to be planned carefully, items need to be written and the initial version of the measures needs to be administered so that the effectiveness of the items can be determined. After this, the final items are chosen and the measure is administered to a representative group of people so that the measure’s validity, reliability and norms can be established finally, the test manual is compiled (Foxcroft and Roodt, 2013).
2.1 The Planning Phase
2.1.1 Specify the aim of the measure (why is the test being developed and what is its purpose)
According to Fox croft and Rood t (2013) the following needs to be clearly stated:
•

the purpose of the measured

•

what attribute, characteristic or construct it will measure ( for example: verbal ability

•

whether the measure is to be used for screening purposes, in-depth diagnostics

•

assessment or for competency-based selection and training purposes

•

the type of decisions that could be made on the basis of the test scores

•

which population (group of people) the measure is intended as well as whether

introversion or extroversion, fine or gross motor skills)

the measure is to be used in a multicultural and multilingual context
•

whether the measure can be individually administered and/or administered in a group context

•

whether it is paper-based or computer-based

•

whether it is a normative, causative or criterion-referenced measure

5
For example:
Name: South African Developmental Screening Test (SADST)
Target Population and Age range:
Children aged ten to sixteen from all South African cultural groups.
Aim/Purpose: Psychological assessment of teenagers to measure self-esteem.
Type of measure: Normative (individual’s performance is compared to that of an external reference or norm group).
2.1.2 Define the content of the measure
The content of a measure is directly related to the purpose of the measure (Foxcroft and Roodt, 2013). Firstly you need to operationally define the construct (content domain) to be tapped by the measure. The rational method can be used for tests used in clinical settings by under-taking a thorough literature study of the main theoretical viewpoints regarding the construct that is to be measured. In educational settings the construct can be defined through an analysis of learning outcomes and in organisational settings through a job analysis. It is very important that the construct is comprehensively defined, and according to Wilson (as sited in Fox croft and Rood t,
2013) a construct map will insure this which will make it easier to operational more concretely. Secondly, the purpose for which the measure is developed must be considered. If it needs to discriminate between different groups of individuals information will have to be gathered about the aspects of the construct on which these two groups usually differ criterion keying for example when it comes to a construct such as self-esteem children with low self-esteem find it difficult to socialize while children with high self-esteem find it easier to socialize, therefore if a measure aims to identify low self-esteem children an item related to social skills should be included.
The more rational or analytical approach is often combined with the empirical criterion-keying approach to ensure that the resultant measure is theoretically grounded, as well as linked to an important criterion furthermore, when items are tried out in the subsequent phases, factor analysis can be used to assist in defining the underlying dimensions (constructs) being tapped by the measure (Fox croft and Rood t, 2013).

6
2.1.3 Develop the test plan
The format of the test need to be considered. It consists of two aspects:
- a stimulus to which a test taker responds
- a mechanism for response according to McIntyre and Miller (as sited in Fox croft and
Rood t, 2013). Test items provide the stimulus and the more common item formats are the following:
•

open-ended items

•

forced-choice items for example multiple-choice or true/false

•

sentence-completion items

•

performance-based items for example essay must be written or a scientific experiment needs to be performed.

While the choice of item format is linked to what is to be measured, practical consideration also have a bearing on the choice of format for example of large numbers of people are to be evaluated it needs to be easily scored by using multiple choice rather than essay tape or open-ended questions.
Method of responding also entales various formats such as:
•

Objective formats where there is only one response that is either correct (e.g. in multiple-choice options), or is perceived to provide evidence of a specific construct
(e.g. in true/false options).

•

Subjective formats where the test-taker responds to a question verbally (interview) or in writing (open-ended questions or essay-type questions) and the interpretation of the response as providing evidence of the construct depends on the judgement of the assessment practitioner according to McIntyre and Miller, 2006 (as cited in
Foxroft and Rood t, 2013).

Time-limits place a restriction on the number of items to be include, but seems to increase the effectiveness of a test.
It is further more, according to Foxcroft and Rood t (2013) critically important to pay attention to the potential for method and response base, especially when the measure is developed for use with multicultural test-takers. Test-takers are also a potential source of bias themselves as they might respond by using a specific style or response st/bias (e.g.
Agreeing with all the statements) that might result in false or misleading information.

7
Another potential source of bias relates to language in which the measure is developed and administered. If the measure in intent to be used in a multilingual context, then only having the measure available in Xhosa, could be a potential source of bias for test-takers who only speak English or Afrikaans.
The length of the measure should also be considered and is partly influenced by the amount of time that will be available to administer the measure. The length also depends on the purpose of the measure.
Taking into account all these aspects the test-developer now has a clear conceptualisation of the specifications for the measure. This is formalized in the test plan specifications which specify the content domains to be included and the number of items that will be include in each for example emotional intelligence test: four aspects of the construct namely interpersonal relations, intra personal functioning, emotional will-being and stressmanagement will be covered and there will be ten items per aspect.
2.2

Item writing

2.2.1 Write the items
According to Fox croft and Rood t (2013) a few important pointers for item writing are the following:
•

Wording – clear and concise

•

Vocabulary used must be appropriate for the target audience

•

Avoid negative expressions such as “never” or “not”, and double negatives in particular •

Only one central theme in an item

•

Avoid ambiguous items

•

Vary the positioning of the correct answer in multiple-choice measures

•

All distracted for multiple-choice response alternatives should be plausible (i.e. the distracted should be as attractive as the correct answer). According to Linn and
Gronlund (2000) (as cited in Fox croft and Rood t, 2013) it is best to provide four answer choices.

•

True and false statements should be approximately the same length and the number of

8 true verses false should be more or less equal.
•

The nature of the content covered should be relevant to the purpose of the measure
(e.g. you would not expect a personality measure to contain an item asking what the capital of Kenya is).

As stated by Own and Taljaard, 1996 (cited in Foxcroft and Roodt, 2013) the rule of thumb is that test developers usually develop double the number of items than they require in their final item pool and at least one third of the items are discarded when item analysis is performed. 2.2.2 Review the items
After the pool of items has been developed, it should be submitted to a panel of experts for review and evaluation during which they will be asked whether the items sufficiently tap the content domain for dimensions of the construct being assessed. The cultural, linguistic and gender appropriateness of the items are also consider, as well as the wording and nature of the stimulus material.
The pool of items could also be administered to a small number of individuals from the target population to obtain qualitative information regarding items and test instructions that they have difficulty understanding.
Based on the review certain of the items may have to be revised or even rewritten.
2.3 Assemble and pre-test the experimental version of the measure
In getting the measure ready for its first experimental administration, several practical considerations require attention.
2.3.1 Arrange the items
The items need to be arranged in a logical way in terms of the construct being measured.
2.3.2 Finalise the length
Now that the experimental item pool is available the length of the measure needs to be revisited. Their must be sufficient items to sample the construct being measured but the time test-takers will need to read items also has to be considered.
2.3.3 Answer protocols
For paper-based tests, decisions need to be made as to whether items will be completed in the test booklet, or whether a separate answer sheet (protocol) needs to be developed. The

9 protocol must aid the scoring of the measure and must be easy to reproduce.
2.3.4 Develop administration instruction
Care needs to be taken in developing clear, unambiguous administration instructions for the experimental try-out of the items. The assessment practitioners need to be comprehensively trained. If care is not taken during this step, it could have negative consequences for performance on the items during the experimental pretesting stage. For example, badley worded administration instructions, rather than poorly constructed items can cause poor performance on the item.
2.3.5 Pre-test the experimental version of the measure
The measure should now be administered to a large sample (400 – 500) from the target population. Besides quantitative information regarding performance of each item gathered during this step, qualitative information should also be gathered.
2.4 The item-analysis phase
The item-analysis phase is the phase during which “the best” items are chosen for the final measure, according to Tutorial Letter 201 for PYC 4807 (2014).
This phase adds value to the item development and the development of the measure in general. According to Fox croft and Rood t (2013) the purpose of item analysis is to examine each item to see whether it serves the purpose for which it was designed. It helps to determine how difficult an item was, whether it discriminates between good and poor performers, whether it is biased against certain groups and what the shortcomings are.
Certain statistics are computed to evaluate the characteristics of each item namely the classical test theory or item response theory (IRT) or both. The resultant statistics are then used to guide the final item selection and the organisation of the items in the measure.
2.4.1 Classical test-theory item analysis: Determine item difficulty (p)
The difficulty of an item (p) is the proportion or percentage of individuals who answer the item correctly. That is: p – value = Number of people who answered correctly
Number of people who took the measure

10
The higher the percentage of correct response, the easier the item and vice versa.
2.4.2 Classical test-theory item analysis: Determine discriminating power
One of the purposes of item analysis is to discover which of the items best measure the construct or content domain that the measure aims to assess. Good items consistently measured the same aspect that the total test is measuring.
The discriminating power of an item can be determined by means of the discrimination index
(D) the method of extreme groups are used. Performance on an item is compared between the upper 25% and the lower or bottom 25% of the sample. If the item is a good discriminator, more people in the upper group will answer the item correctly. To complete the discrimination index, the upper and lower 25% of test-takers need to be identified. Next the percentage of test-takers in the upper and lower groups who passed the item is computed and the information I substituted in to the following formula:
D=U–L
nᵤ

nı

where:
U

=

number of people in the upper group who passed the item

Nu =

number of people in the upper group

L

number of people in the lower group who passed the item

=

n1 =

number of people in the lower group

A positive D

=

item that discriminates between the extreme groups

A negative D

=

indicates an item with poor discriminatory power

An item – total correlation can be performed between the score on an item and performance on the total measure. A positive item – total correlation indicates that the item discriminates between those who do well and poorly on the measure. Close to Zero indicates that the item does not discriminate between high and low total scores. A negative item-total correlation is indicative of an item with poor discriminatory power. Correlations of .20 are considered to
10
be the minimum acceptable discrimination value to use when it comes to item selection.

2.4.3 Item-response theory (IRT)
By using item-response theory, the difficulty level and discriminatory power of an item can be even more accurately determined.
2.4.4 Identify items for final pole
Classical test theory, IRT and DIF analyse can be used to determine item parameters based on which items should be included in the final version of the measure or discarded.
2.5 Revise and standardize the final version of the measure
2.5.1 Revise the items and test
Items identified as being problematic during the item-analysis phase need to be considered whether it should be discarded or revised.
2.5.2 Select items for the final version
The test developer now has a pool of items that has been reviewed by experts and on which empirical information regarding item difficulty, discrimination and bias has been obtained. based on this information the selection of items for the final measure takes place.
2.5.3 Refine administration instructions and scoring procedures
Based on the experience and feedback gain during the pretesting phase, the administration and scoring instructions might need to be modified.
2.5.4 Administer the final version
The final version is now administered to a large representative sample of individuals for the purposes of establishing the psychometric properties (Validity and reliability) and norms.
2.6 Technical evaluation and establishing norms
2.6.1 Establish validity and reliability
The psychometric properties of the measure need to be established.
2.6.1.1 Reliability
According to Tutorial Letter 301 for PYC 4807 (2014). Reliability refers to the consistency with which a psychometric test measures whatever it measures. However consistency always implies, according to Fox croft and Rood t (2013), amount of error in measurement a
11
person's performance in ons administration of measure does not reflect with complete accuracy the “true” amount of trait that the individual possesses. Emotional state of mind,

fatigue, etc. may effect his or her score on the measure.
We can capture the true and error measurement in equation form as follows:
X=T+E
where:
X

=

observed score (the total score)

T

=

proportion true score (reliability of the measure)

E

=

proportion error score (unexplained variance)

If we obtained the test scores of the target population, the variance of the observed test scores
(X) would be expressed as follows in terms of true (T) and error (E) variance:
Sx² = Sт² + Sᴇ²
Reliability (R) can be defined as the ratio of true score variance to observed score variance:
R = Sт² = Sx² - Sᴇ²
Sx²

Sx²

The numerical expression for reliability is a reliability coefficient, which is nothing more than a correlation coefficient.
Types of reliability
There are five types of reliability coefficient:
•

Test-retest

•

Alternate-form

•

Split-half

•

Inter-item
12

•

Inter-and intra- scorer reliability

2.6.1.2 Validity
Validity refers to appropriateness of the inferences made from test scores or the according to tutorial Letter 102 (2014) the more traditional definition that a “test measures what is is supposed to measure”. We need to make sure, if we claim to measure intelligence, that we measure intelligence and not memory or aptitude.
We should have a clear conception of at least three aspects, namely:
•

What the entity is that we want to measure (e.g. a persons weight of height)

•

What the exact nature of the measure is (e.g. the scale or measuring tape) that we want to use i.e. how well it would measure the construct

•

Finally, the application of the rules on how to measure the object (e.g. with or without clothes, shoes)

Type of validity:
1) Content -description procedures
- involves essentially the systematic examination of the test content to determine weather it covers a representative sample of the behaviour domain to be measured (Tutorial Letter
103, 2014). Such a validation procedure is commonly used to tests designed to measure how well the individual has mastered a specific skill or course of study.
There are two important aspects when considering the content validity of a measure:
•

Face validity – type of validity in non-psychometric or non-statistical terms. It has to do with whether the measure 'looks valid' to test-takers who have to undergo testing.

•

Content validity – involves determining whether the content of the measure covers a representative sample of the behaviour domain.

2) Construct identification procedures
•

Construct validity involves a quantitative, statistical analysis procedures. The extent to which it measures the theoretical construct or trail it Is supposed to measure.(e.g. intelligence, personality, eye-hand coordination ext)
13

3) Criterion – prediction procedures
•

Criterion – prediction validity involves the calculation of a correlation coefficient between predictors and criterion.

2.6.2 Establish norms, set performance standards or cut-scores
A norm is described as “a measurement against which the individuals raw score is evaluated sot that the individual's position relative to that of the norm sample can be determined”
(Foxcroft and Roodt 2013 p 40).
If a norm-referenced measure is developed appropriate norms need to be established. This represents the final step in standardising the measure. An individual’s test score has little meaning on its own. However, by comparing it to that of a similar group of people (the norm group) the individual's score can be more meaningfully interpreted (Foxcroft and Roodt,
2013).
Types of test norms:
•

Developmental scales

•

Percentiles

•

Standard scores

•

Deviation IQ scales

Setting standards and cut-off scores
Another way in which test scores can be interpreted is to compare performance to external criterion (standard). For example, in the psychometrics examination that is set by the
Professional Board for Psychology, the passing standard is 70%. Someone who obtains a mark of 60% may have performed better than a classmate who obtained 50% (norm referenced comparison), but would fail the examination nonetheless, as his/her percentage in blow that of the 70% standard (cut-off) set.

14
2.7

Publish and refine continuously

2.7.1 Compile the test manual
Test manual should:

•

Specify the purpose of the measure

•

Indicate to whom it can be administered

•

Provide practical information

•

Specify administration and scoring instructions

•

Outline in detail the test-development process followed

•

Provide detailed information on types of reliability and validity information established, how this was done, and what findings are

•

Provide information about cultural appropriateness of the measure and extent to which test and item bias has been investigated.

•

Provide detailed information about when and how norms were established

•

Where appropriate, provide information about how local norms and cut-off scores could be established

•

Indicate how performance on the measure should be interpreted

2.7.2 Submit the measure for classification
As the use of psychological measures is restricted to the psychology profession, it is important to submit the measure to the Psychometrics Committee of the Professional Board for Psychology so it can de determined whether it should be classified as a psychological measure or not.
2.7.3 Publish and market the measured
As the use and purchase of psychological measures is restricted to the psychology profession in South Africa, test developers and publishers need to market the measures to the appropriate target group/market.
2.7.4 Revise and refine continuously
Test developers usually wait until a substantial amount of information has been gathered over a five to ten year period regarding how the measure needs to be revised before they undertake a full-scale revision and reforming process.
15
3. Evaluate a measure
When evaluating a measure assessment practitioners should, among other things, conceder the following (Foxcroft and Roodt, 2013):

•

How long age measure was developed, as the appropriateness of the item content and the norms changes over time

•

The quality and appeal of test materials

•

The quality of the manual content

•

The clarity of the test instructions

•

The cultural appropriateness of the item content and the constructs being topped

•

If the measure is to be used in a multicultural and/or multilingual context, whether bias investigations proved evidence that the performance of various groups on the measure is equivalent •

The adequacy of psychometric properties

•

The nature of the norm groups and recency of the norm

Section B
4. Steps in the adaptation of an assessment measure for cross-cultural application
According to Van Ede (1996) cross-cultural assessment is the evaluation of person's attributes and behaviour by obtaining measures of these under different cultural conditions and by comparing them in order to establish cross-cultural uniformities and differences.
Steps in the adaptation process:
1) Selection of an appropriate instrument to adapt:
The test/measuring instrument should be suitable for the purpose for which it is intended.
We will consider the purpose for which the measuring instrument is going to be used, establish the transportability of constructs, consider language evaluation to avoid different interpretations, consider psychometric characteristics that should have test reliability and validity. 2) Translation of the source instrument
Here we consider the selection of translators to ensure that they are expert in both the
16
language of the source instrument and the language into which the instrument is translated.
We also consider translation procedures where an attempt should be made to achieve translation equivalence between the two instrument. According to Van Ede (1996) there are

various translation procedures for example the team approach, back-translation, forward translation etc.
3) Deciding on an experimental design
4) Administration of adapted instrument
– Equivalence of context eliminate the effect of cultural differences which are not under investigation, but which could influence scores
- Eliminate bias in instrument administration.
Be aware of :
•

rudeness bias (not sensitive to norms of target culture)

•

“I-can-answer-any-question-bias” (believe they must answer all the questions even if they do not have answers)

•

courtesy bias (trying to find out what the researcher want and then answers accordingly •

hidden premises bias (trying to discover who the researcher really represent and really trying to learn).

- Familiarity of testing techniques, presentation mode, item formats, test conventions and procedures. - Influence of the test administrator
Gender, age, ethnicity, language can influence the outcome of the measurements
5) Pilot testing
After translating/adapting an instrument it should be pilot tested to confirm its factor structure in the target culture and to establish whether it exhibits measurement equivalence.
6) Determining psychometric equivalence
To establish whether the adapted measuring instrument is equivalent to the original, item equivalence as well as the equivalence of the factor structure should be establish.
17
5. Conclusion
The development of a psychological measure is a time-consuming and people intensive process. According to Fox croft and Rood t (2013) it takes a minimum three to five years to

develop a measure from the pont at which the idea is conceived to the moment when the measure is published and becomes available to the consumer.
According to Van Ede (1996) we should be aware of the problems involved in cross-cultural assessment since we have eleven official languages in South Africa. By following the indicated steps, unnecessary mistakes in adapting measuring instruments or test for application among the various culture/language groups in South-Africa may be avoided.
6. References
Foxcroft, C.S Roodt, G (2013). Introduction to Psychological Assessment in the South
African context.
Turorial Letter 301, PYC 4807 (2014). Department of Psychology, University of South
Africa, Pretoria.
Tutorial Letter 201, PYC 4807 (2014). Department of Psychology, University of South
Africa, Pretoria.
Van Ede, D.M. (1996). How to adapt a measuring instrument for use with various cultural groups: a practical step-by-step introduction, 10 (2), 153-159.

RESULTS
Section A = 4
Theory section A = 16
Steps section B = 4
Theory section B = 6
Application = 12
Structure of the e = 8
Referencing = 8
-----------------------------Total = 58 / 100 (58%)

Finalize