1

Student Number: 42858798
Module PYC 4807
Assignment 1

Title: Developing a Psychological Measure.

2

Contents Page
Title Page
1. Introduction : What is Psychological Testing
2. Development of Psychological Measure
2.1
The planning phase
2.2
Item Writing
2.3
Assembly and Pretesting of measure
2.4
Item Analysis
2.5
Standardisation of the Final version
2.6
Technical Evaluation and establishing norms
2.7
Publishing and ongoing refinement
Cross Cultural test adaptation
3.
3.1 Initial Translation
3.2 Synthesis of the Translation
3.3 Back Translation
3.4 Expert Committee
3.5 Test of the prefinal version
3.6 Submission of Documentation
4.
Conclusion
5.
References

1
3
4
5
6
6
7
8
9
10
11
11
11
12
12
12
13
14

3

1. Introduction: What is Psychological Testing

Psychological testing refers to the administration of psychological tests. A psychological test is an objective and standardized measure of sample behaviour . The term sample of behaviour refers to an individual s performance on tasks that have usually been prescribed beforehand. The samples of behaviour that make up a paper and pencil test, the most common type of test, are a series of items. Performances on these items produce a test score. A score on a well constructed test is believed to reflect a psychological construct such as achievement in a school subject , cognitive ability , aptitude, emotional functioning, personality , etc. differences in test scores are thought to reflect individual differences in the construct the test is meant to measure. The technical term for the science behind this is psychometrics.
Proper psychological testing is conducted after vigorous research and development. Proper testing should include the following :
1. Standardisation : All procedures and steps must be conducted with consistency and under the same environment to achieve the same testing performance form those being tested.
2. Objectivity : Scoring must be done such that subjective judgements and biases are minimized, with results for each test taker obtained in the same way. 3. Test Norms : The average test score within a large group of people where the performance of one individual can be compared to the results of others by establishing a point of comparison.
4. Reliability : Obtaining the same result after multiple testing
5. Validity : The type of test being administered must be measured for what it is intended to measure.
The development of a psychological measure is very complex and in-depth. The process involves 7 different stages each comprised of different sub-sections.
Through the essay each stage with its respective parts will be discussed. The adaptation of psychological measure to accommodate multi-cultural testing will also be discussed.
2. Development of Psychological Measure
2.1 The Planning Phase

4

This stage of the process is the most crucial and often time s sufficient importance is not placed on this stage.
At this point the developer needs to clearly outline the purpose of the measure.
The characteristics to be measured, what the measure will be used for and the target group for the measure must also be defined and stated. Once this is determined the developer can now have an indication on what decisions can be based on tests scores.
Another important aspect in the planning phase of development is determining whether the performance will be compared to a criterion or group norm.
To define the content of a measure , the developer needs to have a defined purpose of the measure. The construct has to be operationally defined. The purpose of the measure is very important as it serves as a basis for constructing the measure. In clinical settings , the rational method is used , whereby an undertaking of the thorough literature study of the main theoretical viewpoints of the construct being measured is done. In educational settings , learning outcomes in specific learning areas define the constructs needing to be tapped.
Regardless of whether the dimensions of a construct are identified by theoretical review or analysis of learning areas, the construct must be comprehensively defined.
Secondly the purpose of the measure must be considered. A measure will use criterion keying when the different groups of individuals need to be identified according to aspects of the construct with which the individuals differ. An example provided by Foxcroft and Roodt in An Introduction to Psychological
Assessment , of criterion keying ; when a construct such as academic aptitude is being measured , high-risk students find it very difficult to thick critically compared to low risk counterparts. Therefore the measure will have to include items related to critical thinking.
In developing the test plan , the format of the test must be considered. The format comprises of two aspects namely a stimulus (how the test taker responds) and a mechanism of response. Test items form the stimulus and the most common item formats are open-ended items ( no limitations are imposed on the response ) , forced-choice items ( these include multiple choice questions where careful consideration needs to be given to the alternative options) and performance based items ( apparatus needs to be manipulated by the test taker

5

e.g. a student taking physics practical exam and must identify readings and how these readings influence final outcome.
The choice of the item format is linked to what is being measured , practical considerations also bear an influence on the format. In the case of a large number of people being tested , their tests need to be scored easily.
With regard to the method of responding to test items , the various formats include objective formats and subjective formats. Objective formats are used when there is only one response that is either correct or perceived to provide evidence of a construct. Examples of this type of format are multiple choice test and matching exercise. Subjective formats are where the test taker responds to questions either verbally or in writing , such as an interview or essay-type questions .
Time limits imposed are also linked to response of a stimulus. If a test is to be timed , there are restrictions on the number of items that can be used.

2.2 item Writing
A team of experts or specialists are usually tasked with writing of items. The purpose of the measure will help keep focus on validity. Existing measures, theories, textbooks etc. provide resources for ideas on test items. When writing a measure, the wording must be clear , concise and allows the test taker easy comprehension of what is required. Negative expressions and ambiguity are never to be used. This only serves to create confusion and may invalidate results. The central theme in an item should only be covered and appropriate vocabulary for the target audience must be used. In the case of multiple choice questions the positioning of correct answers and the ratio of true to false items should be varied and equivalent. Once the items have been developed it is review by panel experts. The panel will judge items on relevance, appropriateness for target group , wording of items and the nature of the stimuli.
Based on their findings certain items could be re-written or excluded.

2.3 Assembly and Pretesting of experimental version

6

Items need to be arranged logically and this would include grouping items together. Once the measure is in logical format, the length of the measure needs to be reassessed. Time must be given to the test takers to read and comprehend instructions. After consideration, either the time allocated for completion of the measure needs to be increased or decreased or certain items need to be further eliminated. Answer protocol should be predetermined as it allows for ease of administration. A booklet or answer sheet should be developed in such a way that scoring of the measure, and reproduction of the booklet can be done in the most uncomplicated way.
Administration instructions are then developed. The instructions should be clear and unambiguous. Usually the instructions are pretested on a sample group of the target audience. To eliminate the risk of poor performance , training on administration the test can be done.
The benefit of the pretest phase is the feedback from the test takers on the level of difficulty ,ease of comprehension , sequencing of the items as well as the length of the measure.

2.4 Analysis of items
The item analysis phase adds value to item development and in general the development of the measure. The main purpose of the items analysis is to examine each item to see whether is serves the purpose for which it was designed. The difficulty of the item is directly proportional to the percentage of the individuals who answer it correctly. This is represented in an equation as the p value . Clearly the higher the percentage of correct responses , the more difficult the item. The p value provides us with information to the frequency of correct answers, however it does not tell us anything about the characteristics of an item. Different sample will yield different values for p . The value for p does provide us with a generalised view on item difficulty.

7

Good test items will measure the same thing as a total test is measuring ( An
Introduction to Psychological Assessment, Foxcroft and Roodt). A good item score will predict a good score on the total test and vice versa. The discriminating power of an item is determined by a discriminating index (D).
Where a good discriminator has the top percentage of test takers answering correctly. A positive D value will indicate an item discriminating between upper and lower groups.
Item correlation is performed between the score on the item and the total performance on a measure. Correlations of 0.20 are the minimum acceptable discrimination value allowed when considering item selection.

2.5 Standardising the final version
After qualitative and quantitative information on the items have been determined as well as the format of the experimental version of the test, the focus becomes on revising items and administrating the final version of the test on a large group of test takers.

Items that are deemed to problematic during the previous phase need to be omitted. Whenever items are revised the same qualitative review procedures by panel of experts and item test runs should be done.

The selection of the items for the final measure takes place based on the empirical data that include item difficulty and discrimination as well as biases.
The existing database is used after final selection to confirm reliability and validity coefficients. In some cases the administration and scoring instructions might need to be revised after the above process.
Thereafter the final version is now complete and ready to be tested on s large representative sample. The aim of the tests is to establish validity and reliability as well as norms.

8

2.6 Technical evaluation and establishing norms
Validity and reliability coefficients can be computed, depending on the nature and purpose of the measure. Thereafter norms should be established as well as the performance standard and/or cut-scores set.
If a norm-referenced measure is developed it stands to reason the appropriate norms need to be used. A test takers rest scores have little meaning on its own, however when compared to that of a similar group, the results are more meaningful and better interpreted.
With a measure that uses criterion-references, cut-scores or performance standards need to be set before interpretation of test data can be done. This also guides decision making.

2.7 Publish and Refine continuously

Before a measure can be published, a test manual must be compiled and be submitted for classification.
When compiling a test manual the following should be kept in mind; the manual must specify the purpose of the measure and indicate who the measure can be administered to. The manual should also include practical information such as duration of measure and particular reading level required of the test taker.
The manual must also stipulate if training is required by those who administer the measure. Scoring instructions as well as detailed outline of the test

9

development processes used. How performance on the measure should also be interpreted. Measures must be submitted to the Psychometrics Committee of the
Professional Board for Psychology. The Committee will determine whether the measure should be classified as a psychological measure or not.

After classification, the measure is ready to publish. When publishing, test developers and publishers need to be aware of making claims that cannot be supported and misrepresenting any information.

Clear distinctions must be made between marketing material and test manual. A test manual contains factual information about the measure instead of acting as a selling device. Marketing material should not contain examples of actual test items or content. This would invalidate their use in the measure.
The use and purchase of psychological measures are restricted to psychology profession; the appropriate target market must be used in marketing campaigns.
Test measures, depending on the content of the measure should be revised.
Popularity of the measure also determines the rate at which a measure dates.

Test developers usually await a substantial amount of information has been gathered before the measure needs to be renormed and a full scale revision done. 3. Cross cultural test adaptation.
Due to the increasing the number of multinational as well as multicultural research projects, the need to adapt measures for other than the source language has also grown.
Most measures were developed in English-speaking countries; however researchers needed to for example immigrant populations in studies of health.

10

The cross-cultural adaptation of measures for use in a new country, culture or language necessitates the use of a unique method. The equivalence between the original source and the target versions of the measure must be maintained.
If measures are to be used across cultures, items must not only be translated well linguistically but be adapt to maintain content validity of the instrument at conceptual level across different cultures.
The term cross- cultural adaptation is used to encompass a process that looks at both language and cultural adaptation issues in the process of preparing a questionnaire for use in another setting.
For the purpose of this essay, the guidelines for cross-cultural adaptation process are detailed below. The information below is taken form the method used by the American Association of Orthopaedic Surgeons
3.1 Initial Translation
The first stage in adaptation is the forward language. It has been recommended that at least two forward translations be made of the instrument from the original language to the target language. The translations need to be compared and discrepancies in the translation process need to be noted. The discrepancies and poorer wording choices are identified and resolved in a discussion between the translators.
Translator one needs to be aware of the constructs being measured. Their adaptations are intended to provide equivalency from a clinical perspective and the production of a translation providing a more reliable equivalence from a measurement perspective.
Translator two should be neither aware nor informed of the constructs being quantified and should preferably have no clinical background. He/she is the uninformed translator and is more likely to detect different meaning of the original than the first translator. This translator is less influenced by an academic goal.
3.2 Synthesis and Translations
In this next phase the two translators and a recording observer synthesize the results of the translations. Working from the original measure, the two translator s versions, a common translation is produced. A written report is

11

carefully documenting the synthesis process. It is very important that consensus is reached rather than one comprising his or her feelings. The synthesis of this common translation (now called T12) marks the end of this phase.
3.3 Back Translation
Working from T-12 synthesis and totally blind from the original version, a translator then translates the measure back into the original language. This is a process of validity checking and ensures the translated version has the same item content as the original. This stage also brings to light any unclear wording in the translations. Back translation is however only one method of validity checking. Agreement between back translation and the original source version does not guarantee a satisfactory forward translation. Back translation only serves to highlight gross inconsistencies and conceptual errors in the translations. Two back translations are considered a minimum. The back translations are produced by two persons with the source language, in most cases English, as their mother tongue. The two translators should be neither informed and again preferably be without any clinical background. Information bias is avoided in this manner as well as aid in eliciting unexpected meanings of the items in the translated measure (T-12). Thereby also increasing the likelihood of highlighting the imperfections.

3.4 Expert Committee
The composition of this committee is crucial in achieving a cross-cultural equivalence. The original developers of the measure are in close contact with the expert committee. The role of the committee is to consolidate all the versions of the measure. They will therefore review all translations and reach a consensus on any discrepancies. The material used by the committee includes the original measure, each translation (including the two back translations) and all corresponding written reports. The written reports explain the reasons for each decision made at the earlier stages.
The decisions made by the committee will aim at achieving equivalence between the source and target version in four areas. Namely Sematic, Idiomatic, experiential and conceptual equivalence.

12

The committee uses the source and back translations for all the above equivalences. Consensus again must be reached on all items. In some cases translation and back translation processes are repeated to clarify how another wording of an item would work. Items, instructions and response options are considered. 3.5 Test of the Prefinal Version
This phase includes the pretest. This test of the new measure seeks to use the prefinal version on test takers from the target group.
Each test taker is interviewed after they have taken the pretest. Feelings and thoughts about the test items are noted as well as the meaning of items and responses are explored. This ensure the adapted version is has retained its equivalence. However it should be noted that this stage provides useful insight into how the test taker interprets the items. It does not address construct validity and reliability. This process provides to a certain extent the quality of the content validity. Additional testing must be done to ensure the retention of the psychometric properties.

3.6 Submission of documentation to the developers
The final stage in the adaptation process is the submission of all reports and forms to the developers. The developers then determine if the recommended stages were followed and verified. What follows is the conclusion that a reasonable translation has been achieved.

The adaptation of a measure for use in a new setting is time consuming and costly. However it is believed that this is the best possible way to achieve an equivalent metric for whatever construct is being measured. The adaptation allows data collection efforts to be the same in cross-national studies that must exclude all test takers who were unable to complete the form in English. It therefore avoids the selection bias that may occur as a result.

13

4. Conclusion
There are varying methods of validating a measure; each can be applied to specific constructs. The factors included in validity testing comprise of tests for content criterion and construct identification. The end result will be the completion of a measure and test manual as well as the classification of a measure. This essay also aimed at outlining the important factors to consider when developing a test plan for cross cultural tests. As well as highlight the importance of test team with appropriate knowledge of culture and language groups. 14

5. References
1.
2.
3.
4.
5.
6.

Anastasi, A & Urbina, S (1997) Psychological testing (7 th Ed).
Shultz & Shultz, Duane (2010) Psychology and work today. New
York : Prentice Hall P99-102 ISBN 0-205-68358-4
Van Ede, DM (1996) How to adapt a measuring instrument for use with various culture groups: practical step by step introduction P153159
Cross-Cultural Adaptation of Self Report Measures,Beaton,et al
(2000) Volume 25 Number 24 P 3186 3191
Foxcroft,CD (2004) Planning a psychological test in the multicultural South African context (2004) SA Journal of Industrial
Psychology 30 P8-15
An Introduction to Psychological Assessment, (2013) 4th Ed.
Foxcroft, CD & Roodt, G