UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Topic 6 – Post-hoc contrast analysis, controlling Familywise error rate
What inferential procedure should follow a significant ANOVA test? When ANOVA F test is significant, and have rejected homogeneity hypothesis, what should happen next is to make inferences regarding hypotheses implied by homogeneity hypothesis make confident inferences for contrasts (and comparisons).
The ANOVA F Simultaneous Test Procedure (Scheffé method) An overall test (a test of the homogeneity of population means) can be used as the basis for a simultaneous test procedure (STP), allowing for tests on all hypotheses implied by the homogeneity hypothesis (including contrasts)
1
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
The critical value for the F STP is the ANOVA Fc: Fc = F; 1, 2 , where
1 = J – 1, 2 = N – J = J(n – 1)
The decision rule for the ANOVA F test can be written as: Reject H0 : 1 2 J if
F MSB F ; 1, 2 MSE
ie Rearranging we get, ie
SSB F ;1 , 2 1 MSE
SSB > 1 F; 1, 2 MSE
(1)
2
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
We can use the RHS of (1) as the basis for a decision rule, allowing for a test of any contrast null hypothesis, H0: = 0 (implied by the homogeneity hypothesis).
ˆ We substitute SS ( ) for SSB in (1), giving the decision rule:
Reject H0: = 0, if
ˆ SS 1 F ;1 , 2 MSE
(2)
The RHS of (2) is the critical sum of squares (SSc) for tests of all contrast null hypotheses in the analysis, For the ANOVA F STP (Scheffé method)
SSc = 1 × F; 1, 2 × MSE
3
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Alternatively, we can express the F STP (Scheffé) decision rule as reject H0: = 0, if
Fˆ Fˆ critical (contrast Fc) where contrast F: and contrast Fc
ˆ SS Fˆ MSE
Fˆ critical 1 F ;1, 2
4
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Example J = 3, n = 11, N = 33
(SPSS data file: RM3 Lecture Topic 6.sav)
(Treatment 1, Treatment 2, Control; dependent measure = performance) (i) Test of homogeneity hypothesis, H0: 1 = 2 = 3
ANOVA y Sum of Squares 682.000 472.000 1154.000 df 2 30 32 Mean Square 341.000 15.733 F 21.674 Sig. .000
We can reject H0: 1 = 2 = 3 , since F = 21.674 > 3.316 What inference follows? [heterogeneity inference ...... ]
5
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
(ii) Follow-up inferences on contrasts – post-hoc analysis The ANOVA F STP allows for tests of any H0 implied by the homogeneity hypothesis. This means we are at liberty to choose any number of contrasts for our follow-up analysis, and contrasts can be based on the pattern of sample means. We call this a post-hoc analysis strategy. For follow-up contrasts, Scheffé SSc: SSc = 1 F; 1, 2 MSE = 2 3.316 15.733 = 104.34 Decision rule for contrast H0’s: Reject H0: g = 0, if
ˆ SS g SS c 104.34
6
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Sample Means T1 Mj M1 = 39 T2 M2 = 46 C M3 = 35 (M = 40)
Suppose the following contrasts are to be included in the analysis: contrasts coefficients: 1 = ½1 + ½2 - 3 (average of Treatment ES) 2 = 1 – 2 3 = 1 – 3 4 = 2 – 3 (T1 ES vs T2 ES) (T1 ES) (T2 ES)
1 2 1 1 0
1 2
1 0 1
1 0 1 1
Each contrast null hypothesis states that the population value (parameter) of contrast is 0, i.e., H0: = 0.
7
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
SS Inference 682.00 Population means are heterogeneous 412.50 269.50 88.00 665.50 ½ (1 + 2) > 3 1 < 2 None 2 > 3
SSc = 104.34; significant test outcomes for 1, 2 and 4. Directional inference (referring to population of interest): Average performance is higher for treatments compared to no treatment (1), higher for T2 compared C (4) and higher for T2 compared to T1 (2). No inference can be made regarding T1 Es (3). Note: Each contrast SS is less than SSB, and so accounts for some of the variation between groups.
8
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
INSERT
ˆ Calculating SS (1)
ˆ 2 Sample value of 1 : 1 M 1 M 2 M 3 1 39 46 35 7.5 2
c
2 j
.52 .52 1 1.5
n = 11
n( )2 11(7.5) 2 ˆ SS (1) ˆ 2 412.5 1.5 c
9
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
The Scheffé Simultaneous Confidence Interval (SCI) procedure In general, the limits of 100(1-)% CI for the gth contrast in an analysis can be written as:
g
ˆ ˆˆ g CC g
where CC is a critical constant that depends on the error rate chosen, and whether the analysis is unrestricted (post hoc) or restricted to planned contrasts. The CC is a function of a critical value of a theoretical probability distribution (often a t or F distribution). For Scheffé SCI method: CC = 1 F; 1, 2 (square root of contrast Fc ). Thus Scheffé SCI limits are calculated as follows
ˆ g g ˆˆ 1 F; 1, 2 g .
10
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Example (continued) 95% Scheffé SCIs
ˆ g g ˆˆ 1 F; 1, 2 g .
What are 95% Scheffé SCI limits for 1?
ˆ 2 Sample value of 1 (mean difference scaling): 1 M 1 M 2 M 3 1 39 46 35 7.5 2
ˆ ˆ Standard error of 1 : ˆ g MSE c 2 j j n
c
2 j
.52 .52 1 1.5 ;
n = 11
ˆ ˆ g 15.733 1.5 1.465 11
11
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Putting this all together: 1 7.5 2.575 1.465 1 (3.728, 11.272) Inference: Receiving a treatment increases performance, on average, by at least 3.7 units and at most 11.3 units, compared to control condition. (similarly, can obtain 95% SCIs for 2, 3 and 4, and any other contrast of interest)
12
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
NOTE: Between-group contrasts are labelled ‘B’ p-values for contrast Fs are not provided. can use PSY probability calculator or F tables to obtain contrast Fc.
where F (ˆ ) c = 1 F; 1, 2 = 2× F.05, 2, 30 = 2 ×3.316 = 6.632 [Alternatively,
ˆ reject H0 if SS ( ) SSc where SSc = 1 F; 1, 2 × MSE = 6.632 ×15.733 =104.34 ]
14
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Test outcomes: Significant contrast Fs for B1, B2 and B4. T1 Mj M1 = 39 T2 M2 = 46 C Contrasts:
1 -2 B2: 1 -1 0 B3: 1 0 -1 B4: 0 1 -1 Tmts-C T1-T2 T1-C T2-C
M3 = 35 B1: 1
Directional inference: B1: Average performance better with treatments than without (positive Treatment effect) B2: Average performance higher with T2 than T1. B4: Average performance better with T2 than no treatment (T2 effect). but B3: no inference can be made regarding T1 effect.
15
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
For 95% post-hoc SCIs: select ‘post hoc’ [SCIs will be calculated with CC = 2.575] Note: Default values for Confidence Level (%) = 95 Scaling Options = Mean Difference Contrasts (PSY automatically rescales integer contrast coefficients to mean difference coefficients when calculating CI limits.)
16
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Post hoc 95% Simultaneous Confidence Intervals ----------------------------------------------The CIs refer to mean difference contrasts, with coefficients rescaled if necessary. Rescaled Between contrast coefficients Contrast Group... 1 2 3 Tmts-C B1 0.500 0.500 -1.000 T1-T2 B2 1.000 -1.000 0.000 T1-C B3 1.000 0.000 -1.000 T2-C B4 0.000 1.000 -1.000 Raw CIs (scaled in Dependent Variable units) -----------------------------------------------Contrast Value SE ..CI limits.. Lower Upper -----------------------------------------------Tmts-C T1-T2 T1-C B1 B2 B3 7.500 -7.000 4.000 1.465 1.691 1.691 3.728 -11.356 -0.356 11.272 -2.644 8.356
SCI output is commensurate with test outcomes (B1, B2 and B4 all signif. as CIs do not cover 0; B3 n.s as CI does cover 0.)
Inference: (from approx standardised CIs) 1: Average treatment effect is positive and at least large (and possibly very large) 2: Difference between treatment ESs (favouring T2 over T1 ) is at least medium (and possibly very large) 3: T1 ES cannot be established by CI. T1 ES may be negative (but at most trivial), or may be zero, or may be positive (and if so, possibly very large). 4: T2 ES is positive and at least very large.
18
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
How satisfactory is this analysis? Reasonably satisfactory because we can infer a large T2 ES. What about T1 ES? Test outcomes do not allow for a satisfactory inference for T1 effect: On the one hand, B1 is significant. We are not sure if both T1 and T2 contribute to positive treatment effect, but this outcome allows for the possibility of T1 effect). But B3 is not significant, so although (strictly speaking) no inference can be made for T1 vs C, this outcome allows for the possibility of no T1 effect. However, CI output provides more information regarding T1 ES. 95% post-hoc SCI limits for B3 indicate that o either trivial T1 ES (in which case T1 effect same as ‘no treatment’) o or positive T1 ES o but rules out the possibility of a non-trivial negative T1 ES.
19
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Redundancy and Contradiction Main purpose of data analysis is to give a reasonable and thorough account of the data. Post-hoc analysis – allows for any number of contrasts to be included and because of this, the analysis outcome can lead to redundancy and contradiction. What is meant by redundancy? A contrast inference is redundant if it is implied by the inference for two (or more) other contrasts in the analysis. From above example, for 1 infer that ½(1 + 2) > 3 for 2 infer that 1 < 2 o together these inferences imply that 2 > 3 (which is inference for 4). o 4 inference is redundant, given 1 and 2.
20
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
What is meant by contradiction? A contrast inference may be contradicted by the inference for two (or more) other contrasts. Consider a different set of sample means for the above example: J = 3, n = 11, 1 M1 = 39, M2 = 43, M3 = 35 and SSc = 104.34 SS 264 sig. 88 n.s 88 n.s 352 sig. inference ½ (1 + 2) > 3 1 = 2?? 1 = 3?? 2 > 3 strong inference of ‘no difference’ for 2 and 3 contradicts directional inference for 1 , and directional inference for 4. contrast coeffs
½ ½ -1 0 Ts-C T1-T2 T1-C T2-C
2: 1 -1 3: 1 4: 0 ie
0 -1 1 -1
if 1 = 2 and 1 = 3, then it cannot be that 2 > 3; or the other way around,
if 2 > 3, then it cannot be case that both 1 = 2 and 1 = 3.
21
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
How to resolve contradiction? Would have to consider either B1 and B4 test outcomes as Type I errors OR consider test outcomes for B2 and/or B3 as Type II errors. Either way, not desirable to report contradictions in findings. Post-hoc analysis affords flexibility in choice of contrasts to be included in the analysis, and because the ANOVA-F STP controls for the FWER for all conceivable contrasts, it allows the E to choose what contrasts to report when writing up the findings.
22
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Contrast complexity and precision In general, greater precision for complex contrast CI than comparison CI, because SE for complex contrast (eg B1: ½ ½ -1) is smaller than SE for comparison (eg B2: 1 -1 0). Data from RM3 Lecture Topic 6.in
..CI limits.. Lower Upper ---------------------------------------------Tmts-C T1-T2 B1 B2 7.500 -7.000 1.465 1.691 3.728 -11.356 11.272 -2.644 Contrast Value SE
ie SEB1 = 1.465 < SEB2 = 1.691 For any contrast, MSE and n are constants, but c2 varies as a function of contrast coefficients.
Consider:
ˆ SEˆ ˆ g
MSE c 2 j j n
Thus, SE varies as a function of type of {m, r} mean difference contrast.
23
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
What is the relationship between contrast complexity, sample size and c2 ? Consider example for J = 4, with n Ps per group: Type of {m, r} contrast {1, 1} {2, 1} {3, 1} {2, 2} example [ c 1 c2 c3 c4 ] [ 1 -1 0 0 ] [ 1 -½ -½ 0 ] [ -⅓ 1 -⅓ -⅓] [ ½ ½ -½ -½] c2 2 1.5 1.33 1 No. of Ps being compared n vs n 2n vs n 3n vs n 2n vs 2n
Complex contrasts compare data across three or more groups and hence include more Ps overall than a comparison.
Complex contrasts compared to a comparison: c2 smaller, SE smaller and hence greater precision (narrower CI). [NOTE: similar argument can be made regarding statistical power for tests of complex contrasts vs tests of comparisons more power to detect TE with complex contrast than with comparison.]
24
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Maximal contrast – what is it? Can define a contrast that accounts for all of the variation between groups, such a contrast
ˆ is called the maximal contrast ( max ).
ˆ In other words, the maximal contrast is a contrast that has SS( max) = SSB
ˆ Coefficients of maximal contrast ( max ) are proportional to group deviation ˆ means ( M j M ) , thus max is necessarily defined post hoc.
Maximal contrast is usually uninterpretable (and so of little interest to experimenter), but can suggest an interpretable contrast of interest.
25
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Example - For previous data set: M1 = 39, M2 = 46, M3 = 35 Coefficients of maximal contrast: M1 – M = M2 – M = M3 – M = -1 6 -5 and
A test of H0 for maximal contrast is ‘equivalent’ to ANOVA F test, with the following
ˆ SS max SSc 1 F ; 1, 2 MSE
(SSc = 104.34)
Inference: 62 > (1 + 53), which is uninterpretable; so no interpretable directional inference can be made. Only inference that can be made is heterogeneity of means.
26
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
HOWEVER We can look to the coefficients of the maximal contrast to suggest interpretable {m,r} contrasts. Logic of this approach: o if maximal contrast is significant, then o {m, r} contrasts with largest SS will be those with coefficients similar (close in magnitude and sign) to coefficients of maximal contrast.
ˆ For our example, coefficient vector for
max
is [-1 6 -5]
This suggests the following interpretable contrasts, likely to be declared significant:
ˆ o [-.5 1 -.5] which is based on sign of coefficients of
max
ie [ – + – ] max ˆ o [ 0 1 -1] which is based on sign and magnitude of coefficients of .
27
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
For B1, o SS(Psymax) = SS Between (from ANOVA F test); o but F statistic for Psymax is not same as ANOVA F – why? Also, CI output for PSYmax is not of interest - why?
28
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
The ANOVA F simultaneous test procedure and the Scheffé (SCI) procedure are parts of the same approach to simultaneous statistical inference, and are appropriate for a post-hoc analysis that controls the FWER. Properties of the F STP o It controls the maximum FWER for any contrast analysis (post hoc or otherwise). o The analysis is coherent - an implied hypothesis cannot be rejected unless the implying hypothesis can also be rejected. I.e., if H0 : 1 2 J is not rejected, then every H0: = 0 cannot be rejected. o The analysis is consonant – if the overall H0 can be rejected, then at least one follow-up H0 can be as well (maximal contrast). o The order of tests has no bearing on the outcome. o STP inferences on contrasts are compatible with simultaneous confidence intervals (SCIs) produced by the F-based Scheffé SCI procedure.
29
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
Properties of the Scheffé SCI procedure o It controls the noncoverage FWER for CIs on all conceivable contrasts (an infinite set). eg, if noncoverage FWER of = .05 then for 95% SCIs, for a set of contrasts defined post-hoc, noncoverage errors for one or more CIs (in the set) will occur on at most 5% of replications. o It is completely compatible with the ANOVA F test; if the F test cannot reject the homogeneity hypothesis, then all Scheffé CIs will include zero.
30
UNSW PSYC3001 Research Methods 3 ‐ Topic 6 Lecture Slides
Dr Melanie Gleitzman
To Conclude: Main advantage of post-hoc analysis strategy (which we will label as ‘Scheffé procedure’) is o its flexibility o no restriction on number and nature of contrasts; o can define contrasts based on pattern of group means and/or coefficients of maximal contrast When contrasts are defined based on how the data turn out, then we are ‘capitalising on chance’, and it is important to control the familywise error rate appropriately. o This is what the Scheffé procedure does – it allows for ‘capitalising on chance’, and controls maximum FWER at .
