50 Years of SEM in 50 Minutes??
Karl G J¨reskog o Norwegian Business School & Uppsala University
May 11, 2015

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

1 / 42

Factor Analysis before 1964

Although its roots can be traced back to the work of Francis Galton, it is generally considered that factor analysis began with the celebrated article by Spearman (1904). In the first half of the 20th century factor analysis was mainly developed by psychologists for the purpose of identifying mental abilities by means of psychological testing.
Various theories of mental abilities and various procedures for analyzing the correlations among psychological tests emerged. The most prominent factor analysts in the first half of the 20th century seem to be Godfrey Thomson, Cyril Burt, Raymond Cattell, Karl
Holzinger, Louis Thurstone and Louis Guttman. A later generation of psychological factor analysts that played important roles are Ledyard
Tucker, Ray Cattell, Henry Kaiser, and Chester Harris.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

2 / 42

Through the 1950’s factor analysis was characterized by a set of ad hoc procedures for analyzing the correlation matrix R of the tests.
Four problems of factor analysis emerged:
Number of factors
Communalities
Factor extraction
Factor rotation

The focus was om computation. Computers were very rare and consisted of large mainframes that filled whole rooms.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

3 / 42

A few statisticians had began to be interested in factor analysis, notably Lawley, D.N. (1940) The estimation of factor loadings by the method of maximum likelihood. Proceedings of the Royal Society Edinburgh,
60, 64–82.
Anderson, T.W., and Rubin, H. (1956) Statistical inference in factor analysis. In Proceedings of the Third Berkeley Symposium, Volume
V. Berkeley: University of California Press.
J¨reskog, K.G. (1962) On the statistical treatment of residuals in o factor analysis. Psychometrika, 27, 335-345.
J¨reskog, K.G. (1963) Statistical Estimation in Factor Analysis: A o New Technique and its Foundation. Stockholm: Almqvist & Wiksell.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

4 / 42

Communalities
Guttman (1953) defined the factor analysis problem as follows. What numbers should be put in the diagonal of R such that this matrix is
Gramian and of smallest possible rank k. This is equivalent to finding a matrix Λ of order p × k, where k < p, such that
Rc ≈ ΛΛ ,

(1)

where Rc is R with communalities in the diagonal.
The problem of communalities was involved in much discussion of factor analysis in the 1950’s.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

5 / 42

Factor Extraction
Once the communalities have been determined, one could determine Λ in
(1). The most common method in the early literature is one which chooses the columns of Λ proportional to the eigenvectors of Rc corresponding to the k largest eigenvalues.
After Λ has been determined, the communalities can be re-estimated as the sum of squares of each row in Λ. Putting these new communalities in the diagonal of R gives a new matrix Rc from which a new Λ can be obtained. This process can be repeated. In this process it can happen that one or more of the communalities exceed 1, so called Heywood cases.
Such Heywood cases occurred quite often in practice and caused considerable problems.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

6 / 42

The Factor Analysis Model
The basic idea of factor analysis is the following. For a given set of observed response variables x1 , . . . , xp one wants to find a set of underlying latent factors ξ1 , . . . , ξk , much fewer than the observed variables. These factors are supposed to account for the correlations of the response variables. This leads to the linear factor analysis model of
Thurstone (1947): xi = µi + λi1 ξ1 + λi2 ξ2 + · · · + λik ξk + δi , i = 1, 2, . . . , p ,

(2)

where δi , the unique part of xi , is uncorrelated with ξ1 , ξ2 , . . . , ξk and with δj for j = i. In matrix notation (2) is x = µ + Λξ + δ ,

Σ = ΛΛ + Ψ .

(3)

The objective of factor analysis is to estimate the number of factors k and the factor loadings Λ = (λij ) from a random sample of observations x1 , x2 , . . . , xN .
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

7 / 42

Guttman’s (1953) Image Theory

Guttman (1953) considered a different system than (2), namely the regression of xi on all the other x’s: xi = µi +βi1 x1 +βi2 x2 +· · ·+βi,i−1 xi−1 +βi,i+1 xi+1 +· · ·+βp xp +zi , (4) that is xi = µi + β )i( x)i( + zi ,

(5)

x = µ + Bx + z ,

(6)

or in matrix form where B is a matrix of order p × p with βii = 0.
What does (4) has to do with (2)?
Guttman (1956) showed that the squared multiple correlation in the regression (4) is a lower bound for communality. Let’s consider some statements which are equivalent to this.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

8 / 42

For this purpose consider some notation
Σ=

σii σi Σii

Σ−1 =

σ ii σi (7)

Σii

σ ii = (σii − σ i Σ−1 σ i )−1 ii Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

(8)

May 11, 2015

9 / 42

i = 1, 2, . . . , p xi = c i + δ i , δ i ⊥ c i , δ i ⊥ δ j j = i xi = pi + zi , zi ⊥ pi

(10)

σii = Var (ci ) + Var (δi )

(11)

σii = Var (pi ) + Var (zi )

Ri2 =

(9)

(12)

Var (pi )
Var (ci )
≤
⇔ Var (pi ) ≤ Var (ci ) ⇔ Var (δi ) ≤ Var (zi ) σii σii
(13)

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

10 / 42

But
Var (zi ) = σii − σ i Σ−1 σ i = 1/σ ii ii (14)

ψi ≤ 1/σ ii ⇔ ψii σ ii ≤ 1

(15)

so

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

11 / 42

This leads to the model
Σ = ΛΛ + θ(diag Σ−1 )−1 ,

(16)

which is to be interpreted as an implicit equation defining Σ as a function of Λ and θ. In my dissertation I developed a simple non-iterative method for estimating Λ and θ.
1
Pre- and postmultiplying (16) by (diag Σ−1 ) 2 and defining
1

1

Σ = (diag Σ−1 ) 2 Σ(diag Σ−1 ) 2 , and 1

Λ = (diag Σ−1 ) 2 Λ gives Σ = Λ Λ + θI , which shows that p − k of the eigenvalues of Σ are equal to θ.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

12 / 42

Let S be a consistent estimate of Σ. Then
1

1

S = (diag S−1 ) 2 S(diag S−1 ) 2 , is a consistent estimate of Σ . Let γ1 , γ2 , . . . , γp be the eigenvalues of S
ˆ ˆ
ˆ
in descending order and let ω 1 , ω 2 , . . . , ω k be unit-length eigenvectors
ˆ ˆ
ˆ
corresponding to the k largest eigenvalues. Furthermore, let
ˆ
Γk = diag (ˆ1 , γ2 , . . . , γk ) , γ ˆ
ˆ
and
ˆ
Ωk = (ω 1 , ω 2 , . . . , ω k ) .
ˆ ˆ
ˆ
Then the simple solution is
ˆ
θ=

1
(ˆk+1 + γk+2 + · · · + γp ) , γ ˆ
ˆ
p−k

1
ˆ
ˆ ˆ
ˆ 1
Λ = (diag S−1 )− 2 Ωk (Γk − θI) 2 U ,

where U is an arbitrary orthogonal matrix of order k × k. This solution also offers a solution to the number of factors problem. Choose the
ˆ
smallest k such that θ < 1.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

13 / 42

This simple solution has several obvious advantages:
It is non-iterative and very fast to compute.
It does not require estimates of communalities.
Heywood-cases cannot occur, i.e., the estimates of uniquenesses
ˆ
which are the diagonal elements in θdiag S−1 are always positive.
It is scale-free in the sense that if x is replaced by Dx, where D is a
ˆ
ˆ diagonal matrix of scale factors, then Λ will be replaced by DΛ while
ˆ is unchanged. θ Note that the matrix S is independent of D, yet it is not a correlation
ˆ ˆ
ˆ 1 matrix. The part Ωk (Γk − θI) 2 U of the solution is also independent of D.
Later I also developed a maximum likelihood method for this model. see
J¨reskog, K.G. (1969) Efficient estimation in image factor analysis. o Psychometrika, 34, 51–75. But this went unnoticed, why?

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

14 / 42

Maximum Likelihood Factor Analysis

J¨reskog, K.G. (1967) Some contributions to maximum likelihood factor o analysis. Psychometrika, 32, 443–482.
*****
Let x1 , x2 , . . . , xN , be iid with xi ∼ N(µ, Σ) with Σ positive definite. If µ is unconstrained and
Σ = ΛΛ + Ψ .
(17)
then maximizing ln L is equivalent to minimizing
F (Λ, Ψ) = log Σ + tr (SΣ−1 ) − log S − p ,

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

(18)

15 / 42

J¨reskog (1967) approached the computational problem by focusing on the o concentrated fit function f (Ψ) = min F (Λ, Ψ) ,

(19)

Λ

which could be minimized numerically.
If one or more of the ψi gets close to zero, this procedure becomes unstable, a problem that can be circumvented by reparameterizing: θi = ln ψi , ψi = +e θi .

(20)

This leads to a very fast and efficient algorithm.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

16 / 42

Rotation
When k > 1, the factor loadings in Λ are not uniquely defined.
Geometrically the factor loadings may be viewed as p points in a k-dimensional space. In this space the points are fixed but their coordinates can be referred to different factor axes. If the factor axes are orthogonal we say we have an orthogonal solution; if they are oblique we say that we have an oblique solution where the cosine of the angles between the factor axes are interpreted as correlations between the factors.
In statistical terminology, an orthogonal solution corresponds to uncorrelated factors and an oblique solution corresponds to correlated factors. One can also have solutions in which some factors are uncorrelated and some are correlated.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

17 / 42

Rotation is illustrated in the following figures

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

18 / 42

To facilitate the interpretation of the factors one makes an orthogonal or oblique rotation of the factor axes. This rotation is usually guided by
Thurstone’s principle of simple structure which essentially states that only a small fraction of the loadings in each row and column should be large.
Geometrically, this means that the factor axes pass through or near as many points as possible.
*****
J¨reskog, K.G. (1966) Testing a simple structure hypothesis in factor o analysis. Psychometrika, 31, 165-178.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

19 / 42

In exploratory factor analysis it is usually assumed that the factors ξ1 , . . . , ξk are uncorrelated and have variances 1. These assumptions can be relaxed and the factors may be correlated and they need not have variance 1. If ξ has covariance matrix Φ, the covariance matrix of x is
Σ = ΛΦΛ + Ψ .

(21)

Let T be an arbitrary non-singular matrix of order k × k and let ξ ∗ = Tξ

Λ∗ = ΛT−1

Φ∗ = TΦT .

Then
Λ∗ ξ ∗ ≡ Λξ

Λ∗ Φ∗ Λ∗ ≡ ΛΦΛ

Since T has k 2 independent elements, this shows that at least k 2 independent conditions must be imposed on Λ and/or Φ to make these identified. Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

20 / 42

Factor analysis was typically done in two steps. In the first step, one obtains an arbitrary orthogonal solution in which Φ = I in (21). In the second step, this is rotated orthogonally or obliquely to achieve a simple structure. For the rotated factors to have unit variance, T must satisfy diag (TT ) = I ,

(22)

TT = I ,

(23)

for an oblique solution and for an orthogonal solution.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

21 / 42

Confirmatory Factor Analysis

J¨reskog, K.G. (1969) A general approach to confirmatory maximum o likelihood factor analysis. Psychometrika, 34, 183-202.
*****
In contrast to exploratory factor analysis, a confirmatory factor analysis begins by defining the latent variables one would like to measure. This is based on substantive theory and/or previous knowledge. One then constructs observable variables to measure these latent variables. Thus, in a confirmatory factor analysis, the number of factors is known and equal to the number of latent variables. The confirmatory factor analysis is a model that should be estimated and tested.
Exploratory and confirmatory factor analysis are illustrated in Figures 1 and 2.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

22 / 42

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

23 / 42

In a confirmatory factor analysis the investigator has such knowledge about the factorial nature of the variables that he/she is able to specify that each measure xi depends only on a few of the factors ξj . If xi does not depend on ξj , λij = 0 in (2). In many applications, the latent variable ξj represents a theoretical construct and the observed measures xi are designed to be indicators of this construct. In this case there is only one non-zero λij in each equation (2). In general, assuming that Φ is a correlation matrix, one needs to specify at least k − 1 zero elements in each column of Λ but in a confirmatory factor analysis there are usually many more zeros in each column.
The possibility of a priori specified zero elements in Λ was mentioned in
Anderson & Rubin (1956) and in J¨reskog & Lawley (1968), but the term o confirmatory factor analysis was first used in J¨reskog (1969). o Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

24 / 42

To estimate a confirmatory factor analysis model one can minimize any of the fit function (18) with respect to all free elements of Λ, Φ, and Ψ. In most cases no analytic solution is available so the minimization must be done numerically. By contrast to exploratory factor analysis, no eigenvalues and eigenvectors are involved and the solution is obtained in one step. No factor rotation is needed.
In a way, confirmatory factor analysis shifts the focus from the problems of factor extraction and rotation to the problem of testing a specified model.
With the ML method, the most common way of testing the model is to use N times the minimum value of the fit function FML as a χ2 with
1
degrees of freedom equal to 2 p(p + 1) minus the number of independent parameters in Λ, Φ, and Ψ.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

25 / 42

Covariance Structures
J¨reskog, K.G. (1974) Analyzing psychological data by structural analysis o of covariance matrices. In R.C. Atkinson et al. (Eds.): Contemporary
Developments in Mathematical Psychology - Volume II. San Francisco:
W.H. Freeman, 1–56.
*****
Equation (21) can be extended in various ways, for example,
Σ = Λy (ΓΦΓ + Ψ)Λy + Θ .

(24)

This can accommodate second-order factor analysis, where Λy is the first-order factor loadings and Γ are the second-order factor loadings, see next slide.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

26 / 42

- y1 k Q
Q
Q
?
Q
Q ηm
 1

o
S

S

+
- y3 
S
S
S
- y4
S
k
Q
Q
Q
S
?
Q
S ξm
- y5 
Q ηm

 2





+

- y6



- y7

k
Q
Q
Q

/

?
Q
- y8 
Q ηm
 3



+
- y9 
- y2 

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

27 / 42

Equation (21) can also accommodate various test theory models shown here Model
Parallel
Tau-equivalent
Variable-length
Congeneric

Karl G J¨reskog ( ) o Covariance Structure
Σ= λ2 jj + θI
Σ= λ2 jj + Θ
Σ= Dλ (λλ + ψI)Dλ
Σ= λλ + Θ

50 Years of SEM in 50 Minutes??

No. of Parameters
2
p+1 p+1 2p

May 11, 2015

28 / 42

Multi-Group Factor Analysis
J¨reskog, K.G. (1971) Simultaneous factor analysis in several populations. o Psychometrika, 57, 409–426.
*****
Consider data from several groups or populations of individuals. These may be different nations, states, or regions, culturally or socioeconomically different groups, groups of individuals selected on the basis of some known selection variables, groups receiving different treatments, and control groups, etc. In fact, they may be any set of mutually exclusive groups of individuals that are clearly defined. It is assumed that a number of variables have been measured on a number of individuals from each population. This approach is particularly useful in comparing a number of treatment and control groups regardless of whether individuals have been assigned to the groups randomly or not.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

29 / 42

Consider the situation where the same tests have been administered in G different groups and the factor analysis model is applied in each group: xg = Λg ξ g + δ g , g = 1, 2, . . . , G ,

(25)

where, as before, ξ g and δ g are uncorrelated. The covariance matrix of xg in group g is
Σg = Λg Φg Λg + Ψ2 .
(26)
g

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

30 / 42

The hypothesis of factorial invariance is:
Λ1 = Λ2 = · · · = ΛG .

(27)

This states that the factor loadings are the same in all groups. Group differences in variances and covariances of the observed variables are due only to differences in variances and covariances of the factors and different error variances. The idea of factorial invariance is that the factor loadings are attributes of the tests and they should therefore be independent of the population sampled, whereas the distribution of the factors themselves could differ across populations. A stronger assumption is to assume that the error variances are also equal across groups:
Ψ1 = Ψ2 = · · · = ΨG .

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

(28)

May 11, 2015

31 / 42

Structural Equation Models(SEM)

J¨reskog, K.G. (1973) A general method for estimating a linear structural o equation system. In A.S. Goldberger and O.D. Duncan (Eds.): Structural
Equation Models in the Social Sciences. New York: Seminar Press,
85–112.
*****
Factor analysis is used to investigate latent variables that are presumed to underlie a set of manifest variables. Understanding the structure and meaning of the latent variables in the context of their manifest variables is the main goal of traditional factor analysis. After a set of factors has been identified, it is natural to go on and use the factors themselves as predictors or outcome variables in further analyses. Broadly speaking, this is the goal of structural equation modeling.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

32 / 42

A further extension of the classical factor analysis model is to allow the factors not only to be correlated, as in confirmatory factor analysis, but also to allow some latent variables to depend on other latent variables.
Models of this kind are called structural equation models and there are many examples of this in the literature.

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

33 / 42

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

34 / 42

η = α + Bη + Γξ + ζ ,

(29)

y = τ y + Λy η + ,

(30)

x = τ x + Λx ξ + δ ,

(31)

and

µ=

Σ=

τ y + Λy (I − B)−1 (α + Γκ) τ x + Λx κ

Λy B (ΓΦΓ + Ψ)B Λy + Θ
Λx ΦΓ B Λy + Θδ

,

(32)

Λy B ΓΦΛx + Θδ
Λx ΦΛx + Θδ

,

(33)

B = (I − B)−1
.

fixed parameters that have been assigned specified values, constrained parameters that are unknown but linear or non-linear functions of one or more other parameters, and free parameters that are unknown and not constrained.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

35 / 42

The LISREL model combines features of both econometrics and psychometrics into a single model. The first LISREL model was a linear structural equation model for latent variables, each with a single observed, possibly fallible, indicator, see J¨reskog (1973). This model was o generalized to models with multiple indicators of latent variables, to simultaneous structural equation models in several groups, and to more general covariance structures. J¨reskog & S¨rbom developed the LISREL o o program. Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

36 / 42

Some History of LISREL

Goldberger

1970 J¨reskog o The idea of combining features of both econometrics and psychometrics into a single mathematical model was born in February 1970.
The first version of LISREL was a linear structural equation model for latent variables, each with a single observed, possibly fallible, indicator . This model was presented at the conference on Structural Equation Models in the Social Sciences held in Madison, Wisconsin, in November 1970. The proceedings of this conference, edited by Professors Goldberger and Duncan, were published in 1973.
This LISREL model was generalized in 1971-72 to include models previously developed for multiple indicators of latent variables
The basic form of the LISREL model has remained the same ever since and is still the same model as used today.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

37 / 42

The first two computer versions of LISREL were written in 1970–71. The program was completely rewritten in 1974-75 by Dag S¨rbom. This version, o called LISREL III, was the first made generally available with a written manual. It had fixed column input, fixed dimensions, only the maximum likelihood method, and users had to provide starting values for all parameters. The versions that followed demonstrated an enormous development in both statistical methodology and programming technology:
LISREL IV (1978) had Keywords, Free Form Input, and Dynamic Storage
Allocation
LISREL V (1981) had Automatic Starting Values, Unweighted and
Generalized Least Squares, and Total Effects
LISREL VI (1984) had Parameter Plots, Modification Indices, and
Automatic Model Modification
LISREL 7 (1988) had PRELIS, Weighted Least Squares, and Completely
Standardized Solution
LISREL 8 (1994) had SIMPLIS, Path Diagrams, and Non-linear Constraints
LISREL 9 (2013) with FIML for Missing Values and Adaptive Quadrature for
Ordinal Variables
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

38 / 42

Two More Recent Important Developments

Robust Estimation (Browne, 1984, Satorra & Bentler, 1988)
Ordinal Variables
Underlying variables approach (Muthen ,1984, J¨reskog, 1990,1994) o Latent trait models (J¨reskog & Moustaki, 2001) estimated with o adaptive quadrature (Schilling & Bock, 2005)

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

39 / 42

The Growth of Structural Equation Modeling
SEM became became very popular in multivariate analysis much because of the
LISREL program. As witnessed by the literature, there has been an enormous development of both the statistical theory and computer technology, Hershberger
350
(2003).
300

250

200

150

100

50

0
1994 1995 1996 1997 1998 1999 2000 2001

Number of Journals and Articles by Year
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

40 / 42

Main Virtues of SEM Methodology

SEM has the power to test complex hypotheses involving causal relationships among construct or latent variables
SEM unifies several multivariate methods into one analytic framework
SEM specifically expresses the effects of latent variables on each other and the effect of latent variables on observed variables
SEM can be used to test alternative hypotheses.
SEM gives social and behavioral researchers powerful tools for stating theories more exactly, testing theories more precisely, generating a more thorough understanding of observed data.
Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

41 / 42

Karl G J¨reskog ( ) o 50 Years of SEM in 50 Minutes??

May 11, 2015

42 / 42