Alfred DeMaris - Regression with Social Data, Modeling Continuous and Limited Response Variables

Подождите немного. Документ загружается.

in X

) on all of the regressors in X

. Suppose that we denote the alias matrix as G.

Then it has the form

G 

冋册



冋册

where each g

is the partial regression coeﬃcient for the impact of the kth regressor

in X

on the ith regressor in X

. Hence, g

is the partial eﬀect of regressor 1 in X

on regressor 1 in X

, g

is the partial eﬀect of regressor 2 in X

on regressor 1 in

,...,g

is the partial eﬀect of regressor K in X

(in this case, the cross-product

term) on regressor 1 in X

, and so on. The bias in b

as an estimator of

ββ

is then

(X

)

1

X

ββ



冋册冋册

where β

in the rightmost vector is the partial eﬀect of the qth variable in X

on Y.

The bias in the kth coeﬃcient in X

is therefore



ββ



冱

q1

(6.5)

which is the sum of the products of the elements in the kth row of G with the ele-

ments in the vector

ββ

. That is, there is bias in the kth coeﬃcient whenever the kth

regressor in X

is associated with the qth regressor in X

, net of the other regressors

in X

(i.e., g

is nonzero), and the qth regressor in X

is a predictor of Y, net of the

other regressors in X

(i.e., β

is nonzero).

Bias in the Cross-Product Term

The bias in the Kth term in X

, the cross-product term, takes the same form as

equation (6.5): namely,



ββ



冱

q1

This suggests that the cross-product term is biased whenever X

and X

interact in their

eﬀects on some excluded regressor that also aﬀects Y. (The same applies to quadratic

eﬀects: They are biased whenever X

has a quadratic eﬀect on some excluded regres-

sor that also aﬀects Y.) Although researchers are usually very careful to include con-

trols to preclude omitted-variable bias in the main eﬀects of focus variables, they



βQy



...



...









OMITTED-VARIABLE BIAS IN A MULTIVARIABLE FRAMEWORK 215

c06.qxd 8/27/2004 2:53 PM Page 215

typically treat interaction terms as though they are impervious to the same phenome-

non. The foregoing shows that they are not. This suggests that analysts should also be

aware of potential confounds or suppressors when they are investigating interaction (or

quadratic) eﬀects. Equally important, they may want to consider covariates that could

mediate these eﬀects. That is, certain covariates may be the mechanisms that are

responsible for bringing about the interaction (or nonlinearity).

Example: Bias in Models for Faculty Salary

Table 6.5 presents the results of several models that examine the main eﬀect of gen-

der and the interaction of gender with college in their eﬀects on faculty salary for our

725 faculty members in the faculty salary dataset. Model 1 contains the main eﬀects

of gender and college, both factors represented by dummy variables. Here we see

that the “ﬁrelands college” and “other departments” are signiﬁcantly lower, and the

“business college” is signiﬁcantly higher, in average salary than “arts and sciences,”

controlling for gender. We see also that holding college constant, female faculty

members’ salaries are, on average, $11,799 lower than males’. Model 2 adds the

block of four cross-product terms representing the interaction of gender with col-

lege. The test for the signiﬁcance of the interaction is

(4,715)





(



)



 5.671.

216 ADVANCED ISSUES IN MULTIPLE REGRESSION

Table 6.5 Omitted-Variable Bias in the Interaction of Gender and College in Their

Eﬀects on Salary for 725 Faculty Members

Predictor Model 1 Model 2 Model 3 Model 4

Intercept 51793.000*** 52420.000*** 2313.589 1803.918

Female 11799.000*** 14641.000*** 690.883 927.114

Firelands 8552.429*** 9934.029*** 60.204 561.723

Business 5179.857*** 6156.490*** 5309.482*** 6095.228***

Education 1057.819 2603.822 1173.765 1295.265

Other departments 3840.726** 7588.529*** 960.892 1724.595*

Female  Firelands 6142.285 2681.444

Female  Business 3974.483 3230.073

Female  Education 4529.827 300.709

Female  other

departments 10704.000*** 1908.234

Prior experience 348.234*** 335.251***

Years in rank 720.781*** 723.296***

Years at BG 146.391* 158.452*

Marketability 19638.000*** 19294.000***

Rank 9530.083*** 9546.362***

.213 .237 .796 .798

* p  .05. ** p  .01. *** p  .001.

c06.qxd 8/27/2004 2:53 PM Page 216

With a p-value of .00017, this is quite signiﬁcant. Tests of the individual terms indi-

cate that the eﬀect is driven largely by the interaction of gender with “other depart-

ments.” Ignoring the other interaction terms, the gender gap in salary can be written

14641  10704 other departments. This suggests that the gender gap is 14641 in

“arts and sciences” compared to 3937 in “other departments.”

Missing from these models, however, are several key factors in salary determination

that are also related to both gender and college, as well as to their interaction. Model 3

adds the covariates prior experience, years in rank, years at BG, marketability, and

rank to model 1. Rank is an ordinal variable (being treated as continuous) with four val-

ues: 1 (“instructor/lecturer”), 2 (“assistant professor”), 3 (“associate professor”), and 4

(“professor”). Among other associations in the data, it is well known that gender is

strongly related to rank, with females generally occupying lower ranks than males (see,

e.g., Balzer et al., 1996; Boudreau et al., 1997). For this reason, we would expect the

gender eﬀect to be at least somewhat diminished when rank is held constant. In fact, it

is diminished substantially when these additional covariates are added in model 3 and

is no longer signiﬁcant. The eﬀect of college is also reduced when adding the covari-

ates, as was seen in Chapter 4. Now only “business” is signiﬁcantly diﬀerent from “arts

and sciences” in average salary. More important, when the interaction terms are reen-

tered, in model 4, the block is no longer signiﬁcant (F  2.112, p  .077). The interac-

tion is apparently accounted for by the covariates. But which ones?

Subsequent analyses (not shown) reveal that both years at BG and years in rank

exhibit signiﬁcant gender * college interaction eﬀects. The pattern of relationships is

exhibited in Figure 6.3. The gender * college cross-product term has a positive eﬀect

on salary net of the covariates and has positive eﬀects on both years at BG and years

in rank. Years at BG has a negative eﬀect on salary, while years in rank has a positive

eﬀect. Of the two covariates, years in rank has a much stronger eﬀect on salary, 723.296

versus 158.452 in model 4 (as these covariates are in the same metric, the magnitudes

of the unstandardized coeﬃcients can be directly compared). What this means is that

OMITTED-VARIABLE BIAS IN A MULTIVARIABLE FRAMEWORK 217

Female

{Other Regressors}

Female x Other Departments

Years at BG

Years in Rank

Salary

−

Figure 6.3 Model illustrating omitted-variable bias in the interaction of gender with membership in

other departments in their eﬀects on faculty salary.

c06.qxd 8/27/2004 2:53 PM Page 217

when these covariates are ignored, the positive indirect path from the cross-product to

salary through years in rank overrides the negative indirect path through years at BG to

combine with the positive direct path from the cross-product to salary. The result is a

much larger positive “direct” path that manifests itself as a signiﬁcant interaction. In the

current case, since gender is causally antecedent to other variables, we would say that

the covariates mediate, or account for, the interaction eﬀect. Speciﬁcally, the gender gap

in years in rank is virtually nil in “other departments” vis-à-vis “arts and sciences,” and

more years in rank translate into higher salaries. Therefore, the smaller gender gap in

salary in “other departments” versus “arts and sciences” (the interaction eﬀect) is due

primarily to the smaller gender gap in years in rank in “other departments,” in combi-

nation with years in rank’s positive eﬀect on salary.

REGRESSION DIAGNOSTICS I: INFLUENTIAL OBSERVATIONS

In the ﬁnal sections of the chapter I take up the issue of regression diagnostics and

begin by outlining the discovery and treatment of inﬂuential observations. Inﬂuential

observations are cases that exert an “undue” amount of inﬂuence on the estimated

regression model. That is, these are one or more cases that are essentially “driving”

the results, in the sense that the estimated model might be substantially diﬀerent

were these cases deleted. Interest in the discovery of such cases was stimulated

largely by the work of Belsley et al. (1980). Today most regression software oﬀers

several tools that allow the analyst to comb the data for such observations.

Why is it important to know which cases are especially inﬂuential? There are a

couple of reasons. First, it is possible that an inﬂuential data point is a coding error, a

bogus response, or some other such anomaly that was not caught in the data-cleaning

process. We certainly don’t want this type of case to remain in the analysis as is. As

an example, in DeMaris (1997) I discovered a highly inﬂuential case in my investi-

gation of heightened sexual activity in violent intimate relationships, using the NSFH

data from wave 1. A key hypothesis in that work was that greater sexual activity in

violent romantic partnerships was due partially to the climate of fear created by male

violence. I argued that females in these liaisons were likely to accede to sexual activ-

ity more often than they preferred, out of fear of the consequences of displeasing the

partner. Given that this sexual activity would in a sense be coerced, due to women’s

fear of violence by a displeased partner, I expected an increase in depression among

such women. To marshal evidence for this, I tested whether sex in the presence of

male violence was associated with an increase in the female’s depressive symptoma-

tology. Although this ﬁnding was supported, I also found that the most inﬂuential case

in the data appeared to have a bogus value of 93 for the number of times the couple

had “had sex” in the past month. However, deleting this observation only strength-

ened the original ﬁnding. In this case the ﬁndings appeared robust to the inﬂuence of

the “rogue” observation. Such may not always be the case, however.

Another possibility is that the existence of a few very inﬂuential observations reveals

a ﬂaw in the model speciﬁcation that the researcher may be able to correct. A handy

example of this can be found by examining inﬂuence diagnostics for model 3 in

218 ADVANCED ISSUES IN MULTIPLE REGRESSION

c06.qxd 8/27/2004 2:53 PM Page 218

Table 6.5. One will ﬁnd that the three most inﬂuential faculty members in the dataset

have one element in common: They are all “eminent scholars.” These are professors

with particularly luminous reputations in their ﬁelds who had been recently hired

under a statewide program to enhance the quality of scholarship at Ohio universities.

Their salaries were, as a consequence, considerably higher than those of other faculty

with comparable rank and experience at BGSU. In this case, however, it is easy to

model this phenomenon by including in the model a dummy variable that identiﬁes

these types of scholars (see, e.g., Balzer et al., 1996). Once the dummy is included,

these cases no longer exert such inﬂuence, because they are better ﬁtted by the model.

If inﬂuential observations represent neither “bad” data nor ﬂaws in model

speciﬁcation, nothing further can be done about them. There is no rationale for delet-

ing legitimate data from the analysis, regardless of their inﬂuence. It may nevertheless

be fruitful to know that the results are largely due to perhaps only a few “interesting”

cases in the data. The analyst may then wish to be more cautious in attempting to gen-

eralize beyond the current sample until the ﬁndings can be replicated in other datasets.

At any rate, let’s consider some tools to use in the exploration of such cases.

Building Blocks of Inﬂuence: Outliers and Leverage

The degree to which a case has the ability to inﬂuence the regression analysis

depends on two characteristics: the extent to which it is an outlier, and the extent to

which it has leverage. An outlier is a case that is far from the regression trend exhib-

ited by the other data points. That is, if the regression of Y on the X’s is represented

by a “swarm” of points in p-dimensional space, an outlier is a point that is at a

noticeable distance from this swarm, in the Y direction. An outlier is typically

identiﬁed by having a comparatively large residual, indicating that its actual Y value

is nowhere near where it is “supposed to be” [i.e., the regression line (plane) that

runs through the swarm]. An example of an outlier can be seen in Figure 2.1, dis-

cussed earlier, showing the regression of the ﬁrst exam score on the math diagnostic

for 213 students. It is the lowest point in this swarm of points and occurs in the mid-

dle of the plot, the point with an X score of 37 and a Y score of 17. It is also the point

having the largest residual in the data (in absolute value), or the largest standardized

residual (see also Figures 2.6 and 2.8).

However, being an outlier by itself does not give an observation the power to

aﬀect the regression. The location of this particular outlier in the middle of the data

does not allow it to exert much “pull” on the regression line. We say that it lacks

leverage to aﬀect the regression. An observation has leverage to the extent that its

covariate pattern is atypical. That is, its combination of X scores is far from the

centroid, or vector of means, of the X’s. For the data in Figure 2.1, in which there

is only one X, the centroid is 40.925, the mean on the math diagnostic. Since a

score of 37 is relatively close to this, the outlier has little leverage. The two obser-

vations with math diagnostic scores of 28 (at the far left in the ﬁgure), on the other

hand, have considerable leverage. However, their location in the Y direction is con-

sistent with the trend in the swarm of points—their Y scores are 37 and 45, respec-

tively. Hence one can imagine that the regression line would not change much

REGRESSION DIAGNOSTICS I: INFLUENTIAL OBSERVATIONS 219

c06.qxd 8/27/2004 2:53 PM Page 219

were they omitted from the analysis. In other words, the two cases with diagnos-

tic scores of 28 have leverage but are not outliers and thus have little inﬂuence, too.

In sum, the only types of cases with real power to inﬂuence the analysis are out-

liers with leverage.

Measuring Inﬂuence

There are several measures available that can help the analyst identify potentially

inﬂuential observations. First, let’s consider simply how to ﬁnd such observations.

Then we’ll take up the issue of assessing how much inﬂuence they exert.

Externally Studentized Residuals. A handy diagnostic for ﬂagging inﬂuential

observations is based on the residual for a given case. Now, the residual by itself is

not always eﬀective in signaling inﬂuence. The reason for this is that an outlier with

lots of leverage will typically cause the ﬁtted line to be shifted toward it, thereby

reducing the magnitude of the residual. What is needed is a residual that is sensitive

both to an observation’s distance from the ﬁtted line and to its degree of leverage.

The ideal measure is called the externally studentized residual (Myers, 1986) and

symbolized by t

. Its formula is





i

兹



苶



苶

yˆ

苶



. (6.6)

The numerator of this measure is e

, the ordinary residual. The ﬁrst product in the

denominator, s

i

, is the standard error of estimate (i.e., the square root of MSE) from

a regression that leaves out the ith observation. If the ith observation reﬂects a model

misspeciﬁcation, s

i

is a better estimate of σ than is the square root of MSE using all

of the cases (Myers, 1986). The second term involves the hat diagonal, h

, discussed

earlier. As mentioned, h

is a measure of the leverage exerted by the ith case. For mod-

els with an intercept, 1/n  h

 1, and h

represents the standardized squared distance

from the ith case’s covariate pattern to the centroid of the X’s (Myers, 1986). In SLR,

the formula for h











苶

)



As is evident, h

reaches its minimum in SLR when X is at its mean. Similarly, h

reaches its minimum in a MULR model when x

is equal to x

苶苶

, the vector of covari-

ate means. A rough guideline for evaluating the amount of leverage exerted by an

observation is that h

’s greater than 2p/n represent cases with noticeable leverage.

In sum, t

is a kind of standardized residual adjusted for a case’s leverage. Large

values of t

will be observed whenever cases have large residuals and/or high lever-

age values, and t

will be especially high for cases with both properties. This statis-

tic also represents a formal test for outlier status. Under the null hypothesis that the

ith case is not an outlier, and given standard assumptions on the ε

, including nor-

mality, t

follows a t distribution with n  p  1 degrees of freedom, where p  K  1

is the number of model parameters (Myers, 1986). This suggests that t

values of

220 ADVANCED ISSUES IN MULTIPLE REGRESSION

c06.qxd 8/27/2004 2:53 PM Page 220

about 2 or higher signal outliers. (Note that t

is usually referred to as “rstudent” in

software packages such as SAS.)

Dffits

. Several measures are available to tap the degree of inﬂuence exerted by the

ith case on various regression results. The inﬂuence on the ﬁtted values is best tapped

by a measure called Dﬃts

Dﬃts







兹

yˆ

苶



苶



(6.7)

The numerator of (6.7) is the diﬀerence between the ﬁtted value for the ith case with

and without the ith case in the analysis (i.e., yˆ

i

is the ﬁtted value for the ith case based

on a regression that omits the ith case). What about the denominator? The standard

error of prediction (i.e., the standard error of the ith ﬁtted value) is σ兹x

苶



苶

X

苶

)



苶

and recall that h

 x

(XX)

1

. Therefore, the denominator is the estimated standard

error of the ith ﬁtted value. Dﬃts

can therefore be intepreted as the estimated number

of standard errors the ﬁtted value changes when the ith case is omitted from the regres-

sion. A size-adjusted cutoﬀ for Dﬃts

, above which the value suggests noticeable

inﬂuence, is 2兹p

苶

(Belsley et al., 1980).

Dfbetas

. The inﬂuence of the ith case on the jth parameter estimate is tapped by

Dfbetas







兹

,

苶



. (6.8)

The numerator of (6.8) is the diﬀerence in the jth parameter estimate with and with-

out the ith case in the regression. The denominator is the estimated standard error of

the jth coeﬃcient, since c

is the jth diagonal element of (XX)

1

. Dfbetas

is inter-

preted as the number of standard errors the jth parameter estimate changes when the

ith case is omitted from the regression. A recommended cutoﬀ for Dfbetas

is 2/兹n

苶

Cook’s D

. When there are many regressors in the model and n is large, it may be

very tedious to scrutinize all of the Dfbetas

, particularly when no single parameter

estimate is of paramount importance. With this in mind, a convenient summary

measure of the inﬂuence of the ith case on the collection of parameter estimates is

Cook’s distance measure, D

 . (6.9)

As b

i

represents the vector of parameter estimates based on a regression that omits

the ith case, (6.9) is the standardized distance between the vector of parameter esti-

mates with and without the ith case in the regression. It may be easier to get an intu-

itive feeling for D

by regarding its expression in the SLR model:

i,SLR



n(b

 b

0,i

)

 2nx

苶

 b

0,i

)(b

 b

1,i

)  (b

 b

1,i

)

冱



2MSE

(b  b

i

)(XX)(b  b

i

)



pMSE

REGRESSION DIAGNOSTICS I: INFLUENTIAL OBSERVATIONS 221

c06.qxd 8/27/2004 2:53 PM Page 221

The larger the diﬀerence in parameter estimates from regressions with and without

the ith case, the larger is D

. In the extreme opposite case in which there is no change

in the parameter estimates after deleting the ith case, D

is clearly zero. Cook’s D

has been described as an “F-like statistic” with degrees of freedom p and n  p

(Myers, 1986). However, the usual critical values of F used for hypothesis testing are

not applicable here. Neter et al. (1985) suggest instead that the 50th (rather than, say,

the 95th) percentile of the F distribution should be the cutoﬀ for declaring a case

inﬂuential. In practice, it may be prudent to investigate any case more closely if its

value is markedly larger than all others (see, e.g., DeMaris, 1997).

Covratio

. Finally, a measure that assesses the inﬂuence of the ith case on the esti-

mated variance–covariance matrix of parameter estimates is Covratio

Covratio





冟s



冟s

(







)



)



冟



(6.10)

Recalling that s

is MSE, the estimate of σ

, this represents the ratio of the determinants

of the estimated variance–covariance matrix without, versus with, the ith case in the

regression. The rationale for this measure is that the determinant of the variance–

covariance matrix is a scalar measure of the generalized variance of the regression

coeﬃcients (Graybill, 1976; Myers, 1986). All else equal, a smaller generalized vari-

ance implies regression coeﬃcients that have greater precision. A value of Covratio

greater than 1 suggests that the ith data point brings about a reduction in the general-

ized variance (the determinant is smaller with the ith case in the analysis), while a value

less than 1 implies that the ith case increases the generalized variance. Belsley et al.

(1980) suggest that inﬂuential observations will be indicated by Covratio

being either

greater than 1  3p/n or less than 1  3p/n. [It should be mentioned here that although

the diagnostic measures are all based on “omitting the ith case from the regression,” all

measures are computed in one pass through the data. In other words, all measures can

be calculated from one regression run. There is no need to actually run the regression

n times, each time omitting the ith case. See Myers (1986) for details.]

Illustration of Inﬂuence Diagnosis

As an example of the evaluation of inﬂuence, I examine the regression of the ﬁrst exam

score on the math diagnostic score, college GPA, and attitude toward statistics for 214

students in my introductory statistics classes. The results are shown in Table 6.6. Panel

A presents regression results, and panel B presents diagnostics for the four most

inﬂuential observations in the dataset. Cutoﬀs for all of the diagnostic measures are

shown in parentheses. For example, since p is 4, the cutoﬀ for leverage is 2(4)/214 

.037. The cutoﬀ for Dﬃts

is 2兹4

苶

 .273, the cutoﬀ for Dfbetas

is 2/兹2

苶



.137, and so on.

The single most inﬂuential observation is case number 214. Its t

value is over 4, and

its Dﬃts, Covratio, and D values are all the most extreme of any of the cases. With the

exception of the coeﬃcient for college GPA, case 214’s inﬂuence on the ﬁtted values

and the parameter estimates is especially noteworthy. This particular student, the same

222 ADVANCED ISSUES IN MULTIPLE REGRESSION

c06.qxd 8/27/2004 2:53 PM Page 222

one responsible for the outlier in Figure 2.1, represents somewhat of an atypical data

point. He or she is fairly strong on the regressor values—with math diagnostic, college

GPA, and attitude scores of 37, 3.00, and 11, respectively—but surprisingly weak on

the exam, with a score of 17. On the other hand, his or her leverage is below the cutoﬀ,

with a value of only .021. Nevetheless, the combination of what leverage there is plus

the case’s outlier status adds up to a fair amount of inﬂuence.

Panel A shows the regression estimates with (b

all

) and without (b

214

) this case in

the analysis. To get an intuitive feeling for the coeﬃcient inﬂuence measures, regard

the Dfbetas

for, say, the coeﬃcient for the math diagnostic score. The value of .433

suggests that the coeﬃcient should drop about four-tenths of a standard error when

case 214 is omitted from the regression. The estimated standard error of the

coeﬃcient is .279 without this case in the analysis. Thus, the actual drop is (2.151 

2.031)/.279  .430, which is about what was predicted. Even deleting this most

inﬂuential case, however, there is little substantive change in the model. All regres-

sors have signiﬁcant, positive eﬀects on the ﬁrst exam score. In fact, as the diagnos-

tics indicate, no case has the ability to alter any of the coeﬃcients by more than

one-half of a standard error. In this example, then, the degree of inﬂuence exerted by

any given case is less than dramatic. For case number 214, even though he or she

exhibits a relatively unusual pattern of data values, there would be no compelling

reason to delete his or her data from the analysis.

REGRESSION DIAGNOSTICS I: INFLUENTIAL OBSERVATIONS 223

Table 6.6 Eﬀect of Inﬂuential Observations on the OLS Regression for Score on the

First Exam for 214 Students in Introductory Statistics

Panel A:

Predictor b

all

214

Intercept 52.892*** 47.879***

Math diagnostic 2.151*** 2.031***

College GPA 12.929*** 12.872***

Attitude toward statistics .374* .441*

.432 .446

Panel B:Inﬂuential Observations

Diagnostic (Benchmark) No.58 No.92 No.158 No.214

(  2 in absolute value) 2.936 2.649 2.220 4.287

(  .037) .017 .025 .050 .021

Dﬃts

(  .273 in absolute value) .381 .425 .511 .628

Dfbetas

(  .137 in absolute value) .275 .191 .461 .437

Dfbetas

(  .137 in absolute value) .179 .143 .477 .433

Dfbetas

(  .137 in absolute value) .159 .093 .044 .032

Dfbetas

(  .137 in absolute value) .105 .273 .019 .417

Covratio

( .944 or  1.056) .882 .916 .978 .743

Cook’s D

(  .84) .035 .044 .064 .091

* p  .05. ** p  .01. *** p  .001.

c06.qxd 8/27/2004 2:53 PM Page 223

REGRESSION DIAGNOSTICS II: MULTICOLLINEARITY

Multicollinearity was touched on brieﬂy in Chapter 3, in which I deﬁned the prob-

lem and suggested some easy remedies. In this section I consider the topic in much

greater technical detail and also discuss alternatives to OLS when the problem can-

not easily be resolved. Recall that multicollinearity is a condition in which one or

more of the independent variables is almost exactly determined by the other regres-

sors. Another way of saying this is that there are one or more “near linear depend-

encies” among the columns of X, that is, among the regressors in the X matrix

(Myers, 1986, p. 76). A complete understanding of the problem requires that we dis-

sect the X matrix—in a manner of speaking. However, let’s work with the centered

and scaled matrix X* (discussed earlier), since this is the basis for the standardized

regression coeﬃcients, from which the unstandardized versions are easily recovered.

We will refer to X* as the design matrix. Remember that X*X* is R

, the correla-

tion matrix for the X’s.

Linear Dependencies in the Design Matrix

I begin by reminding the reader of the deﬁnition of linear dependence with respect

to the columns of a matrix. In Section V.G of Appendix A I deﬁned the linear

dependence of the columns of a matrix A as follows: If there is a nonnull vector x

such that Ax = 0, then provided that no column of A is null, the columns of A are lin-

early dependent. Let’s see how this principle applies here. I will show that a linear

dependence in the design matrix is associated with a zero eigenvalue of the matrix.

Since the K  K matrix X*X* is symmetric, it can be spectrally decomposed

(See Section V.H of Appendix A for an explanation of spectral decomposition).

Recall that the spectral decomposition of a symmetric matrix A allows us to write

the matrix as

A  冱λ

u

where λ

is the jth eigenvalue of A and u

is the jth eigenvector for j  1,2,...,K.

This is equivalent to writing A  UD

U, where U is the matrix whose columns are

the normalized eigenvectors of A and D

is the diagonal matrix of eigenvalues of A

(veriﬁcation that these expressions for A are equivalent is left as an exercise). Pre-

and postmultiplication of A by U and U, respectively, thereby produces UAU  D

(since U, being an orthogonal matrix, has the property that UU  UUI). This is

called the eigenvalue decomposition of A (Myers, 1986). Now substitute X*X* for

A, and we have that U(X*X*)U  D

, where λ now refers to the eigenvalues of

X*X*, and U to the matrix of its normalized eigenvectors.

Recall from Appendix A that a matrix has an inverse only if its columns are lin-

early independent, in which case its determinant is not zero. Moreover, its determi-

nant is the product of its eigenvalues. So one or more zero eigenvalues imply that the

determinant is zero and are therefore indicative of exact linear dependence among the

columns of the matrix. Similarly, eigenvalues that are near zero reﬂect near linear

224 ADVANCED ISSUES IN MULTIPLE REGRESSION

c06.qxd 8/27/2004 2:53 PM Page 224