Alfred DeMaris - Regression with Social Data, Modeling Continuous and Limited Response Variables

Подождите немного. Документ загружается.

The diﬀerence in the amount of error made in using, versus ignoring, the regression

model is TSS – SSE. As a proportion of the original amount of error, it is



TSS



SSE



 1 



 r

Another way to view r

is to consider how it reﬂects the extent to which the linear

function of X (i.e., the regression) explains the variation in Y. Figure 2.5 reveals this

aspect of r

. Depicted is a perfect linear relationship between Y and X, since Y is

exactly determined by the line 1  2X. The r

in this case is 1.0, since there is no error

in predicting Y using the regression line. Suppose for the moment that we ignore X and

simply examine the unconditional variation in Y. This is represented by the horizontal

lines that map the Y values from the line 1  2X back to the Y axis. The Y values range

from 1 to 21, exhibiting considerable variation. However, the reason for this variation

is that Y is determined completely by its linear relationship with X. That is, the reason

Y varies is that X varies from 0 to 10, and Y—being determined completely by X—

varies accordingly. The conditional variation in Y given X is zero. That is, given any

value x of X, Y is exactly 1  2x, and there is no variability in Y at this x value. Here it

is clear that the variation in Y is caused or explained totally by the regression on X,

since Y exhibits no variance once X is held constant at any particular value. Even

ESTIMATION USING SAMPLE DATA 55

Figure 2.5 Regression of Y on X producing an r

of 1.0, showing how the regression completely

“explains” the variation in Y.

c02.qxd 8/27/2004 2:47 PM Page 55

though r

is virtually never equal to 1.0, the closer it is to 1.0, the more the variation

in Y is explained by the regression.

Adjusted r

. Although r

provides a convenient comparative tool for assessing the rel-

ative discriminatory power of diﬀerent models for Y, it has a tendency to overestimate

the population ρ

(Stevens, 1986). If estimation of this parameter is one’s goal, a better

estimator of ρ

is the adjusted r

or r

adj

. Its formula is

adj

 1 



 ,

or 1 minus the ratio of MSE to the sample variance of Y. Notice that this is the pre-

cise sample analog of 1  σ

/V(Y). In fact, r

adj

is a consistent estimator of r

because

plim(r

adj

)  plim



1 



 plim(1) 



(

)



(by the Slutsky theorem)

 1 



)





since the MSE and sample variance of Y are consistent for their respective popula-

tion parameters. (Recall that the plim, consistency, and Slutsky’s theorem were all

discussed in the Chapter 1 appendix.)

Tables 2.2 to 2.4 show values of r

and r

adj

for the regressions of exam 1 on math

diagnostic score, couple modernism on male’s schooling, and frequency of sex on

going to bars. The values of r

and r

adj

for the same response are quite close. This

will typically be the case in large samples, especially with only one regressor in the

model. (I shall have more to say about the properties of r

adj

when we get to multiple

regression models.) As is evident, the discriminatory power of the model for exam

scores is fairly impressive, at least from a social science standpoint. Fully 26.8% of

the variance in exam scores is accounted for by math diagnostic scores alone.

Another interpretation of this value is that we would improve our ability to predict

exam scores by 26.8% when using, as opposed to ignoring, its linear relationship to

math diagnostic scores. The prediction of couple modernism using male’s schooling

is considerably less powerful. Only about 7% of the variance in couple modernism

is accounted for by this predictor. And the predictive eﬃcacy of frequenting bars as

a determinant (or correlate) of having sex is less impressive still: A mere 3.8% of the

variation in sexual frequency in the past year is explained by going to bars.

Limitations of r

. The discriminatory power of one’s model is certainly an important

consideration to data analysts. However, we should be careful not to put too much

weight on this dimension of regression. First, a high r

does not necessarily imply a cor-

rectly speciﬁed model. That is, high discriminatory power does not automatically imply

authenticity. Second, high r

values are fairly easy to achieve if the regressor is con-

ceptually close to the response. For example, a regression of someone’s depressive

56 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 56

symptomatology on his or her self-esteem is likely to produce an impressive r

. But

both attributes are typically tapped with paper-and-pencil scales that are aﬀected by cur-

rent mood. So, those who tend to be depressed are most likely also “down on them-

selves,” suggesting that their self-esteem rating will also be diminished. Rather than this

suggesting that self-esteem has high discriminatory power in predicting depression, or

vice versa, a high r

value might only be indicating that people are consistent in answer-

ing mood-related items. Third, some criteria just lend themselves to better prediction

than others. Exam scores of students in statistics, for example, can probably be nicely

explained by college GPA, math aptitude, attitude toward the subject matter, and the

number of other classes students are also taking. On the other hand, imagine trying to

predict marital happiness. Certainly, there are a number of social factors that would be

expected to aﬀect this attribute: number of years married, number and ages of children,

income level of the household, and so on. But ultimately, happiness is a fairly elusive

feeling that may also have a large random component. One day we’re happier with our

spouse, one day we’re not as happy with him or her, and some days we’re just not sure.

To expect a high r

value in predicting this type of variable is probably unrealistic unless

one employs predictors such as satisfaction with communication in the marriage, which

are themselves conceptually close to the response. In that case we might wonder

whether we are really learning anything about the response variable, after all. In sum,

the importance placed on r

—and on discriminatory power, in general—depends on the

researcher’s goal. If the model is to be used for accurate prediction of responses for new

observations, a high r

value is especially desirable. But if the central aim is to under-

stand the nature of the “eﬀect” of a regressor on the response in order to test theoreti-

cally driven hypotheses, discriminatory power may well be of secondary importance.

Correlation Coefficient and Standardized Slope. The r

is also the square of r, the

Pearson correlation coeﬃcient discussed in Section III.C of Appendix A. The corre-

lation coeﬃcient is calculated as

r 





(



)





The correlation coeﬃcient ranges from 1.0 to  1.0 and is a measure of linear asso-

ciation between X and Y. When Y is a perfect linear function of X (i.e. all of the points

lie exactly on the regression line), r is 1.0 in absolute value (this is proved in Section

III.C of Appendix A). The correlation coeﬃcient is also the standardized slope in sim-

ple linear regression. This terminology stems from the fact that if we standardize X

and Y and then regress Y on X, the slope will be the correlation coeﬃcient and the

intercept will be zero (see also Exercises 2.17 and 2.18, where you’re asked to verify

this principle). The sample regression equation in standardized form is Z

 b



, where b

is the standardized slope, or r. The standardized slope can be recovered

from the unstandardized slope, b

, via the formula

 b





i1

 x



)(y

 y



)











i



(







)













i



(







)







ESTIMATION USING SAMPLE DATA 57

c02.qxd 8/27/2004 2:47 PM Page 57

The standardized slope has a rather cumbersome interpretation: it is the expected

change in Y, in standard deviation units, for a 1-standard-deviation increase in X. Why?

Well, if b

is the expected change in Y for each unit increase in X, b

is the expected

change in Y for each 1-standard-deviation increase in X. Dividing again by s

 b

) is the expected change in Y for each 1-standard-deviation increase in

X, but now it is in terms of the number of standard deviations in Y. Standardized slopes

for our three substantive regression examples are shown in Tables 2.2 to 2.4. They

range in value from .518 for the regression of exam scores down to .194 for the regres-

sion of sexual frequency. No standardized value is shown for the intercept, since the

intercept in the standardized equation is always zero.

INFERENCES IN SIMPLE LINEAR REGRESSION

In this section I discuss inferences connected with the simple linear regression model.

In particular, I discuss three equivalent tests for the signiﬁcance of the sample slope,

as well as a test for the sample intercept. I also consider conﬁdence intervals for the

values of the population slope and intercept. Moreover, to justify inferences about the

slope, I derive its expectation and variance, and its distribution, under general

assumptions about the equation errors.

Tests about the Population Slope

Once we have estimated the parameters of the regression equation, the next question

is: Is there a linear relationship between Y and X in the population? There is, pro-

vided that the value of the population slope is not zero. Hence, we wish to test the

null hypothesis that β

equals zero against the alternative that β

is not zero. To do

this, we need to ﬁnd the sampling distribution of b

, the sample estimator of the

slope. It turns out that if n is suﬃciently large, b

is normally distributed with mean

equal to β

and variance equal to σ

/

i 1

 x



)

, regardless of the distribution of

ε. (In small samples it is necessary that ε be normally distributed with mean equal to

zero and variance equal to σ

in order for b

to have the same distributional proper-

ties.) Given these properties, a t—or in large samples, z—test for H

: β

 0 against

: β

 0 is t  b

/σ

, where σ

 σ







i 







)



is the estimated standard

deviation, or standard error, of the sample slope. Under the null hypothesis that the

population slope is zero, this statistic has a t distribution with n  2 degrees of freedom

(or, equivalently, a z distribution) in large samples. As an example, the t test for the

slope of the regression of exam 1 score on math diagnostic score is t  2.749/.313 

8.788. This t has a p-value less than .001 under the null hypothesis that the corre-

sponding population slope is zero. In fact, t tests for all of the sample slopes in

Tables 2.2 to 2.4 suggest signiﬁcant linear relationships in the population. That is,

we can reject the null hypothesis of zero population slope in each case and conclude

that there are signiﬁcant positive linear relationships between math diagnostic score

and exam 1 score, between male’s schooling and couple modernism, and between

going to bars and having sex.

58 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 58

Justification. How does one derive the distribution of b

? First, we rewrite the for-

mula for b

to show that it is a weighted sum of the y







[The last step in this sequence is justiﬁed by exercise (1) in Section II.E of Appen-

dix A.] Now, if we let s

 

i 1

 x



)

, we can write each person’s weight as w



 x



)/s

, and we can write b

as b

 

i 1

. This shows that b

is a weighted

sum of the y

. At this point we make use of the CLT, outlined in the Chapter 1 appen-

dix, which states that a weighted sum of independent and identically distributed ran-

dom variables is distributed approximately normal in large samples, with mean equal

to µ

i 1

and variance equal to σ



i 1

, where µ and σ

are the mean and vari-

ance, respectively, of the random variables. In this case, the random variables, Y

,are

not exactly identically distributed, since although they all have the same variance of

, their means are a function of X. In particular E(Y

)  β

 β

. No matter, how-

ever. This just means that the mean of the weighted sum is written as 

i 1

instead of µ

i 1

, where µ

 β

 β

. At any rate, by the CLT, b

is normally dis-

tributed with mean

E(b

) 



i1





i1

(β

 β

)















i1

[β

 x



)  β

 x



)]







i1

 x



) 





i1

 x



)









i1

 x





i1











i1

 nx







i1

x





i1

x



)





i1

x



)



i1

x



)



i1

x





i1

x



)



i1

[(x

 x



 (x

 x



]







i1

x



)



i1

x



)(y

y



)





i1

x



)

INFERENCES IN SIMPLE LINEAR REGRESSION 59

c02.qxd 8/27/2004 2:47 PM Page 59

and recalling from Appendix A that



i1

 nx







i1

 x



)

 s

we have that

E(b

) 



 β

It is therefore clear that b

is an unbiased estimator of β

. Moreover, the variance of

V(b

)  σ



i1

 σ







i1

(

x



)



which agrees with the result stated above. The point of this demonstration is that in

large samples, the sample slope is normally distributed regardless of the distribution

of the errors. This allows us to make inferences about the population slope using its

sample counterpart, without the requirement of normally distributed errors. As men-

tioned previously, however, the normality of errors is a requirement in small sam-

ples. Provided that the errors are normally distributed, Y is also normally distributed.

Hence the sample slope—shown above to be a weighted sum of the Y’s—would then

be normally distributed for any sample size.

Equivalent Tests for the Slope. Two other tests are available that are mathemati-

cally equivalent to the t test for the slope. The ﬁrst is the t test for the correlation

coeﬃcient. The test statistic is

t 



(1







)







)





This test also has n  2 degrees of freedom under the null hypothesis that the popu-

lation slope is zero—or, equivalently, that the population correlation, ρ, between X

and Y, is zero. In fact, this is exactly the same statistic as the test for the slope. For

example, the coeﬃcient for the correlation of exam score with math diagnostic score

is .518. Testing this value, we get

t 8.797,

which, within rounding error, is the same value as that reported for the t test of the

slope in Table 2.2 (both values round to 8.8).

The other test is an F test. This has the form F  RSS/MSE. Under the null

hypothesis of zero population slope, this statistic follows the F distribution with 1

and n  2 degrees of freedom. Again, for the regression of exam scores, the test is

F  16509.689/213.756  77.236, a highly signiﬁcant result. In fact, this test statis-

tic is also mathematically equivalent to the t test for the slope, as in simple linear

regression, F  t

. For example, the square root of 77.236 is 8.788, which agrees

with the t test for the slope as reported in Table 2.2.

.518



(1











)





i1

x



)

60 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 60

Testing the Intercept

As it is also a weighted combination of the Y

(Neter et al., 1985), the sample esti-

mate, b

, of the equation intercept is also normally distributed in large samples (or

with any sample size provided that ε is normal). Its expected value is β

, which means

that it is also unbiased for the population parameter. Its variance is (Neter et al., 1985)

V(b

)  .

Tests about the population intercept are almost never of interest, especially when zero

is not a meaningful value for X. Nonetheless, a t test of H

: β

 0 against H

: β

 0

takes the form t  b

/σ

, where σ





















is the estimated standard

error of the sample intercept. Under the null hypothesis that the population intercept

is zero, this statistic has a t

n  2

(or equivalently, a z) distribution in large samples.

As is evident in Table 2.2, the t-test statistic for the intercept is 35.494/

12.840 2.764. Hence we would reject the null hypothesis that the population

intercept is zero at a p-value of .006. As indicated above, however, this test is not par-

ticularly meaningful, due to the potential nonlinearity of the relationship at diagnos-

tic scores of zero.

Conﬁdence Intervals for

ββ

and

ββ

The dominant vehicle for inferences in the social sciences appears to be the hypothesis

test. Yet we may, as well, be interested in trying to pin down the value of population

parameters as closely as possible. For this purpose, we can form conﬁdence intervals

for values of β

or β

. For example, a 95% conﬁdence interval for β

is b

 t

(.025,n  2)

(σ

), where t

(.025,n2)

is the 97.5th percentile of the t distribution corresponding to n  2

degrees of freedom. In the regression of exam scores, n  2  211 and t

(.025,211)

is 1.97.

For this regression, a conﬁdence interval for the slope is 2.749  1.97(.313)  (2.132,

3.366). Similarly, a 95% conﬁdence interval for β

is b

 t

(.025,n  2)

(σ

). For the regres-

sion of exam 1, the interval is 35.494  1.97(12.840)  (60.789, 10.199). Only

the interval for the slope is particularly meaningful. It implies that we can be 95%

conﬁdent that those who are a unit higher on the math diagnostic score perform, on

average, between 2.13 and 3.37 points higher on the ﬁrst exam in statistics compared to

others.

Additional Examples

Additional examples of simple linear regressions are shown in Table 2.5. Here, I

present the regression of having sex on going to bars separately by gender and

then separately by marital status. The reason for estimating the regression

separately within diﬀerent subgroups of the population (male vs. female, or



i1

INFERENCES IN SIMPLE LINEAR REGRESSION 61

c02.qxd 8/27/2004 2:47 PM Page 61

married vs. unmarried) is that it is reasonable to expect that frequenting bars

would have a stronger or weaker association with having sex, depending on the

subgroup.

For example, one might expect going to bars to have a stronger eﬀect on sexual

activity for women than for men. Why? Given that men are still expected to be the

initiators of social contact in mixed-sex environments, the mere presence of women

at bars is likely to garner more potential “suitors” for women than it is for men. That

is, women can remain quite passive and still be approached by men at bars, whereas

the passive male is less likely to end up with female companionship during the

course of an evening. Also, given the rather sexualized atmosphere of many bars,

such a venue may be especially selective of sexually willing females but not espe-

cially selective with respect to males. At any rate, I expect going to bars to bear a

stronger association with sexual activity for women. This appears to be borne out

by the results. Although the eﬀ ect of going to bars on sexual frequency is signiﬁcant

for both genders, it is substantially larger for women, with a value of .254, com-

pared to .155 for men.

In a similar vein, I expect the eﬀect of going to bars to be stronger for the unmar-

ried than for the married. Married people tend to have more regular sex than do the

unmarried since they already have a presumably willing sex partner in their spouse.

Going to bars should not, therefore, make as much diﬀerence in their sexual activity

level as it might for single persons. Consistent with this notion, the eﬀect of going

to bars on sexual frequency is more than twice as great for the unmarried than for

the married (slopes for the married and unmarried are .167 and .379, respectively).

The reader should notice that I’m comparing unstandardized rather than stan-

dardized slopes when comparing the impact of the same predictor on the same

response across subpopulations. This is not just coincidental. For comparing across

subpopulations in this manner, one should always use the unstandardized slope. The

62 SIMPLE LINEAR REGRESSION

Table 2.5 Parameter Estimates from the Regression of Y ⴝ Frequency of Having Sex

in the Past Year on X ⴝ Frequency of Going to Bars for 1515 Respondents in the 1998

GSS, Separately by Gender and Marital Status

Explanatory Variable b σ

tpR

A. Males

Constant 2.682 .131 20.403 .001

Frequency of going to bars .155 .041 3.824 .001 .022

B. Females

Constant 2.001 .113 17.645 .001

Frequency of going to bars .254 .042 6.078 .001 .041

C. Married

Constant 3.053 .100 30.494 .001

Frequency of going to bars .167 .040 4.215 .001 .024

D. Unmarried

Constant 1.185 .128 9.280 .001

Frequency of going to bars .379 .039 9.807 .001 .110

c02.qxd 8/27/2004 2:47 PM Page 62

reason for this is that the standardized slope is a function not only of the unstan-

dardized slope but also of the standard deviations of X and Y in each subpopulation.

Therefore, the standardized slopes can be diﬀerent for diﬀerent subgroups solely

because of diﬀerences in the distributions of X and Y in each group rather than

because the eﬀect of X is diﬀerent in each group. Comparing the unstandardized

slopes avoids this problem.

Once we have discovered that the eﬀect of going to bars on having sex is diﬀerent

in samples from diﬀerent subpopulations, a natural question is: Do these sample

diﬀerences reﬂect real diﬀerences in the eﬀect of X in the subpopulations themselves?

We must be careful not to assume automatically that diﬀerent slope estimates from

diﬀerent subsamples imply diﬀerent eﬀects in the corresponding subpopulations.

What is needed is a statistical test to tell whether, say, the eﬀect of going to bars is

weaker for men than it is for women. That is, if β

is the eﬀect for males and β

the eﬀect for females, we need a test for H

: β

 β

against H

: β

 β

. This is a

test for what is referred to as statistical interaction. If the eﬀect of X depends on the

level of another explanatory variable—in this case, gender—we say that gender and

X interact in their eﬀect on Y. We take up the issue of statistical interaction in regres-

sion in Chapter 3 and discuss such a test at that time.

ASSESSING EMPIRICAL CONSISTENCY OF THE MODEL

Above I discussed how we can assess the discriminatory power of a regression

model using either r

or r

adj

. What about empirical consistency? This issue is some-

what more complex. An empirically consistent model should conform to the model

assumptions as well as accurately forecast the behavior of the response variable. In

this section I discuss some informal and formal techniques for evaluating this dimen-

sion of regression.

Conforming to Assumptions

Recall that the equation disturbances, the ε

’s, are subject to some critical assump-

tions. First, in small samples, they should be normally distributed. In large samples

this assumption is not as important. Second, they should have a mean of zero at each

x. Third, they should have constant variance at each x. As our best estimates of the

’s, the sample residuals can be used to check the ﬁrst and third of these assump-

tions. For example, the normality-of-errors assumption for the regression of exam

scores in Table 2.2 was evaluated using SAS by saving the residuals from the regres-

sion and then requesting a test of normality via PROC UNIVARIATE. The test sta-

tistic, which was devised by Shapiro and Wilk (1965), had a p-value below .0001,

resulting in rejection of the null hypothesis of normality. However, in that the sam-

ple size is relatively large, nonnormal errors are not of great concern here.

The constant variance assumption can be checked visually by plotting the residu-

als against X. Such a plot for the residuals from the regression of exam scores is

shown in Figure 2.6. What we are looking for is a scatter of points of approximately

ASSESSING EMPIRICAL CONSISTENCY OF THE MODEL 63

c02.qxd 8/27/2004 2:47 PM Page 63

constant width spread about the line e  0—the dark line shown in the ﬁgure. The

point scatter seems roughly to have this appearance. Nonconstant variance typically

shows up as a point scatter in the shape of a wedge, in which the variance of e

appears

either to decrease or to increase with increasing X. As an example, Figure 2.7 shows

a plot of residuals against X for the regression of couples’ coital frequency in the last

month on male partner’s age from the couples dataset. Coital frequency is the aver-

age of the male’s and female’s report of how many times the couple “had sex” in the

past month. As is evident here, the width of the band of points appears to taper down

considerably with increasing age. In fact, a statistical test suggests that we should

reject the null hypothesis of constant variance for these disturbances. The test is dis-

cussed in Chapter 6.

The assumption that the mean of the errors at each X is zero—the orthogonality

condition—cannot be checked using the residuals. The reason for this is that it is an

artifact of OLS estimation that the mean of the residuals is always zero, and moreover,

that the covariance of the residuals with X is always zero. This is easy to show. Recall

once again the normal equations 

i 1

( y

 yˆ

)  0 and 

i 1

( y

 yˆ

)  0. Now,









1







0.



i1

( y

 yˆ

)

64 SIMPLE LINEAR REGRESSION

Figure 2.6 Plot of e (residuals) against X (math diagnostic score) from OLS regression of score on

exam 1 on math diagnostic score for 213 students in introductory statistics.

c02.qxd 8/27/2004 2:47 PM Page 64