Alfred DeMaris - Regression with Social Data, Modeling Continuous and Limited Response Variables

Подождите немного. Документ загружается.

Thus, we see that β

is also the ﬁrst derivative of E(Y

) with respect to X

. Recall

(from Section IV of Appendix A) that the ﬁrst derivative, also called the slope of

E(Y

) with respect to X

, represents the instantaneous rate of change in E(Y

) with an

increase in X

at point x

. It also represents the slope of the line tangent to the curve

relating Y to X at point x. Because the curve in this case is really a straight line, the

slope of the tangent line is just the slope of the regression line itself (there is no tan-

gent line in this case, since it is impossible for a line to touch a straight line at only

one point). Hence, the instantaneous rate of change in E(Y) with increase in X is the

same as the change in E(Y) per unit increase in X. In other words, in regression mod-

els in which E(Y) is a linear function of X, the ﬁrst derivative and the unit impact are

identical quantities. This is no longer the case for models in which E(Y) is modeled

as a nonlinear function of X (discussed in subsequent chapters).

ESTIMATION USING SAMPLE DATA

Estimation of the regression equation and associated parameters using sample data is

most often accomplished by employing ordinary least squares estimation (OLS). The

assumptions enumerated above are largely assumptions required for unbiased estima-

tion of the regression parameters using OLS. Let’s review the assumptions. They are:

1. Y is a linear function of X; that is, Y

 β

 β

 ε

for i  1,2,...,n.

2. The observations are sampled independently.

3. X and Y are approximately continuous variables.

4. The X-values are ﬁxed over repeated sampling and measured with only negli-

gible error.

5. E(ε

)  0 at each x

6. V(ε

)  σ

at each x

7. Cov(ε

,ε

)  0 for i  j (i.e., the errors are uncorrelated with each other). (Note

that this is usually equivalent to assumption 2.)

8. The errors are normally distributed.

Assumption 5 also ensures the orthogonality condition that Cov(X

,ε

)  0. The

reason for this is straightforward. If there were a linear relationship between ε

and

,reﬂected by a nonzero covariance, it would take the form E(ε

)  γ

 γ

. That

is, the mean of the errors would be a linear function of X. If the mean of the errors

is the same (in particular, it is zero) at each X, this implies that there is no linear rela-

tionship between the error and X. This, in turn, implies that Cov(X

,ε

) is zero.

Rationale for OLS

To develop the rationale for OLS, we consider the following example. Table 2.1

presents ﬁctitious data for ﬁve currently married persons; X is the number of years

married and Y is a continuous measure of marital happiness. A scatterplot of these

ESTIMATION USING SAMPLE DATA 45

c02.qxd 8/27/2004 2:47 PM Page 45

(x,y) values is shown in Figure 2.4. There appears to be somewhat of a positive lin-

ear trend in the points, such that those married longer are, on average, more happily

married. Assuming that there is a linear relationship between X and Y in the popula-

tion, how do we go about estimating the population regression line representing this

relationship? Well, suppose that we start with a visual approximation of the line that

best ﬁts the points. One possibility is the lightly drawn line in the ﬁgure, which

touches the rightmost data point. The equation for this line is yˆ  3  .25x ( yˆ is

shown as “y” in the ﬁgure). Here, yˆ, called the ﬁtted, or predicted, value of Y,

denotes the sample estimate of E(Y) (I have dropped the i subscript here for econ-

omy of expression). The sample regression equation is therefore Y  3  .25x  e,

where e, called a residual, is our sample estimate of the equation error for a partic-

ular case. Notice that e

 y

 yˆ

is the diﬀerence between observed and ﬁtted Y-

values for the ith case. That is, e is the vertical distance from the point to the regres-

sion line for the ith data point. For example, the residual for the second observation

is shown as e

 in Figure 2.4. Table 2.1 shows the ﬁve ﬁtted values of Y using the

visual approximation (VA) approach, along with the residuals for each case.

46 SIMPLE LINEAR REGRESSION

Table 2.1 Visual Approximation versus Ordinary Least Squares Regression Lines Fit

to Fictitious Data on Marital Happiness (Y) and Number of Years Married (X) for Five

Persons

Years Marital

Person Married Happiness yˆ

yˆ

OLS

1 1 5 3.25 1.75 3.58 1.42

2 3 2 3.75 1.75 3.96 1.96

3 4 6 4.00 2.00 4.15 1.85

4 9 3 5.25 2.25 5.10 2.10

5 20 8 8.00 .00 7.19 .81

e .25 .02

e

15.1875 14.3466

Calculation of OLS Estimates

Case XY(X  X



)(Y  Y



)[(X  X



)(Y  Y



)] (X  X



)

11 56.4 .2 1.28 40.96

23 24.4 2.8 12.32 19.36

34 63.4 1.2 4.08 11.56

4 9 3 1.6 1.8 2.88 2.56

5 20 8 12.6 3.2 40.32 158.76

Sum 44.4 233.2





 .1904

 4.8  .1904(7.4)  3.3911

c02.qxd 8/27/2004 2:47 PM Page 46

Measuring Prediction Error. How well does this line ﬁt the data? One way to tell is

to examine the total prediction error made in using this equation to predict Y. Our ﬁrst

impulse might simply be to sum the e’s for the ﬁve cases to assess total error.

However, that doesn’t work well, since large positive errors and large negative errors

tend to cancel each other out. We might therefore end up with a small total error even

with a poorly ﬁtting line. Instead, the errors are ﬁrst squared and then summed. The

resulting quantity is called (ﬁttingly) the sum of squared errors and denoted SSE:

SSE 



i1





i1

( y

 yˆ

)

The idea behind ordinary least squares is to choose as our estimate of the population

regression line that sample line that minimizes SSE (hence the resulting b

and b

are

called the ordinary least squares estimates). The least squares line for these data is

shown as the darker, thicker line in Figure 2.4 and is also the ﬁtted line drawn

through the point scatters in Figures 2.1 to 2.3. The least squares equation is Y 

3.39  .19x  e. The residual for the fourth observation, based on the least squares

line, is shown as e

in Figure 2.4. Table 2.1 also shows the ﬁtted values and residu-

als based on the OLS regression equation. Notice that SSE

OLS

, at 14.3466, is smaller

than SSE

, which is 15.1875. In fact, the SSE based on the OLS estimates of b

and

ESTIMATION USING SAMPLE DATA 47

Figure 2.4 Scatterplot of Y (marital happiness) against X (number of years married) for ﬁve ﬁctitious

married persons.

c02.qxd 8/27/2004 2:47 PM Page 47

is guaranteed to be smaller than SSE based on any other estimates. How is this

condition satisﬁed?

Mathematics of OLS

Finding the b

and b

that minimize SSE is a minimization problem in two variables,

which is readily solved using the techniques of diﬀerential calculus. First, we rewrite

SSE to highlight the fact that it is a function of b

and b

SSE 



i 1

 yˆ

)





i1

 (b

 b

)]

To ﬁnd the b

and b

that minimize this function using calculus, we take the ﬁrst

derivative of SSE with respect to, ﬁrst, b

and then b

. We then set each of these

expressions to zero and solve the resulting simultaneous equations for the b

and b

that satisfy the pair of equations. (Okay, this sounds complicated, but bear with me

here. Of course, it will help immensely if you have read and absorbed Section IV of

Appendix A before proceeding.) By a theorem in calculus, if SSE has a maximum or

minimum at some value of (b

), it occurs where these ﬁrst derivatives equal zero

(Anton, 1984).

Taking the ﬁrst derivative of SSE with respect to b

gives us



∂







i1

( y

 b

)



2



i1

( y

 b

which equals zero when



i1

 b

)  0

or when



i1

 nb

 b



i1

 0

or when





i1

 b



i1

Dividing both sides by n, we have

 y



 b



. (2.3)

So we have the solution for b

, but of course it requires the solution for b

to com-

pute. Therefore, we solve for b

. Again we ﬁrst take the derivative of SSE with respect

to b



∂







i1

( y

 b

)



2



i1

 b

48 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 48

which equals zero when



i1

( y

 b

)  0

or when



i1

 b



i1

 b



i1

 0.

At this point we can substitute the solution for b

from equation (2.3) for b

, and after

some mildly tedious algebra, we arrive at



(2.4)

Dividing both numerator and denominator of the right-hand side of equation (2.4) by

n  1, we see that the slope can also be expressed as





cov

(

x,y)



The equations



i1

 b

)  0 and



i1

 b

)  0 are called

the normal equations. Once we have found the values of b

and b

that satisfy these

equations, we must then show that plugging these particular values into

SSE 



i1

 yˆ

)





i1

 b

)

produces a minimum value for this function. Formally, by the second partials test

(Anton, 1984), this can be shown using second partial derivatives of the function

f 



i1

 b

)

. In particular, let f

, b

represent the second partial deriva-

tive of f with respect to b

, f

, b

represent the second partial derivative of f with

respect to b

, and f

, b

represent the mixed second partial derivative of f with respect

to b

, followed by b

. Then, by the second partials test, if the expression D 

( f

)( f

, b

) f

is greater than zero and f

is greater than zero when these

expressions are evaluated at a particular value of b

and b

, then f has a minimum at

those particular values of b

and b

. In fact, f

evaluated at the value in expression

(2.3) is 2n, f

, b

evaluated at the value in expression (2.4) is 2x

, and f

, b

evalu-

ated at expressions (2.3) and (2.4) is 2x. Hence,

D  2n



2







2



 4n





(x  x



)



which is greater than zero, as is f

, b

. Therefore, the OLS solutions to the normal

equations are, in fact, the values that minimize SSE.

Intuitively, we can arrive at the same conclusion by simply regarding the function

f 



i1

 b

)

. If there is an extreme value of this function, it has to be a

minimum, since the maximum value is unbounded. That is, there is no ﬁnite maximum

since we can make f as large as we wish simply by choosing b

, b

values—and,



i1

x



)(y

y



)



i1

x



)

ESTIMATION USING SAMPLE DATA 49

c02.qxd 8/27/2004 2:47 PM Page 49

therefore a line—that is as far from the points as we want to make it. By the princi-

ples of calculus, the solutions to the normal equations show us the b

and b

that lead

to a critical value of the function. That critical value, by this reasoning, must there-

fore be a minimum.

The bottom panel of Table 2.1 illustrates the OLS computations for b

and b

for

our ﬁctitious data. As shown, b

, the estimate of the equation intercept, is 3.3911. This

implies that the average happiness level for newlyweds (i.e., those with zero years

married) is 3.3911. The intercept is not typically very meaningful whenever zero is

outside the range of observed X-values, as it is here. In fact, zero may not be a mean-

ingful value for many explanatory variables, rendering the intercept uninterpretable

much of the time. The estimate of the equation slope, b

, is .1904. This suggests that

those who are a year apart in marital duration are about .1904, or approximately two-

tenths of a unit, apart in marital happiness, on average. Or, in somewhat stronger

causal language, each additional year of marriage would be expected to increase mar-

ital happiness, on average, by about two-tenths of a point. Using the estimated equa-

tion, we can generate predicted marital happiness scores based on the number of years

that a person has been married. Hence, for someone married, say, 15 years, the pre-

dicted, or ﬁtted, value of marital happiness is yˆ  3.3911  .1904(15)  6.2471.

According to the model, this value is the estimated mean of marital happiness for all

those who have been married for 15 years. It is generally safe to generate predicted

scores on Y for X-values within the range observed in one’s sample. However, we

would not want to try to predict Y for values outside that range—say, for someone

married 25 years in the current example. The reason for this is that the relationship

may or may not be linear beyond the range observed; hence the model may no longer

hold for more extreme levels of X.

OLS Estimates for the Examples. OLS estimates of the regression equations for the

regression of exam performance on math diagnostic score, the regression of couple

modernism on male’s education, and the regression of sexual frequency on going to

bars are shown in Tables 2.2, 2.3, and 2.4, respectively. (Shown also are several other

statistics discussed below.)

50 SIMPLE LINEAR REGRESSION

Table 2.2 Parameter Estimates from the Regression of Y ⴝ Score on the First

Exam on X ⴝ Math Diagnostic Score for 213 Students in Introductory Statistics

Explanatory Variable b σ

Constant 35.494 12.840 2.764 .006

Math diagnostic score 2.749 .313 8.788 .518  .001

Model Summary Measure Value

F 77.236

.268

adj

.265

213.756

c02.qxd 8/27/2004 2:47 PM Page 50

Exam Scores. The slope for the regression of exam performance on math diagnostic

score is 2.749. This implies that those who are 1 unit apart on the diagnostic are, on

average, about 2



points apart on exam performance. Moreover, someone with a perfect

score of 45 on the diagnostic is expected to get a grade of yˆ 35.494  2.749(45) 

88.2 on the ﬁrst exam. Notice the uninterpretability of the intercept in this case, since it

is impossible to have a negative score on the exam. Moreover, as the lowest observed

score on the diagnostic was 28, trying to predict exam scores for those with diagnostic

scores of zero—albeit a plausible value for the diagnostic—would be extrapolating

beyond the range of observed X-values for these data. In all likelihood, the relationship

between exam performance and diagnostic score becomes nonlinear at lower diagnos-

tic scores, since zero on the exam is a “ﬂoor” below which no one can go.

Couple Modernism. From Table 2.3 we see that the estimated intercept for the regres-

sion of couple modernism on male’s years of schooling is 24.274. In this case, the

intercept is somewhat meaningful, since there is a case in which the male has zero

years of schooling. In this instance, the predicted score for couple modernism is

24.274. Given that there is only one observation in which the male has zero years of

ESTIMATION USING SAMPLE DATA 51

Table 2.3 Parameter Estimates from the Regression of Y ⴝ Couple

Modernism Score on X ⴝ Male’s Years of Schooling for 416 Couples

from the National Survey of Families and Households, 1987–1988

Explanatory Variable b σ

Constant 24.274 .490 49.522 .001

Male’s years of schooling .210 .038 5.586 .265 .001

Model Summary Measure Value

F 31.204

.070

adj

.068

5.751

Table 2.4 Parameter Estimates from the Regression of Y ⴝ Frequency of

Having Sex in Past Year on X ⴝ Frequency of Going to Bars for 1515

Respondents from the General Social Survey, 1998

Explanatory Variable b σ

Constant 2.252 .086 26.206 .001

Frequency of going to bars .224 .029 7.708 .194 .001

Model Summary Measure Value

F 59.417

.038

adj

.037

3.696

c02.qxd 8/27/2004 2:47 PM Page 51

schooling, however, the prediction of couple modernism at this level is probably not

very robust. On the other hand, for males with four years of college—an X-value with

several observations on Y—the predicted modernism score is 24.274  .21(16) 

27.634, a more robust estimate. The slope estimate of .21 suggests that each additional

year of schooling for the male partner is expected to increase the couple’s level of mod-

ernism by about two-tenths of a point.

Frequency of Sex. The estimates for the regression of frequency of sex on going to

bars (Table 2.4) suggest that those who are 1 unit apart in frequenting bars are about

.224 unit apart on sexual frequency during the previous year, on average. As neither

Y nor X in this case is measured in a meaningful metric, the value of this slope is not

very intuitive. Perhaps it suﬃces just to notice that there is, indeed, a positive rela-

tionship between going to bars and having sex. Notice, again, that the intercept is not

particularly meaningful, since zero is not a possible value for the independent vari-

able. Those who go to bars almost every day are predicted to have a sexual frequency

during the past year, on average, of 2.252  .224(7)  3.82. This corresponds roughly

to having sex about once per week.

Estimating

σσ

and

ρρ

. Two other quantities connected to the linear regression model

are of interest to estimate. The ﬁrst is σ

, the variance of the error terms at each X-

value. The second is denoted ρ

(pronounced “rho-squared”) and is called the

coeﬃcient of determination of the regression equation. The latter is the primary index

of discriminatory power for a regression model. To develop estimators of these quan-

tities, let’s consider partitioning the variability in Y, based on the assumption that it is

a linear function of X in the population. Hence, if Y  β

 β

X  ε (again, omitting

the i subscript for simplicity), then

V(Y)  V(β

 β

X  ε)  Cov[(β

 β

X)  ε,(β

 β

X)  ε]

 Cov(β

 β

X, β

 β

X)  Cov(β

 β

X, ε)

 Cov(ε, β

 β

X)  Cov(ε,ε)

 V(β

 β

X)  V(ε),

since Cov(X,ε)  0 by assumption [and, of course, Cov(β

,ε)  0, since the covari-

ance of a constant with a variable is always zero]. What this shows is that the vari-

ability in Y can be neatly partitioned into two parts. The ﬁrst, V(β

 β

X), is the

variability of the regression line itself, or the variance of the set of points that forms

that line. This is the variance that is due to the linear function involving X, or the

variance of the structural part of the model. The second, V(ε), is the variance around

that line, and reﬂects the variability in Y due to the stochastic part of the model—all

of the presumably random inﬂuences on Y other than its linear relationship with X.

Now if we divide both sides of the equation V(Y )  V(β

 β

X)  V(ε) by V(Y),

we have



(

)



 1 



V(β



)







(

ε)

)



 ρ





)



. (2.5)

52 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 52

Equation (2.5) shows that the total variability in Y consists of two proportions: ρ

(which can also be written as 1  σ

/V(Y )) is the proportion of variability in Y

accounted for by the linear regression on X (i.e., by the structural part of the model)

and σ

/V(Y ) is the proportion accounted for by error. Thus, ρ

reﬂects our ability to

account for variation in Y using a linear function of X, and as such, is our ideal meas-

ure of discriminatory power for linear regression. The value of ρ

ranges from 0, for

the case in which X has absolutely no ability to account for Y, to 1.0, for the case in

which Y is perfectly determined by X (i.e., all the points lie exactly on the regression

line and there is no error). Typically, ρ

will range somewhere between these two

extremes.

Decomposition of Variability in the Sample. The variability in Y among the sample

observations can be decomposed in a similar manner. Here, however, the variability

in Y is expressed in terms of the sum of squares in Y, or



i1

 y



)

. This expres-

sion is the same as the numerator of the sample variance of Y and is referred to as the

total sum of squares in Y, denoted TSS. It can be decomposed as follows:



i1

 y



)





i1

 yˆ

 yˆ

 y



)

Now, a word about this ﬁrst step. You’ll notice that I’ve added and subtracted yˆ

from

the term inside the parentheses of TSS. This is a well-known mathematical “trick,” if

you will, that is equivalent to adding zero to an expression. Naturally, adding zero

doesn’t change anything. Now notice, however, what this facilitates. Continuing yields





i1

[(yˆ

 y



)  ( y

 yˆ

)]





i1

[(yˆ

 y



)

 ( y

 yˆ

)

 2( yˆ

 y



)(y

 yˆ

)]





i1

(yˆ

 y



)



i1

 yˆ

)

 2



i1

(yˆ

 y



)(y

 yˆ

The last term in this expression is zero. Why? Well, ﬁrst this expression can be

written



i1

( yˆ

 y



)( y

 yˆ

)  2



i1

[yˆ

 yˆ

)  y



( y

 yˆ

)]

 2



i1

yˆ

 yˆ

)  2y





i1

 yˆ

)

 2



i1

 b

)( y

 yˆ

)  2y





i1

 yˆ

)

 2b



i1

 yˆ

)  2b



i1

( y

 yˆ

)  2y





i1

 yˆ

). (2.6)

ESTIMATION USING SAMPLE DATA 53

c02.qxd 8/27/2004 2:47 PM Page 53

At this point, recall the normal equations



i1

 b

)  0 and



i1



 b

)  0, or



i1

( y

 yˆ

)  0 and



i1

( y

 yˆ

)  0. In that the OLS b

and

solve these equations, they also make both equations true. Therefore, since



i1

( y

 yˆ

)  0 and



i1

( y

 yˆ

)  0, expression (2.6) is also zero. Thus, we

have the following decomposition of the total sum of squares in Y:



i1

 y



)





i1

(yˆ

 y



)



i1

 yˆ

)

(2.7)

That is, TSS is the sum of two independent, or orthogonal, sums of squares. The ﬁrst,



i1

(yˆ

 y



)

, is called the regression sum of squares and denoted RSS. This is the

squared deviation of the regression line, represented by yˆ

, around the overall sample

mean of Y. This is the contribution of the linear relationship with X to the variability in

Y. The second sum of squares is the sum of squared errors, or SSE. This is the squared

deviation of the individual Y scores around the regression line. This is the component

of variability in Y that is not accounted for by Y’s linear relationship with X.

Dividing both sides of equation (2.7) by



i1

 y



)

, we have

 1 

(2.8)

1 







Hence, the decomposition of the variability in the sample Y-values mirrors the decom-

position of the variability in Y in the population. Equation (2.8) shows that the sample

variability in Y consists of two proportions, analogous to the proportions in equation

(2.5). Our estimate of ρ

is therefore RSS/SSE, or 1  (SSE/TSS), and is denoted r

(pronounced “r-squared”). Like its population counterpart ρ

, r

ranges from 0 to 1.

Also, our unbiased estimator of σ

is σ

 SSE/(n  2) and is typically denoted MSE

for mean-squared error. This estimate is shown at the bottom of Tables 2.2 to 2.4.

Interpreting r

. Let’s consider the sample coeﬃcient of determination, r

, in more

detail, as it is relied on heavily as a measure of how well the model performs. (r

often referred to as a measure of model ﬁt. However, assessing ﬁt, or empirical con-

sistency, is better accomplished via other approaches considered below.) The r

can be

considered a proportional reduction in error measure, indicating the degree to which

we improve our prediction of Y when using, as opposed to ignoring, the regression

model. To see this, we reason as follows. First, we try to predict the individual Y scores

while ignoring the model, assessing prediction error via squared deviation from the

predicted value. Our best prediction, lacking any other information, is achieved by

using the sample mean of Y as the predicted value for every case. Our total error is then

TSS. Then, we try to predict Y using the regression model. Our total error is now SSE.



i1

yˆ

)





i1

y



)



i1

(yˆ

y



)





i1

y



)



i1

y



)





i1

y



)

54 SIMPLE LINEAR REGRESSION

c02.qxd 8/27/2004 2:47 PM Page 54