Alfred DeMaris - Regression with Social Data, Modeling Continuous and Limited Response Variables
Подождите немного. Документ загружается.
whereas
x
13
[].
However, most of the time, the dimensions of a matrix or a vector will be omitted
from the symbol, for expressive economy.
A matrix can be shown in more compact notation by showing only its columns,
written as vectors. For example, if the columns of X are denoted
x
1
冤冥
, x
2
冤冥
, and x
3
冤冥
,
we can write X as
X [x
1
x
2
x
3
],
where in this case, the subscripts on the columns indicate the column numbers rather
than their dimensions. Notice that when we write X as [x
1
x
2
x
3
], it has the form
of a row vector. In fact, a matrix could be described as a vector whose individual
elements are themselves vectors. Writing X as [x
1
x
2
x
3
] is also referred to as
partitioning X by its columns. This is not the only way to partition a matrix. It can
also be partitioned by submatrices within the overall matrix. However, we will fre-
quently find it useful to partition X by its columns (or its rows) and then to treat it in
subsequent operations as though it is a vector, which, in fact, it is. Examples will be
seen shortly.
Similarly, if the rows of X are denoted
x
1
[],x
2
[ ], and x
3
[],
we can also write X as
X
冤冥
.
In this case, X is shown as partitioned by its rows and has the appearance of a col-
umn vector, which, again, it is. It is a column vector whose individual elements are
themselves row vectors. In general, I use x
i
to denote the ith column of a matrix, X,
and x
i
to denote the ith row of that matrix. This notation allows an unequivocal lan-
guage for representing rows and columns of a matrix, as well as their transposes,
without ambiguity. As noted above, the transpose of a column vector is that column
written as a row. Hence,
x
1
[].
The transpose of a row vector is, conversely, that vector written as a column. In that
the row vector already has a transpose sign, transposing it again eliminates the
123
x
1
x
2
x
3
369258147
7
8
9
4
5
6
1
2
3
123
MATHEMATICS TUTORIALS 475
bapp01.qxd 30.8.04 11:46 Page 475
transpose sign:
冢
x
1
冣
x
1
冤冥
.
Note that for any matrix X, x
i
, the ith column of the matrix, and x
i
, the transpose
of the ith row of the matrix, are not generally equal. This is easily seen in the exam-
ple matrix X above, since
x
1
冤冥
冤冥
x
1
.
Two vectors are equal only if they consist of exactly the same elements. Two matri-
ces are equal only when they have the same elements and these elements are in the
same positions in the matrix. For example, X
冤冥
is not equal to
Y
冤冥
, even though both matrices consist of the same elements.
The transpose of a matrix X, denoted X, is a matrix whose rows are the columns
of X, or equivalently, whose columns are the rows of X. For example, Y immediately
above is the transpose of X. That is, Y X, since the first column of X,
冤冥
, forms
the first row of Y, and so on.
The transpose of a partitioned matrix is the transposed matrix of transposed sub-
matrices. This sounds complicated. Let’s see what it means, by finding the transpose of
X again, but this time writing X partitioned by its columns. That is, X [x
1
x
2
x
3
],
so
X
冤冥
冤冥
Y.
Notice that X is created by transposing the matrix X from a row vector to a column
vector, while at the same time transposing all of the submatrices, which are column
vectors, into row vectors. We could achieve the same result by partitioning X by its
rows and then transposing it:
X
冤冥
implies that Xx
1
x
2
x
3
冤冥
.
Here we see that X has been created by transposing X from a column to a row
vector while transposing the rows into columns.
3
6
9
2
5
8
1
4
7
x
1
x
2
x
3
3
6
9
2
5
8
1
4
7
x
1
x
2
x
3
1
2
3
3
6
9
2
5
8
1
4
7
7
8
9
4
5
6
1
2
3
1
4
7
1
2
3
1
4
7
476 MATHEMATICS TUTORIALS
bapp01.qxd 30.8.04 11:46 Page 476
A square matrix is one with the same number of rows as columns. X and X
above are square matrices since they each have three rows and three columns. In
general, a square matrix is of the form A
nn
, where n is the number of rows and
columns in the matrix. The diagonal of a square matrix consists of the elements in
the ith row and ith column for i 1, 2, ..., n. For example, the diagonal of X is
冤冥
. (We follow the convention of denoting the diagonal of a matrix as a column
vector.) Notice that these are the elements in, respectively, the first row, first column;
the second row, second column; and the third row, third column, of X. The trace of
a square matrix, or tr(A), for A square, is the sum of the diagonal elements. Hence
tr(X) 1 5 9 15.
C. Working with Matrices
Vectors. Adding or subtracting two vectors consists of the elementwise addition or
subtraction of the components of two vectors, provided that they are of the same
order. For example, if a
冤冥
and b
冤冥
, then a b
冤冥
冤冥
and
a b
冤冥
冤冥
. However, addition and subtraction cannot be performed
for vectors of different orders, or between vectors and matrices, or between scalars
and either vectors or matrices.
Multiplication of vectors takes place in either of two ways. The inner product of
two vectors is the product of a row vector with a column vector of the same order. It
is formed by summing the elementwise products of the corresponding components of
each vector. The result is a scalar. For example, ab [3 5]
冤冥
(3)(2) (5)(4) 26.
The outer product, on the other hand, is the product of a column vector with a row
vector of the same order. The result is a matrix whose elements are all possible
products of the elements of the column vector with the elements of the row vector.
Hence, ab
冤冥
[2 4]
冤冥
冤冥
. Finally, any vector can
be multiplied or divided by a scalar. The result is a vector whose elements are all
multiplied or divided by that scalar. For example, .5a
冤冥
冤冥
.
Matrices. Adding or subtracting two matrices consists of the elementwise addition
or subtraction of the components of two matrices, provided that they are of the same
order. Hence if
A
冤冥
and B
冤冥
,
9
4
3
12
5
8
2
7
1.5
2.5
(.5)(3)
(.5)(5)
12
20
6
10
(3)(4)
(5)(4)
(3)(2)
(5)(2)
3
5
2
4
1
1
3 2
5 4
5
9
3 2
5 4
2
4
3
5
1
5
9
MATHEMATICS TUTORIALS 477
bapp01.qxd 30.8.04 11:46 Page 477
then
A B
冤冥
冤冥
,
and
A B
冤冥
冤冥
.
Matrix multiplication is not simply the elementwise product of corresponding
components in two matrices (although one type of matrix product, the Hadamard
product, is formed in this fashion, but is not used very much). To begin, matrix mul-
tiplication can only take place between two matrices that are conformable for multi-
plication. A and B are conformable if B has the same number of rows as A has
columns. That is, A
rc
and B
cq
are conformable for multiplication since A has c
columns and B has c rows. The result of the product AB is a matrix with r rows and
q columns. Hence, two matrices are conformable if the column dimension of the first
is equal to the row dimension of the second, and the order of their product is then the
row dimension of the first by the column dimension of the second. As an example,
A
32
冤冥
and B
24
冤冥
are conformable, since A has two columns
and B has two rows. Their product, AB, will be a 3 4 matrix.
The product is constructed by taking the inner products of each row vector in A with
each column vector in B. That is, the (i, j)th element of AB is the inner product of the
ith row of A with the jth column of B. Let’s see how this works by computing AB:
The (1,1)th element of AB is the inner product of the first row of A with the first
column of B: [1 2]
冤冥
4. The (2,1)th element of AB is the inner product of the
second row of A with the first column of B: [3 4]
冤冥
10. The (3,1)th element
of AB is the inner product of the third row of A with the first column of
B: [5 6]
冤冥
16. The (1,2)th element of AB is the inner product of the first row
of A with the second column of B: [1 2]
冤冥
10. We continue in this fashion until
we get to the last element of AB, the (3,4)th: [5 6]
冤冥
94. The complete prod-
uct is
AB
冤冥
冤冥
冤冥
.
26
60
94
16
38
60
10
24
38
4
10
16
8
9
6
5
4
3
2
1
2
4
6
1
3
5
8
9
4
3
2
1
2
1
2
1
8
9
6
5
4
3
2
1
2
4
6
1
3
5
4
4
1
5
5 9
8 4
2 3
7 12
14
12
5
19
5 9
8 4
2 3
7 12
478 MATHEMATICS TUTORIALS
bapp01.qxd 30.8.04 11:46 Page 478
Another way to understand this operation is to partition A by its rows and B by
its columns. This treats A like a column vector and B like a row vector. AB is then
an outer product whose individual elements are inner products of rows of A with
columns of B. In particular, if A
冤冥
and B [b
1
b
2
b
3
b
4
] then
AB
冤冥
[b
1
b
2
b
3
b
4
]
冤冥
This result demonstrates the method of matrix multiplication we just articulated.
The product of a matrix, A, with a vector, c, is formed in a similar fashion, except
that a vector consists of only one column, so the process is somewhat simpler. Again,
the process is easy to see if A is partitioned by its rows. In this case, A takes the form
of a column vector and c takes the form of a constant. The result is treated like the
product of a vector with a constant. Hence,
Ac
冤冥
c
冤冥
.
That is, the result is a vector whose elements are inner products of the rows of A with
the vector c. For example, if c
冤冥
, then
Ac
冤冥
冤冥
冤冥
.
Notice that A
32
and c
21
are conformable for multiplication, and the result, Ac, is a
3 1 vector. Finally, the product or the quotient of a matrix with a scalar is formed
by multiplying (dividing) each element of the matrix by that scalar.
D. Special Matrices
A variety of special matrices are used in matrix algebra, and these will be defined
here.
(1) Diagonal Matrix. A diagonal matrix is a square matrix having zeros for all of
its off-diagonal elements. For example:
D
冤冥
is a diagonal matrix.
Premultiplying a matrix A by a diagonal matrix D results in a matrix whose rows
are multiplied by the corresponding diagonal elements in D.
0
0
9
0
5
0
3
0
0
19
43
67
5
7
2
4
6
1
3
5
5
7
a
1
c
a
2
c
a
3
c
a
1
a
2
a
3
a
1
b
4
a
2
b
4
a
3
b
4
a
1
b
3
a
2
b
3
a
3
b
3
a
1
b
2
a
2
b
2
a
3
b
2
a
1
b
1
a
2
b
1
a
3
b
1
a
1
a
2
a
3
a
1
a
2
a
3
MATHEMATICS TUTORIALS 479
bapp01.qxd 30.8.04 11:46 Page 479
Example:
冤冥冤冥
冤冥
冤冥
.
Postmultiplying a matrix A by a diagonal matrix D results in a matrix whose
columns are multiplied by the corresponding diagonal elements in D.
Example:
冤冥
冤冥
冤冥
冤冥
.
(2) Null Matrix. A null matrix is a matrix with all elements zero. It will be distin-
guished from the scalar zero by placing it in boldface type.
Example: 0
冤冥
is a null matrix.
Analogous to the rule in scalar algebra, A 0 A and A0 0A 0 for any
matrix A, provided that A and 0 are conformable for these respective operations.
(3) Identity Matrix. An identity matrix, denoted by I, is a square matrix with 1’s on
the diagonal and zeros everywhere else.
Example: I
冤冥
is an identity matrix of order 3.
Identity matrices are analogous to the scalar 1: AI IA A for any matrix A.
(4) Idempotent Matrix. An idempotent matrix A is one with the property that
A
2
A.
Example. Identity matrices are all idempotent, since I
2
II I. The second equality
in this last expression follows from the property that any matrix is unchanged when
multiplied by an identity matrix.
(5) Symmetric Matrix. A symmetric matrix A has the property that A A.
Example:
冤冥
is symmetric, since if you transpose it, you get the same
matrix again. Covariance and correlation matrices for a set of K variables are always
symmetric matrices.
(6) Orthogonal Matrix. An orthogonal matrix A has the property that A
A AAI. To elaborate further, we must first define normality and orthogonality
with respect to vectors. A vector x is normal if xx 1. First, note that xx is simply
1234
2569
3678
4981
100
010
001
0
0
0
0
4(
1
2
)
8(
1
2
)
2(
1
3
)
6(
1
3
)
2(0) 4
(
1
2
)
6(0)
1
8
(
1
2
)
2(
1
3
) 4(0)
6(
1
3
)
1
8(0)
0
1
2
1
3
0
4
8
2
6
(
1
3
)4
(
1
2
)8
(
1
3
)2
(
1
2
)6
(
1
3
)4 (0)8
(0)4 (
1
2
)8
(
1
3
)2 (0)6
(0)2 (
1
2
)6
4
8
2
6
0
1
2
1
3
0
480 MATHEMATICS TUTORIALS
bapp01.qxd 30.8.04 11:46 Page 480
the sum of squared elements of the vector x (as the reader can easily verify for him-
self or herself, using any vector x). Thus, a normal vector is such that the sums of
squares of its elements equals 1. Two vectors, x and y, are orthogonal if xy
yx 0. A set of vectors is orthonormal if each vector is normal and they are all
pairwise orthogonal. A square matrix whose columns constitute an orthonormal set
of vectors is an orthogonal matrix.
Example:
P
冤冥
is orthogonal, as is readily verified. The reader can also verify that PP PPI.
(7) Centering Matrix. Centering matrices are used to represent quantities such as
冱
(X X
苶
)
2
for a set of variable scores x
1
, ..., x
n
. To describe this matrix, we must
first define the summing vector. A vector whose elements are all 1’s is represented
as 1 and is referred to as a summing vector. This label arises from the fact that
1x x1
冱
x, as the reader can easily verify with any vector x. For a 1-vector of
order n (i.e., having n elements), the square matrix J
nn
11 is a matrix of order n, all
of whose elements are 1’s. Further, the matrix J
苶
(1/n)J
nn
is a square matrix all of
whose elements are 1/n. A centering matrix, C, is then defined as C (I J
苶
) , where
I is of the same order as J
苶
. Let’s see where the centering matrix gets its appellation.
For a vector x of variable scores, Cx is such that Cx (I J
苶
)x Ix J
苶
x. Now
Ix x [see V.D(3)]. But what is J
苶
x? J
苶
x (1/n)11x 1[(1/n)1x] 1x
苶
, which equals
a vector all of whose elements are x
苶
. Let’s denote this vector of means by x
苶
. Thus,
Cx (I J
苶
)x x x
苶
, a vector consisting of x scores that have been centered or devi-
ated from their means. This formulation is shown below to provide the basis of
matrix formulas for sample variances and covariances.
E. Rules for Matrix Expressions
Rules for matrix expressions are analogous to those for algebraic expressions, with
some key differences. They are:
(1) Commutative property:A B B A.
(2) Associative property 1:A (B C) (A B) C.
(3) Associative property 2: (AB) C A(BC) ABC.
(4) Scalar property: cAx Acx Axc for any scalar c. That is, the order of
multiplication is invariant with respect to scalars.
Rules V.E(1) to V.E(4) are similar to rules in scalar algebra. The following
properties, however, are unique to matrix algebra:
兹
1
3
苶
兹
1
3
苶
兹
1
3
苶
兹
1
2
苶
兹
1
2
苶
0
兹
1
6
苶
兹
1
6
苶
兹
2
6
苶
MATHEMATICS TUTORIALS 481
bapp01.qxd 30.8.04 11:46 Page 481
(5) Distributive property 1:A(B C) AB AC. Note that A is on the left in
both terms here. This is not just coincidental. The result is not generally
equal to AB CA or BA AC or BA CA. In fact, given conformability
requirements, BA and CA may be undefined.
(6) Distributive property 2:(B C) A BA CA, which is not generally equal
to AB AC.
(7) AB CA cannot be factored.
(8) In scalar algebra a ca is factored as (1 c)a. Similarly, in matrix algebra,
A CA, for A and C being square matrices, is factored as (I C)A, where
I is an identity matrix of the same order as A and C. That is, I is analogous
to the scalar 1 in factoring operations. Also, the expression xx xCx is
factored as x(I C)x. In the same vein, BA 2B is factored as B(A 2I).
(9) Unlike the situation in scalar algebra, AB 0 does not imply that either A
or B is a null matrix.
Example: AB 0 for A
冤冥
and B
冤冥
. Moreover, for
X
冤冥
,X
2
0 even though X is not 0.
(10) (ABC)CBA. That is, the transpose of a product of matrices is the
product of the transposed matrices, in reverse order.
(11) (A B)AB.
(12) tr(A B) tr(A) tr(B).
(13) tr(A) tr(A).
(14) tr(ABC) tr(CAB) tr(BCA). That is, the trace of a matrix product is
invariant to elementwise cycling of individual matrices in the product. Note
that elementwise cycling means the last matrix in the product is “cycled” to
the front of the product, and this process is repeated to produce all possible
nonredundant orderings of the product. The product BAC, for example,
would not constitute an elementwise cycling of ABC.
(15) A scalar equals its own trace [i.e., tr(c) c].
(16) A scalar equals its own transpose [i.e., cc].
F. Matrix Equations and Their Solutions
One of the most valuable uses of matrix algebra is in solving systems of linear equa-
tions. As an example, the following is a system of linear equations in two unknowns,
x
11
and x
21
:
x
11
x
21
20
x
11
x
21
12.
125
2410
1 2 5
1
1
1
1
1
1
1
1
482 MATHEMATICS TUTORIALS
bapp01.qxd 30.8.04 11:46 Page 482
This system of equations can be written as a matrix equation of the form Ax y,
where A is a matrix of constants, x is the vector of unknowns, x
11
and x
21
, and y is
the vector of constants on the right-hand side of the equation. For this system we
have A
冤冥
, x
冤冥
, and y
冤冥
. The reader can verify that the
left-hand side of this system can be expressed as the vector Ax.
The Inverse. Solving this system amounts to finding the inverse matrix of A,
denoted A
1
, if it exists. (Below we consider conditions necessary for the inverse of
a matrix to exist.) The inverse of a square matrix A (the inverse exists only for square
matrices) is the matrix A
1
, with the following property: AA
1
A
1
A I. This is
analogous to the inverse, a
1
, of a scalar, a, since a a
1
a
1
a 1. Multiplication
by the inverse matrix is the scalar analog of division. (Note that dividing by a matrix
is an undefined operation.) The equation ax y has solution x a
1
y, in scalar alge-
bra, because we can multiply both sides of the equation by a
1
to isolate x on the
left-hand side. That is, ax y implies that a
1
ax a
1
y or x a
1
y. similarly, the
matrix equation Ax y is solved by premultiplying both sides of the equation by A
1
:
Ax y implies that A
1
Ax A
1
y, or Ix A
1
y, or x A
1
y. In the example,
A
1
冤冥
(the reader can verify that AA
1
A
1
A I in this case). The
solution is therefore x
冤冥冤冥
冤冥
.
The Determinant. Finding the inverse matrix can be complex and time consuming.
Fortunately, we can let the computer do the actual work for us. Nevertheless, to fur-
ther our understanding, we’ll discuss how to find the inverse of a second-order
matrix, as that is quite simple. First, we need to find the determinant of the matrix.
The determinant of a square matrix A, denoted 冟 A冟, is a scalar value that is related
to the degree of linear independence among the columns of the matrix. (We will
take up the issue of linear independence below.) For any 2 2 matrix A, where
A
冤冥
, 冟A冟 is equal to (a
11
)(a
22
) (a
21
)(a
12
). For higher-order matrices, the
idea is the same, except that the determinant becomes a weighted sum of determi-
nants of lower-order submatrices within the larger matrix.
Example. The determinant of A
冤冥
is 冟A冟 (1)(1) (1)(1) 2.
Finding the Inverse. The inverse of a second-order matrix A, having the form
冤冥
,is
A
1
冟A
1
冟
冤冥
a
11
a
22
1
a
21
a
12
冤冥
.
a
12
a
11
a
22
a
21
a
12
a
11
a
22
a
21
a
11
a
12
a
21
a
22
1
1
1
1
a
11
a
12
a
21
a
22
4
16
20
12
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
20
12
x
11
x
21
1
1
1
1
MATHEMATICS TUTORIALS 483
bapp01.qxd 30.8.04 11:46 Page 483
Here we see that the inverse is the reciprocal of the determinant times a matrix in which
the positions of a
11
and a
22
are interchanged, while the signs of a
12
and a
21
are reversed.
Example. The determinant of A
冤冥
was already found to be 2; hence,
A
1
is
1
2
冤冥
冤冥
,
which agrees with our previous result.
Multiplication Rule. The inverse of a product of matrices is the product of their
individual inverses, in reverse order, provided that the individual inverses exist. That
is, (ABC)
1
C
1
B
1
A
1
, provided that A
1
,B
1
, and C
1
all exist. It is easy to
see why this is true, since (ABC)
1
(ABC) C
1
B
1
A
1
ABC C
1
B
1
IBC
C
1
B
1
BC C
1
IC C
1
C I.
Transpose Rule. The inverse of the transpose is the transpose of the inverse. That
is, (A)
1
(A
1
).
G. Linear Dependence and Rank
Let’s consider again the product Ax. The result is a vector. For a 3 3 matrix A and
a 3 1 vector x,Ax can be represented as
[a
1
a
2
a
3
]
冤冥
[a
1
x
11
a
2
x
21
a
3
x
31
].
This formulation exploits the fact that A can be partitioned by its columns and then
treated like a row vector (which it is). Hence the resulting product, Ax, can be treated
like an inner product of two vectors, giving us the result on the right-hand side of the
equation. The resulting vector, [a
1
x
11
a
2
x
21
a
3
x
31
], is therefore seen to be a lin-
ear combination, or weighted sum, of the columns of A, with the weights being the
components of x. Now if there is a nonnull (i.e., nonzero) vector x such that Ax 0,
then provided that no column of A is null, the columns of A are said to be linearly
dependent. What this means is that one column of A is a linear combination of the
other columns, since Ax 0 implies that a
1
x
11
a
2
x
21
a
3
x
31
0,or
x
11
a
1
x
21
a
2
x
31
a
3
, or a
1
冢
x
x
2
1
1
1
冣
a
2
冢
x
x
3
1
1
1
冣
a
3
.
If there is no nonnull x such that Ax 0, the columns of A are linearly independent.
Linear dependence is especially important in linear regression, where the columns of
the “design matrix,” X, represent one’s independent variables (as we will shortly show).
It is assumed in linear regression that the columns of X are linearly independent. If they
x
11
x
21
x
31
1
2
1
2
1
2
1
2
1
1
1
1
1
1
1
1
484 MATHEMATICS TUTORIALS
bapp01.qxd 30.8.04 11:46 Page 484