svdfinal

Matrices as Arrays

The simplest way we think of matrices is as arrays of numbers, with an associated way to 'multiply' them, yeilding new such arrays:

[\begin{matrix} □ & □ & □ \\ □ & □ & □ \\ □ & □ & □ \end{matrix}] \otimes [\begin{matrix} □ & □ & □ \\ □ & □ & □ \\ □ & □ & □ \end{matrix}] = [\begin{matrix} □ & □ & □ \\ □ & □ & □ \\ □ & □ & □ \end{matrix}]

This 'multiplication' can also 'apply' a matrix

M

to a vector

\vec{V}

M \otimes \vec{V} = [\begin{matrix} □ & □ & □ \\ □ & □ & □ \\ □ & □ & □ \end{matrix}] \otimes (\begin{matrix} □ \\ □ \\ □ \end{matrix}) = (\begin{matrix} □ \\ □ \\ □ \end{matrix})

Provide an 'inner product' mapping two vectors to a scalar:

(\begin{matrix} □ & □ & □ \end{matrix}) \otimes [\begin{matrix} □ \\ □ \\ □ \end{matrix}] = [□]

Or even an Outer Product, seldom discussed but very useful and interesting, which takes two vectors and returns a matrix:

[\begin{matrix} □ \\ □ \\ □ \end{matrix}] \otimes (\begin{matrix} □ & □ & □ \end{matrix}) = [\begin{matrix} □ & □ & □ \\ □ & □ & □ \\ □ & □ & □ \end{matrix}]

Matrices as Functions

Another way to think of a matrix is as a 'mapping' from vectors to vectors. This can be thought of an actual transform (generally, a scaling or rotation) or as a change in coordinate system.
For example, the following matrix 'projects' a two-dimensional plane onto the line

y = - x

[\begin{matrix} 1 & - 1 \\ - 1 & 1 \end{matrix}] \otimes [\begin{matrix} x \\ y \end{matrix}] = [\begin{matrix} x - y \\ y - x \end{matrix}]

A very important point is that information is being lost in this transform. For example, the point

(2,2)

is the 'image' of both

(5,3)

and

(12,10)

in the original space. The fact that this mapping 'loses' a dimension is reflected in the fact that the two rows of the matrix are (trivially) linearly dependant. This matrix has 'rank' ONE, since it can only transform ONE dimension without losing information.

Note that a rectangular matrix of size

m \times n, m > n

can have at most rank n.

Matrices as Datasets

Another useful way to think of matrices is as observations of Random Variables. For example, if we have two random variables

x

and

y

, with three observations each, we can tabulate our data as

\vec{X} = (\begin{matrix} x_{1} \\ x_{2} \\ x_{3} \end{matrix})

or even put both variables together as

D = [\begin{matrix} x_{1} & y_{1} \\ x_{2} & y_{2} \\ x_{3} & y_{3} \end{matrix}]

. It is common and convenient to 'center' these data matrices by substracting off the mean for each variable (the vector of means is easily added back in when appropriate. With data arranged this way, our familiar matrix operations become very useful; Our inner product yeilds the variance (a fact used to great effect in the geometry of statistics):

X' \otimes X = (\begin{matrix} (x_{1} - μ_{x}) & (x_{2} - μ_{x}) & (x_{3} - μ_{x}) \end{matrix}) \otimes (\begin{matrix} x_{1} \\ x_{2} \\ x_{3} \end{matrix}) = \sum_{i} {(x_{i} - μ_{i})}^{2} = cov (x)

Extending this to multiple variables brings us into the land of covariance matrices, the fundamental object in applied statistics:

D' \otimes D = (\begin{matrix} x_{1} & x_{2} & x_{3} \\ y_{1} & y_{2} & y_{3} \end{matrix}) \otimes [\begin{matrix} x_{1} & y_{1} \\ x_{2} & y_{2} \\ x_{3} & y_{3} \end{matrix}] = [\begin{matrix} var (x) & cov (x, y) \\ cov (y, x) & var (y) \end{matrix}]

Note that these covariance matrices are always real, symmetric, and positive semi-definate.

Eigenvalues and Eigenvectors

An eigenvector

V

associated with a matrix

M

is a vector whose image under

M

is a simple scaling, thus:

M \otimes V = λ V

Note that every eigenvalue is thus associated with a 1-dimensional linear subspace defined by it\'s eigen vector, since

\forall k \in ℝ

M \otimes \vec{V} = λ \vec{V} \Rightarrow \vec{V} \otimes (k V) = k λ \vec{V}

Examples

Another way to think of the line defined by the eigenvector is as an invariant subspace - every 'point' along that line is mapped to that line by the action of

M

.

In dynamical systems, the 'acceleration' or 'flow' is often represented as a field of vectors, each assigned to a position

\vec{X} = (x_{1}, y_{1})

by the equation

\vec{X} = M \otimes \vec{X}

, usually approximated as

\vec{X_{i + 1}} = \vec{X_{i}} + M \otimes \vec{X}

This means the dynamics along these lines is known - and by 'continuity', they must be 'similar nearby'. Thus, understanding the eigenstuff yields understanding about the overall behavior of the system:

M = [\begin{matrix} - 2 & 1 \\ 1 & - 2 \end{matrix}]

has eigenvectors

\vec{V_{1}} = (1,1)

and

\vec{V_{2}} = (1, - 1)

, with eigenvalues

λ_{1,2} = - 1

(why?). The phase portrait is 'fixed' by the dynamics of these subspaces:

Diagonalization

A fundamental result is that, for every n-dimensional real, symmetric matrix

M

, there exists an orthogonal matrix

U

and a diagonal matrix

D

such that

M = U' \otimes D \otimes U

and the diagonal elements of

D

are

λ_{1} \geq . . . \geq λ_{n}

(the ordered eigenvalues of

M

) corresponding to

(\vec{U_{1}}, . . ., \vec{U_{n}})

the columns of

U

(the eigenvectors of

M

).

This is huge.

One way to think of this is that every (full-rank) matrix has a coordinate system (

U

is essentially a change of coordinates) in which the action of the matrix is a simple scaling along the coordinate axes.

Another is as a way of 'decomposing' a matrix. Reflection and hand-waving will show that the above 'factoring' of

M

allows us to write M as the sum of simple (

n \times n

)-matrices determined by the eigensystem:

M = λ_{1} \vec{U_{1}}' \otimes \vec{U_{1}} + . . . + λ_{n} \vec{U'_{n}} \otimes \vec{U_{n}}

Note further that since the

{U_{i}}_{i \in 1 . . n}

are all the same 'length' (part of the orthonormal condition on

U

), the relative contributions of the matrices are determined by the size of the eigenvalues

{λ_{i}}_{i \in 1 . . n}

.
Thus, an excellent approximation to

M

may be achievable using only the first few vectors of the eigensystem. This is especially useful if the eigenvectors describe important behaviors of the system, as they always seem to.

Quadratic Forms

A common construction with a matrix

M

is in a Quadratic Form, thusly:

y = \vec{X}' \otimes M \otimes \vec{X}

note that

y

is a scalar. Thus we are generating a real-valued function of the vector

\vec{X}

. This can be a wieghted norm or average, or arise as a 'squared-distance' function.

An important and useful result is that the maximum and minimum of this function of

\vec{X}

are

λ_{1}

and

λ_{2}

, acheived at their corresponding eigenvectors

U_{1}

and

U_{n}

, respectively.
More specifically,

\frac{\vec{X}}{2}

{sup}_{\vec{X}} \frac{\vec{X'} \otimes M \otimes \vec{X}}{\vec{X'} \otimes \vec{X}} = U_{1} for some k\inℝ

Since the

{U_{i}}

form a basis, we can write any

X

as in terms of them:

\vec{X} = C_{1} \vec{U_{1}} + . . . + C_{N} \vec{U_{n}}

which allows us to us to write the value of the (normalized) Quadratic Form as:

\frac{\vec{X'} \otimes M \otimes \vec{X}}{\vec{X'} \otimes \vec{X}} = \frac{C_{1} λ_{1}^{2} + . . . + C_{n} λ_{n}^{2}}{λ_{1}^{2} + . . . + λ_{n}^{2}}

It is clear upon examination this is maximized as

λ_{1}

C = (1,0,0, . . .)

; that is, when

\vec{X} = \vec{U_{1}}

. Similar treatments show the minimum is acieved at

\vec{U_{n}}

Least Squares Theory

Many classical and useful problems in statistics reduce to the system

\vec{Y} = M \otimes \vec{X}

\vec{Y}

may be a vector of observations, and we seek to estimate the value of

\vec{X}

for known parameters

M

. Note that we may well have many more observations than Random Variables, so this system may be underdetermined; that is, the

M

may not be a real symmetric matrix. The Least Squares Estimate of

\vec{X}

will minimize

(Y - M \otimes \vec{X})' \otimes (Y - M \otimes \vec{X})

Taking derivatives w.r.t.

\vec{X}

yields the Normal Equation:

(M' \otimes M) \otimes \vec{X} = M' \otimes \vec{Y}

Since

(M' \otimes M)

is (generally) a real, symmetric matrix, we can write its spectral decompostion

(M' \otimes M) = λ_{1} U_{1}' U_{1} + . . . + λ_{n} U_{n}' U_{n}

which in turn provides us with a (psuedo) inverse:

{(M' \otimes M)}^{-} = \frac{1}{λ_{1}} U_{1}' U_{1} + . . . + \frac{1}{λ_{n}} U_{n}' U_{n}

so we can solve the system:

\vec{X} = {(M' \otimes M)}^{-} M' \otimes \vec{Y}

Fisher Linear Discriminant

A classic problem in statistics involves finding a linear dsicriminant, that is, a linear combination of the features of a datapoint whose sign classifies it into one of two (or more) categories. For example, a datapoint

\vec{X}

might represent a particular tissue sample, the features

(x_{1}, . . ., x_{n})

the expression levels of various genes, and the classes a clinical outcome. We seek a

\vec{w}

such that

\vec{w}' \otimes \vec{X} is {\begin{matrix} > 0 \forall \vec{X} \in Class A \\ < 0 \forall \vec{X} \in Class B \end{matrix}

or as close as can be found.

Fisher's measure of the discriminating ability of such a

\vec{w}

is given by

J (\vec{w}) = \frac{{(μ_{Class A} - μ_{Class B})}^{2}}{σ_{A}^{2} + σ_{B}^{2}}

that is, the ratio of the (squared) distance between class means to the projected variance for the classes determined by choice of

\vec{w}

. If

\vec{X_{A}}

is the centered datapoints in class A, define the Scatter Matrix of A as

S_{A} : = \vec{X_{A}}' \otimes \vec{X_{A}} = \sum_{\vec{x} \in A} (\vec{x_{i}} - μ_{A})' \otimes (\vec{x_{i}} - μ_{A}) = n_{A} Σ_{A}

IE, simple the covariance matrix for the class weighted by the number of elements in the class

n_{A}

. If we then define the Scatter-Within matrix as

S_{W} = S_{A} + S + A

to represent the scatter within the classes, and the Scatter Between matrix relating the two classes as

S_{B} = (\vec{μ_{A}} - \vec{μ_{B}})' \otimes (\vec{μ_{A}} - \vec{μ_{B}})

then we can wewrite the discriminant function as

J (\vec{w}) = \vec{w}' (S_{B} \otimes {S_{W}}^{-}) \vec{w}

but this is just a quadratic form. So we know the best linear discriminant

\vec{w}

is given by the first eigenvector of

S_{B} \otimes S^{-_{W}}

. This is given by

\vec{w} = {S_{W}}^{-} (μ_{A} - μ_{B})

which can be seen as intuitive.

Singular Value Decomposition

So, eigensystems tell you everything you need to go. What else could be needed? Well, nothing. Unfortunately, most matrices encountered in the real world are not real and symmetric. For example, in statistical applications, we often have many more observations than variables, so the data matrices are not square.
So, we cannot generally diagonalize our matrix and work with the Spectral Decompostion. The Singular Value Decomposition should be seen as a way to come as close as possible to this enviaable state of things.

In short, for any matrix

M

, we can find orthogonal matrices

U

and

V

, and diagonal matrix

D

, such that:

M = U \otimes D \otimes V

where the rows of

V

are an orthonormal basis for the rows of

M

, the columns of

U

are an orthonormal basis for the coloumns of

M

, and the diagonal elements of

D

are the Singular Values of

M

. The singular values are the closest to eigenvalues we can get for a singular matrix

M

; they are the eigenvalues of

\sqrt{M' \otimes M}

, and correspond to the Left and Right Singular Vectors making up

U

and

V

respectively, just as do eigenvalues and eigenvectors.

So the situation is akin to a diagonalized matrix, but we are unable to use a single coordinate tranform to get into the 'optimized' space where the action of the matrix is easy to identify. Instead, we have two coordinate transforms - one for the rows, and one for the columns. It's an elegant solution to the Curse of Dimensionality, and a powerful one.

There is an interesting and useful duality between

U

and

V

. Suppose we wish to find out what linear combination of solumns from

M

gave rise to one of the right singular vectors (the columns of

U

). We'd like to get

U

in terms of

M

M \otimes {(D V)}^{-} = U

In other words,

{(D V)}^{-}

will tell us how to build the columns of

U

from those of

M

. This is useful when we find something interesting and want to relate it back to the raw data. Note that

(D V)

is very easy to invert.

For any matrices

A

and

B

{(A B)}^{-} = B^{-} A^{-}

- multiply them out and see. Now,

D^{-}

is easy: for any diagonal matrix

D = diag (d_{1}, . . ., d_{n}) \Rightarrow D^{-} = diag (1 / d_{1},...,1 / d_{n})

.
The inverse of

V

brings out the beauty of the SVD. We know that

V

is orthogonal. This implies lots of nice things, but the definition is that

V^{-} = V'

. In other words, the linear combination of columns from

M

used to make the

i^{t h}

left singular vector (column of

U

) is given by the i^{th} row of

V

, scaled by the

i^{t h}

singular value.

Finally, it is worth noting that if the original matrix

M

is centered along the rows, then the LSVs are the Principal Components of the rows of

M

, with

D V

giving the PCA 'coordinates'. Similarly, centering the columns yields the PCAs in

V

Image Processing Example

A color-indexed or RGB digital image is essentially a big matrix (or three), and as such provides an excellent example of some of the properties of the SVD. We can make use of the ordering of the singular values to reduce the data needed to describe the most 'important' aspects of the system. Specifically, we can write the following SVD version of the Spectral Decomposition:

M = d_{1} \vec{U_{1}} \otimes \vec{V_{1}} + d_{2} \vec{U_{2}} \otimes \vec{V_{2}} + . . . + d_{n} \vec{U_{n}} \otimes \vec{V_{n}}

where

d_{i}

\vec{U_{i}}

, and

\vec{V_{i}}

are the i^{th} Singular Value, Left Singular Vector (column of

U

), and Right Singular Vector (row of

V

), respectively.
Just as with the eigenvalues in a Spectral Decomposition, the first few Singular Values will be the largest, and thus these terms will make the largest 'contributions' to

M

. In fact, the matrix constructed by adding the first

k

such terms is the rank-k matrix closest to

M

, in terms of least-squares distance between elements. This provides a simple compression or feature extraction technique.
The following image is the blue channel from a 650 \times 600 RGB image. We convert the elements to floats, contruct the SVD, and then recreate the image using the approximation above for various values of

k

. Note that for, say,

k = 10

, we are using (at most)

k * 600 * 2 + 1 = 1201

numbers, rather then the original

600 * 650 = 390,000

, a significant compression.

SVD for Microarray Data

Microarray data, with its massive datasets and high dimensionality, can be a fruitful domain for the application of SVD techniques. We demonstrate here one such approach.
This example uses a synthetic dataset for clarity of exposition. We simulate 100 hybridizations, the last 10 of which are associated with some phenotype of interest. We have 1000 genes, the last 900 of which are slightly upregulated in the above 10 samples. Note that the increase in mean expression in this set is less than the general variances of the datapoints, so the upregulation is very difficult to detect naively.
We then calculate the SVD. Note that the columns of

M

are arrays (chips, hybridizations).
We would like to find an eigen-array (Left Singular Value) that focuses on the behavior we are interested in. Recall that the 'recipe' for each LSV is given by the corresponding row of

V

. So we calculate the correlation of each of these columns to our 'phenotype vector'. This correlation is maximal for the second eigen-array, and examination of its 'recipe' shows that it is indeed capturing the overall differences between the two pheontype classes.
So we know that the second eigenarray is a coordinate axis for the information we're looking for. We now need only find genes whose 'component' along this axis is large, indicating that they may be differentially expressed between the classes.