\\section{Least Squares Theory}

Many classical and useful problems in statistics reduce to the system

$$\\vec{Y} = M \\CircleTimes \\vec{X}$$

$\\vec{Y}$ may be a vector of observations, and we seek to estimate the value of $\\vec{X}$ for known parameters $M$. Note that we may well have many more observations than Random Variables, so this system may be underdetermined; that is, the $M$ may not be a real symmetric matrix. The \\it{Least Squares Estimate} of $\\vec{X}$ will minimize 

$$( Y - M \\CircleTimes \\vec{X})\' \\CircleTimes ( Y - M \\CircleTimes \\vec{X} )$$

Taking derivatives w.r.t. $\\vec{X}$ yields the \\it{Normal Equation}:

$$(M\'\\CircleTimes M)\\CircleTimes \\vec{X} = M\' \\CircleTimes \\vec{Y}$$

Since $(M\'\\CircleTimes M)$ is (generally) a real, symmetric matrix, we can write its spectral decompostion 
$$(M\'\\CircleTimes M) = \\lambda_1 U_1\' U_1 + ... + \\lambda_n U_n\' U_n$$ which in turn provides us with a (psuedo) inverse:
$$(M\'\\CircleTimes M)^{-} = \\frac{1}{\\lambda_1} U_1\' U_1 + ... + \\frac{1}{\\lambda_n} U_n\' U_n$$ so we can solve the system:
$$\\vec{X} = (M\'\\CircleTimes M)^- M\' \\CircleTimes \\vec{Y}$$