Like any other vector space, 
has a dual space of functionals (ie, mappings from a vector
space to its set of scalars) 
. In fact, since it is an infinite-dimensional Hilbert Space, it
is self dual. One very important functional is the expectation. This can be thought of as the
center of mass (or moment) of a function on the state space. That is,
Where P(x) is the measure used in the definition of x as a Random variable. In the usual terminology
P(x) is called the Probability Density Function, and 
.
We can define an inner product in by taking the expected value of the standard inner
product in 
,
Note that this is called the Covariance by statisticians,and that the Norm in induced by
this inner product is simply the standard
deviation.
We will generally just work with the variance, 
. Also of great interest is the induced distance
function,
though again, we generally encounter 
in this context. This is because, if we consider x to
be in some sense an estimate of y, then a natural measure of error is the distance function. We
use its square for convenience, since it works just as well. That is, minimizing (called the ``Sum Of
Squares'' error) minimizes d(x,y) as well.
Another construction of great utility here is the Outer Product, which is a function of two random variables yielding their covariance matrix. This is defined as
Note that this is simply the covariance matrix of x with y, though we often also write, for notational convenience,
which is, of course, the variance matrix as it is defined in standard statistics texts .
It is worth making explicit some properties of this operation which follow directly from
the definition. Given 
and A,B matrices of appropriate dimension to pre-multiply
them with X and Y, respectively:
These properties will be used again
and again in what follows.
In fact, it is this outer product we will use here to define orthogonality.
We call two vectors 
orthogonal if
[x,y]=0
Why are we revising our definition? Because it covers more general situations, and frees us from worrying unneccessarily about dimensionality. We really aren't changing anything, just generalizing. In the case where the two vectors corresspond to vectors of equal dimension in the state space, we note that the Trace of the Outer Product equals the inner product,
So that if the Outer Product is the Zero Matrix, the Inner Product is the Zero Scalar. So the definition coincides exactly with the usual definition of orthogonality in terms of the inner product.
If we cannot directly calculate the inner product due to seemingly conflicting dimensionality, this more general definition still
makes sense. Consider our original construction
of as the Cartesian Product of 
, or 
. The 
element
of [x,y] gives the (scalar) covariance between the 
component of x and the 
of y. So if [x,y] = 0 it means that every component is probabalistically orthogonal to each
component of the other vector. That is, looking at everything as happenning in 
, with one-dimensional
random variables, the vectors are component-wise orthogonal.
Note that independence implies orthogonality. If both RV's are Gaussian, the converse can be shown
to be true as well.
exercise Characterize the unit hypersphere in , 
.
That is, the set of Random Variables with Covariance one, or Trace([x])=1.
Of slightly more interest is a Sphere around a specific point in ,
So we have some (infinite dimensional) manifold in this wild space of random variables. What dynamics can we imagine sliding across the surface of this bubble of causality, shimmering in the piercing light of reason? What atlas maps its surface?
The orthogonal projection of 
on 
(assuming, for simplicity, both are
zero-mean) is the random variable
proof
By the uniqueness of the orthogonal projection on a subspace, we need only demonstrate an
element of 
such that
which requires that
a simple case of the celebrated Wiener-Hopf equation. Now, since 
, we can
express it as Ay where A is some dim(x) by dim(y) matrix.
So we can solve for A, possibly using psuedo-inverses, which yields:
