Of great importance in all that follows are the fundamental concepts of orthogonality and orthogonal projection. We briefly characterize them here, in the familiar euclidean case, in order to more fully appreciate the brain twisting beauty of their natural extension into infinite dimensional probability spaces a little later.
Two vectors (in the same vector space) are orthogonal if
their inner product is zero. This is written 
In a Euclidean space, this simply means the (smallest) angle between them is a right angle. We will also want to talk about orthogonality relations between vectors and entire spaces. Following the above definition naturally,
Given a vector x and a closed subspace S, both in some larger Hilbert Space,
we say x is orthogonal to S, written 
, if
.
Thus, as a quick example, in 
, the x-axis is orthogonal to the yz-plane, but not
to the plane x=y
Given a Hilbert Space 
, a closed subspace 
,
and a vector 
, we may uniquely decompose x into orthogonal components
in 
and its orthogonal
complement. That is,
where 
and 
are uniquely determined by ,
and are orthogonal.
Equivalently, we might say
By 
we will mean the Linear Span of
.
The ability to decompose any vector into orthogonal components leads
naturally to the definition of a special linear transormation: the
orthogonal projection onto a subspace. By the above theorem, we know
that given any vector x and subspace S there exists a unique
element, say y, in S such that x-y is orthogonal to S. We call
it the orthogonal projection of x on S, written 
.
This should seem familiar. We perform these decomositions under various rubrics and aliases all the time. In elementary mechanics, we decompose a particle's momentum and position into independant components along the appropriate axes. Taking the real part of a complex valued function would seem to be no more than the projection of the plane onto the line.
Thomas Pynchon supplies us with a further example of this in his book ``V.''. If we imagine that
we are looking at the rotation of the Earth about the Sun from a point in the plane
of the ecliptic, from far enough away that depth information is lost, we would observe
the rotation as motion in a one dimensional space. We have projected the two-dimensional system of
the Earth's rotation 
onto a one-dimensional subspace - the line crossing through the center of the sun
perpendicular to the line of our observation. Note that when we talk about these projections, we're generally
talking about the relationship with the observer - we're not changing the system at all, just how we're looking at it.
The following facts about projections will be used quite often:
, where the norm is induced by the
inner product, ie, 
.
.
These results find direct application in so called Gramm-Schmidt
Orthogonalization. Often, the space generated by a set of vectors will
be of greater interest to us than the vectors themselves. This being
the case, we may seek out another set of vectors generating the same
space which is easier to work with. Generally, we will wish to reduce
a given basis down to an orthogonal basis with identical span. We do
this as follows, given 
:
By simple application of the above properties of projections, we see that this definition will indeed yield the fact that
as desired, so that the set is pairwise orthogonal.