Of great importance in all that follows are the fundamental concepts of orthogonality and orthogonal projection. We briefly characterize them here, in the familiar euclidean case, in order to more fully appreciate the brain twisting beauty of their natural extension into infinite dimensional probability spaces a little later.
Two vectors (in the same vector space) are orthogonal if their inner product is zero. This is written
In a Euclidean space, this simply means the (smallest) angle between them is a right angle. We will also want to talk about orthogonality relations between vectors and entire spaces. Following the above definition naturally,
Given a vector x and a closed subspace S, both in some larger Hilbert Space, we say x is orthogonal to S, written , if
Thus, as a quick example, in , the x-axis is orthogonal to the yz-plane, but not to the plane x=y
Given a Hilbert Space , a closed subspace , and a vector , we may uniquely decompose x into orthogonal components in and its orthogonal complement. That is,
where and are uniquely determined by , and are orthogonal.
Equivalently, we might say
By we will mean the Linear Span of .
The ability to decompose any vector into orthogonal components leads naturally to the definition of a special linear transormation: the orthogonal projection onto a subspace. By the above theorem, we know that given any vector x and subspace S there exists a unique element, say y, in S such that x-y is orthogonal to S. We call it the orthogonal projection of x on S, written .
This should seem familiar. We perform these decomositions under various rubrics and aliases all the time. In elementary mechanics, we decompose a particle's momentum and position into independant components along the appropriate axes. Taking the real part of a complex valued function would seem to be no more than the projection of the plane onto the line.
Thomas Pynchon supplies us with a further example of this in his book ``V.''. If we imagine that we are looking at the rotation of the Earth about the Sun from a point in the plane of the ecliptic, from far enough away that depth information is lost, we would observe the rotation as motion in a one dimensional space. We have projected the two-dimensional system of the Earth's rotation onto a one-dimensional subspace - the line crossing through the center of the sun perpendicular to the line of our observation. Note that when we talk about these projections, we're generally talking about the relationship with the observer - we're not changing the system at all, just how we're looking at it.
The following facts about projections will be used quite often:
, where the norm is induced by the inner product, ie, .
These results find direct application in so called Gramm-Schmidt Orthogonalization. Often, the space generated by a set of vectors will be of greater interest to us than the vectors themselves. This being the case, we may seek out another set of vectors generating the same space which is easier to work with. Generally, we will wish to reduce a given basis down to an orthogonal basis with identical span. We do this as follows, given :
By simple application of the above properties of projections, we see that this definition will indeed yield the fact that
as desired, so that the set is pairwise orthogonal.