The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or to view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
OCW.MIT.edu.
PROFESSOR: OK.
We are talking about jointly Gaussian random variables.
One comment through all of this and through all of the
notes is that you can add a mean to Gaussian random
variables, or you can talk about
zero-mean random variables.
Here we're using random variables mostly
to talk about noise.
When we're talking about noise, you really should be
talking about zero-mean random variables, because you can
always take the mean out.
And because of that, I don't like to state everything twice
once for variables of processes without a mean, and
once for variables of processes with a mean.
And after looking at the notes, I think I've been a
little inconsistent about that.
I think the point is you can keep yourself straight by just
saying the only thing important is the
case without a mean.
Putting the mean in is something unfortunately done
by people who like complexity.
And they have unfortunately got in the
notation on their side.
So anytime you want to talk about a
zero-mean random variable.
You have to say zero-mean random variable.
And if you say random variable, it means it could
have a mean or not have a mean.
And of course the way the notation should be stated is
you talk about random variables as things which
don't have means.
And then you can talk about random variables plus means as
things which do have means, which would make life easier
but unfortunately it's not that way.
So anytime you see something and are wondering about
whether I've been careful about the mean or not, the
answer is I well might not have been.
And two, it's not very important.
So anyway.
Here I'll be talking all about zero-mean things.
A k-tuple of zero-mean random variables is said to be
jointly Gaussian if you can express them in this way here.
Namely as a linear combination of IID normal Gaussian again
random variables.
OK.
In your homework, those of you who've done it already, have
realized that just having two Gaussian random variables is
not enough to make those two random
variables be jointly Gaussian.
They can be individually Gaussian
but not jointly Gaussian.
This is sort of important because when you start
manipulating things as we will do when we're generating
signals to send and things like this, you can very easily
wind up with things which look Gaussian and are appropriately
modeled as Gaussian, but which are not jointly Gaussian.
Things which are jointly Gaussian are
defined in this way.
We will come up with a couple of other
definitions of them later.
But you've undoubtedly been taught that any old Gaussian
random variables are jointly Gaussian.
You've probably seen joint densities for them
and things like this.
Those joint densities only apply when you have jointly
Gaussian random variables.
And the fundamental definition is this.
This fundamental definition makes sense, because the way
that you generate these noise random variables is usually
from some very large underlying set of very, very
small random variables.
And the Law of Large Numbers says that when you add up a
very, very large number of small underlying random
variables, you normalize the sum so it has some reasonable
variance then that random variable is going to be
appropriately modeled as being Gaussian.
If you at the same time look at linear combinations of
those things for the same reason, it's appropriate to
model each of the random variables you're looking at as
a linear combination of some underlying set of noise
variables which is very, very large.
Probably not Gaussian.
But here when we're trying to do this, because of the
central limit theorem, we just model these as normal Gaussian
random variables to start with.
So that's where that definition comes from.
It's why when you're looking at noise processes in the real
world, and trying to model them in a sensible way this
jointly Gaussian is the thing you almost
always come up with.
OK.
So one important point here, and the notes point this out.
If each of these random variables Z sub i is
independent of each of the others, and they have
arbitrary variances, then because of this formula, the
set of Z i's are going to be jointly Gaussian.
Why is that?
Well you simply make a matrix here A, which is diagonal.
And the elements of this diagonal matrix are sigma 1
squared, sigma 2 squared, sigma 3 squared, and so forth.
If you have a vector Z, which is then sigma 1 squared down
to sigma k squared times a noise vector N1 to N sub k.
What you wind up with is Z sub i is going to be equal to --
I guess I don't want those squares in there.
Sigma 1 up to sigma k --
Z sub i is going to be equal to sigma i, N sub i; N sub i
is a Gaussian random variable with variance one and
therefore Z sub i as Gaussian random variable with variance
sigma sub i squared.
OK.
So anyway, one special case of this formula is anytime that
you want to deal with a set of independent Gaussian random
variables with arbitrary variances, they are always
going to be jointly Gaussian.
Saying that they're uncorrelated is
not enough for that.
You really need the statement that they're independent of
each other.
OK.
That's sort of where we were last time.
When you look at this formula, you can look at it in terms of
sample values.
And if we look at a sample value what's happening is that
the sample value of the random vector Z, namely little z, is
going to be defined as some matrix A times this sample
value for the normal vector N. OK.
And what we want to look at is geometrically
what happens there.
Well this matrix a is going to map a unit vector, E sub i
into the i'th column of A. Why is that?
Well you look at A, which is whatever it is, A sub 1, 1,
blah, blah, blah up to A sub k, k.
And you look at multiplying it by a vector which is zero only
in the i'th position.
And what's this matrix multiplication going to do?
It's going to simply pick out the i'th column
of this matrix here.
OK.
So a is going to map e sub i into the i'th column of A. OK.
So then the question is what is this matrix A going to do
to some small cube in the n plane?
OK.
If you take a small cube in the n plane from 0 to delta
along the n1 line is going to map into 0 to this point here,
which is A e sub 1.
This point here is going to map into A times e sub 2.
This is just drawing for two dimensions of course.
So in fact all the points in this cube are going to get map
into this little rectangle here.
OK.
Namely, that's what a matrix times a vector is going to do.
Anybody awake out there?
You're all looking at me as if I'm totally insane.
OK.
Every everyone following this?
OK.
So perhaps this is just too trivial.
I hope not.
OK.
So unit cubes get mapped into rectangles here.
If I take a unit cube up here, it's going to get mapped into
the same kind of unit rectangle here.
If I visualize tiling this plane here with little tiny
cubes, delta on a side, what's going to happen?
Each of these little cubes is going to map into a
parallelogram over here, and these parallelograms going to
tile this space here.
OK.
Which means that each little cube here maps into one of
these rectangles.
Each rectangle here maps back into a little cube here.
Which means that I'm looking at a very
special case of a here.
I'm looking at a case where a is non-singular.
In other words, I can get the any point in this plane by
starting with some point in this plane.
Which means for any point here, I can go back here also.
In other words, I can also write n is equal to A to the
minus 1 times Z. And this matrix has to exist.
OK.
That's what you mean geometrically by a
non-singular matrix.
It means that all points in this plane get mapped into
points in this plane.
And get mapped into only one point in this plane, and get
mapped into only one point in this plane.
Where every point in this plane is the map
of some point here.
In other words you can go from here to there.
You can also go back again.
OK.
The volume of a parallelepiped here, and this is in an
arbitrary number of dimensions is going to be the determinant
of A. And you all know how to find determinants.
Namely you program a computer to do it, and the
computer does it.
I mean it used to be we had to do this by an enormous amount
of calculation.
And of course nobody does that anymore.
OK.
So we can find the volume of this parallelepiped, it's this
determinant.
And what does all of that mean?
Well first let me ask you the question.
What's going to happen if the determinant of a
is equaled to zero?
What does that mean geometrically?
What does that mean here in terms of this
two dimensional diagram?
AUDIENCE: [INAUDIBLE]
PROFESSOR: What?
AUDIENCE: Projection onto a line.
PROFESSOR: Yeah.
This little cube here is simply going to get projected
onto some line here.
Like for example that.
In other words it means that this matrix is not invertible
for one thing, but it also means everything here gets
mapped onto some lower dimensional
sub-space here in general.
OK.
Now remember that for a minute, and we'll come back to
that when we start talking about probability densities.
OK because of the picture we were just looking at, the
density of the random variables Z at A times N,
namely the density of Z at this particular value here is
just the density that we get corresponding to the density
of some point here mapped over into here.
And what's going to happen when we take a point here and
map it over into here?
If you have a certain amount of density here, which is
probability per unit volume.
Now when you map it into here, and the determinant is bigger
than zero, what you're doing is mapping a little volume
into a big volume.
And if you're doing that over small enough region where the
probability density is of such essentially fixed, what's
going to happen to the probability density over here?
It's going to get scaled down, and it's going to get scaled
down precisely by that determinant.
OK.
So what this is saying is the probability density of the
random variable z, which is linear combination of these
normal random variables, is in fact the probability density
of this normal vector N, and we know what that probability
density is, divided by this determinant of a.
In fact this is a general formula for any old
probability density at all.
You can start out with anything which you call a
random vector N and you can derive the probability density
of any linear combination of those elements in
precisely this way.
So long as this volume element is non-zero, which means
you're not mapping an entire space into a sub-space.
When you're mapping an entire space into a sub-space and you
define density as being density per unit volume, of
course the density in this map space doesn't exist anymore.
Which is exactly what this formula says.
If this determinant is zero, it means this density here is
going to be infinite in the regions where this z exists at
all, which is just in this linear sub-space, and what do
you do about that?
I mean do you get all frustrated about it?
Or do you say what's going on and treat it in
some sensible way?
I mean the thing is happening here.
And this talks about it a little bit.
If a is singular, then A is going to map Rk
into a proper sub-space.
Determinant A is going to be equal to 0.
The density doesn't exist.
So what do you do about it?
I mean what does this mean if you're mapping
into a smaller sub-space.
What does it mean in terms of this diagram here?
Well in the diagram here it's pretty clear.
Because these little cubes here are getting mapped into
straight lines here.
Yeah?
What?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Some linear combinations of this are being
mapped into 0.
Namely if this straight line is this way any Z which is
going in this direction is being mapped into 0.
Any vector Z which is going in this direction has to be
identically equaled to 0.
In other words some linear combination of n1 and n2 is a
random variable which takes on the value 0 identically.
Why do you try to represent that in a probabilistic sense?
Why don't you just take it out of consideration altogether?
Here what it means is that z1 and z2 are simply linear
combinations of each other.
OK.
In other words once you know what the sample value of z1
is, you can find the sample values z2.
In other words z2 is a linear combination of z1.
It's linearly dependent on z1, which means that you can
identify it exactly once you know what z1 is.
Which means you might as well not call it a
random variable at all.
Which means you might as well view this situation where the
determinant is 0 as where the vector Z is really just one
random variable, and everything else is determined.
So you throw out these extra random variables,
pseudo-random variables, which are really just linear
combinations of the others.
So you deal with a smaller dimensional set.
You find the probability density of the smaller
dimensional set, and you don't worry about all of these
mathematical peculiarities that would arise otherwise.
OK.
So once we do that, A is going to be non-singular.
Because we're going to be left with a set of random
variables, which are not linearly
dependent on each other.
They can be statistically dependent on each other, but
not linearly dependent, OK.
So for all z then, since determinant A is not 0, the
probability density at some arbitrary vector Z is going to
be the normal joint density evaluated at a to the minus 1z
divided by the determinant of A. OK.
In other words we're just working the map backwards.
This formula is the same as this formula, except instead
of writing An here, we're writing z here.
And when An is equal to z then little n has to be equal to A
to the minus 1z.
And what does that say?
It says that the joint probability density has to be
equal to this quantity here.
Which in matrix terms looks rather simple, it
looks rather nice.
You can rewrite this.
This is a norm here, so it's this vector
there times this vector.
Where in fact when you want to multiply vectors in this way,
you're taking inner product of the vector with this cell.
These are real vectors we're looking at.
Because we're trying to model real noise, because we're
modeling the noise on communication channels.
So this is going to be the inner product of A to the
minus 1z with itself, which means you want to look at the
transform of a to the minus 1z.
Now what's the transform of a to the minus 1z?
It's z transform times a to the minus 1 transform.
So we wind up with a to the minus 1 transform times a to
the minus 1 times z.
So we have some kind of peculiar bilinear form here.
So for any sample value of the random vector Z we can find
the probability density in terms of this quantity here.
Which looks a little bit peculiar, but it
doesn't look too bad.
OK.
Now we want to simplify that a little bit.
Anytime you're dealing with zero-mean random variables --
now remember I'm going to forget to say zero-mean half
the time because everything is concerned with zero-mean
random variables.
The covariance of Z1 and Z2 is expected value of Z1 times Z2.
So if you have a k-tuple Z, the covariance is a matrix
whose i,j element is expected value of Zi times Zj.
And what that means is that the covariance matrix, this is
a matrix now, it's going to be the expected value of z times
z transpose.
Z is a random vector, which is a column random vector, Z
transpose is a row-random vector, which is this simply
turned upside down, turned by 90 degrees.
Now when you multiply the components of this vector
together, you can see that what you get is the elements
of this covariance matrix.
In other words this is just standard matrix manipulation,
which I hope most of you or at least partly familiar with.
OK.
When we then talk about the expected values Z times Z
transpose we can write this as the expected value of A times
N, which is what z is.
N transpose times A transpose, which is what Z transpose is.
And miraculously here the N and the N transpose are in the
middle here, where it's easy to deal with them, because
these are normal Gaussian random variables.
And when you look at this column times this row, since
all diagonal elements are independent of each other, and
all of them have variance one, the expected value of N times
N transpose is simply the identity matrix.
All the randomness goes out of this which it obviously has to
because we're looking at a covariance which is just a
matrix and not something random.
So you wind up with this covariance matrix is equal to
a rather arbitrary matrix A, but not singular, times the
transpose of that matrix.
OK.
We've assumed that A is non-singular and therefore
it's not too hard to see the k sub Z is non-singular also.
And explicitly case of Z mainly its co-variance matrix,
the inverse of it, is A to the minus 1 transpose times A to
the minus 1.
Namely you take the inverse and you start flipping things
and you do all of these neat matrix things
that you always do.
And you should review them if you've forgotten that.
So that when we write our probability density, which was
this in terms of this transformation here, what we
get is the density of Z is equal to, in place of
determinant of A, we get the square root of the determinant
in the k sub Z. You probably don't remember that, but is
what you get.
And it's sort of a blah.
Here this is more interesting.
This is minus 1/2 z transpose times Kz to the
minus 1 times z.
What does this tell you?
Look at this formula.
Is it just a big formula or what does it say to you?
You got to look at these things and see what they say.
I mean we've gone through a lot of work to derive
something here.
AUDIENCE: [INAUDIBLE]
PROFESSOR: Well it is Gaussian.
Yes.
I mean that's the way we define jointly Gaussian.
But what's the funny thing about
this probability density?
What does it depend on?
What do I have to tell you in order for you to calculate
this probability density for every z you
want to plug in here?
I have to tell you what this covariance matrix is.
And once I tell you what the covariance matrix is, there is
nothing more to be specified.
In other words, a jointly Gaussian random vector is
completely specified by its covariance matrix.
And it's specified exactly this way by
its covariance matrix.
OK.
There's nothing more there.
So this says anytime you're dealing with jointly Gaussian,
the only thing you have to be interested in is this
covariance here.
Namely all you need to have jointly Gaussian is somebody
has to tell you what the covariance is, and somebody
has to tell you also that it's jointly Gaussian.
Jointly Gaussian plus a given covariance specifies the
probability density.
OK.
What does that tell you?
Well let's look at an example where we just have two random
variables, Z1 and Z2.
then expected value of Z1 squared is the upper.
Left hand element of that covariance matrix, which we'll
call sigma 1 squared.
The lower, right hand side of the matrix is K22, which we'll
call sigma 2 squared.
And we're going to let rho be the normalize covariance.
We're just defining a bunch of things here, because this is
the way people usually define this.
So rho will be the normalized cross covariance.
Then the determinant of the Kz is this mess here.
For A to be non-singular, we have to have rho less than 1.
If rho is equal to 1 then this determinant is going to be
equal to 0, and we're back in this awful case that we don't
want to think about.
So then if we go through the trouble of finding out what
the inverse of K sub z is, we find this 1 over 1 minus row
square times this matrix here.
The probability density plugging this in is this big
thing here.
OK what does that tell you?
Well the thing that it tells me is that I never want to
deal with this, and I particularly don't want to
deal with it if I'm dealing with three or more variables.
OK.
In other words the interesting thing here is the simple
formula we had before, which is this formula.
OK.
And we have computers these days which say given nice
formulas like this, their standard computer routines to
calculate things like this.
And you never want to look at some awful mess like this.
OK.
And if you put a mean into here, which you will see in
every textbook on random variables and probability you
ever look at, this thing becomes so ugly that you were
probably convinced before you took this class that jointly
Gaussian random variables were things you wanted to avoid
like the plague.
And if you really have to deal with explicit formulas like
this, you're absolutely right.
You do want to avoid them like the plague, because you can't
get any insight from what that says, or at least I can't.
So I say OK, we have to deal with this.
But yet we like to get a little more insight about what
this means.
And to do this, we like to find a little bit more about
what these bilinear forms are all about.
Those of you who have taken any course on linear algebra
have dealt with these bilinear forms.
And played with them forever.
And those of you who haven't are probably puzzled about how
to deal with them.
The notes have an appendix, which is about two pages long
which tells you what you have to know about these matrices.
And I will just sort of quote those results as we go.
Incidentally those results are not hard to derive.
And not hard to find out about.
You can simply derive them on your own, or you can look at
Strang's book on linear algebra, which is about the
simplest way to get them.
And that's all you need to do.
OK.
We've said the probability density depends on this
bilinear form z transpose times Kz to the
minus 1 time z.
What is this?
Is this a matrix or a vector or what?
How many people think it's a matrix?
How many people think it's a vector?
You think it's a vector?
OK.
Well in a very peculiar sense it is.
How many people think it's a number?
Good.
OK.
It is a number, and it's a number because
this is a row vector.
This is a matrix.
This is a column vector.
And if you think of multiplying a matrix times a
column vector, you get a column vector.
And if you take a row vector times a column
vector, you got a number.
OK.
So this is just a number which depends on little z.
OK.
Kz is called a positive definite matrix.
And it's called a positive definite matrix, because this
thing is always non-negative.
And it always has to be non-negative because this
refers to the --
if I put capital Z in here, namely if I put the random
vector in here Z, then what this is, is the variance of a
particular random variable.
So it has to be greater than or equal to zero.
So anyway K sub z is always non-negative definite.
Here it's going to be positive definite, because we've
already assumed that the matrix A was non-singular, and
therefore the matrix Kz has to be non-singular.
So this has to be positive definite.
So it has an inverse, K sub z minus 1 is also positive
definite which means this quantity is always greater
than zero, if z is non-zero.
You can take these positive definite matrices and you can
find eigenvectors and eigenvalues for them.
Do you ever actually calculate these eigenvectors and
eigenvalues?
I hope not.
It's a mess to do it.
I mean it's just as bad as writing that awful formula we
had before.
So you don't want to actually do this, but it's nice to know
that these things exist.
And because these vectors exist, and in fact if you have
a matrix which is k by k, little k by little k, then
there are k such eigenvectors and they can be chosen
orthonormal.
OK.
In other words each of these Q sub i are orthogonal to each
of the others.
You can clearly scale them, because you scale this and
scale this together.
And it's going to maintain equality.
So you just scale them down so you can make them normalize.
If you have a bunch of eigenvectors with the same
eigenvalue, then the whole linear sub-space formed by
that set of eigenvectors all have the same eigenvalue
lambda sub i.
Namely you take any linear combination of these things
which satisfy this equation for a particular lambda.
And any linear combination will satisfy the
same the same equation.
So you can simply choose an orthonormal set among them to
satisfy this.
And if you look at Q sub i and Q sub j, which have different
eigenvalues, then it's pretty easy to show that in fact they
have to be orthogonal to each other.
So anyway when you do this you wind up with this form becomes
just the sum over i of lambda sub i to the minus 1.
Namely these eigenvalues to the minus 1.
These eigenvalues are all positive.
Times the inner product of z with Q sub i.
In other words, you take whatever vector Z you're
interested in here, you project it on these k
orthonormal vectors Q sub i.
You get those k values.
And then this form here is just that sum there.
So when you write the probability
density in that way --
we still have this here we'll get rid of that
in the minute --
you have e to the minus sum over i, these inner product
terms squared divided by 2 times lambda sub i.
That's just because this is equal to this.
It's just substituting this for this in the formula for
the probability density.
OK.
What does that say pictorially?
Let me show you a picture of it first.
It says that for any positive definite matrix and therefore
for any covariance matrix, you can always find these
orthonormal vectors.
I've drawn them here for two dimensions.
They're just some arbitrary vector q1; q2 has to be
orthogonal to it.
And if you look at the square root of lambda 1 times q1, you
got a point here.
You look at the square root of lambda 2 times q2, you got a
point here.
If you then look at this probability density here, you
see that all the points on this ellipse here have to have
the same sum of z times Q sub i.
OK.
It looked a little better over here.
Yes.
OK.
Namely the points little z for which this is constant are the
points which form this ellipse.
And it's the ellipse which has the axes square root of lambda
i times Q sub i.
And then you just imagine it if it's lined up this way.
And think of what you would get if you were looking at the
lines of equal probability density for independent
Gaussian random variables with different variances.
I mean we already pointed out if the variances are all the
same, these equal probability contours are spheres.
If you now expand on some of the axes, you get ellipses.
And now we've taken these arbitrary vectors, so these
are not in the directions we started out with, but in some
arbitrary directions.
We have q1 and q2 are orthornormal to each other,
because that's the way we've chosen them.
And then the probability density has this form
which is this form.
And the other thing we can do is to take this form here and
say gee this is just a probability density for a
bunch of independent random variables, where the
independent random variables are the inner product of the
random vector Z with Q sub 1 up to the inner product of z
with Q sub k.
So these are independent Gaussian random variables.
They have variances lambda sub i.
And this is the nicest formula for the probability density of
an arbitrary set of jointly density of jointly Gaussian
random variables.
OK.
In other words what this says is in general if you have a
set of jointly Gaussian random variables and I have this
messy form here, all you're doing is looking at them in a
wrong coordinate system.
If you switch them around, you look at them this way instead
of this way, you're going to have independent Gaussian
random variables.
And the way to look at them is found by solving this
eigenvector eigenvalue equation, which will tell you
what these orthogonal directions are.
And then it'll tell you how to get this nice picture that
looks this way.
OK.
OK so that tells us what we need to know, maybe even a
little more than we have to know, about jointly Gaussian
random variables.
But there's one more bonus we get.
And the bonus is the following.
If you create a matrix here B where the i'th row of B is
this vector Q sub i divided by the square root of lambda sub
i, then what this is going to do is the corresponding
element here is going to be a normalized Gaussian random
variable with variance one and all of these are going to be
independent of each other.
OK.
That's essentially what we were saying before.
This is just another way of saying the same thing.
That when you squish this probability density around and
look at it in a different frame of reference, and then
you scale the random variables down or up, what you wind up
with is IID normal Gaussian random variables.
OK.
But that says that Z is equal to B to the
minus 1 times N prime.
Well so what?
Here's the reason for so what.
We started out with a definition of jointly
Gaussian, which is probably not the definition of jointly
Gaussian you've ever seen if you've seen this before.
Then what we did was to say, OK if you have jointly
Gaussian random variables and they're not linearly
independent and none of them are linearly dependent on the
others, then this matrix A is invertible.
From that we derive the probability density.
From the probability density we derive this.
OK.
The only thing you need to get this is
the probability density.
And this says that anytime you have a random vector Z with
this probability density that we just wrote down.
Then you have the property that N prime is equal to BZ,
and Z is equal to B to the minus 1 N prime.
Which says that if you have this probability density, then
you have jointly Gaussian random variables.
So we have an alternate
definition of jointly Gaussian.
Random variables are jointly Gaussian if they have this
probability density.
You can somehow represent them as linear combinations of
normal random variables.
OK.
Then there's something even simpler.
It says that if all linear combinations of a random
vector Z are Gaussian, then Z is jointly Gaussian.
Why is that?
well If you look at this formula here, it says take any
old random vector at all that has a covariance matrix.
From that covariance matrix, we can solve for all of these
eigenvectors.
If I find the appropriate random variables here from
this transformation, those random variables are then
uncorrelated from each other, they are all statistically
independent of each other, And it follows from that, that if
all these linear combinations are Gaussian then Z has to be
jointly Gaussian.
OK.
So we now have three definitions.
This one is the simplest one to state.
It's the hardest one to work with.
That's life you know.
This one is probably the most straightforward, because you
don't have to know anything to understand this definition.
This original definition is physically the most appealing
because it shows why noise vectors actually
do have this property.
OK.
So here's a summary of all of this.
It says if Kz is singular, you want to remove the linearly
independent random variables.
You just take them away because they're uniquely
specified in terms of the other variables.
And then you take the resulting non-singular matrix,
and Z is going to be jointly Gaussian if and only if Z is
equal to AN for some normal random variable N. If Z has
jointly Gaussian density or if all linear combinations of Z
are Gaussian all of this for 0 mean would mean it applies to
fluctuation.
OK.
So why do we have to know all of that about jointly Gaussian
random variables?
Well because everything about Gaussian processes depends on
jointly Gaussian random variables.
You can't do anything with Gaussian processes without
being able to look at these jointly
Gaussian random variables.
And the reason is that we said that Z of t is a Gaussian
process if for every k and every set of time instance,
every set of e, that's t1 to t sub k.
If Z of t1 up to Z of tk is a jointly
Gaussian random vector.
OK.
So that directly links the definition of a Gaussian
process to Gaussian random vectors.
OK.
Supposed the sample functions of Z of t or L2 with
probability one.
I want to say a little bit about this because otherwise
you can't sort out any of these things about how L2
theory connects with random processes.
OK.
So I'm going to start out by just assuming that all these
sample functions are going to be L2 functions with
probability 1.
One way to ensure this is to look only at processes of the
form Z of t equals some sum of Z sub i times ti of t.
OK remember at the beginning we started looking at this
process the sum of a set of normalized Gaussian random
variables times sinc functions times
displaced sinc functions.
You can also do the same thing with Fourier
coefficients or anything.
You got a fairly general set of processes that way.
unfortunately they don't quite work, because if you look at
the sinc functions and you look at noise, which is
independent and identically distributed in time, then the
sample functions are going to have infinite energy.
I mean that kind of process just runs on forever.
It runs on with finite power forever.
And therefore it has infinite energy.
And therefore the simplest process to look at, the sample
functions are non-L2 with probability one, which is sort
of unfortunate.
So we say OK we don't care about that, because if you
want to look at that process a sum of Z sub i times sinc
functions, what do you care about?
I mean you only care about the terms in that expansion, which
run from the big bang until the next big bang.
OK.
We certainly don't care about it before that or after that.
And if we look within those finite time limits, then all
these functions are going to be L2.
Because they just last for a finite amount of time.
So all we need to do is to truncate these things somehow.
And we're going to diddle around a little bit with a
question of how to truncate these series.
But for the time being we just say we can do that.
And we will do it.
So we can look at any processes the form sum of Zi
times phi i of t, where the Zi are independent and the phi i
of t are orthonormal.
And to make things L2, we're going to assume that the sum
over i of Zi squared bar is less than infinity.
OK.
In other words we only take a finite number of these things,
or if we want to take infinite a number of them, the
variances are going to go off to zero.
And I don't know whether you're proving it in the
homework this time or you will prove it in the homework next
time, I forget, but you're going to look at the question
of why this finite variance condition makes these sample
functions be L2 with probability one.
OK.
So if you had this condition, then all your sample functions
are going to be L2.
I'm going to get off all of this L2
business relatively shortly.
I want to do a little bit of it to start with.
Because if any of you have start doing any research in
this area, at some point you're going to be merrily
working away calculating all sorts of things.
And suddenly you're going to find that none of it exists,
because of these problems of infinite energy.
And you're going to get very puzzled.
So one of the things I tried to do in the notes is to write
them in way that you can understand them at a first
reading without worrying about any of this.
And then when you go back for a second reading, you can pick
up all the mathematics that you need.
So that in fact you won't have the problem of suddenly
finding out that three-quarters of your thesis
has to be thrown away, because you've been dealing with
things that don't make any sense.
OK.
So, we're going to define linear functionals in the
following way.
We're going to first look at the sample functions of this
random process Z. OK.
Now we talked about this last time.
If you have a random process Z than really what you have is a
set of functions defined on some samples space.
So the quantities you're interested in is what is the
value of the random process at time t for sample point omega.
OK.
If we look at that and make it for a given omega, this thing
becomes a function of t.
In fact for a given omega, this is just what we've been
calling a sample element of the random process.
So if we take this sample element, look at the inner
product of that with some function g of t.
In other words we look at the integral of Z of t omega times
g of t, dt.
And if all these sample functions are L2 and if g of t
is L2, what happens when you take the integral of an L2
function times an L2 function, which is the inner product of
something L2 with something L2 which says something with
finite energy inner product with
something with finite energy.
Well the Schwarz inequality tells you that if this has
finite energy and this has finite energy, the inner
product exists.
That's the reason why we went through the Schwarz
inequality.
It's the main reason for doing that.
So these things have finite value.
So V of omega the results of doing this namely V as a
function of the sample space is a real number.
And it's a real number for the sample points of omega with
probability one, which means we can talk about V as a
random variable.
OK.
And now V is a random variable which is defined in this way.
And from now on we will call these things linear
functionals which are in fact the integral of a random
process times a function.
And we can take that kind of integral.
It sort of looks like the linear combinations of things
we were doing before when we were talking about matrices
and random vectors.
OK.
If we restrict the random process to have the following
form, where these are independent and these are
orthonormal, then one of these linear functionals is given by
the random variable V is going to be the integral of Z of t
times g of t, but Z of t is this.
And at this point we're not going to fuss about
interchanging integrals with summations.
You have the machinery to do it, because we're now dealing
with an L2 space.
We're not going to fuss about it.
And I advise you not to fuss about it.
So we have a sum of these random variables here times
these integrals here.
These integrals here are just projections of g of t on this
space of orthonormal functions.
So whatever space of orthonormal functions gives
you your jollies, use it talk about the inner products on
that space.
This gives you a nice inner products space of
sequences of numbers.
And then if the z i are jointly Gaussian, then V is
going to be Gaussian.
And then to generalize this one little bit further, if you
take a whole bunch of L2 functions, g1 of t g2 of t and
so forth, you can talk about a whole bunch of random
variables V1 up to V sub j, 0.
And V sub j is going to be the integral of Z of
t times gj of tdt.
Remember this thing looks very simple.
It looks like the --
like the convolutions you've been doing all your life.
It's not.
It's really a rather peculiar quantity.
This in fact is what we call a linear functional, and is the
integral of a random process times this.
Which we have defined in terms of the sample functions of the
random process.
And now we said OK now that we understand what it is, we will
just write this all the time.
But I just caution you not to let
familiarity breed contempt.
Because this is a rather peculiar notion.
And a rather powerful notion.
OK.
So these things are jointly Gaussian.
We want to take the expected value of V sub i times V sub
j, and now we're going to do this without worrying about
being careful at all.
We have the expected value of the integral of Z of t, gi of
t, td times the integral of Z tau gj of tau, d tau.
And now we're going to slide this expected value inside of
both of these integrals.
And not worry about it.
And therefore what we're going to have is a double integral
of gi of t expected value of z of t time z of tau times gj of
tau dt, d tau, which is this thing here.
Which you should compare with what we've been dealing with
most of the lecture today.
This is the same kind of form for a covariance function as
we've been dealing with for covariance matrices.
It has very similar effects.
I mean before you were just talking about finite
dimensional matrices, which is all simple mathematically in
eigenfunctions and eigenvalues.
You have eigenfunctions and eigenvalues of
these things also.
And so long as these are defined nicely by these L2
properties we've been talking about.
In fact you can deal with these in virtually the same
way that you can deal with the matrices we were
dealing with before.
If you just remember what the results are from matrices, you
can guess what they are for these covariance functions.
OK.
But anyway you can find the expected value of Vi times Vj
by this formula.
Again we're dealing with zero-mean and therefore we
don't have to worry about the mean, put that in later.
And that all exists.
OK.
So the next thing we want to deal with, hitting
you with a lot today.
But I mean the trouble is a lot of this is half familiar
to most of you.
People who have taken various communication courses at
various places have all been exposed to random processes in
some highly trivialized sense.
But the major results are the same as the results we're
going through here.
And all we're doing here is adding a little bit of
carefulness about what works and what doesn't work.
Incidentally in the notes which is towards the end of
lectures 14 and 15, we give three examples which let you
know why in fact we want to look primarily at random
processes which are defined in terms of a sum of independent
Gaussian random variables time orthonormal functions.
And if you look at those three examples, some
of them have problems.
Because of the fact that everything you're dealing with
has infinite energy.
And therefore it doesn't really make any sense.
And one of them I should talk about this just a
little bit in class.
And I think I still have a couple of minutes, is a very
strange process where Z of t is IID.
In fact just let it be normal.
And independent for all t.
OK.
In other words you generate a random process by looking at
an uncountably infinite collection of
normal random variables.
How do you deal with such a process?
I don't know how to deal with it.
I mean it sounds like it's simple.
If I put this on a quiz, three-quarters of you would
say oh that's very simple.
What we're dealing with is a family of impulse functions.
Spaced arbitrarily closely together.
This is not impulse function.
Impulse functions are even worse than this, but this is
bad enough.
When we start talking about spectral density, we can
explain this a little bit better by thinking this kind
of process it doesn't make any sense.
But this kind of process, if you look at its spectral
density, it's going to have a spectral density which is zero
everywhere, but whose integral over all frequencies is one.
OK.
In other words it's not something you want to wish on
your on your worse friend.
It makes a certain amount of sense as a limit of things.
You can look at a very broadband process where in
fact you spread the process out enormously.
You can make pseudo noise which looks sort of like this.
And you make the process broader and broader and lower
and lower intensity everywhere.
But it still has this energy of one.
It still has a power of one everywhere.
And it just is ugly.
OK.
Now if you never worried about these questions of L2, you
would look at a process like that and say, gee there must
be some easy way to handle that because it's probably the
easiest process you can define.
I mean everything is normal.
If you look at any set at different times, you get a set
of IID normal Gaussian variables.
You try to put it together, and it doesn't mean anything.
If you pass it through a filter, the filter is going to
cancel it all out.
So anyway, that's one reason why we want to look at this
restricted class of random processes we're looking at.
OK.
What we're interested in now is we want to take a Gaussian
random process really, but you can take any random process.
We want to pass it through a filter, and we want to look at
the random process that comes out.
OK.
And that certainly is a very physical kind of operation.
I mean any kind of communication system that you
build is going to have noise on the channel.
And one of the first things you're going to do is you're
going to filter what you've received.
So you have to have some way of dealing with this.
OK.
And the way we sort of been dealing with it all along in
terms of the transmitted way forms we've been dealing with
is to say OK.
What we're going to do is to first look what happens when
we take sample functions of this, pass them through the
filter, and then what comes out is going to be some
function again.
And then we're back into what you studied as an
undergraduate talking about functions through filters.
We're going to jazz it up a little bit by saying these
functions are going to be L2 the filters
are going to be L2.
So that in fact you know you'd get something
out that make sense.
So what we're doing is looking at these sample functions V
the output at time tau for sample point omega is going to
be the convolution of the input at time t and sample
point omega.
Remember this one sample point exists for all time.
That's why these sample points and sample spaces are so damn
complicated.
Because they have everything in them.
OK.
So there's one sample point which exists for all of them.
This is a sample function.
You're passing the sample function through a filter,
which is just normal convolution.
What comes out then.
If in fact we express this random process in terms of
this orthonormal sum the way we've been doing before.
Is you get the sum over j of this, which is a sample value
of the j'th random variable coming out times the integral
of pj of t, h of tau minus t, d tau.
OK.
For each tau that you look at, this is just a sample value of
a linear functional.
OK.
If I want to look at this at one value of tau, I have this
integral here which is a random process.
A sample value of a random process at
omega time a function.
This is just a function of t for given tau.
OK.
So this is a linear functional.
As a type we've been talking about.
And that linear functional is then given by
this for each tau.
This is a sample value of a linear
functional we can talk about.
OK.
These things are then, if you look over the whole sample
space omega, V of tau becomes a random variable.
OK.
V of tau is the random variable whose sample values
are V of t omega, and they're given by this.
So if z of t is Gaussian process, you get jointly
Gaussian linear functionals at each of any set of times tau
1, tau 2 up to tau sub k.
So this just gives you a whole set of linear functionals.
And if z of t is a Gaussian process, then all these linear
functionals are going to be jointly Gaussian also.
And bingo.
What we have then is an alternate way to generate
Gaussian processes.
OK.
In other words you can generate a Gaussian process by
specifying at each set of k times, you have jointly
Gaussian random variables.
But once you do that, once you understand one Gaussian
process, you're off and running.
Because then you can pass it through any L2 filter
that you want to.
And you generate another Gaussian process.
So for example, if you start out with this sinc type
process, I mean we'll see that that has a spectral density,
which is flat over all frequencies.
And we'll talk about spectral density tomorrow.
But then you pass it through a linear filter, and you can
give it any spectral density that you want.
So at this point, we really have enough to talk about
arbitrary covariance functions just by starting out with
random processes, which are defined in terms of some
sequence of orthonormal random variables.
OK.
Now we can get to covariance function of a filter process
in the same way as we got the matrix for linear functionals.
OK.
And this is just computation.
OK.
So what is it?
The covariance function of this output process V
evaluated at one time r another time s.
One of the nasty things about notation when you start
dealing with a covariance function of the input and the
output to a linear filter is you suddenly need to worry
about two times at the input and two other
times at the output.
OK.
Because this thing is then expected value of V sub r
times the expected value of V subs.
OK.
This is a random variable, which is the process V of r
evaluated at time r.
This is the random variable, which is the process V of t
evaluated at a particular time s.
This is going to be the expected value of what this
random variable now is the integral of the random process
z of t times this function here, which is now
a function of t.
Because we're looking at a fixed value of r.
So this is a linear functional,
which is a random variable.
This is another linear functional.
This is evaluated at some time s, which is the output of the
filter at time s.
Then we will throw caution to the wind interchange integrals
with expectation and everything.
And then in the middle we'll have expected value of z of t
times z of tau, which is the covariance function of z.
OK.
So the covariance function of z then specifies what the
covariance function of the output is.
OK.
So whenever you pass a random process through a filter if
you know what the covariance function of input to the
filter is, you can find the covariance function of the
output of the filter.
That's a kind of a nasty formula, it's not very nice.
But anyway the thing that it tells you is that whether this
random process is Gaussian or not.
The only thing that determines the covariance function of the
output of the filter is the covariance function of the
input to the filter plus of course the filter response,
which is needed OK.
And this is just the same kind of bilinear form that we were
dealing with before.
Next time we will talk a little bit about the fact that
when you're dealing with a bilinear form like this, you
can take these covariance functions and they have the
same kind of eigenvalues and eigenvectors that we had
before for a matrix.
Namely this is again going to be positive definite as a
function, and we will be able to find its eigenvectors and
its eigenvalues.
We can't calculate them.
Computers can calculate them.
People who've spend their lives doing this
can calculate them.
I wouldn't suggest that you spend your life doing this.
Because again you would be setting yourself up as a
second class computer, and you don't make any
profit out of that.
But anyway, we can find this in principle from this.
OK.
One of the things that we haven't talked about at all
yet, and which we will start talking about next time in
which the next set of lecture notes, lecture 16, we'll deal
with is the question of stationarity.
Let me say just a little bit about that to get into it.
And then we'll talk a lot more about it next time.
The notes will probably be on the web some time tomorrow.
I hope before noon if you want to look at them.
Physically if you look at a stochastic process, and you
want to model it.
I mean suppose you want to model a noise process.
How do you model a noise process?
Well you look at it over a long period of time.
You start taking statistics about it over a
long period of time.
And somehow you want to model it in such a way, I mean the
only thing you can look at is statistics over a
long period of time.
So if you're only looking at one process, you can look at
it for a year and then you can model it, and then you can use
that model for the next 10 years.
And what that's assuming is that the noise process looks
the same way this year as it does next year.
You can go further than that and say, OK I'm going to
manufacture cell phones or some other kind of widget.
And what I'm interested in then is what these noise wave
forms are to the whole collection of my widgets.
Namely different people will buy my widgets.
They will use them in different places.
So I'm interested in modeling the noise over this whole set
of widgets.
But still if you're doing that you're still almost forced to
deal with models which have the same statistics over at
least a broad range of times.
Sometimes when we're dealing with wireless communication we
say, no the channel keeps changing in time. and the
channel keeps changing slowly in time.
And therefore you don't have the same statistics
now as you have then.
If you want to understands that, believe me, the only way
you're going to understand it is to first understand how to
deal with statistics for the channel which
stay the same forever.
And once you understand those statistics, you will then be
in a position to start to understand what happens when
these statistics change slowly.
OK.
In other words, what our modeling assumption is in this
course, and I believe it's the right modeling assumption for
all engineers, is that you never start with some physical
phenomena and say, I want to test the hell out of this
until I find an appropriate statistical model for it.
You do that only after you know enough
about random processes.
That you know how to deal with an enormous variety of
different home cooked random processes.
OK.
So what we do in a course like this is we deal with lots of
different home cooked random processes, which is in fact
why we've done rather peculiar things like saying, let's look
at a random process which comes from a sum of
independent random variables multiplied
by orthonormal functions.
And you see what we've accomplished by that already.
Namely by starting out that way we've been able to define
Gaussian processes.
We've been able to define what happens when one of those
Gaussian processes goes through a filter.
And in fact, that gives a way of generating a lot more
random processes.
And next time what we're going to do is not to say how is it
that we know that processes are stationary.
How do you test whether processes are
stationary or not.
But we're just going to assume that they're stationary.
In other words if they had the same statistics now as they're
going to have next year.
And those statistics stay the same forever.
And then see what we can say about it.
To give you a clue as to how to start looking at this,
remember what we said quite a long time ago
about Markov chains.
OK.
And now when you look at Markov chains, remember that
what happens at one time is statistically a function of
what happened at the unit of time before it.
OK.
But we can still model those Markov change as being
stationary.
Because the dependents at this time on the previous sample of
time is the same now as it will be five years from now.
OK.
In other words you can't just look at the process at one
instant of time and say this is independent
of all other times.
That's not what stationary means.
What stationary means is the way the process depends on the
past at time t is the same is the way it depends on the past
at some later time tau.
And that in fact is the way we're going to define
stationarity.
It's at these joint sample times that we're going to be
looking at.
Which I had a better word for that.
Join sets of epochs that we'll be looking at.
The joint statistics over a set of epics is going to be
the same now as it will be at sometime in the future.
And that's the way we're going to define stationarity.
A prelude of what we're going to find when we do that is
that this covariance function, if the covariance function at
time t and time tau is the same if you translate it up to
t plus t1 and tau plus t1, then in fact this function is
going to be a function only if the difference t minus tau.
It's going to be a function of one variable
instead of two variables.
OK.
So as soon as we get this function being a function of
one variable instead of two variables, first thing we're
going to do is to take the Fourier transform of this.
Because then we'll be taking the Fourier transform of a
function of a single variable.
We're going to call that the spectral
density of the process.
And we're going to find that for stationary processes the
spectrum densities tell you
everything if they're Gaussian.
Why is that?
Well the inverse Fourier transform is
this covariance function.
And we've now seen that the covariance function tells you
everything about a Gaussian process.
So if you know the spectral density for a stationary
process, it will tell you everything.
We will also have to fiddle around a little bit about how
we define stationarity.
But at the same time don't have this
infinite energy problem.
And the way we're going to do it is the way
we've done it all along.
We're going to take something that looked stationary, we're
going to truncate it over some long period of time, and we're
going to have our cake and eat it too that way.
OK.
So we'll do that next time.