WEBVTT 00:00:01.600 --> 00:00:04.220 Our discussion of least mean squares estimation 00:00:04.220 --> 00:00:06.590 so far was based on the case where 00:00:06.590 --> 00:00:09.400 we have a single unknown random variable 00:00:09.400 --> 00:00:11.990 and a single observation. 00:00:11.990 --> 00:00:14.300 And we're interested in a point estimate 00:00:14.300 --> 00:00:16.720 of this single unknown random variable. 00:00:16.720 --> 00:00:20.880 What happens if we have multiple observations or parameters? 00:00:20.880 --> 00:00:22.850 For example, suppose that instead 00:00:22.850 --> 00:00:27.540 of a single observation, we have a whole vector of observations. 00:00:27.540 --> 00:00:29.220 And, of course, we assume that we 00:00:29.220 --> 00:00:32.119 have a model for these observations. 00:00:32.119 --> 00:00:35.510 Once we observe our data, a numerical value 00:00:35.510 --> 00:00:39.300 for this vector, or what is the same numerical values 00:00:39.300 --> 00:00:42.930 for each one of these observation random variables. 00:00:42.930 --> 00:00:46.030 Then we're placed in the conditional universe where 00:00:46.030 --> 00:00:48.790 these values have been observed. 00:00:48.790 --> 00:00:51.780 Then, we notice that the arguments that we carried out 00:00:51.780 --> 00:00:54.660 did not rely in any way on the fact 00:00:54.660 --> 00:00:56.870 that X was one-dimensional. 00:00:56.870 --> 00:00:58.990 Exactly the same argument goes through 00:00:58.990 --> 00:01:02.970 for the multi-dimensional case, and simply, the answer 00:01:02.970 --> 00:01:06.060 is again, that the optimal estimate, the one that 00:01:06.060 --> 00:01:09.120 minimizes the mean squared error, is again, 00:01:09.120 --> 00:01:12.590 the conditional expectation of the unknown random variable 00:01:12.590 --> 00:01:15.810 given the values of the observations. 00:01:15.810 --> 00:01:19.110 So this gives us a simple and much more general 00:01:19.110 --> 00:01:22.070 solution that also applies to the case 00:01:22.070 --> 00:01:24.580 of multiple observations. 00:01:24.580 --> 00:01:28.780 Now, what if we have multiple parameters? 00:01:28.780 --> 00:01:32.950 Once more, the argument is exactly the same, 00:01:32.950 --> 00:01:35.320 and we obtain that the optimal estimate 00:01:35.320 --> 00:01:37.740 of any particular parameter is going 00:01:37.740 --> 00:01:41.070 to be the conditional expectation of that parameter 00:01:41.070 --> 00:01:43.060 given the observations. 00:01:43.060 --> 00:01:50.729 So if our parameter vector is something of this form, 00:01:50.729 --> 00:01:53.670 consisting of several components, 00:01:53.670 --> 00:01:59.700 then the LMS estimate of the jth component of our parameter 00:01:59.700 --> 00:02:03.950 vector is going to be simply the conditional expectation 00:02:03.950 --> 00:02:09.850 of this parameter given the data that we have obtained. 00:02:09.850 --> 00:02:13.180 And this gives us the most general solution 00:02:13.180 --> 00:02:15.480 to the program of least mean squares 00:02:15.480 --> 00:02:20.370 estimation when we have multiple parameters 00:02:20.370 --> 00:02:22.950 and multiple observations. 00:02:22.950 --> 00:02:27.565 One very simple concept that applies to all possible cases. 00:02:30.670 --> 00:02:35.680 Unfortunately, however, our worries are not over. 00:02:35.680 --> 00:02:39.970 Even though LMS estimation has such a simple and such 00:02:39.970 --> 00:02:45.360 a general solution, things are not always easy. 00:02:45.360 --> 00:02:48.180 Let us see what's happening. 00:02:48.180 --> 00:02:52.590 No matter what, we have to first find out 00:02:52.590 --> 00:02:55.680 the posterior distribution of Theta 00:02:55.680 --> 00:02:58.480 given the observations that we have obtained. 00:02:58.480 --> 00:03:02.150 And this is done using the Bayes rule, which we have written 00:03:02.150 --> 00:03:04.530 here, and this is how you evaluate 00:03:04.530 --> 00:03:08.380 the denominator in Bayes' rule. 00:03:08.380 --> 00:03:11.390 What are the difficulties that we may encounter? 00:03:11.390 --> 00:03:15.020 One first difficulty is that in many applications, 00:03:15.020 --> 00:03:18.520 we do not necessarily have a good model 00:03:18.520 --> 00:03:21.950 or we're not very confident about our model 00:03:21.950 --> 00:03:23.890 of the observations. 00:03:23.890 --> 00:03:26.710 If X and Theta are multi-dimensional, 00:03:26.710 --> 00:03:31.560 such a model might be difficult to construct. 00:03:31.560 --> 00:03:36.594 Setting this issue aside, there's a further issue. 00:03:36.594 --> 00:03:39.640 The conditional expectation of Theta 00:03:39.640 --> 00:03:44.160 given X may be a complicated non-linear function 00:03:44.160 --> 00:03:46.300 of the observations. 00:03:46.300 --> 00:03:49.890 This means that it may be difficult to analyze, 00:03:49.890 --> 00:03:52.720 but even more important, it may be 00:03:52.720 --> 00:03:56.140 very difficult to calculate even after you 00:03:56.140 --> 00:03:58.140 have obtained your data. 00:03:58.140 --> 00:04:01.790 Let us understand why this might be the case. 00:04:01.790 --> 00:04:06.100 Suppose that Theta is a multi-dimensional parameter. 00:04:06.100 --> 00:04:09.680 Then in order to calculate the denominator that's involved 00:04:09.680 --> 00:04:13.780 here in the Bayes rule, when you integrate with respect to theta 00:04:13.780 --> 00:04:20.279 , you have to actually carry a multi-dimensional integral, 00:04:20.279 --> 00:04:24.190 and this can be very challenging or sometimes, 00:04:24.190 --> 00:04:27.110 practically impossible. 00:04:27.110 --> 00:04:31.040 Even if you had this denominator term in your hands, 00:04:31.040 --> 00:04:36.420 still, in order to calculate a conditional expectation, 00:04:36.420 --> 00:04:41.230 you would have to calculate once more 00:04:41.230 --> 00:04:46.430 an integral of theta j integrated 00:04:46.430 --> 00:04:49.780 against the posterior distribution of the vector 00:04:49.780 --> 00:04:52.190 theta. 00:04:52.190 --> 00:04:59.040 But this integral should be once more, over all the parameters. 00:04:59.040 --> 00:05:02.640 So it would be a multi-dimensional integral 00:05:02.640 --> 00:05:06.090 in the general case, and that's one additional source 00:05:06.090 --> 00:05:07.940 of difficulty. 00:05:07.940 --> 00:05:10.570 And this is the reason why we will also 00:05:10.570 --> 00:05:14.490 consider an alternative to least mean squares estimation, which 00:05:14.490 --> 00:05:17.890 is much simpler computationally and much less 00:05:17.890 --> 00:05:21.100 demanding in terms of the model that we 00:05:21.100 --> 00:05:23.610 need to have in our hands.