Lecture 9: Four Ways to Solve Least Squares Problems

Course Info

Instructor

Prof. Gilbert Strang

Departments

Mathematics

As Taught In

Spring 2018

Level

Undergraduate

Topics

Learning Resource Types

Lecture Videos

Problem Sets

Instructor Insights

Download Course

Video Lectures

Description

In this lecture, Professor Strang details the four ways to solve least-squares problems. Solving least-squares problems comes in to play in the many applications that rely on data fitting.

Summary

Solve $A^{T} A x = A^{T} b$ to minimize $‖ A x - b ‖^{2}$
Gram-Schmidt $A = Q R$ leads to $x = R^{- 1} Q^{T} b$ .
The pseudoinverse directly multiplies $b$ to give $x$ .
The best $x$ is the limit of $(A^{T} A + δ I)^{- 1} A^{T} b$ as $δ \to 0$ .

Related section in textbook: II.2

Instructor: Prof. Gilbert Strang

Problems for Lecture 9
From textbook Section II.2

2. Why do $A$ and $A$

have the same rank? If

A

is square, do

A

and

A

have the same eigenvectors? What are the eigenvalues of

A

8. What multiple of $a = [\begin{matrix} 1 \\ 1 \end{matrix}]$ should be subtracted from $b = [\begin{matrix} 4 \\ 0 \end{matrix}]$ to make the result $A_{2}$ orthogonal to $a$ ? Sketch a figure to show $a$ , $b$ , and $A_{2}$ .

9. Complete the Gram-Schmidt process in Problem 8 by computing $q_{1} = a / ‖ a ‖$ and $q_{2} = A_{2} / ‖ A_{2} ‖$ and factoring into $Q R$ : $[\begin{matrix} 1 & 4 \\ 1 & 0 \end{matrix}] = [\begin{matrix} q_{1} & q_{2} \end{matrix}] [\begin{matrix} ‖ a ‖ & ? \\ 0 & ‖ A_{2} ‖ \end{matrix}]$ The backslash command $A ∖ b$ is engineered to make $A$ block diagonal when possible.