Calculating the Least Squares Line in 2D
by Nick V. Flor (nickflor@unm.edu) • March 17, 2015 • @ProfessorF
It can be difficult for people with no math backgrounds to follow the math (written by mathematicians on the internet) for the equation of the best line that passes through a set of N points. So, I decided to do this math proof.
PRELIMINARIES
For starters, imagine a cloud of points, and a line that passes through those points. In your imagination, you should see some points hit the lines, while most of the points do not fall exactly on the line. Recall that the equation of a line is:
y=mx+b
Our goal is to calculate m, the slope of the line, and to calculate b, the y-intercept.
For any point that does not fall exactly on the line—say x’,y’— The error (e) is:
e=y’-(mx’+b)
Why? Because the equation of the line would calculate y as mx’+b. However, we know that the true value that corresponds to x’ is y’. Thus, the difference is y’-y, which is e above.
Okay, so next let’s square the error to get a positive value, and let’s get rid of the ‘ to reduce clutter:
e²=(y-mx-b)*(y-mx-b)
e²=y²-2xym-2yb+m²x²+2xbm+b²
The idea of squaring the error is attributed to Gauss in the 19th century. And it’s a genius move as you’ll see later. As an aside: it’s amazing what “old” scientists were able to accomplish without the aid of calculators and computers.
Now let’s add up all the errors in the set of points, and call this total error E:
E=Σy²-2mΣxy-2bΣy+m²Σx²+2bmΣx+Nb²
What makes squaring the error and summing all errors a genius move by Gauss?
Well if you know calculus, you know that if you take derivatives of formulas and set them equal to zero, you can find minimal points on the curve (think parabola). Thus, we can find that point in the equation where the error is minimized for m and for b, by taking partial derivatives with respect to m and b.
CALCULATE M
To calculate m, take the partial derivative of E with respect to m and set it equal to 0, then solve for m:
∂E/∂m(Σy²-2mΣxy-2bΣy+m²Σx²+2bmΣx+Nb²)=0
-2Σxy+2mΣx²+2bΣx =0
m=(2Σxy-2bΣx)/2Σx²
m=(Σxy-bΣx)/Σx² or b=(Σxy-mΣx²)/Σx
CALCULATE B
To calculate b, take the partial derivative of E with respect to b and set it equal to 0, then solve for b:
∂E/∂b(Σy²-2mΣxy-2bΣy+m²Σx²+2bmΣx+Nb²)=0
-2Σy+2mΣx+2bN=0
b=(2Σy-2mΣx)/2N
b=(Σy-mΣx)/N or
m=(Σy-Nb)/Σx
We’re not done because we have a kind of catch-22—our m equation contains b and our b equation contains m.
RECALCULATE M
To fix this and find m, set the two b equations equal to one another:
(Σxy-mΣx²)/Σx=(Σy-mΣx)/N
NΣxy-NmΣx²=ΣxΣy-m(Σx)²
m(Σx)²-NmΣx²=ΣxΣy-NΣxy
m
((Σx)²-NΣx²)=ΣxΣy-NΣxy
m=(ΣxΣy-NΣxy)/((Σx)²-NΣx²)
RECALCULATE B
Next to find b, set the two m equations equal to one another:
(Σxy-bΣx)/Σx²=(Σy-Nb)/Σx
ΣxΣxy-b(Σx)²=Σx²Σy-NbΣx²
b(NΣx²-(Σx)²)=Σx²Σy-ΣxΣxy
b=(Σx²Σy-ΣxΣxy)/(NΣx²-(Σx)²)
And we are done! So to find the equation of a line that passes through your point cloud consisting of N points, calculate
Sx=Σx
Sy=Σy
Sxy=Σxy
Sxx=Σx²
m=(SxSy-NSxy)/((Sx)²-NSxx)
b=(SxxSy-SxSxy)/(nSxx-(Sx)²)
y=mx+b
