Least Squares in R
Finding the best coefficients, making predictions, and computing MSE
- Making predictions from a model yourself, with
mutate() - Computing errors, squared errors, and MSE, one step at a time
- Letting R find the best coefficients with
lm() - Pulling out the coefficients with
coef() - Making predictions with
predict() - Getting the errors (residuals) with
residuals() - Drawing the least squares line with
geom_smooth() - Bonus: fitting the other models from the notes with
lm()
In the Alternate Notes, we set up a general form of a model, like happiness = ? + ? * gdp, then found good ? values by dragging lines around and watching the MSE go up and down. In this tutorial, we’ll do the same things in R, quickly finding the best coefficients (? values). First we’ll do it the long way, one small step at a time, so you can see exactly what’s going on. Then we’ll spend time with the R function, lm that does it all for us.
Be sure to run all the code cells as you go. The code below is meant to be run in order, and later cells depend on earlier ones.
The data
We’ve already loaded the happiness data from the notes into a data frame called countries. Run the cell below to take a look:
Each row is one country, with four columns:
country: the country’s namecontinent: which continent it’s ongdp: GDP per capita, in thousands of dollars. So agdpof 50 means $50,000.happiness: the country’s average answer to a survey question rating their life from 0 to 10
There are 140 countries in all. Let’s make the same scatter plot from the notes, with ggplot:
Making predictions yourself
Every country gets the same prediction
Remember the very first model from the notes?
happiness = 5
It predicts a happiness of 5 for every country, no matter what. Let’s add those predictions to our data frame as a new column. You’ve used mutate() to add columns before:
(For now, we’re not saving the results back to countries. We’ll do that later when we need to.)
There’s our new prediction column, with a 5 in every row. We gave mutate() a single number, so R used that same number for every row.
Errors
Next, how wrong is each prediction? Remember from the notes, the error is the actual value minus the prediction:
error = actual happiness - predicted happiness
That’s just one more column. Run this:
Look at the first row, Albania. Its actual happiness is about 5.66 and we predicted 5, so its error is about 5.66 - 5 = 0.66. You see that in the first row of the new error column.
Notice: inside mutate(), you can refer to a column you just made (prediction) in a later line of the same mutate(). R makes the columns in order, top to bottom.
Some errors are positive (the country is happier than we predicted) and some are negative (less happy than we predicted).
Squared errors
Next, we square each error:
Squaring makes every value positive (a negative times a negative is a positive). It also makes big errors much bigger. That’s exactly the “punish really bad predictions” idea from the notes. An error of 0.5 becomes 0.25, but an error of 3 becomes 9.
Mean squared error
Last step: take the mean of the squared errors. Turning a whole column into one number is a job for summarize(). We’ll take the data frame we built and pipe it (|>) into summarize():
The MSE of the model happiness = 5 is about 1.60.
That’s the whole recipe for MSE:
- Make a prediction for every row.
- Subtract to get each error.
- Square each error.
- Take the mean.
You’ll use this recipe again and again, so make sure each step makes sense to you before moving on.
Finding the best horizontal line
Try it! In the cell below, change the 5 to other values and run it each time. Can you find a value with a smaller MSE than 1.60?
You’re doing in code exactly what you did by dragging the line in the notes: guess, check the MSE, adjust, repeat.
In the notes, we said the best horizontal line is at the average happiness. Let’s check. Instead of typing in a number, we can make the prediction mean(happiness):
An MSE of about 1.14, our baseline model from the notes. If you’d like to see the average itself:
About 5.68, just like the notes said.
Predictions that depend on gdp
Now the more interesting model from the notes:
happiness = ? + ? * gdp
The notes told us the best coefficients (? values) were about 4.90 and 0.028. Let’s plug those in and use the same recipe. The only thing that changes is the prediction line:
This time, the prediction is different for every country, because each country has its own gdp. For Albania, R took the formula 4.90 + 0.028 * gdp, swapped in Albania’s gdp (about 22.7), and got 4.90 + 0.028 * 22.7, which is about 5.54. Other countries had other gdp values and therefore different predictions.
And now the MSE:
About 0.59. A big improvement over the baseline’s 1.14, just like in the notes.
Try it! What values did you try on in Plot 7 of the notes? Plug them in above in place of 4.90 and 0.028, and see if the MSE matches (it should).
Letting R find the best coefficients: lm()
Where did 4.90 and 0.028 come from?
So far, we’ve been given the coefficients (or guessed them). But how do you actually find the very best ones?
You could keep guessing and checking, like you did when dragging lines in the notes. But that’s tedious, and you’d never know if you’d found the very best values or just pretty good ones. Fortunately, there’s a mathematical shortcut (with a little calculus and linear algebra) that jumps straight to the coefficients with the smallest possible MSE. You don’t need to know the math here. You just need to know how to ask R to do it.
The R function for this is lm(), short for linear model. You might remember it briefly from the Summarizing Numerical Associations tutorial, but we’ll start from scratch here.
Your first lm()
Run this:
Notice the happiness ~ gdp equation doesn’t perfectly parallel the happiness = ? + ? * gdp form we’ve been using. By default, lm assumes you want an intercept. You don’t have to ask for it. So this line is effectively saying, “R, find the best coefficients for the model happiness = ? + ? * gdp.”
You see them in the output above, under Coefficients. (R tends to print lots of decimal places. We’ll round to a couple.)
(Intercept): about 4.90. That’s the first?.gdp: about 0.028. That’s the second?, the number we multiplygdpby.
So R found the best line in this family:
happiness = 4.90 + 0.028 * gdp
That’s the line you were hunting for in Plot 7, found exactly, in one line of code.
Reading the code
Let’s slow down and look at what we typed:
lm(happiness ~ gdp, data = countries)Remember, lm() is a function (the parentheses are a dead giveaway), and functions take arguments (inputs). The arguments are inside the parentheses, separated by commas.
There are two arguments here.
The first argument, happiness ~ gdp, is a special R objectcalled a formula. The squiggle ~ is called a “tilde” (on most keyboards it’s in the top-left corner, with Shift). Read ~ as “as a function of” or “explained by”:
“Model
happinessas a function ofgdp.”
The thing you want to predict (the target variable) goes on the left of the ~. The things you’re predicting it from (the features) go on the right.
Notice that you don’t write any ?s, or the intercept, or the *. When you write happiness ~ gdp, R knows you mean happiness = ? + ? * gdp, and its job is to fill in the ?s.
The second argument, data = countries, says which data frame to look in. R needs to know where to find columns called happiness and gdp.
IMPORTANT: The order matters! happiness ~ gdp and gdp ~ happiness are different models. The second one would predict a country’s gdp from its happiness: gdp = ? + ? * happiness. That’s not what we want, but R will happily fit either one, so remember to put the target variable on the left.
Saving your model
You’ll almost always want to save the model as a variable, so you can use it again:
Nothing printed! Just like when you saved a number as a variable back in the first week, R did the work quietly and stored the result under the name model. To see it, just type its name:
Getting just the numbers: coef()
The printout is nice for reading, but sometimes you want the ? values themselves, to use in more code. That’s what coef() (short for “coefficients”) is for:
This gives you a vector with the two values, named (Intercept) and gdp.
Making predictions with predict()
Predicting for a new country
In the notes, we predicted the happiness of a brand new country with a GDP per capita of $40,000 by plugging into the formula by hand. You could do that in R too:
But typing in the coefficients yourself is tedious and easy to get wrong. Instead, predict() does the plugging in for you, using the model’s exact ? values.
It needs the new data in a dataframe, though. Usually you want to make a bunch of predictions at once, so newdata is a dataframe with one row per prediction. For just one country, make a tiny dataframe with one row and the right column name:
Or even more succinctly, just make the dataframe inline, and only make the one column it needs to do its prediction:
About 6.01. (The hand calculation gave 6.02 because we rounded the coefficients to 4.90 and 0.028. predict() uses the unrounded ones.)
Let’s break down that line:
- The function is
predict(). - The first argument is the model to use:
model. - The
newdataargument is the data you want predictions for. It has to be a data frame, with a column named exactly like the feature in your model (gdp). data.frame(gdp = 40)makes a tiny data frame with one column,gdp, and one row, 40.
(Ignore the little 1 printed above the answer. It’s just labeling this as the prediction for the first row of newdata.)
Predicting for several countries at once
Since newdata is a data frame, it can have as many rows as you like.
Let’s say we learn about three newly-minted countries: Marianas, Atlantis, and El Dorado. We might load this data from a csv file, but for this demo we’ll just make it on the fly. Then we’ll pass it as newdata to predict().
One prediction per row: about 5.18, 6.01, and 7.66.
Errors (residuals) and MSE, the quick way
Predictions for every country
If you leave out newdata, predict() makes a prediction for every row of the data the model was fit on. That’s exactly what we need for the MSE recipe:
Same recipe as before. The only change is that predict(model) makes the predictions, instead of us typing in a formula. The MSE is about 0.59, the smallest MSE any line can get on this data. That’s what makes it the least squares line.
residuals()
Errors are so useful that R keeps them for you, inside the model. They’re called residuals (what’s left over after the prediction), and residuals() pulls them out:
That’s 140 numbers, one error per country, in the same order as the rows of countries. The first one should match the error for Albania.
So now our whole MSE recipe fits on one line. Square every residual, then take the mean:
It’s worth reading that from the inside out:
residuals(model): get the 140 errorsresiduals(model)^2: square each onemean(residuals(model)^2): average the squares
Same result as before, just a lot less effort.
Drawing the line
To draw the least squares line on the scatter plot, add geom_smooth(method = "lm"). Behind the scenes, it fits the exact same lm() model and draws its line:
The se = FALSE turns off a shaded band around the line. That band is about uncertainty in the line, which we’re not getting into here.
Bonus: the other models from the notes
lm() can fit the other two models from the notes too. Both use a small twist on the formula.
The baseline model: happiness = ?
Remember from the notes that the intercept is really a coefficient on a feature that’s always 1? You can say exactly that to lm(), with a 1 on the right of the ~:
The best ? is about 5.68: the average happiness. Least squares agrees with what you found by dragging in Plot 5.
Through the origin: happiness = ? * gdp
Normally, R adds an intercept feature (1) for you without being asked. To tell it not to, add - 1 to the right side.
Now there’s only one coefficient, on gdp, and no intercept. This is the best possible line through the origin, (0, 0). You’ll get to check how good it is in the exercises below.
Summary
| To do this | Use this |
|---|---|
| Fit a least squares line | model <- lm(happiness ~ gdp, data = countries) |
See the ? values (coefficients) |
model or coef(model) |
| Predict for every row of the data | predict(model) |
| Predict for new values | predict(model, newdata = new_data_frame) |
| Get the errors (residuals) | residuals(model) |
| Compute MSE | mean(residuals(model)^2) |
| Fit the baseline (intercept-only) model | lm(happiness ~ 1, data = countries) |