# 2.5 Assignment 2

## State Data

We often take data for granted. However, one of the hardest parts about analyzing a problem you’re interested in can be to find good data to answer the questions you want to ask. As you’re learning R, though, there are many datasets that R has built in that you can take advantage of.

In this problem, we will be examining the “state” dataset, which has data from the 1970s on all fifty US states. For each state, the dataset includes the population, per capita income, illiteracy rate, murder rate, high school graduation rate, average number of frost days, area, latitude and longitude, division the state belongs to, region the state belongs to, and two-letter abbreviation.

Load the dataset and convert it to a data frame by running the following two commands in R:

data(state)

statedata = cbind(data.frame(state.x77), state.abb, state.area, state.center,  state.division, state.name, state.region)

If you can’t access the state dataset in R, here is a CSV file with the same data that you can load into R using the read.csv function: statedata (CSV).

After you have loaded the data into R, inspect the data set using the command: str(statedata)

This dataset has 50 observations (one for each US state) and the following 15 variables:

• Population - the population estimate of the state in 1975
• Income - per capita income in 1974
• Illiteracy - illiteracy rates in 1970, as a percent of the population
• Life.Exp - the life expectancy in years of residents of the state in 1970
• Murder - the murder and non-negligent manslaughter rate per 100,000 population in 1976
• Frost - the mean number of days with minimum temperature below freezing from 1931–1960 in the capital or a large city of the state
• Area - the land area (in square miles) of the state
• state.abb - a 2-letter abreviation for each state
• state.area - the area of each state, in square miles
• x - the longitude of the center of the state
• y - the latitude of the center of the state
• state.division - the division each state belongs to (New England, Middle Atlantic, South Atlantic, East South Central, West South Central, East North Central, West North Central, Mountain, or Pacific)
• state.name - the full names of each state
• state.region - the region each state belong to (Northeast, South, North Central, or West)

## Problem 1.1 - Data Exploration

We begin by exploring the data. Plot all of the states’ centers with latitude on the y axis (the “y” variable in our dataset) and longitude on the x axis (the “x” variable in our dataset). The shape of the plot should look like the outline of the United States! Note that Alaska and Hawaii have had their coordinates adjusted to appear just off of the west coast.

In the R command you used to generate this plot, which variable name did you use as the first argument?

Exercise 1

statedata$y statedata$x

I used a different variable name.

Explanation

To generate the described plot, you should type plot(statedata$x, statedata$y) in your R console. The first variable here is statedata$x. CheckShow Answer ## Problem 1.2 - Data Exploration Using the tapply command, determine which region of the US (West, North Central, South, or Northeast) has the highest average high school graduation rate of all the states in the region: Exercise 2 West North Central South Northeast Explanation You can find the average high school graduation rate of all states in each of the regions by typing the following command in your R console: tapply(statedata$HS.Grad, statedata$state.region, mean) The highest value is for the West region. CheckShow Answer ## Problem 1.3 - Data Exploration Now, make a boxplot of the murder rate by region (for more information about creating boxplots in R, type ?boxplot in your console). Which region has the highest median murder rate? Exercise 3 Northeast South North Central West Explanation To generate the boxplot, you should type boxplot(statedata$Murder ~ statedata$state.region) in your R console. You can see that the region with the highest median murder rate (the one with the highest solid line in the box) is the South. CheckShow Answer ## Problem 1.4 - Data Exploration You should see that there is an outlier in the Northeast region of the boxplot you just generated. Which state does this correspond to? (Hint: There are many ways to find the answer to this question, but one way is to use the subset command to only look at the Northeast data.) Exercise 4 Delaware Rhode Island Maine New York Explanation The correct answer is New York. If you first use the subset command: NortheastData = subset(statedata, state.region == “Northeast”) You can then look at NortheastData$Murder together with NortheastData$state.abb to identify the outlier. CheckShow Answer ## Problem 2.1 - Predicting Life Expectancy - An Initial Model We would like to build a model to predict life expectancy by state using the state statistics we have in our dataset. Build the model with all potential variables included (Population, Income, Illiteracy, Murder, HS.Grad, Frost, and Area). Note that you should use the variable “Area” in your model, NOT the variable “state.area”. What is the coefficient for “Income” in your linear regression model? Exercise 5 Numerical Response Explanation You can build the linear regression model with the following command: LinReg = lm(Life.Exp ~ Population + Income + Illiteracy + Murder + HS.Grad + Frost + Area, data=statedata) Then, to find the coefficient for income, you can look at the summary of the regression with summary(LinReg). CheckShow Answer ## Problem 2.2 - Predicting Life Expectancy - An Initial Model Call the coefficient for income x (the answer to Problem 2.1). What is the interpretation of the coefficient x? Exercise 6 For a one unit increase in income, predicted life expectancy increases by |x| For a one unit increase in income, predicted life expectancy decreases by |x| For a one unit increase in predicted life expectancy, income decreases by |x| For a one unit increase in predicted life expectancy, income increases by |x| Explanation If we increase income by one unit, then our model’s prediction will increase by the coefficient of income, x. Because x is negative, this is the same as predicted life expectancy decreasing by |x|. CheckShow Answer ## Problem 2.3 - Predicting Life Expectancy - An Initial Model Now plot a graph of life expectancy vs. income using the command: plot(statedata$Income, statedata$Life.Exp) Visually observe the plot. What appears to be the relationship? Exercise 7 Life expectancy is somewhat positively correlated with income. Life expectancy is somewhat negatively correlated with income. Life expectancy is not correlated with income. Explanation Although the point in the lower right hand corner of the plot appears to be an outlier, we observe a positive linear relationship in the plot. CheckShow Answer ## Problem 2.4 - Predicting Life Expectancy - An Initial Model The model we built does not display the relationship we saw from the plot of life expectancy vs. income. Which of the following explanations seems the most reasonable? Exercise 8 Income is not related to life expectancy. Multicollinearity Explanation Although income is an insignificant variable in the model, this does not mean that there is no association between income and life expectancy. However, in the presence of all of the other variables, income does not add statistically significant explanatory power to the model. This means that multicollinearity is probably the issue. CheckShow Answer ## Problem 3.1 - Predicting Life Expectancy - Refining the Model and Analyzing Predictions Recall that we discussed the principle of simplicity: that is, a model with fewer variables is preferable to a model with many unnnecessary variables. Experiment with removing independent variables from the original model. Remember to use the significance of the coefficients to decide which variables to remove (remove the one with the largest “p-value” first, or the one with the “t value” closest to zero), and to remove them one at a time (this is called “backwards variable selection”). This is important due to multicollinearity issues - removing one insignificant variable may make another previously insignificant variable become significant. You should be able to find a good model with only 4 independent variables, instead of the original 7. Which variables does this model contain? Exercise 9 Income, HS.Grad, Frost, Murder HS.Grad, Population, Income, Frost Frost, Murder, HS.Grad, Illiteracy Population, Murder, Frost, HS.Grad Explanation We would eliminate the variable “Area” first (since it has the highest p-value, or probability, with a value of 0.9649), by adjusting our lm command to the following: LinReg = lm(Life.Exp ~ Population + Income + Illiteracy + Murder + HS.Grad + Frost, data=statedata) Looking at summary(LinReg) now, we would choose to eliminate “Illiteracy” since it now has the highest p-value of 0.9340, using the following command: LinReg = lm(Life.Exp ~ Population + Income + Murder + HS.Grad + Frost, data=statedata) Looking at summary(LinReg) again, we would next choose to eliminate “Income”, since it has a p-value of 0.9153. This gives the following four variable model: LinReg = lm(Life.Exp ~ Population + Murder + HS.Grad + Frost, data=statedata) This model with 4 variables is a good model. However, we can see that the variable “Population” is not quite significant. In practice, it would be up to you whether or not to keep the variable “Population” or eliminate it for a 3-variable model. Population does not add much statistical significance in the presence of murder, high school graduation rate, and frost days. However, for the remainder of this question, we will analyze the 4-variable model. CheckShow Answer ## Problem 3.2 - Predicting Life Expectancy - Refining the Model and Analyzing Predictions Removing insignificant variables changes the Multiple R-squared value of the model. By looking at the summary output for both the initial model (all independent variables) and the simplified model (only 4 independent variables) and using what you learned in class, which of the following correctly explains the change in the Multiple R-squared value? Exercise 10 We expect the “Multiple R-squared” value of the simplified model to be slightly worse than that of the initial model. It can’t be better than the “Multiple R-squared” value of the initial model. We expect the “Multiple R-squared” value of the simplified model to be slightly better than that of the initial model. It can’t be worse than the “Multiple R-squared” value of the initial model. We expect the “Multiple R-squared” of the simplified model to be about the same as the intial model (we have no way of knowing if it will be slightly worse or slightly better than the Multiple R-squared of the intial model). Explanation When we remove insignificant variables, the “Multiple R-squared” will always be worse, but only slightly worse. This is due to the nature of a linear regression model. It is always possible for the regression model to make a coefficient zero, which would be the same as removing the variable from the model. The fact that the coefficient is not zero in the intial model means it must be helping the R-squared value, even if it is only a very small improvement. So when we force the variable to be removed, it will decrease the R-squared a little bit. However, this small decrease is worth it to have a simpler model. On the contrary, when we remove insignificant variables, the “Adjusted R-squred” will frequently be better. This value accounts for the complexity of the model, and thus tends to increase as insignificant variables are removed, and decrease as insignificant variables are added. CheckShow Answer ## Problem 3.3 - Predicting Life Expectancy - Refining the Model and Analyzing Predictions Using the simplified 4 variable model that we created, we’ll now take a look at how our predictions compare to the actual values. Take a look at the vector of predictions by using the predict function (since we are just looking at predictions on the training set, you don’t need to pass a “newdata” argument to the predict function). Which state do we predict to have the lowest life expectancy? (Hint: use the sort function) Exercise 11 South Carolina Mississippi Alabama Georgia Explanation If your simplified 4-variable model is called “LinReg”, you can answer this question by typing sort(predict(LinReg)) in your R console. The first state listed has the lowest predicted life expectancy, which is Alabama. Which state actually has the lowest life expectancy? (Hint: use the which.min function) Exercise 12 South Carolina Mississippi Alabama Georgia Explanation You can find the row number of the state with the lowest life expectancy by typing which.min(statedata$Life.Exp) into your R console. This returns 40. The 40th state name in the vector statedata$state.name is South Carolina. CheckShow Answer ## Problem 3.4 - Predicting Life Expectancy - Refining the Model and Analyzing Predictions Which state do we predict to have the highest life expectancy? Exercise 13 Massachusetts Maine Washington Hawaii Explanation If your simplified 4-variable model is called “LinReg”, you can answer this question by typing “sort(predict(LinReg))” in your R console. The last state listed has the highest predicted life expectancy, which is Washington. Which state actually has the highest life expectancy? Exercise 14 Massachusetts Maine Washington Hawaii Explanation You can find the row number of the state with the highest life expectancy by typing which.max(statedata$Life.Exp) into your R console. This returns 11. The 11th state name in the vector statedata$state.name is Hawaii. CheckShow Answer ## Problem 3.5 - Predicting Life Expectancy - Refining the Model and Analyzing Predictions Take a look at the vector of residuals (the difference between the predicted and actual values). For which state do we make the smallest absolute error? Exercise 15 Maine Florida Indiana Illinois Explanation You can look at the sorted list of absolute errors by typing sort(abs(model$residuals))

into your R console (where “model” is the name of your model). Alternatively, you can compute the residuals manually by typing

sort(abs(statedata$Life.Exp - predict(model))) in your R console. The smallest absolute error is for Indiana. For which state do we make the largest absolute error? Exercise 16 Hawaii Maine Texas South Carolina Explanation You can look at the sorted list of absolute errors by typing sort(abs(model$residuals))

into your R console (where “model” is the name of your model). Alternatively, you can compute the residuals manually by typing

sort(abs(statedata\$Life.Exp - predict(model)))

in your R console. The largest absolute error is for Hawaii.