Ctrl+K

Homework 10 : Linear Regression

Homework 10 : Linear Regression#

Logistics#

Due date: The homework is due 23:59 on Monday, March 25th.

You will submit your work on MarkUs. To submit your work:

  1. Download this file (Homework_10.ipynb) from JupyterHub. (See our JupyterHub Guide for detailed instructions.)

  2. Submit this file to MarkUs under the hw10 assignment. (See our MarkUs Guide for detailed instructions.) All homeworks will take place in a Jupyter notebook (like this one). When you are done, you will download this notebook and submit it to MarkUs.

Introduction#

In this homework we explore:

  • Preparing data from two different sources and merging before analysis

  • Performing linear regression with multiple dependent variables

  • Checking performance of linear regression

Question: How do mental health visits correlated with different socioeconomic factors in men and woman 65+ living in Toronto?#

You may use the lecture notebook to help you conduct the following analysis.

Task 1a: Pre-processing independent variables#

Sociodemographic data#

To perform ordinary least squares (i.e. linear regression) analysis, we will need a set of independent variables and one dependent variable.

  1. Read sociodemographic data from the excel file 1_socdem_neighb_2006-2.xls with sheet name socdem_2006 into pandas using read_excel. Name the dataframe socdem_neighb.

  2. Select columns/variables relevant for your analysis.

  3. Rename dataframe columns according to table.

  4. Check first few entires of dataframe.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from statsmodels.formula.api import ols

Step 1: Read in data, name the dataframe socdem_neighb. Set header to row 10.#

# Write your code here

Step 2: Select columns and save the results to a new dataframe called socdem_neighb_important sconsisting of the important columns.#

For our analysis, select the columns Neighbourhood id, Neighbourhood Name, % Not in labour force *, % Rented Dwellings, % Aged 65+ living alone.

# Write your code here

Step 3: Rename dataframe columns inplace according to table below:#

Old name

New name

Neighbourhood id

neighbid

Neighbourhood Name

name

% Not in labour force *

pct_retired

% Rented Dwellings

pct_rent

% Aged 65+ living alone

live_alone

 
# Write your code here

Step 4: Check your dataframe by using head() function#

# Write your code here

Task 1b: Processing dependent variable and merging dataframes#

Mental health visit data#

  1. Read data from the excel file 1_ahd_neighb_db_ast_hbp_mhv_copd_2007.xls with sheet name 1_ahd_neighb_mentalHV_2007 into pandas using read_excel. Name this dataframe mh_neighb.

  2. Only select age 65+ to filter our dataset for the older population. Include both sexes.

  3. Rename dataframe according to table

  4. Merge socioeconomic data with mental health visit data

Step 1: Loading in data and name dataframe mh_neighb. Set header to row 11.#

# Write your code here

Step 2: Selecting age 65+ of both sexes, name resulting dataframe mh_visit_rates.#

Since this dataframe is a little complicated, a hint is given: columns 0, 1, and 53 correspond to Neighbourhood ID, Neighbourhood Name, and percentage of mental health visits in both sexes 65+ respectively.

# Write your code here

Step 3: Renaming the columns of the dataframe inplace using the following table:#

Old name

New name

Unnamed: 0

neighbid

Unnamed: 1

name

% With mental health visits.11

mh_visit_pct

 
# Write your code here

# Check you dataframe!
mh_visit_rates.head()

Step 4: Merge Sociodemographic and mental health visit data and call the merged dataframe mh_socdem#

  • To examine the relationship between mental health visits and socioeconomic status, we will need to merge socdem_neighb_important to mh_visit_rates based on neighbid.

  • Note: merge dataframe A to dataframe B is B.merge(A). This is for passing the rigid autotesting :)

# Write your code here

Task 2: Fit the regression model#

  1. Fit the model by using the ols function. See lecture notebook for examples.

  2. Create a summary of key values by using the .summary method on the fitted model.

Step 1: Fit the regression model and call the fitted model reg_mh_fit#

  • Dependent variable: mh_visit_pct

  • Independent variables: pct_retired , pct_rent , live_alone

# Write your code here

Step 2: Generate statistical summary of the regression model and name result reg_mh_summary#

# Write your code here

Task 3: Determining accuracy of the regression model#

  1. Use the .rsquared method to check the rsquared value. Assign this variable to the name reg_rsquared.

  2. Create a scatter plot of residuals vs. fitted values.

Step 1: Check rsquared value and name result reg_rsquared#

# Write your code here

Step 2: Create scatter plot using matplotlib to visualize the residuals vs. fitted values#

# Write your code here

Task 4: Answering some questions about the regression model you have just fitted#

Question 1: How is the accuracy of the regression model according to its r-squared value and residual plot?

Question 2: What is the intercept of the model? What does it mean?

Question 3: What are the coefficients and p-value of each independent variable? What do they mean?

Contents