Week 3: Reading Files, Dictionaries, Stats

Week 3: Reading Files, Dictionaries, Stats#

Recap#

Last lecture, we learned about:

  • Lists

  • String methods

  • Loops

# Discuss with your neighbors to write comments for each line of the following code.
# In particular, what values does "word" take on?

sentence = "Measures of central tendency are summary statistics that represent the center point or typical value of a dataset."

words = sentence.split()

long_words = []

for word in words:

    if len(word) > 4:
        
        long_words.append(word)

print(long_words)

This week#

Our topics for this week are:

  • How to read csv files

  • Storing data from a file into a list

  • Introducing dictionaries: storing key, value pairs

  • Introducing central tendency and dispersion statistics

Reading data from files#

So far, we’ve been working entirely with data we defined inside our notebook.

But in practice, we’ll be reading in large quantities of data from files. Let’s learn how to do this in JupyterHub and Python!

Running example: Canada’s Electoral Districts#

Today we’ll work with a small dataset of Canada’s 338 electoral districts, sourced from Elections Canada.

We’ve downloaded this data in a file called ED-Canada_2016.csv and added it to course website in the same folder as this notebook.

Let’s take a look now. The easiest way to get to the file is to go to File -> Open… in the menu. Demo time!

CSV files#

ED-Canada_2016.csv is a type of file called a comma-separated values (CSV) file, which is a common way of storing tabular data.

In a CSV file:

  • Each line of the file represents one row

  • Within one line, each column entry is separated by a comma

Reading file data: open and readlines#

Now that we’ve seen the file, let’s learn how to read the file’s contents into Python.

Formally, we do this in two steps:

  1. Open the file.

  2. Read the file data into Python, line by line.

# Step 1
district_file = open("ED-Canada_2016.csv", encoding="utf-8")

district_file

<_io.TextIOWrapper name='ED-Canada_2016.csv' mode='r' encoding='utf-8'>

district_data = district_file.readlines()

district_data

['Avalon,86494\n',
 'Bonavista--Burin--Trinity,74116\n',
 'Coast of Bays--Central--Notre Dame,77680\n',
 'Labrador,27197\n',
 'Long Range Mountains,86553\n',
 "St. John's-Est,85697\n",
 "St. John's-Sud--Mount Pearl,81979\n",
 'Cardigan,36615\n',
 'Charlottetown,36094\n',
 'Egmont,34168\n',
 'Malpeque,36030\n',
 'Cape Breton--Canso,71913\n',
 'Nova-Centre,71962\n',
 'Cumberland--Colchester,80590\n',
 'Dartmouth--Cole Harbour,92301\n',
 'Halifax,94610\n',
 'Halifax-Ouest,96255\n',
 'Kings--Hants,83465\n',
 'Sackville--Preston--Chezzetcook,86498\n',
 'South Shore--St. Margarets,91830\n',
 'Sydney--Victoria,72148\n',
 'Nova-Ouest,82026\n',
 'Acadie--Bathurst,77791\n',
 'Beauséjour,82292\n',
 'Fredericton,83303\n',
 'Fundy Royal,79943\n',
 'Madawaska--Restigouche,60378\n',
 'Miramichi--Grand Lake,57405\n',
 'Moncton--Riverview--Dieppe,92666\n',
 'Nouveau-Brunswick-Sud-Ouest,65287\n',
 'Saint John--Rothesay,79363\n',
 'Tobique--Mactaquac,68673\n',
 'Abitibi--Baie-James--Nunavik--Eeyou,87787\n',
 'Abitibi--TĂ©miscamingue,103491\n',
 'Ahuntsic-Cartierville,117447\n',
 'Alfred-Pellan,101373\n',
 'Argenteuil--La Petite-Nation,95781\n',
 'Avignon--La Mitis--Matane--Matapédia,71897\n',
 'Beauce,108746\n',
 'Beauport--Limoilou,96029\n',
 'BĂ©cancour--Nicolet--Saurel,94588\n',
 'Bellechasse--Les Etchemins--LĂ©vis,114966\n',
 'Beloeil--Chambly,117343\n',
 'Berthier--Maskinongé,100371\n',
 'Thérèse-De Blainville,101840\n',
 'Pierre-Boucher--Les Patriotes--Verchères,99787\n',
 'Bourassa,101032\n',
 'Brome--Missisquoi,103457\n',
 'Brossard--Saint-Lambert,107582\n',
 'Rimouski-Neigette--TĂ©miscouata--Les Basques,84918\n',
 'Charlesbourg--Haute-Saint-Charles,107254\n',
 "Beauport-Côte-de-Beaupré-Île d'Orléans-Charlevoix,93674\n",
 'Châteauguay--Lacolle,97887\n',
 'Chicoutimi--Le Fjord,81639\n',
 'Compton--Stanstead,105459\n',
 'Dorval--Lachine--LaSalle,112866\n',
 'Drummond,103397\n',
 'Gaspésie--Les Îles-de-la-Madeleine,75850\n',
 'Gatineau,107464\n',
 'Hochelaga,106496\n',
 'Honoré-Mercier,103592\n',
 'Hull--Aylmer,105419\n',
 'Joliette,104136\n',
 'Jonquière,89818\n',
 "La Pointe-de-l'ĂŽle,106336\n",
 'La Prairie,105496\n',
 'Lac-Saint-Jean,104911\n',
 'Lac-Saint-Louis,108579\n',
 'LaSalle--Émard--Verdun,106766\n',
 'Laurentides--Labelle,113815\n',
 'Laurier--Sainte-Marie,111835\n',
 'Laval--Les ĂŽles,108003\n',
 'Longueuil--Charles-LeMoyne,106583\n',
 'Lévis--Lotbinière,113528\n',
 'Longueuil--Saint-Hubert,108703\n',
 'Louis-HĂ©bert,103346\n',
 'Louis-Saint-Laurent,117238\n',
 'Manicouagan,92518\n',
 "Mégantic--L'Érable,87233\n",
 'Mirabel,117652\n',
 'Montarville,97811\n',
 'Montcalm,107981\n',
 "Montmagny--L'Islet--Kamouraska--Rivière-du-Loup,95527\n",
 'Mont-Royal,103320\n',
 'Notre-Dame-de-Grâce--Westmount,104974\n',
 'Outremont,102088\n',
 'Papineau,110750\n',
 'Pierrefonds--Dollard,108587\n',
 'Pontiac,115531\n',
 'Portneuf--Jacques-Cartier,115313\n',
 'Québec,97143\n',
 'Repentigny,116066\n',
 'Richmond--Arthabaska,107242\n',
 'Rivière-des-Mille-Îles,102346\n',
 'Rivière-du-Nord,112156\n',
 'Rosemont--La Petite-Patrie,110677\n',
 'Marc-Aurèle-Fortin,101750\n',
 'Saint-Hyacinthe--Bagot,102693\n',
 'Saint-Jean,111190\n',
 'Saint-Laurent,98828\n',
 'Saint-LĂ©onard--Saint-Michel,113212\n',
 'Saint-Maurice--Champlain,110264\n',
 'Salaberry--Suroît,112826\n',
 'Shefford,111139\n',
 'Sherbrooke,111176\n',
 'Vaudreuil--Soulanges,119227\n',
 'Terrebonne,111575\n',
 'Trois-Rivières,110515\n',
 'Ville-Marie--Le Sud-Ouest--ĂŽle-des-Soeurs,114659\n',
 'Vimy,111867\n',
 'Ajax,119677\n',
 'Algoma--Manitoulin--Kapuskasing,79483\n',
 'Aurora--Oak Ridges--Richmond Hill,115227\n',
 'Barrie--Innisfil,109286\n',
 'Barrie--Springwater--Oro-Medonte,100788\n',
 'Baie de Quinte,109735\n',
 'Beaches--East York,109468\n',
 'Brampton-Centre,102270\n',
 'Brampton-Est,122000\n',
 'Brampton-Nord,118180\n',
 'Brampton-Sud,121188\n',
 'Brampton-Ouest,130000\n',
 'Brantford--Brant,130296\n',
 'Bruce--Grey--Owen Sound,107679\n',
 'Burlington,123180\n',
 'Cambridge,115463\n',
 'Chatham-Kent--Leamington,109619\n',
 'Davenport,108473\n',
 'Don Valley-Est,94579\n',
 'Don Valley-Nord,110076\n',
 'Don Valley-Ouest,102508\n',
 'Dufferin--Caledon,128237\n',
 'Durham,130872\n',
 'Eglinton--Lawrence,114395\n',
 'Elgin--Middlesex--London,115052\n',
 'Essex,125442\n',
 'Etobicoke-Centre,118022\n',
 'Etobicoke--Lakeshore,129081\n',
 'Etobicoke-Nord,118040\n',
 'Flamborough--Glanbrook,111065\n',
 'Glengarry--Prescott--Russell,109975\n',
 'Guelph,131794\n',
 'Haldimand--Norfolk,109652\n',
 'Haliburton--Kawartha Lakes--Brock,113956\n',
 'Hamilton-Centre,100103\n',
 'Hamilton-Est--Stoney Creek,107848\n',
 'Hamilton Mountain,104877\n',
 'Hamilton-Ouest--Ancaster--Dundas,113024\n',
 'Hastings--Lennox and Addington,94333\n',
 'Huron--Bruce,106570\n',
 'Kanata--Carleton,110960\n',
 'Kenora,62556\n',
 'King--Vaughan,131995\n',
 'Kingston et les ĂŽles,117543\n',
 'Kitchener-Centre,105258\n',
 'Kitchener--Conestoga,100709\n',
 'Kitchener-Sud--Hespeler,105309\n',
 'Lambton--Kent--Middlesex,105331\n',
 'Lanark--Frontenac--Kingston,101630\n',
 'Leeds-Grenville-Thousand Islands et Rideau Lakes,100546\n',
 'London--Fanshawe,119467\n',
 'London-Centre-Nord,125362\n',
 'London-Ouest,126110\n',
 'Markham--Stouffville,126064\n',
 'Markham--Thornhill,99078\n',
 'Markham--Unionville,123318\n',
 'Milton,114093\n',
 'Mississauga-Centre,124849\n',
 'Mississauga-Est--Cooksville,120205\n',
 'Mississauga--Erin Mills,122560\n',
 'Mississauga--Lakeshore,117444\n',
 'Mississauga--Malton,118240\n',
 'Mississauga--Streetsville,118301\n',
 'Nepean,119110\n',
 'Newmarket--Aurora,117418\n',
 'Niagara-Centre,109067\n',
 'Niagara Falls,136292\n',
 'Niagara-Ouest,90838\n',
 'Nickel Belt,93772\n',
 'Nipissing--Timiskaming,88813\n',
 'Northumberland--Peterborough-Sud,112412\n',
 'Oakville,120923\n',
 'Oakville-Nord--Burlington,129078\n',
 'Oshawa,126764\n',
 'Ottawa-Centre,118038\n',
 'Orléans,128281\n',
 'Ottawa-Sud,121058\n',
 'Ottawa--Vanier,111508\n',
 'Ottawa-Ouest--Nepean,111837\n',
 'Oxford,113790\n',
 'Parkdale--High Park,108805\n',
 'Parry Sound--Muskoka,94398\n',
 'Perth--Wellington,107908\n',
 'Peterborough--Kawartha,118176\n',
 'Pickering--Uxbridge,112947\n',
 'Renfrew--Nipissing--Pembroke,103495\n',
 'Richmond Hill,110177\n',
 'Carleton,102918\n',
 'St. Catharines,111691\n',
 "Toronto--St. Paul's,107900\n",
 'Sarnia--Lambton,105337\n',
 'Sault Ste. Marie,80371\n',
 'Scarborough--Agincourt,105542\n',
 'Scarborough-Centre,112603\n',
 'Scarborough--Guildwood,102386\n',
 'Scarborough-Nord,98800\n',
 'Scarborough--Rouge Park,102275\n',
 'Scarborough-Sud-Ouest,110278\n',
 'Simcoe--Grey,129944\n',
 'Simcoe-Nord,111332\n',
 'Spadina--Fort York,115506\n',
 'Stormont--Dundas--South Glengarry,103320\n',
 'Sudbury,91532\n',
 'Thornhill,112719\n',
 'Thunder Bay--Rainy River,82805\n',
 'Thunder Bay--Supérieur-Nord,82651\n',
 'Timmins--Baie James,83257\n',
 'Toronto-Centre,103805\n',
 'Toronto--Danforth,106875\n',
 'University--Rosedale,104311\n',
 'Vaughan--Woodbridge,105228\n',
 'Waterloo,110134\n',
 'Wellington--Halton Hills,120981\n',
 'Whitby,128377\n',
 'Willowdale,118801\n',
 'Windsor--Tecumseh,117429\n',
 'Windsor-Ouest,122988\n',
 'York-Centre,104319\n',
 'York--Simcoe,104010\n',
 'York-Sud--Weston,116686\n',
 'Humber River--Black Creek,108037\n',
 'Brandon--Souris,88170\n',
 'Charleswood--St. James--Assiniboia--Headingley,82574\n',
 'Churchill--Keewatinook Aski,87027\n',
 'Dauphin--Swan River--Neepawa,87527\n',
 'Elmwood--Transcona,92738\n',
 'Kildonan--St. Paul,84077\n',
 'Portage--Lisgar,97354\n',
 'Provencher,99946\n',
 'Saint-Boniface--Saint-Vital,89818\n',
 'Selkirk--Interlake--Eastman,94778\n',
 'Winnipeg-Centre,85949\n',
 'Winnipeg-Nord,95676\n',
 'Winnipeg-Sud,99678\n',
 'Winnipeg-Centre-Sud,93053\n',
 'Battlefords--Lloydminster,73506\n',
 'Cypress Hills--Grasslands,68353\n',
 'Desnethé--Missinippi--Rivière Churchill,70891\n',
 'Sentier Carlton--Eagle Creek,80662\n',
 'Moose Jaw--Lake Centre--Lanigan,79733\n',
 'Prince Albert,79625\n',
 'Regina--Lewvan,92426\n',
 "Regina--Qu'Appelle,76017\n",
 'Regina--Wascana,84153\n',
 'Saskatoon--Grasswood,82946\n',
 'Saskatoon--University,82663\n',
 'Saskatoon-Ouest,83711\n',
 'Souris--Moose Mountain,72635\n',
 'Yorkton--Melville,71031\n',
 'Banff--Airdrie,135762\n',
 'Battle River--Crowfoot,110223\n',
 'Bow River,115022\n',
 'Calgary-Centre,119176\n',
 'Calgary Confederation,122023\n',
 'Calgary Forest Lawn,111830\n',
 'Calgary Heritage,112087\n',
 'Calgary Midnapore,121844\n',
 'Calgary Nose Hill,115795\n',
 'Calgary Rocky Ridge,131823\n',
 'Calgary Shepard,147520\n',
 'Calgary Signal Hill,121392\n',
 'Calgary Skyview,135730\n',
 'Edmonton-Centre,109941\n',
 'Edmonton Griesbach,112287\n',
 'Edmonton Manning,121048\n',
 'Edmonton Mill Woods,118561\n',
 'Edmonton Riverbend,120863\n',
 'Edmonton Strathcona,106066\n',
 'Edmonton-Ouest,121869\n',
 'Edmonton--Wetaskiwin,158749\n',
 'Foothills,113227\n',
 'Fort McMurray--Cold Lake,110230\n',
 'Grande Prairie--Mackenzie,117327\n',
 'Lakeland,108451\n',
 'Lethbridge,117394\n',
 'Medicine Hat--Cardston--Warner,106896\n',
 'Peace River--Westlock,109965\n',
 'Red Deer--Mountain View,119019\n',
 'Red Deer--Lacombe,128786\n',
 'St. Albert--Edmonton,121313\n',
 'Sherwood Park--Fort Saskatchewan,122193\n',
 'Sturgeon River--Parkland,120784\n',
 'Yellowhead,101979\n',
 'Abbotsford,101814\n',
 'Burnaby-Nord--Seymour,102486\n',
 'Burnaby-Sud,111973\n',
 'Cariboo--Prince George,108907\n',
 'Central Okanagan--Similkameen--Nicola,110293\n',
 'Chilliwack--Hope,100126\n',
 'Cloverdale--Langley City,117640\n',
 'Coquitlam--Port Coquitlam,123576\n',
 'Courtenay--Alberni,114647\n',
 'Cowichan--Malahat--Langford,108052\n',
 'Delta,103064\n',
 'Fleetwood--Port Kells,116958\n',
 'Kamloops--Thompson--Cariboo,124358\n',
 'Kelowna--Lake Country,119388\n',
 'Kootenay--Columbia,112354\n',
 'Langley--Aldergrove,117017\n',
 'Mission--Matsqui--Fraser Canyon,94825\n',
 'Nanaimo--Ladysmith,122710\n',
 'New Westminster--Burnaby,115340\n',
 'North Okanagan--Shuswap,124605\n',
 'North Vancouver,115344\n',
 'Pitt Meadows--Maple Ridge,101101\n',
 'Port Moody--Coquitlam,110817\n',
 'Prince George--Peace River--Northern Rockies,110995\n',
 'Richmond-Centre,98396\n',
 'Esquimalt--Saanich--Sooke,120834\n',
 'Saanich--Gulf Islands,107339\n',
 'Skeena--Bulkley Valley,88920\n',
 'Okanagan-Sud--Kootenay-Ouest,114695\n',
 'Surrey-Sud--White Rock,104051\n',
 'Steveston--Richmond-Est,99913\n',
 'Surrey-Centre,120172\n',
 'Surrey--Newton,114605\n',
 'Vancouver-Centre,116443\n',
 'Vancouver-Est,115724\n',
 'Vancouver Granville,103456\n',
 'North Island--Powell River,105466\n',
 'Vancouver Kingsway,104870\n',
 'Vancouver Quadra,105608\n',
 'Vancouver-Sud,102927\n',
 'Victoria,117133\n',
 'West Vancouver--Sunshine Coast--Sea to Sky Country,119113\n',
 'Yukon,35874\n',
 'Territoires du Nord-Ouest,41786\n',
 'Nunavut,35944\n']

# You will often see these two operations chained together
district_data = open("ED-Canada_2016.csv", encoding="utf-8").readlines()
district_data

['Avalon,86494\n',
 'Bonavista--Burin--Trinity,74116\n',
 'Coast of Bays--Central--Notre Dame,77680\n',
 'Labrador,27197\n',
 'Long Range Mountains,86553\n',
 "St. John's-Est,85697\n",
 "St. John's-Sud--Mount Pearl,81979\n",
 'Cardigan,36615\n',
 'Charlottetown,36094\n',
 'Egmont,34168\n',
 'Malpeque,36030\n',
 'Cape Breton--Canso,71913\n',
 'Nova-Centre,71962\n',
 'Cumberland--Colchester,80590\n',
 'Dartmouth--Cole Harbour,92301\n',
 'Halifax,94610\n',
 'Halifax-Ouest,96255\n',
 'Kings--Hants,83465\n',
 'Sackville--Preston--Chezzetcook,86498\n',
 'South Shore--St. Margarets,91830\n',
 'Sydney--Victoria,72148\n',
 'Nova-Ouest,82026\n',
 'Acadie--Bathurst,77791\n',
 'Beauséjour,82292\n',
 'Fredericton,83303\n',
 'Fundy Royal,79943\n',
 'Madawaska--Restigouche,60378\n',
 'Miramichi--Grand Lake,57405\n',
 'Moncton--Riverview--Dieppe,92666\n',
 'Nouveau-Brunswick-Sud-Ouest,65287\n',
 'Saint John--Rothesay,79363\n',
 'Tobique--Mactaquac,68673\n',
 'Abitibi--Baie-James--Nunavik--Eeyou,87787\n',
 'Abitibi--TĂ©miscamingue,103491\n',
 'Ahuntsic-Cartierville,117447\n',
 'Alfred-Pellan,101373\n',
 'Argenteuil--La Petite-Nation,95781\n',
 'Avignon--La Mitis--Matane--Matapédia,71897\n',
 'Beauce,108746\n',
 'Beauport--Limoilou,96029\n',
 'BĂ©cancour--Nicolet--Saurel,94588\n',
 'Bellechasse--Les Etchemins--LĂ©vis,114966\n',
 'Beloeil--Chambly,117343\n',
 'Berthier--Maskinongé,100371\n',
 'Thérèse-De Blainville,101840\n',
 'Pierre-Boucher--Les Patriotes--Verchères,99787\n',
 'Bourassa,101032\n',
 'Brome--Missisquoi,103457\n',
 'Brossard--Saint-Lambert,107582\n',
 'Rimouski-Neigette--TĂ©miscouata--Les Basques,84918\n',
 'Charlesbourg--Haute-Saint-Charles,107254\n',
 "Beauport-Côte-de-Beaupré-Île d'Orléans-Charlevoix,93674\n",
 'Châteauguay--Lacolle,97887\n',
 'Chicoutimi--Le Fjord,81639\n',
 'Compton--Stanstead,105459\n',
 'Dorval--Lachine--LaSalle,112866\n',
 'Drummond,103397\n',
 'Gaspésie--Les Îles-de-la-Madeleine,75850\n',
 'Gatineau,107464\n',
 'Hochelaga,106496\n',
 'Honoré-Mercier,103592\n',
 'Hull--Aylmer,105419\n',
 'Joliette,104136\n',
 'Jonquière,89818\n',
 "La Pointe-de-l'ĂŽle,106336\n",
 'La Prairie,105496\n',
 'Lac-Saint-Jean,104911\n',
 'Lac-Saint-Louis,108579\n',
 'LaSalle--Émard--Verdun,106766\n',
 'Laurentides--Labelle,113815\n',
 'Laurier--Sainte-Marie,111835\n',
 'Laval--Les ĂŽles,108003\n',
 'Longueuil--Charles-LeMoyne,106583\n',
 'Lévis--Lotbinière,113528\n',
 'Longueuil--Saint-Hubert,108703\n',
 'Louis-HĂ©bert,103346\n',
 'Louis-Saint-Laurent,117238\n',
 'Manicouagan,92518\n',
 "Mégantic--L'Érable,87233\n",
 'Mirabel,117652\n',
 'Montarville,97811\n',
 'Montcalm,107981\n',
 "Montmagny--L'Islet--Kamouraska--Rivière-du-Loup,95527\n",
 'Mont-Royal,103320\n',
 'Notre-Dame-de-Grâce--Westmount,104974\n',
 'Outremont,102088\n',
 'Papineau,110750\n',
 'Pierrefonds--Dollard,108587\n',
 'Pontiac,115531\n',
 'Portneuf--Jacques-Cartier,115313\n',
 'Québec,97143\n',
 'Repentigny,116066\n',
 'Richmond--Arthabaska,107242\n',
 'Rivière-des-Mille-Îles,102346\n',
 'Rivière-du-Nord,112156\n',
 'Rosemont--La Petite-Patrie,110677\n',
 'Marc-Aurèle-Fortin,101750\n',
 'Saint-Hyacinthe--Bagot,102693\n',
 'Saint-Jean,111190\n',
 'Saint-Laurent,98828\n',
 'Saint-LĂ©onard--Saint-Michel,113212\n',
 'Saint-Maurice--Champlain,110264\n',
 'Salaberry--Suroît,112826\n',
 'Shefford,111139\n',
 'Sherbrooke,111176\n',
 'Vaudreuil--Soulanges,119227\n',
 'Terrebonne,111575\n',
 'Trois-Rivières,110515\n',
 'Ville-Marie--Le Sud-Ouest--ĂŽle-des-Soeurs,114659\n',
 'Vimy,111867\n',
 'Ajax,119677\n',
 'Algoma--Manitoulin--Kapuskasing,79483\n',
 'Aurora--Oak Ridges--Richmond Hill,115227\n',
 'Barrie--Innisfil,109286\n',
 'Barrie--Springwater--Oro-Medonte,100788\n',
 'Baie de Quinte,109735\n',
 'Beaches--East York,109468\n',
 'Brampton-Centre,102270\n',
 'Brampton-Est,122000\n',
 'Brampton-Nord,118180\n',
 'Brampton-Sud,121188\n',
 'Brampton-Ouest,130000\n',
 'Brantford--Brant,130296\n',
 'Bruce--Grey--Owen Sound,107679\n',
 'Burlington,123180\n',
 'Cambridge,115463\n',
 'Chatham-Kent--Leamington,109619\n',
 'Davenport,108473\n',
 'Don Valley-Est,94579\n',
 'Don Valley-Nord,110076\n',
 'Don Valley-Ouest,102508\n',
 'Dufferin--Caledon,128237\n',
 'Durham,130872\n',
 'Eglinton--Lawrence,114395\n',
 'Elgin--Middlesex--London,115052\n',
 'Essex,125442\n',
 'Etobicoke-Centre,118022\n',
 'Etobicoke--Lakeshore,129081\n',
 'Etobicoke-Nord,118040\n',
 'Flamborough--Glanbrook,111065\n',
 'Glengarry--Prescott--Russell,109975\n',
 'Guelph,131794\n',
 'Haldimand--Norfolk,109652\n',
 'Haliburton--Kawartha Lakes--Brock,113956\n',
 'Hamilton-Centre,100103\n',
 'Hamilton-Est--Stoney Creek,107848\n',
 'Hamilton Mountain,104877\n',
 'Hamilton-Ouest--Ancaster--Dundas,113024\n',
 'Hastings--Lennox and Addington,94333\n',
 'Huron--Bruce,106570\n',
 'Kanata--Carleton,110960\n',
 'Kenora,62556\n',
 'King--Vaughan,131995\n',
 'Kingston et les ĂŽles,117543\n',
 'Kitchener-Centre,105258\n',
 'Kitchener--Conestoga,100709\n',
 'Kitchener-Sud--Hespeler,105309\n',
 'Lambton--Kent--Middlesex,105331\n',
 'Lanark--Frontenac--Kingston,101630\n',
 'Leeds-Grenville-Thousand Islands et Rideau Lakes,100546\n',
 'London--Fanshawe,119467\n',
 'London-Centre-Nord,125362\n',
 'London-Ouest,126110\n',
 'Markham--Stouffville,126064\n',
 'Markham--Thornhill,99078\n',
 'Markham--Unionville,123318\n',
 'Milton,114093\n',
 'Mississauga-Centre,124849\n',
 'Mississauga-Est--Cooksville,120205\n',
 'Mississauga--Erin Mills,122560\n',
 'Mississauga--Lakeshore,117444\n',
 'Mississauga--Malton,118240\n',
 'Mississauga--Streetsville,118301\n',
 'Nepean,119110\n',
 'Newmarket--Aurora,117418\n',
 'Niagara-Centre,109067\n',
 'Niagara Falls,136292\n',
 'Niagara-Ouest,90838\n',
 'Nickel Belt,93772\n',
 'Nipissing--Timiskaming,88813\n',
 'Northumberland--Peterborough-Sud,112412\n',
 'Oakville,120923\n',
 'Oakville-Nord--Burlington,129078\n',
 'Oshawa,126764\n',
 'Ottawa-Centre,118038\n',
 'Orléans,128281\n',
 'Ottawa-Sud,121058\n',
 'Ottawa--Vanier,111508\n',
 'Ottawa-Ouest--Nepean,111837\n',
 'Oxford,113790\n',
 'Parkdale--High Park,108805\n',
 'Parry Sound--Muskoka,94398\n',
 'Perth--Wellington,107908\n',
 'Peterborough--Kawartha,118176\n',
 'Pickering--Uxbridge,112947\n',
 'Renfrew--Nipissing--Pembroke,103495\n',
 'Richmond Hill,110177\n',
 'Carleton,102918\n',
 'St. Catharines,111691\n',
 "Toronto--St. Paul's,107900\n",
 'Sarnia--Lambton,105337\n',
 'Sault Ste. Marie,80371\n',
 'Scarborough--Agincourt,105542\n',
 'Scarborough-Centre,112603\n',
 'Scarborough--Guildwood,102386\n',
 'Scarborough-Nord,98800\n',
 'Scarborough--Rouge Park,102275\n',
 'Scarborough-Sud-Ouest,110278\n',
 'Simcoe--Grey,129944\n',
 'Simcoe-Nord,111332\n',
 'Spadina--Fort York,115506\n',
 'Stormont--Dundas--South Glengarry,103320\n',
 'Sudbury,91532\n',
 'Thornhill,112719\n',
 'Thunder Bay--Rainy River,82805\n',
 'Thunder Bay--Supérieur-Nord,82651\n',
 'Timmins--Baie James,83257\n',
 'Toronto-Centre,103805\n',
 'Toronto--Danforth,106875\n',
 'University--Rosedale,104311\n',
 'Vaughan--Woodbridge,105228\n',
 'Waterloo,110134\n',
 'Wellington--Halton Hills,120981\n',
 'Whitby,128377\n',
 'Willowdale,118801\n',
 'Windsor--Tecumseh,117429\n',
 'Windsor-Ouest,122988\n',
 'York-Centre,104319\n',
 'York--Simcoe,104010\n',
 'York-Sud--Weston,116686\n',
 'Humber River--Black Creek,108037\n',
 'Brandon--Souris,88170\n',
 'Charleswood--St. James--Assiniboia--Headingley,82574\n',
 'Churchill--Keewatinook Aski,87027\n',
 'Dauphin--Swan River--Neepawa,87527\n',
 'Elmwood--Transcona,92738\n',
 'Kildonan--St. Paul,84077\n',
 'Portage--Lisgar,97354\n',
 'Provencher,99946\n',
 'Saint-Boniface--Saint-Vital,89818\n',
 'Selkirk--Interlake--Eastman,94778\n',
 'Winnipeg-Centre,85949\n',
 'Winnipeg-Nord,95676\n',
 'Winnipeg-Sud,99678\n',
 'Winnipeg-Centre-Sud,93053\n',
 'Battlefords--Lloydminster,73506\n',
 'Cypress Hills--Grasslands,68353\n',
 'Desnethé--Missinippi--Rivière Churchill,70891\n',
 'Sentier Carlton--Eagle Creek,80662\n',
 'Moose Jaw--Lake Centre--Lanigan,79733\n',
 'Prince Albert,79625\n',
 'Regina--Lewvan,92426\n',
 "Regina--Qu'Appelle,76017\n",
 'Regina--Wascana,84153\n',
 'Saskatoon--Grasswood,82946\n',
 'Saskatoon--University,82663\n',
 'Saskatoon-Ouest,83711\n',
 'Souris--Moose Mountain,72635\n',
 'Yorkton--Melville,71031\n',
 'Banff--Airdrie,135762\n',
 'Battle River--Crowfoot,110223\n',
 'Bow River,115022\n',
 'Calgary-Centre,119176\n',
 'Calgary Confederation,122023\n',
 'Calgary Forest Lawn,111830\n',
 'Calgary Heritage,112087\n',
 'Calgary Midnapore,121844\n',
 'Calgary Nose Hill,115795\n',
 'Calgary Rocky Ridge,131823\n',
 'Calgary Shepard,147520\n',
 'Calgary Signal Hill,121392\n',
 'Calgary Skyview,135730\n',
 'Edmonton-Centre,109941\n',
 'Edmonton Griesbach,112287\n',
 'Edmonton Manning,121048\n',
 'Edmonton Mill Woods,118561\n',
 'Edmonton Riverbend,120863\n',
 'Edmonton Strathcona,106066\n',
 'Edmonton-Ouest,121869\n',
 'Edmonton--Wetaskiwin,158749\n',
 'Foothills,113227\n',
 'Fort McMurray--Cold Lake,110230\n',
 'Grande Prairie--Mackenzie,117327\n',
 'Lakeland,108451\n',
 'Lethbridge,117394\n',
 'Medicine Hat--Cardston--Warner,106896\n',
 'Peace River--Westlock,109965\n',
 'Red Deer--Mountain View,119019\n',
 'Red Deer--Lacombe,128786\n',
 'St. Albert--Edmonton,121313\n',
 'Sherwood Park--Fort Saskatchewan,122193\n',
 'Sturgeon River--Parkland,120784\n',
 'Yellowhead,101979\n',
 'Abbotsford,101814\n',
 'Burnaby-Nord--Seymour,102486\n',
 'Burnaby-Sud,111973\n',
 'Cariboo--Prince George,108907\n',
 'Central Okanagan--Similkameen--Nicola,110293\n',
 'Chilliwack--Hope,100126\n',
 'Cloverdale--Langley City,117640\n',
 'Coquitlam--Port Coquitlam,123576\n',
 'Courtenay--Alberni,114647\n',
 'Cowichan--Malahat--Langford,108052\n',
 'Delta,103064\n',
 'Fleetwood--Port Kells,116958\n',
 'Kamloops--Thompson--Cariboo,124358\n',
 'Kelowna--Lake Country,119388\n',
 'Kootenay--Columbia,112354\n',
 'Langley--Aldergrove,117017\n',
 'Mission--Matsqui--Fraser Canyon,94825\n',
 'Nanaimo--Ladysmith,122710\n',
 'New Westminster--Burnaby,115340\n',
 'North Okanagan--Shuswap,124605\n',
 'North Vancouver,115344\n',
 'Pitt Meadows--Maple Ridge,101101\n',
 'Port Moody--Coquitlam,110817\n',
 'Prince George--Peace River--Northern Rockies,110995\n',
 'Richmond-Centre,98396\n',
 'Esquimalt--Saanich--Sooke,120834\n',
 'Saanich--Gulf Islands,107339\n',
 'Skeena--Bulkley Valley,88920\n',
 'Okanagan-Sud--Kootenay-Ouest,114695\n',
 'Surrey-Sud--White Rock,104051\n',
 'Steveston--Richmond-Est,99913\n',
 'Surrey-Centre,120172\n',
 'Surrey--Newton,114605\n',
 'Vancouver-Centre,116443\n',
 'Vancouver-Est,115724\n',
 'Vancouver Granville,103456\n',
 'North Island--Powell River,105466\n',
 'Vancouver Kingsway,104870\n',
 'Vancouver Quadra,105608\n',
 'Vancouver-Sud,102927\n',
 'Victoria,117133\n',
 'West Vancouver--Sunshine Coast--Sea to Sky Country,119113\n',
 'Yukon,35874\n',
 'Territoires du Nord-Ouest,41786\n',
 'Nunavut,35944\n']

Data processing#

Let’s look at just the first line from the file:

district_data[0]

'Avalon,86494\n'

There’s two annoying parts about this line:

  1. It’s a single string, but really stores two pieces of data.

  2. There’s a strange \n at the end of the string, representing a line break.

Goal: take the list district_data and extract just the population counts, converting to int. We’ll develop this one together!

# Write this code in class.
# district_data is a list of comma-separated pairs with a newline at the end

Now we can compute!#

num_populations = len(populations)
total_population = sum(populations)
max_population = max(populations)
min_population = min(populations)
avg_population = total_population / num_populations

print(f"Number of population entries: {num_populations}")
print(f"Sum of populations: {total_population}")
print(f"Maximum district population: {max_population}")
print(f"Minimum district population: {min_population}")
print(f"Average district population: {avg_population}")

Dictionaries#

There is a Python data structure that allows us to associate labels (or keys) with values called a dictionary.

It is similar to a list, but instead of a sequential index to access values, we think of a dictionary as containing key value pairs.

Using the district_data lines that we read from ED-Canada_2016.csv, let’s see how we can build a dictionary with this data instead of a list of populations:

# create an empty dictionary
district_populations_dict = {}

# build the dictionary from district_data list

Now let’s use the dictionary to compute the same values as above#

# These are not great variable names, but I wante to create new names rather
# than repeat the ones from above to avoid confusion.
num_districts = len(district_populations_dict)
total_population_dict = sum(district_populations_dict.values())
max_population_dict = max(district_populations_dict.values())
min_population_dict = min(district_populations_dict.values())
avg_population_dict = total_population_dict / num_districts

print(f"Number of population entries: {num_districts}")
print(f"Sum of populations: {total_population_dict}")
print(f"Maximum district population: {max_population_dict}")
print(f"Minimum district population: {min_population_dict}")
print(f"Average district population: {round(avg_population_dict)}")

Dictionary Keys#

There is a dictionary method to get the list of keys in the dictionary: dict.keys() In our example, this would be district_populations_dict.keys(). This is particularly useful because the most common way of looping over a dictionary is iterating over the keys. This is so common that there is kind of a short cut for this.

We could also have used district_populations_dict.keys() instead of district_populations_dict below to get the list of keys.

# the short hand for looping over the keys of a dictionary.
for district in district_populations_dict:
    print(district)

Exercise Break#

Let’s put some of this together. Print the following message for each district that has Toronto in its name. Use district_populations_dict that was defined above. f"{name} has population {district_populations[name]}

In English what we want to do is check each dictionary key to see if the string Toronto is in the key. If it is, then we will print the message.

# write your code here

Central Tendency#

This is a fancy way of saying that when you have some data, you would like to be able to summarize it. In particular we would like to say something about the central value or the typical value of the data set. The three most common measures of central tendency are mean (average), median, and mode.

  1. Mean - The arithmetic average, calculated by summing up all the numbers in a data set and dividing by the number of data points.

  2. Median - The middle value when data is sorted. If the number of data points is even, then the median is the average of the two middle values.

  3. Mode - The most frequently occuring value. Useful for categorical data such as likert scale data.

# Calculating Mean
grades = [47.6, 73.2, 85.4, 54.9, 97.6, 26.8, 0.0, 85.4, 96.3, 82.9, 0.0, 0.0, 90.2, 0.0, 0.0, 0.0, 92.7, 47.6, 31.7, 86.6, 95.1, 90.2, 92.7, 54.9, 95.1, 0.0, 82.9, 84.1, 81.7, 87.8, 85.4, 46.3, 92.7, 96.3, 73.2, 92.7, 54.9, 57.3, 93.9, 95.1, 85.4, 0.0, 98.8, 92.7, 100.0, 73.2, 82.9, 90.2, 72.0, 85.4, 51.2, 92.7, 48.8, 0.0, 89.0, 17.1, 53.7, 95.1, 96.3, 89.0, 89.0, 61.0, 95.1, 0.0, 0.0, 0.0, 97.6, 90.2, 64.6, 69.5, 85.4, 81.7, 63.4, 90.2, 95.1, 86.6, 90.2, 0.0, 84.1, 39.0, 0.0, 48.8, 78.0, 47.6, 0.0, 86.6, 97.6, 0.0, 86.6, 0.0, 61.0, 0.0, 91.5, 100.0, 90.2, 95.1, 87.8, 0.0, 87.8, 92.7, 75.6, 63.4, 97.6, 69.5, 80.5, 68.3, 48.8, 0.0, 87.8, 79.3, 70.7, 97.6, 90.2, 62.2, 84.1, 79.3, 97.6, 35.4, 80.5, 0.0, 80.5, 78.0, 96.3, 97.6, 90.2, 95.1, 0.0, 75.6, 0.0]

mean = round(sum(grades) / len(grades), 1)

print(f"The mean grade is {mean}")

The mean grade is 65.0

# Calculating the median
# Step 1: sort the data
grades_sorted = sorted(grades)

n = len(grades_sorted)
mid = n // 2

# Step 2: compute the median
if n % 2 == 1:
    median = grades_sorted[mid]
else:
    median = (grades_sorted[mid - 1] + grades_sorted[mid]) / 2

print(f"The median grade is {median}")

# Calculating the Mode

favourite_season = ['Fall', 'Summer', 'Summer', 'Summer', 'Fall', 'Summer', 'Summer', 'Fall', 'Summer', 'Fall', 'Summer', 'Fall', 'Fall', 'Spring', 'Summer', 'Spring', 'Winter', 'Fall', 'Fall', 'Summer', 'Spring', 'Summer', 'Winter', 'Summer', 'Summer', 'Winter', 'Spring', 'Spring', 'Summer', 'Fall', 'Fall', 'Spring', 'Fall', 'Summer', 'Summer', 'Summer', 'Spring', 'Fall', 'Spring', 'Summer', 'Fall', 'Spring', 'Winter', 'Spring', 'Summer', 'Fall', 'Summer', 'Summer', 'Winter', 'Spring', 'Fall', 'Fall', 'Fall', 'Summer', 'Fall', 'Fall', 'Fall', 'Summer', 'Summer', 'Fall', 'Summer', 'Spring']

# Step 1: count occurrences
counts = {}

for season in favourite_season:
    # if season is already a key in the counts dictionary
    if season in counts:
        counts[season] += 1
    else:
        counts[season] = 1

print(counts)
# Step 2: find the mode
mode = None
max_count = 0

for season, count in counts.items():
    if count > max_count:
        max_count = count
        mode = season

print(f"Most people's favourite season is {mode}")

Skew#

If the mean and median (and mode) are very close or the same, then the data has a symmetrical distribution.

nomal distribution

However, if the mean and median differ then the data is skewed, and the more skewed the distribution the greater the difference between the mean and the median.

For example, suppose that you learned that the average on the midterm was 79%, and your grade was 80%. How would you feel about it?

Now what if you learned that the median was 85%?

In other words, half the class received a grade higher than 85%.

Let’s look at this histogram:

Grades histogram

Here is one more thought about what the data really means. Consider the following histogram. I imagine you might be quite grumpy to be told that the average grade on the final exam was 57%! But there is something else going on here.

Grades with zeros

Let’s look at the data again with the zeros removed. These are the same grades as in the previous graph but with all the zeros removed. Now the mean and the median are very close, and the data distribution is more meaningful.

Grades with the zeros filtered out

Contents