Professional development is important regardless of your employment status. Last year, I had difficulties finding training that fit into my budget and resigned myself to only reading books and blog posts. This year, though, has been an interesting experiment, Since the pandemic lockdown began, I've noticed an uptick in accessible training at free or low costs. Here are some of the resources that I've used to increase my skills over the past few months.

Associations

Organizations in your interest area are a great place to look for free or low-cost professional development. I’ve recently become a member of several organizations who offer webinars, in-person training (whenever the world turns right-side up again), and YouTube videos: Women Who Code, Women in Data, and Women in Analytics.

Platforms

There are many learning platforms out there and while some are free, there are ways to access paid platforms at a reduced cost.

Lynda.com/LinkedIn Learning
Lynda.com recently became a part of the LinkedIn system. As part of LinkedIn's membership service you get access to the courses for around $30/month. There is another option. In many major metropolitan areas, the public library systems have partnerships with Lynda and you can access the same content for free.

Alison.com
Alison.com is another paid learning platform. Some state job centers have partnerships with Alison and you can access the content for free.

Udemy.com
Udemy.com is the last paid learning platform that I’ve tried. I’ve gotten free classes by shopping during sales and through promotional codes posted on social media.

FreeCodeCamp
A place to learn different types of coding for free. The Python Data Analysis course that I took was through a partnership between FreeCodeCamp and another service.

Google Analytics Academy
Google Analytics Academy provides free training in Google Analytics, Data Studio, and Tag Manager and a certificate in Google Analytics.

Hubspot Academy
Hubspot offers free courses and certificates in marketing, SEO, sales, etc.

I hope that these resources are helpful to wherever you are in your professional journey.

"Data Analysis with Python: Zero to Pandas" is a practical, beginner-friendly and coding-focused introduction to data analysis covering the basics of Python, Numpy, Pandas, data visualization and exploratory data analysis.

I learned data analysis and visualization with Python and NumPy, Pandas, Matplotlib, and Seaborn libraries. During the course, we were tasked with completing three assignments and an overall course project. Since I consider the entire month of October as Halloween season, I thought it would be appropriate to analyze some Halloween related data for my project.

Zero to Pandas Course Project: 2017 Halloween Candy Survey

Dataset

In recent years, a group at The University of British Columbia surveyed people on their Halloween activities. I chose to analyze the data from their 2017 survey, Candy Hierarchy 2017 Survey, and see what I could find. The survey, with over 2000 responses, has information on age, gender, location, and candy preferences.

Data Preparation and Cleaning

As a first step, I uploaded my Jupyter notebook to Jovian.ml, created variables for my project name and the location of my data, and then installed Pandas so that I could work with my data as a dataset.

project_name = "course-project-halloween-candy"

project_file = "https://raw.githubusercontent.com/rdhm113/z-to-p-course-project/main/candyhierarchy2017.csv"

!pip install jovian --upgrade -q

import jovian

jovian.commit(project=project_name)

import pandas as pd

project_file

'https://raw.githubusercontent.com/rdhm113/z-to-p-course-project/main/candyhierarchy2017.csv'

After importing my dataset, I encountered a read error when previewing the dataframe. I imported it again and set an encoding type to eliminate the error.

candy_survey_raw_df = pd.read_csv(project_file, encoding = 'ISO-8859-1')

Previewing and requesting info on the dataframe showed me that there are 115 columns and 2460 rows of data in my dataset.

candy_survey_raw_df

candy_survey_raw_df.info()

RangeIndex: 2460 entries, 0 to 2459
Columns: 120 entries, Internal ID to Click Coordinates (x, y)
dtypes: float64(4), int64(1), object(115)
memory usage: 2.3+ MB

Previewing a list of column headers showed the basic data for the columns. In the absence of a data schema, I used the PDF of the survey to understand the relationship between column headers and possible row values.

candy_survey_raw_df.columns

Index(['Internal ID', 'Q1: GOING OUT?', 'Q2: GENDER', 'Q3: AGE', 'Q4: COUNTRY',
       'Q5: STATE, PROVINCE, COUNTY, ETC', 'Q6 | 100 Grand Bar',
       'Q6 | Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Q6 | Any full-sized candy bar', 'Q6 | Black Jacks',
       ...
       'Q8: DESPAIR OTHER', 'Q9: OTHER COMMENTS', 'Q10: DRESS', 'Unnamed: 113',
       'Q11: DAY', 'Q12: MEDIA [Daily Dish]', 'Q12: MEDIA [Science]',
       'Q12: MEDIA [ESPN]', 'Q12: MEDIA [Yahoo]', 'Click Coordinates (x, y)'],
      dtype='object', length=120)

An important step that I've learned in this course and other data analysis that I've done is to always make a copy of data to work with. This preserves the raw data in case you need to go back and start again or fix errors.

candy_survey_df = candy_survey_raw_df.copy()
candy_survey_df

Next, columns that wouldn't be used for analysis were dropped: Internal ID and survey questions 7-13.

candy_survey_df.drop(['Internal ID', 'Q7: JOY OTHER','Q8: DESPAIR OTHER', 'Q9: OTHER COMMENTS', 'Q10: DRESS', 'Unnamed: 113',
       'Q11: DAY', 'Q12: MEDIA [Daily Dish]', 'Q12: MEDIA [Science]',
       'Q12: MEDIA [ESPN]', 'Q12: MEDIA [Yahoo]', 'Click Coordinates (x, y)'], axis=1, inplace=True)

candy_survey_df.columns

Index(['Q1: GOING OUT?', 'Q2: GENDER', 'Q3: AGE', 'Q4: COUNTRY',
       'Q5: STATE, PROVINCE, COUNTY, ETC', 'Q6 | 100 Grand Bar',
       'Q6 | Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Q6 | Any full-sized candy bar', 'Q6 | Black Jacks',
       'Q6 | Bonkers (the candy)',
       ...
       'Q6 | Three Musketeers', 'Q6 | Tolberone something or other',
       'Q6 | Trail Mix', 'Q6 | Twix',
       'Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein',
       'Q6 | Vicodin', 'Q6 | Whatchamacallit Bars', 'Q6 | White Bread',
       'Q6 | Whole Wheat anything', 'Q6 | York Peppermint Patties'],
      dtype='object', length=108)

Reviewed dateframe info again after the change.

candy_survey_df.info()

RangeIndex: 2460 entries, 0 to 2459
Columns: 108 entries, Q1: GOING OUT? to Q6 | York Peppermint Patties
dtypes: object(108)
memory usage: 2.0+ MB

Calculated the null values to get an idea of how many questions were left unanswered.

candy_survey_df.isnull().sum()

Q1: GOING OUT?                      110
Q2: GENDER                           41
Q3: AGE                              84
Q4: COUNTRY                          64
Q5: STATE, PROVINCE, COUNTY, ETC    100
                                   ... 
Q6 | Vicodin                        789
Q6 | Whatchamacallit Bars           823
Q6 | White Bread                    757
Q6 | Whole Wheat anything           747
Q6 | York Peppermint Patties        705
Length: 108, dtype: int64

The column names needed some clean up for easier use so the following steps were taken. The codes were run multiple times as I kept finding inconsistencies.

Cleaning up column names to remove question numbers: Q1, Q2, etc.

candy_survey_df.columns = candy_survey_df.columns.str[4:] #strip the first 4 leading characters

candy_survey_df.columns

Index(['GOING OUT?', 'GENDER', 'AGE', 'COUNTRY',
       'STATE, PROVINCE, COUNTY, ETC', ' 100 Grand Bar',
       ' Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       ' Any full-sized candy bar', ' Black Jacks', ' Bonkers (the candy)',
       ...
       ' Three Musketeers', ' Tolberone something or other', ' Trail Mix',
       ' Twix',
       ' Vials of pure high fructose corn syrup, for main-lining into your vein',
       ' Vicodin', ' Whatchamacallit Bars', ' White Bread',
       ' Whole Wheat anything', ' York Peppermint Patties'],
      dtype='object', length=108)

Now that the question numbers were removed, I noticed that there were leading white spaces in some of the column names that needed to be removed.

candy_survey_df.columns = candy_survey_df.columns.str.strip() #strip leading and trailing whitespace
candy_survey_df.columns

Index(['GOING OUT?', 'GENDER', 'AGE', 'COUNTRY',
       'STATE, PROVINCE, COUNTY, ETC', '100 Grand Bar',
       'Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Any full-sized candy bar', 'Black Jacks', 'Bonkers (the candy)',
       ...
       'Three Musketeers', 'Tolberone something or other', 'Trail Mix', 'Twix',
       'Vials of pure high fructose corn syrup, for main-lining into your vein',
       'Vicodin', 'Whatchamacallit Bars', 'White Bread',
       'Whole Wheat anything', 'York Peppermint Patties'],
      dtype='object', length=108)

Renamed specific columns to make them easier to understand. For example: "Anonymous brown globs that come in black and orange wrappers (a.k.a. Mary Janes)" was not a column name that I wanted to work with.

candy_survey_df.rename(columns={ 'GOING OUT?':'Going Out','Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)':'Mary Janes','Chick-o-Sticks (we don’t know what that is)':'Chick-o-Sticks','Sourpatch Kids (i.e. abominations of nature)':'Sourpatch Kids','Tolberone something or other':'Tolberone','Vials of pure high fructose corn syrup, for main-lining into your vein':'high fructose corn syrup'}, inplace=True)

candy_survey_df.columns

Index(['Going Out', 'GENDER', 'AGE', 'COUNTRY', 'STATE, PROVINCE, COUNTY, ETC',
       '100 Grand Bar', 'Mary Janes', 'Any full-sized candy bar',
       'Black Jacks', 'Bonkers (the candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'high fructose corn syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat anything', 'York Peppermint Patties'],
      dtype='object', length=108)

Converted all column names to title case.

candy_survey_df.columns = candy_survey_df.columns.str.title()

candy_survey_df.columns

Index(['Going Out', 'Gender', 'Age', 'Country', 'State, Province, County, Etc',
       '100 Grand Bar', 'Mary Janes', 'Any Full-Sized Candy Bar',
       'Black Jacks', 'Bonkers (The Candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'High Fructose Corn Syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat Anything', 'York Peppermint Patties'],
      dtype='object', length=108)

After additional work, I noticed that some columns had an unexpected character: 'õ'.

candy_survey_df.columns = candy_survey_df.columns.str.replace('õ', '\'')

candy_survey_df.columns

Index(['Going Out', 'Gender', 'Age', 'Country', 'State, Province, County, Etc',
       '100 Grand Bar', 'Mary Janes', 'Any Full-Sized Candy Bar',
       'Black Jacks', 'Bonkers (The Candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'High Fructose Corn Syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat Anything', 'York Peppermint Patties'],
      dtype='object', length=108)

After column name work it was time to move on to the Age and Country columns.

Clean up the Age and Country columns

The Age and Country survey questions allowed for answers in free-form text fields so a lot of my time was focused on cleaning these.

Country

I started by viewing how many unique country values existed and the country names and counts. There were 128 which could have been true values or not. I needed to do some more digging.

candy_survey_df.Country.nunique()

128

candy_survey_df.Country.value_counts()

USA                             699
United States                   497
usa                             217
Canada                          179
Usa                             139
                               ... 
Korea                             1
U.S.                              1
Indonesia                         1
The United States of America      1
Ahem....Amerca                    1
Name: Country, Length: 128, dtype: int64

In sampling some of the dataset, I immediately noticed inconsistencies in how countries were named, the most visible was for the United States of America.

candy_survey_df.sample(20)

My first bit of cleanup was to remove any leading or trailing whitespace in the Country column.

candy_survey_df.Country = candy_survey_df.Country.str.strip()

candy_survey_df

The Country text-field allowed for numerous spellings and namings of countries and the majority of my project cleaning was spent here. I repeatedly cycled through replace and value counts functions to find and merge as many inconsistencies as possible.

candy_survey_df.Country.value_counts(ascending=True) #Sorting country values from least amount to greatest.

Ahem....Amerca      1
Indonesia           1
Korea               1
New Jersey          1
spain               1
                 ... 
Usa               140
Canada            187
usa               217
United States     534
USA               772
Name: Country, Length: 117, dtype: int64

#Replacing country names to consolidate names.
candy_survey_df.Country = candy_survey_df.Country.replace(['united States','United Statea','united ststes','murrika','USA? Hard to tell anymore..','U S','usas','u s a','The United States of America','unite states','USA! USA! USA!','america','United Sates','Unied States','United State','America','Unites States','United Stated','USSA','USAUSAUSA','The United States','u.s.','Unites States,''america','US of A','united states of america','USA USA USA!!!!','United ststes','\'merica', 'U.S.A.','u.s.a.','U.S.','United staes','United Statss','Us','us', 'USa', 'USAA', 'Usa', 'USA','usa','US', 'united states','United States of America','United states'],'United States')
candy_survey_df.Country = candy_survey_df.Country.replace(['Canada`','CANADA','canada','Can'], 'Canada')
candy_survey_df.Country = candy_survey_df.Country.replace(['U.K.','endland','uk','Uk','UK','United kingdom','England'],'United Kingdom')
candy_survey_df.Country = candy_survey_df.Country.replace(['germany'],'Germany')
candy_survey_df.Country = candy_survey_df.Country.replace(['australia'],'Australia')
candy_survey_df.Country = candy_survey_df.Country.replace(['france'],'France')

#Creating a new series of only the top 15 countries
top_countries = candy_survey_df.Country.value_counts().head(15)

top_countries

United States     2037
Canada             226
United Kingdom      35
Germany             10
Australia            7
Netherlands          6
Scotland             5
Japan                5
Ireland              4
Mexico               4
Switzerland          3
France               3
China                2
Sweden               2
Denmark              2
Name: Country, dtype: int64

There were so many minuscule country counts that became difficult and timeconsuming to catch them all. I decided that my project would focus on the top 10-20 countries.

Age

I expected that there could be white space in the Age data, as I learned with the Country data, so my first step was to strip that.

candy_survey_df.Age = candy_survey_df.Age.str.strip()
candy_survey_df.Age.value_counts()

40                92
34                90
37                89
43                86
36                79
                  ..
old enough         1
46 Halloweens.     1
77                 1
99                 1
5u                 1
Name: Age, Length: 106, dtype: int64

Reviewing the value counts showed that there were non-numerical values. All non-numerical values were converted to null.

candy_survey_df.Age = pd.to_numeric(candy_survey_df.Age, errors='coerce')

candy_survey_df.Age.value_counts()

40.0    92
34.0    90
37.0    89
43.0    86
36.0    79
        ..
77.0     1
88.0     1
4.0      1
99.0     1
1.0      1
Name: Age, Length: 82, dtype: int64

candy_survey_df.Age.describe()

count    2351.000000
mean       42.605444
std        23.844665
min         1.000000
25%        34.000000
50%        41.000000
75%        50.000000
max      1000.000000
Name: Age, dtype: float64

Upon reviewing a summary of the age values, I noticed a problem because there was a max value of 1000. I'd be very concerned if there were people who are 200+ years old answering the survey. I chose to drop the rows with values greater than 110.

candy_survey_df.drop(candy_survey_df[candy_survey_df.Age>110].index, inplace=True)

candy_survey_df.Age.describe()

count    2349.000000
mean       42.083184
std        12.144721
min         1.000000
25%        34.000000
50%        41.000000
75%        50.000000
max       102.000000
Name: Age, dtype: float64

I reviewed the dataframe again for any final touchups and noticed that the very first row contains null values.

candy_survey_df

candy_survey_df = candy_survey_df.drop(candy_survey_df.index[0]) #Dropping first row as it's filled with null values

candy_survey_df

After all of that work, I felt that my dataset was clean enough to go forward with visualization.

Exploratory Analysis and Visualization

My plan going forward was to use Matplotlib and Seaborn for visualization.

I started by importing matplotlib and seaborn and then setting my chart styles.

import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

# chart styling
sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 12
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = 'white'

# Setting a Halloween themed colored palette
colors = ['#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e']
sns.set_palette(colors)

Going out or staying in?

The first question on the survey asked if the respondents had plans to go out to celebrate Halloween or to just stay home.

Let's start by seeing how many responses are left after clean-up.

candy_survey_df['Going Out']

1        No
2       NaN
3        No
4        No
5        No
       ... 
2455     No
2456     No
2457     No
2458     No
2459     No
Name: Going Out, Length: 2457, dtype: object

Next, let's look at how the 2,457 responses were spread.

candy_survey_df['Going Out'].value_counts() #View results of values

No     2037
Yes     311
Name: Going Out, dtype: int64

I used the seaborn .countplot() function to count the number of values since "Yes" and "No" aren't numerical values.

#Visualize in a bar chart
plt.figure(figsize=(12,6))
plt.xticks(rotation=0)
plt.title('Are you actually going trick or treating yourself?')
sns.countplot(y = 'Going Out', data = candy_survey_df);

The overwhelming majority of respondents said that they would not be going out trick or treating.

Gender

The gender question allowed for four answers: Male, Female, I'd Rather Not Say, and Other.

candy_survey_df.Gender.value_counts() #View results of values

Male                  1467
Female                 839
I'd rather not say      82
Other                   29
Name: Gender, dtype: int64

#Visualize in a bar chart
plt.figure(figsize=(16,6))
plt.xticks(rotation=0)
plt.title('Your gender')
sns.countplot(x = 'Gender', data = candy_survey_df);

I was surprised that the majority of respondents chose "Male" as their gender.

Age

Let's take a look at the ages reported by respondents.

candy_survey_df.Age.value_counts() # View counts of Ages of respondents

40.0     92
34.0     90
37.0     89
43.0     86
36.0     79
         ..
4.0       1
39.4      1
74.0      1
102.0     1
99.0      1
Name: Age, Length: 80, dtype: int64

#Import NumPy
import numpy as np

#Display age range in a histogram
plt.figure(figsize=(12,6))
plt.xticks(rotation=0)
plt.title('How old are you? (in years)')
plt.xlabel('Age')
plt.ylabel('Number of respondents')

plt.hist(candy_survey_df.Age, bins=np.arange(0,105,5), color='#ff9936')

(array([  2.,  10.,  27.,  30.,  62., 151., 340., 391., 407., 326., 255.,
        161., 108.,  41.,  25.,   6.,   0.,   1.,   2.,   3.]),
 array([  0,   5,  10,  15,  20,  25,  30,  35,  40,  45,  50,  55,  60,
         65,  70,  75,  80,  85,  90,  95, 100]),
 )

The answers to the Age survey question were interesting because the data included ages that were too young to fill out the survey themselves. It led me to believe that an older parent/guardian answered in a younger person's place.

Country

What country do you live in?

With the knowledge that the respondents' countries weighed heavily amongst the two countries, I decided to narrow my analysis to only the top 10 countries.

#Creating a series of only the countries in the top 10 of responses
top_countries = candy_survey_df.Country.value_counts().head(10)

top_countries

United States     2037
Canada             225
United Kingdom      35
Germany             10
Australia            7
Netherlands          6
Scotland             5
Japan                5
Ireland              4
Mexico               4
Name: Country, dtype: int64

#Visualize the top 10 countries of responses in a bar chart
plt.figure(figsize=(16, 8))
plt.xticks(rotation=0)
plt.title('What country do you live in?')
sns.countplot(y = 'Country', data = candy_survey_df, order=candy_survey_df.Country.value_counts().iloc[0:10].index);

Respondents were primarily located in the United States which I found interesting since the UBC is a Canadian institution.

Candy Ratings: Joy or Despair?

The primary focus of the survey was to collect data on whether people liked, disliked, or had no strong opinion on candies. The ratings were: Joy, Despair, and Meh.

I started by creating a subset of the dataframe that only included candy ratings.

candy_df = candy_survey_df.iloc[0: , 5:]

candy_df