Titanic Disaster: Survivability Parameters¶
In [1]:
# Analysis performed with Python 3.5.1
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import os
%pylab inline
script_dir = os.getcwd()
file = "titanic_data.csv"
abs_file_path = os.path.join(script_dir, file)
titanic_df = pd.read_csv(abs_file_path)
Problem Statement¶
- Dataset Description
- Using data analysis methods, predict which metric or combination of metrics best predict passenger survivability.
- A combination of data visualizations and statistics will be used to determine the most significant predictors of survivability.
Dataset Exploration¶
In [2]:
# Head of the dataset
titanic_df.head()
Out[2]:
In [3]:
# Tail of the dataset
titanic_df.tail()
Out[3]:
In [4]:
# Determine which parameters have missing values
titanic_df.info()
- Name, SibSp, Parch, Ticket and Fare will not be used
- Cabin will not be used because less the 25% of passengers have cabin data
- Missing Age data will be filled in the Age section
- Missing Embarked data will be ignored
In [5]:
# Give gender a numeric value; 0 = male, 1 = female
titanic_df['Sex_Numeric'] = (titanic_df['Sex'].astype('category')).cat.codes
In [6]:
grouped_survived = titanic_df.groupby(['Sex_Numeric', 'Pclass', 'Age', 'Embarked'])
In [7]:
grouped_survived['Survived'].describe()
Out[7]:
In [8]:
# Create Survival Label Column
titanic_df['Survival'] = titanic_df.Survived.map({0 : 'Died', 1 : 'Survived'})
titanic_df.Survival.head()
Out[8]:
In [9]:
# Create Pclass Label Column
titanic_df['Class'] = titanic_df.Pclass.map({1 : '1st Class', 2 : '2nd Class', 3 : '3rd Class'})
titanic_df.Class.head()
Out[9]:
In [10]:
# Create Sex Label Column
titanic_df['Gender'] = titanic_df.Sex.map({'female' : 'Female', 'male' : 'Male'})
titanic_df.Gender.head()
Out[10]:
In [11]:
# Replace blanks with NaN
titanic_df['Embarked'].replace(r'\s+', np.nan, regex=True).head()
Out[11]:
In [12]:
# Create Port Label Column
titanic_df['Ports'] = titanic_df.Embarked.map({'S' : 'Southhampton', 'C' : 'Cherbourg', 'Q' : 'Queenstown', NaN : 'unknown'})
titanic_df.Ports.head()
Out[12]:
Dataset Plots¶
In [13]:
# Mosaic Chart
plt.rc('figure', figsize=(17, 5))
from statsmodels.graphics.mosaicplot import mosaic
mosaic(titanic_df,['Survival', 'Class', 'Gender'], axes_label=False, title='Survival: Red=Died, Green=Survived')
plt.xlabel('Gender: Male & Female')
plt.ylabel('Passenger Class: 1st, 2nd & 3rd Class')
plt.show()
In [14]:
# Figure size
plt.rc('figure', figsize=(15, 15))
# Histogram bin size
bin_size = 10
# Setup a figue of plots
#fig = plt.figure()
fig_size = (3, 2)
# Plot mortality counts
plt.subplot2grid(fig_size, (0, 0))
titanic_df['Survival'].value_counts().plot(kind='bar', title='Mortality')
plt.xticks(rotation=0)
plt.ylabel('Count')
# Plot Pclass counts
plt.subplot2grid(fig_size, (0, 1))
titanic_df['Class'].value_counts().plot(kind='bar', title='Passenger Class')
plt.xticks(rotation=0)
plt.ylabel('Count')
# Plot Gender counts
plt.subplot2grid(fig_size, (1, 0))
titanic_df['Gender'].value_counts().plot(kind='bar', title='Gender')
plt.xticks(rotation=0)
plt.ylabel('Count')
# Plot Port counts
plt.subplot2grid(fig_size, (1, 1))
titanic_df['Ports'].value_counts().plot(kind='bar', title='Ports of Embarkation')
plt.xticks(rotation=0)
plt.ylabel('Count')
# Plot Age histogram
plt.subplot2grid(fig_size, (2, 0))
titanic_df['Age'].hist()
plt.title('Age')
plt.ylabel('Count')
plt.xlabel('Age Categories by Decade (years)')
Out[14]:
Age¶
In [15]:
# Passangers with no age
ageisnull = titanic_df[titanic_df['Age'].isnull()]
ageisnull.head()
Out[15]:
In [16]:
print('Total passengers with no age: ', len(ageisnull))
In the Dataset Exploration section, it was determined there were only 714 of 891 valid age related records. We can see there are 177 NaN entries for Age.
In [17]:
# Mean age
titanic_df['Age'].mean()
Out[17]:
In [18]:
# Mean age by Sex
(titanic_df.groupby(['Gender']))['Age'].mean()
Out[18]:
In [19]:
# Mean age by Pclass and Sex
(titanic_df.groupby(['Class', 'Gender']))['Age'].mean()
Out[19]:
In [20]:
# Mean age by Pclass, Survived and Sex
(titanic_df.groupby(['Class', 'Survival', 'Gender']))['Age'].mean()
Out[20]:
In [21]:
# General statistics of Age by Class, Survival and Gender
(titanic_df.groupby(['Class', 'Survival', 'Gender']))['Age'].describe()
Out[21]:
In [22]:
# Survival count by Sex, Pclass and Age < 20
sex = titanic_df['Gender']
survived = titanic_df['Survival']
pclass = titanic_df['Class']
age_youth = titanic_df['Age'] < 20
pd.crosstab([sex, pclass, age_youth], survived)
Out[22]:
A decision is required to determine the best method of dealing with NaN values.
- The NaN values can be ignored
- NaN can be filled in with a value, typically a mean
- Comparing the counts for various groups leads to the conclusion, simply using the overall mean will heavily weigh one specific age and skew any age dependant results.
- For the remainder of this analytic process, the NaN values data will be replaced with a mean age based upon Pclass, Survived and Sex.
In [23]:
# Maintain Age and create Age_Fill (populate missing ages)
titanic_df['Age_Fill'] = titanic_df['Age']
# Populate Age_Fill
titanic_df['Age_Fill'] = titanic_df['Age_Fill'] \
.groupby([titanic_df['Pclass'], titanic_df['Survived'], titanic_df['Sex']]) \
.apply(lambda x: x.fillna(x.mean()))
Create a new category called Age_Fill and fill NaN with an age based upon the mean of Pclass, Survived and Sex.
In [24]:
# Example of Age_Fill - #5, 17 & 19
print(titanic_df['Age'].head(20))
print(titanic_df['Age_Fill'].head(20))
Age Histogram Comparison¶
In [25]:
# Set figure size
plt.rc('figure', figsize=(10, 10))
# Setup a figue of plots
df1 = titanic_df[titanic_df['Survived'] == 0]['Age']
df2 = titanic_df[titanic_df['Survived'] == 1]['Age']
df3 = titanic_df[titanic_df['Survived'] == 0]['Age_Fill']
df4 = titanic_df[titanic_df['Survived'] == 1]['Age_Fill']
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Age Group')
plt.xlabel('Age')
plt.ylabel('Count')
max_age = max(titanic_df['Age_Fill'])
plt.subplot(311)
plt.hist([df1, df2],
bins=8,
range=(1, max_age),
stacked=True)
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Age Group (not filled)')
plt.ylabel('Count')
plt.subplot(312)
plt.hist([df3, df4],
bins=8,
range=(1, max_age),
stacked=True)
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Age Group (filled)')
plt.xlabel('Age')
plt.ylabel('Count')
plt.show()
In [26]:
# Maximum age
titanic_df['Age'].max()
Out[26]:
In [27]:
# Create a new column that has all ages by bin category: 0-10:10, 10-20:20, 20-30:30, 30-40:40
# 40-50:50, 50-60:60, 60-70:70, 70-80:80
bins = [0, 10, 20, 30, 40, 50, 60, 70, 80]
group_names = [10, 20, 30, 40, 50, 60, 70, 80]
titanic_df['Age_Categories'] = pd.cut(titanic_df['Age_Fill'], bins, labels=group_names)
titanic_df[['Age', 'Age_Fill', 'Age_Categories']].head()
Out[27]:
In [28]:
titanic_df['Age_Categories'] = pd.to_numeric(titanic_df['Age_Categories'])
An Age_Categories column has been inserted into the dataframe to simplify certain visualizations and calculations, as there are to many individual ages to easily draw conclusions or see patterns.
In [29]:
# Survival Count by Age_Categories
titanic_df.groupby('Survival')[['Age_Categories']].count()
Out[29]:
Age Mosaic¶
In [30]:
# Mosaic Plot
plt.rc('figure', figsize=(18, 6)) # figure size
from statsmodels.graphics.mosaicplot import mosaic
mosaic(titanic_df,['Survival', 'Class', 'Age_Categories'], axes_label=False, title='Survival: Red=Died, Green=Survived')
plt.xlabel('Age Categories by Decades (years)')
plt.ylabel('Passenger Class: 1st, 2nd & 3rd Class')
plt.show()
In [31]:
# Mosaic Plot
mosaic(titanic_df,['Survival', 'Gender', 'Age_Categories'], axes_label=False, title='Survival: Red=Died, Green=Survived')
plt.xlabel('Age Categories by Decades (years)')
plt.ylabel('Gender: Male & Female')
plt.show()
Pclass¶
In [32]:
# Survival count by Pclass
pclass_ct = titanic_df.groupby('Class')['Survival'].value_counts().unstack()
pclass_ct
Out[32]:
In [33]:
# Survival Rate
titanic_df.groupby('Class')['Survival'].value_counts(normalize = True).unstack()
Out[33]:
In [34]:
# Set figure size
plt.rc('figure', figsize=(10, 5))
# Setup a figue of plots
pclass_ct.plot(kind='bar',
stacked=True)
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Pclass')
plt.xlabel('Pclass')
plt.ylabel('Count')
plt.xticks(rotation=0)
plt.show()
Pclass is not a strong indicator for surviving, however 3rd Class is a stong indicator for dying.
Sex¶
In [35]:
# Survival count by sex
sex_ct = titanic_df.groupby('Gender')['Survival'].value_counts().unstack()
sex_ct
Out[35]:
In [36]:
# Survival rate by sex
titanic_df.groupby('Gender')['Survival'].value_counts(normalize = True).unstack()
Out[36]:
In [37]:
plt.rc('figure', figsize=(10, 5))
sex_ct.plot(kind='bar',
stacked=True)
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Gender')
plt.xlabel('Gender')
plt.ylabel('Count')
plt.xticks(rotation=0)
plt.show()
Gender is a strong indicator for survivability, with a significant portion of females (74%) surviving and males 81% dying.
Embarked¶
In [38]:
# Survival count by Embarked
embarked_ct = titanic_df.groupby('Ports')['Survival'].value_counts().unstack()
embarked_ct
Out[38]:
In [39]:
# Survival rate by embarked
titanic_df.groupby('Ports')['Survival'].value_counts(normalize = True).unstack()
Out[39]:
In [40]:
plt.rc('figure', figsize=(10, 5))
embarked_ct.plot(kind='bar',
stacked=True)
plt.legend(('Died', 'Survived'), loc='best')
plt.title('Survivors by Embarked')
plt.xlabel('Port of Embarkation')
plt.ylabel('Count')
plt.xticks(rotation=0)
plt.show()
Statistics¶
In [41]:
# Survival count by Sex, Embarked_Numeric, Pclass and Age Category
embarked = titanic_df['Ports']
sex = titanic_df['Gender']
survived = titanic_df['Survival']
pclass = titanic_df['Class']
age_cat = titanic_df['Age_Categories']
pd.crosstab([sex, embarked, pclass], [survived, age_cat])
Out[41]:
OLS Regression Models¶
In [42]:
import statsmodels.formula.api as sm
In [43]:
# OLS modeling for Survived and Gender
result_1 = sm.ols(formula='Survived ~ Gender', data=titanic_df).fit()
result_1.summary()
Out[43]:
In [44]:
# OLS modeling for Survived and Class
result_2 = sm.ols(formula='Survived ~ Class', data=titanic_df).fit()
result_2.summary()
Out[44]:
In [45]:
# OLS modeling for Survived and Ports
result_3 = sm.ols(formula='Survived ~ Ports', data=titanic_df).fit()
result_3.summary()
Out[45]:
In [46]:
# OLS modeling for Survived and Age_Fill
result_4 = sm.ols(formula='Survived ~ Age_Fill', data=titanic_df).fit()
result_4.summary()
Out[46]:
In [47]:
# OLS modeling for Survived and Gender + Class + Age_Fill + Ports
result_5 = sm.ols(formula='Survived ~ Gender + Class + Age_Fill + Ports', data=titanic_df).fit()
result_5.summary()
Out[47]:
In [48]:
# OLS modeling for Survived and Gender + Class + Age_Fill
result_6 = sm.ols(formula='Survived ~ Gender + Class + Age_Fill', data=titanic_df).fit()
result_6.summary()
Out[48]:
In [49]:
# Dataframe for statistical data
comp_index_4 = 'Gender + Class + Age_Fill + Ports'
comp_index_3 = 'Gender + Class + Age_Fill'
statistics_df = pd.DataFrame(
data=[[result_1.rsquared_adj, np.sqrt(result_1.rsquared_adj)],
[result_2.rsquared_adj, np.sqrt(result_2.rsquared_adj)],
[result_3.rsquared_adj, np.sqrt(result_3.rsquared_adj)],
[result_4.rsquared_adj, np.sqrt(result_4.rsquared_adj)],
[result_5.rsquared_adj, np.sqrt(result_5.rsquared_adj)],
[result_6.rsquared_adj, np.sqrt(result_6.rsquared_adj)]],
index=['Gender', 'Class', 'Ports', 'Age_Fill', comp_index_4, comp_index_3],
columns=['R-squared', 'Correlation to Survival']
)
statistics_df
Out[49]:
Ordinary least squares (OLS) regression modeling has been used to determine which metric or combination of metrics provides the best prediction of survival. As can be determined by reviewing the coefficient of determination (R-squared), the individual models for Ports and Age_Fill indicate a large proportion of variance for survival. Gender and a combination of metrics are better models. The square root of R-squared equals the Pearson correlation coefficient of predicted to actual values; Gender is the single metric with the strongest correlation. However, the combination of metrics, Gender + Class + Age_Fill + Ports, shows the strongest correlation to survival for the model used.