{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Movie Body Counts\n",
    "\n",
    "**Author**: [Ramiro Gómez](http://ramiro.org/)\n",
    "\n",
    "A look at movie body counts based on information from the Website [Movie Body Counts](http://www.moviebodycounts.com/).\n",
    "\n",
    "## About the data source\n",
    "\n",
    "Movie Body Counts is a forum where users collect on-screen body counts for a selection of films and the characters and actors who appear in these films. The dataset currently contains counts for 545 films from 1949 to 2013, which is a very small sample of all the films produced in this time frame.\n",
    "\n",
    "To be counted a kill and/or dead body has to be visible on the screen, implied deaths like those died in the explosion of the Death Star are not counted. For more details on how counts should be conducted see [their guidelines](http://moviebodycounts.proboards.com/index.cgi?board=general&action=display&thread=6), the first one reads:\n",
    "\n",
    ">The \"body counts\" for this site are mostly \"on screen kills/deaths\" or fatal/critical/mortal shots/hits of human, humanoid, or creatures (ie monsters, aliens, zombies.) The rule of thumb is \"do they bleed\" which will leave the concept of cyborgs somewhat open and decided per film. The human and creature counts should be separate. These will be added together for a final tally.\n",
    "\n",
    "Apart from the small number of films in this dataset, we can safely assume a [selection bias](https://en.wikipedia.org/wiki/Selection_bias). So take this exploration with a grain of salt and don't generalize any of the results. This is mainly a demonstration of some of things you can to with the tools being used and a fun dataset to look at.\n",
    "\n",
    "The [CSV dataset](http://figshare.com/articles/On_screen_movie_kill_counts_for_hundreds_of_films/889719) is kindly provided by [Randal Olson](http://www.randalolson.com/) ([@randal_olson](https://twitter.com/randal_olson)), who took the effort of collecting the death toll data from Movie Body Counts and added MPAA and IMDB ratings as well as film lengths."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages\n",
    "\n",
    "To explore and visualize the data I'll be using several Python packages that greatly facilitate these tasks, namely: [NumPy](http://numpy.org/), [pandas](http://pandas.pydata.org/) and [matplotlib](http://matplotlib.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "No module named 'signature'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-94d5594567ac>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'load_ext signature'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'matplotlib inline'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mmagic\u001b[0;34m(self, arg_s)\u001b[0m\n\u001b[1;32m   2305\u001b[0m         \u001b[0mmagic_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marg_s\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpartition\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m' '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2306\u001b[0m         \u001b[0mmagic_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmagic_name\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefilter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mESC_MAGIC\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2307\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2308\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2309\u001b[0m     \u001b[0;31m#-------------------------------------------------------------------------\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_line_magic\u001b[0;34m(self, magic_name, line)\u001b[0m\n\u001b[1;32m   2226\u001b[0m                 \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'local_ns'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_getframe\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf_locals\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2227\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2228\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2229\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2230\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/magics/extension.py\u001b[0m in \u001b[0;36mload_ext\u001b[0;34m(self, module_str)\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    191\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    192\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 193\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    194\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    195\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/magics/extension.py\u001b[0m in \u001b[0;36mload_ext\u001b[0;34m(self, module_str)\u001b[0m\n\u001b[1;32m     61\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mmodule_str\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mUsageError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Missing module name.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m         \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshell\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextension_manager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_extension\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule_str\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'already loaded'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/jstrick/.pyenv/versions/anaconda3-2.3.0/lib/python3.4/site-packages/IPython/core/extensions.py\u001b[0m in \u001b[0;36mload_extension\u001b[0;34m(self, module_str)\u001b[0m\n\u001b[1;32m     83\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mmodule_str\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mprepended_to_syspath\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mipython_extension_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m                     \u001b[0m__import__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule_str\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m             \u001b[0mmod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mmodule_str\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_load_ipython_extension\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mImportError\u001b[0m: No module named 'signature'"
     ]
    }
   ],
   "source": [
    "%load_ext signature\n",
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.style.use('ramiro')\n",
    "\n",
    "chartinfo = 'Author: Ramiro Gómez - ramiro.org • Data: Movie Body Counts - moviebodycounts.com'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data and first look\n",
    "\n",
    "We can directly download the CSV file from the Web and read it into a pandas [DataFrame](http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.html) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('http://files.figshare.com/1332945/film_death_counts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a grasp of the data let's look at the first few lines of the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset looks pretty well suited for doing some explorations and visualizations. I'll rename some columns to have shorter and a little nicer labels later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.columns = ['Film', 'Year', 'Body count', 'MPAA', 'Genre', 'Director', 'Minutes', u'IMDB']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also add a `Film count` column to keep track of the number of films when grouping and the body count per minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['Film count'] = 1\n",
    "df['Body count/min'] = df['Body count'] / df['Minutes'].astype(float)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Body counts over time\n",
    "\n",
    "Next we look at how the number of body counts has evolved over the time frame covered by the dataset. To do so the `DataFrame` is grouped by year calculating the means, medians, and sums of the numeric columns. Also for a change print the last few records. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "group_year = df.groupby('Year').agg([np.mean, np.median, sum])\n",
    "group_year.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `group_year` `DataFrame` now contains several columns, that are not useful, like the mean and median film count. We simply don't use them, but instead look at the film and body counts.\n",
    "\n",
    "With matplotlib's `subplots` function multiple graphs can be combined into one graphic. This allows comparing several distributions that have differing scales, as is the case for the film, total and average body counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_bc = pd.DataFrame({'mean': group_year['Body count']['mean'],\n",
    "                      'median': group_year['Body count']['median']})\n",
    "\n",
    "df_bc_min = pd.DataFrame({'mean': group_year['Body count/min']['mean'], \n",
    "                          'median': group_year['Body count/min']['median']})\n",
    "\n",
    "fig, axes = plt.subplots(nrows=4, ncols=1, figsize=(16, 22))\n",
    "\n",
    "group_year['Film count']['sum'].plot(kind='bar', ax=axes[0]); axes[0].set_title('Film Count')\n",
    "group_year['Body count']['sum'].plot(kind='bar', ax=axes[1]); axes[1].set_title('Total Body Count')\n",
    "df_bc.plot(kind='bar', ax=axes[2]); axes[2].set_title('Body Count by Film')\n",
    "df_bc_min.plot(kind='bar', ax=axes[3]); axes[3].set_title('Body Count by Minute')\n",
    "\n",
    "for i in range(4):\n",
    "    axes[i].set_xlabel('', visible=False)\n",
    "    \n",
    "plt.annotate(chartinfo, xy=(0, -1.2), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we can safely say is, that most films in our dataset are from 2007. What this also shows quite well is the selection bias. There is only one film reviewed for each of the years 1978 and 2013, both have a pretty high body count.\n",
    "\n",
    "## Most violent films\n",
    "\n",
    "Now lets see which films have the highest total body counts and body counts per minute. This time we plot two horizontal bar charts next to each other, again using the `subplots` function.\n",
    "\n",
    "Note that sorting is ascending by default, so we call `tail` to get the top 10 films each with the highest total body count and the highest body counts per minute. We could set `ascending` to `False` in the `sort` call and use `head`, but this would plot the highest value on the bottom. Also the y-axis labels of the right chart a printed on the right, so they don't overlap with the left one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_film = df.set_index('Film')\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 8))\n",
    "\n",
    "bc = df_film.sort('Body count')['Body count'].tail(10)\n",
    "bc.plot(kind='barh', ax=axes[0])\n",
    "axes[0].set_title('Total Body Count')\n",
    "\n",
    "bc_min = df_film.sort('Body count/min')['Body count/min'].tail(10)\n",
    "bc_min.plot(kind='barh', ax=axes[1])\n",
    "axes[1].set_title('Body Count per Minute')\n",
    "axes[1].yaxis.set_ticks_position('right')\n",
    "\n",
    "for i in range(2):\n",
    "    axes[i].set_ylabel('', visible=False)\n",
    "    \n",
    "plt.annotate(chartinfo, xy=(0, -1.07), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a considerable gap between **Lord of the Rings** and the runner-up **Kingdom of Heaven** in the left chart, but when you take runtime into account, the later is slightly more violent. Both of them are surpassed by **300** when one looks at deaths by minute, which shouldn't surprise anyone who saw it. Below you can see why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('HdNn5TZu6R8', width=800, height=450)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most violent directors\n",
    "\n",
    "Now let's look at directors. As you may have noticed above the `Genre` column can contain multiple values separated by `|` characters. This also applies to the `Director` column, here are some examples.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df[df['Director'].apply(lambda x: -1 != x.find('|'))].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I want to group by directors later, I have to decide what to do with these multi-value instances. So what I'll do is create a new data frame with one new row for a single director and multiple new rows with the same count values for films that have more than one director. I also considered dividing the body counts by the number of directors, but decided against it.\n",
    "\n",
    "The following function does this. I feel that there is a more elegant way with pandas, but it works for arbitrary columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def expand_col(df_src, col, sep='|'):\n",
    "    di = {}\n",
    "    idx = 0\n",
    "    for i in df_src.iterrows():\n",
    "        d = i[1]\n",
    "        names = d[col].split(sep)\n",
    "        for name in names:\n",
    "            # operate on a copy to not overwrite previous director names\n",
    "            c = d.copy()\n",
    "            c[col] = name\n",
    "            di[idx] = c\n",
    "            idx += 1\n",
    "\n",
    "    df_new = pd.DataFrame(di).transpose()\n",
    "    # these two columns are not recognized as numeric\n",
    "    df_new['Body count'] = df_new['Body count'].astype(float)\n",
    "    df_new['Body count/min'] = df_new['Body count/min'].astype(float)\n",
    "    \n",
    "    return df_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now similar to the film ranking let's plot a director ranking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_dir = expand_col(df, 'Director')\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(14, 8))\n",
    "\n",
    "bc_sum = df_dir.groupby('Director').sum().sort('Body count').tail(10)\n",
    "bc_sum['Body count'].plot(kind='barh', ax=axes[0])\n",
    "axes[0].set_title('Total Body Count')\n",
    "\n",
    "bc_mean = df_dir.groupby('Director').agg(np.mean).sort('Body count/min').tail(10)\n",
    "bc_mean['Body count/min'].plot(kind='barh', ax=axes[1])\n",
    "axes[1].set_title('Body Count per Minute')\n",
    "axes[1].yaxis.set_ticks_position('right')\n",
    "\n",
    "for i in range(2):\n",
    "    axes[i].set_ylabel('', visible=False)\n",
    "\n",
    "plt.annotate(chartinfo, xy=(0, -1.07), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Body counts in film genres\n",
    "\n",
    "As mentioned above `Genre` is a multi-value column too. So let's create a new data frame again, where each film can account for multiple genres and look at the frequency distribution of films by genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_genre = expand_col(df, 'Genre')\n",
    "df_genre['Genre'].value_counts().plot(kind='bar', figsize=(12, 6), title='Genres by film count')\n",
    "\n",
    "plt.annotate(chartinfo, xy=(0, -1.28), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the total body counts for genres doesn't make much sense since some genres occur much more frequently, instead let's see genres by body counts per minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bc_mean = df_genre.groupby('Genre').agg(np.mean).sort('Body count/min', ascending=False)\n",
    "ax = bc_mean['Body count/min'].plot(kind='bar', figsize=(12, 6), title='Genres by body count per minute')\n",
    "ax.set_xlabel('', visible=False)\n",
    "plt.annotate(chartinfo, xy=(0, -1.32), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not a huge surprise to see war movies on top, and since many of them are also classified as history movies this genre comes in 2nd place. Also several of the most deadly films are counted in these two genres, see some examples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_genre[(df_genre['Genre'] == 'War') | (df_genre['Genre'] == 'History')].sort('Body count/min', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MPAA and IMDB Ratings\n",
    "\n",
    "Finally let's look at the MPAA and IMDB ratings and how they relate to the movie body counts by creating two scatter plots.\n",
    "\n",
    "Since MPAA ratings are not numeric, their values need to be mapped to numbers in some way to produce a scatter plot. We can use the `value_counts` method to get a sorted series of different MPPA ratings and their counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ratings = df['MPAA'].value_counts()\n",
    "ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next the different rating names are used as keys of a dictionary mapped to a list of integers of the same length. This dictionary is then used to map the different rating values of the MPAA column to the corresponding integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rating_names = ratings.index\n",
    "rating_index = range(len(rating_names))\n",
    "rating_map = dict(zip(rating_names, rating_index))\n",
    "mpaa = df['MPAA'].apply(lambda x: rating_map[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a scatter plot with the following few lines of code, where the MPAA ratings are show on the x-axis, the body counts per minute on the y-axis and the circle sizes are determined by the total body counts of the movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(14, 10))\n",
    "ax.scatter(mpaa, df['Body count/min'], s=df['Body count'], alpha=.5)\n",
    "ax.set_title('Body counts and MPAA ratings')\n",
    "ax.set_xlabel('MPAA Rating')\n",
    "ax.set_xticks(rating_index)\n",
    "ax.set_xticklabels(rating_names)\n",
    "ax.set_ylabel('Body count per minute')\n",
    "plt.annotate(chartinfo, xy=(0, -1.12), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the things this diagram shows is that the film with highest body count and also a pretty high body count/min is rated PG-13. Looking back at the film rankings above, we know that it is **Lord of the Rings: Return of the King**, but wouldn't it be nice to have labels for at least some of the circles? Yes, so annotating graphs will be demonstrated in the next scatter plot, which shows body counts and IMDB ratings.\n",
    "\n",
    "To not mess up the graph, only the 3 movies with the highest body count will be labeled. They go into a list of lists called `annotations`, where the inner lists are made up of the label text and the x and the y positions of the labels. Then after setting up the basic plot the annotations are added to it in a loop. The label positions can be adjusted with the `xytext`, `textcoords`, `ha`, and `va` arguments to the `annotate` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bc_top = df.sort('Body count', ascending=False)[:3]\n",
    "annotations = []\n",
    "for r in bc_top.iterrows():\n",
    "    annotations.append([r[1]['Film'], r[1]['IMDB'], r[1]['Body count/min']])\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(14, 10))\n",
    "ax.scatter(df['IMDB'], df['Body count/min'], s=df['Body count'], alpha=.5)\n",
    "ax.set_title('Body count and IMDB ratings')\n",
    "ax.set_xlabel('IMDB Rating')\n",
    "ax.set_ylabel('Body count per minute')\n",
    "\n",
    "for annotation, x, y in annotations:\n",
    "    plt.annotate(\n",
    "        annotation,\n",
    "        xy=(x, y),\n",
    "        xytext=(0, 30),\n",
    "        textcoords='offset points',\n",
    "        ha='center',\n",
    "        va='bottom',\n",
    "        size=12.5,\n",
    "        bbox=dict(boxstyle='round,pad=0.5', fc='yellow', alpha=0.5),\n",
    "        arrowprops=dict(arrowstyle='-'))\n",
    "\n",
    "plt.annotate(chartinfo, xy=(0, -1.12), xycoords='axes fraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook demonstrates some of the basic features of pandas, NumPy and matplotlib for processing, exploring and visualizing data.\n",
    "\n",
    "Due to the dataset's limitations mentioned in the introduction, I refrained from interpreting the results too much. The focus of this notebook is how you can use these tools to get to know a dataset. They offer many more possibilities and advanced features are all free and open source. I can only recommend using them and will certainly keep on doing so myself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%signature"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
