{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Analysis in Python\n",
    "\n",
    "In this section, we introduce a few useful methods for analyzing your data in Python.\n",
    "Namely, we cover how to compute the mean, variance, and standard error of a data set.\n",
    "For more advanced statistical analysis, we cover how to perform a\n",
    "Mann-Whitney-Wilcoxon (MWW) RankSum test, how to perform an Analysis of variance (ANOVA)\n",
    "between multiple data sets, and how to compute bootstrapped 95% confidence intervals for\n",
    "non-normally distributed data sets.\n",
    "\n",
    "### Python's SciPy Module\n",
    "\n",
    "The majority of data analysis in Python can be performed with the SciPy module. SciPy\n",
    "provides a plethora of statistical functions and tests that will handle the majority of\n",
    "your analytical needs. If we don't cover a statistical function or test that you require\n",
    "for your research, SciPy's full statistical library is described in detail at:\n",
    "http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html\n",
    "\n",
    "### Python's pandas Module\n",
    "\n",
    "The pandas module provides powerful, efficient, R-like DataFrame objects capable of\n",
    "calculating statistics en masse on the entire DataFrame. DataFrames are useful for when\n",
    "you need to compute statistics over multiple replicate runs.\n",
    "\n",
    "For the purposes of this tutorial, we will use Luis Zaman's parasite data set (available from https://raw.github.com/briandconnelly/BEACONToolkit/master/analysis/data/parasite_data.csv):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-1-4108366ca6bd>, line 6)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-4108366ca6bd>\"\u001b[0;36m, line \u001b[0;32m6\u001b[0m\n\u001b[0;31m    print experimentDF\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "from pandas import *\n",
    "\n",
    "# must specify that blank space \" \" is NaN\n",
    "experimentDF = read_csv(\"parasite_data.csv\", na_values=[\" \"])\n",
    "\n",
    "print(experimentDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing data in pandas DataFrames\n",
    "\n",
    "You can directly access any column and row by indexing the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-2-1e82fbf995ee>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-2-1e82fbf995ee>\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m    print experimentDF[\"Virulence\"]\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# show all entries in the Virulence column\n",
    "print( experimentDF[\"Virulence\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-3-751c68983807>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-3-751c68983807>\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m    print experimentDF[\"ShannonDiversity\"][12]\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# show the 12th row in the ShannonDiversity column\n",
    "print experimentDF[\"ShannonDiversity\"][12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also access all of the values in a column meeting a certain criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-4-2a1a3d66b230>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-4-2a1a3d66b230>\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m    print experimentDF[experimentDF[\"ShannonDiversity\"] > 2.0]\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# show all entries in the ShannonDiversity column > 2.0\n",
    "print experimentDF[experimentDF[\"ShannonDiversity\"] > 2.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Blank/omitted data (NA or NaN) in pandas DataFrames\n",
    "\n",
    "Blank/omitted data is a piece of cake to handle in pandas. Here's an example data\n",
    "set with NA/NaN values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-5-734e8205f92a>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-5-734e8205f92a>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    print experimentDF[isnan(experimentDF[\"Virulence\"])]\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "print experimentDF[isnan(experimentDF[\"Virulence\"])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrame methods automatically ignore NA/NaN values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-6-e28638fbfe34>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-6-e28638fbfe34>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    print \"Mean virulence across all treatments:\", experimentDF[\"Virulence\"].mean()\u001b[0m\n\u001b[0m                                                ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "print \"Mean virulence across all treatments:\", experimentDF[\"Virulence\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, not all methods in Python are guaranteed to handle NA/NaN values properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-7-34f65a37d36d>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-7-34f65a37d36d>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"Mean virulence across all treatments:\", stats.sem(experimentDF[\"Virulence\"])\u001b[0m\n\u001b[0m                                                ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "print \"Mean virulence across all treatments:\", stats.sem(experimentDF[\"Virulence\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, it behooves you to take care of the NA/NaN values before performing your analysis. You can either:\n",
    "\n",
    "**(1) filter out all of the entries with NA/NaN**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-8-97f02fa335f0>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-8-97f02fa335f0>\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m    print experimentDF.dropna()\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# NOTE: this drops the entire row if any of its entries are NA/NaN!\n",
    "print experimentDF.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you only care about NA/NaN values in a specific column, you can specify the\n",
    "column name first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-9-7bcdbde5649f>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-9-7bcdbde5649f>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    print experimentDF[\"Virulence\"].dropna()\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "print experimentDF[\"Virulence\"].dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(2) replace all of the NA/NaN entries with a valid value**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-10-14c081e6dcf1>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-10-14c081e6dcf1>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    print experimentDF.fillna(0.0)[\"Virulence\"]\u001b[0m\n\u001b[0m                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "print experimentDF.fillna(0.0)[\"Virulence\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take care when deciding what to do with NA/NaN entries. It can have a significant\n",
    "impact on your results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-11-e573a4dba811>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-11-e573a4dba811>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    print \"Mean virulence across all treatments w/ dropped NaN:\", experimentDF[\"Virulence\"].dropna().mean()\u001b[0m\n\u001b[0m                                                               ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "print \"Mean virulence across all treatments w/ dropped NaN:\", experimentDF[\"Virulence\"].dropna().mean()\n",
    "\n",
    "print \"Mean virulence across all treatments w/ filled NaN:\", experimentDF.fillna(0.0)[\"Virulence\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean\n",
    "\n",
    "The mean performance of an experiment gives a good idea of how the experiment will\n",
    "turn out *on average* under a given treatment.\n",
    "\n",
    "Conveniently, DataFrames have all kinds of built-in functions to perform standard\n",
    "operations on them en masse: `add()`, `sub()`, `mul()`, `div()`, `mean()`, `std()`, etc.\n",
    "The full list is located at: http://pandas.sourceforge.net/generated/pandas.DataFrame.html\n",
    "\n",
    "Thus, computing the mean of a DataFrame only takes one line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-12-2e1f3e657096>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-12-2e1f3e657096>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"Mean Shannon Diversity w/ 0.8 Parasite Virulence =\", experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"].mean()\u001b[0m\n\u001b[0m                                                             ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "from pandas import *\n",
    "\n",
    "print \"Mean Shannon Diversity w/ 0.8 Parasite Virulence =\", experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance\n",
    "\n",
    "The variance in the performance provides a measurement of how consistent the results\n",
    "of an experiment are. The lower the variance, the more consistent the results are, and\n",
    "vice versa.\n",
    "\n",
    "Computing the variance is also built in to pandas DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-13-9d187b3a8556>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-13-9d187b3a8556>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"Variance in Shannon Diversity w/ 0.8 Parasite Virulence =\", experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"].var()\u001b[0m\n\u001b[0m                                                                    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "from pandas import *\n",
    "\n",
    "print \"Variance in Shannon Diversity w/ 0.8 Parasite Virulence =\", experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"].var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Error of the Mean (SEM)\n",
    "\n",
    "Combined with the mean, the SEM enables you to establish a range around a mean that\n",
    "the majority of any future replicate experiments will most likely fall within.\n",
    "\n",
    "pandas DataFrames don't have methods like SEM built in, but since DataFrame\n",
    "rows/columns are treated as lists, you can use any NumPy/SciPy method you like on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-14-afe39d04d992>, line 4)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-14-afe39d04d992>\"\u001b[0;36m, line \u001b[0;32m4\u001b[0m\n\u001b[0;31m    print \"SEM of Shannon Diversity w/ 0.8 Parasite Virulence =\", stats.sem(experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"])\u001b[0m\n\u001b[0m                                                               ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "from pandas import *\n",
    "from scipy import stats\n",
    "\n",
    "print \"SEM of Shannon Diversity w/ 0.8 Parasite Virulence =\", stats.sem(experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single SEM will usually envelop 68% of the possible replicate means\n",
    "and two SEMs envelop 95% of the possible replicate means. Two\n",
    "SEMs are called the \"estimated 95% confidence interval.\" The confidence\n",
    "interval is estimated because the exact width depend on how many replicates\n",
    "you have; this approximation is good when you have more than 20 replicates.\n",
    "\n",
    "### Mann-Whitney-Wilcoxon (MWW) RankSum test\n",
    "\n",
    "The MWW RankSum test is a useful test to determine if two distributions are significantly\n",
    "different or not. Unlike the t-test, the RankSum test does not assume that the data\n",
    "are normally distributed, potentially providing a more accurate assessment of the data sets.\n",
    "\n",
    "As an example, let's say we want to determine if the results of the two following\n",
    "treatments significantly differ or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-15-5bc897ed99ee>, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-15-5bc897ed99ee>\"\u001b[0;36m, line \u001b[0;32m5\u001b[0m\n\u001b[0;31m    print \"Data set 1:\\n\", treatment1\u001b[0m\n\u001b[0m                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# select two treatment data sets from the parasite data\n",
    "treatment1 = experimentDF[experimentDF[\"Virulence\"] == 0.5][\"ShannonDiversity\"]\n",
    "treatment2 = experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"]\n",
    "\n",
    "print \"Data set 1:\\n\", treatment1\n",
    "print \"Data set 2:\\n\", treatment2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A RankSum test will provide a P value indicating whether or not the two\n",
    "distributions are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-16-ca21d47d2538>, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-16-ca21d47d2538>\"\u001b[0;36m, line \u001b[0;32m5\u001b[0m\n\u001b[0;31m    print \"MWW RankSum P for treatments 1 and 2 =\", p_val\u001b[0m\n\u001b[0m                                                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "z_stat, p_val = stats.ranksums(treatment1, treatment2)\n",
    "\n",
    "print \"MWW RankSum P for treatments 1 and 2 =\", p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If P <= 0.05, we are highly confident that the distributions significantly differ, and\n",
    "can claim that the treatments had a significant impact on the measured value.\n",
    "\n",
    "If the treatments do *not* significantly differ, we could expect a result such as\n",
    "the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-17-f7d837ada734>, line 4)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-17-f7d837ada734>\"\u001b[0;36m, line \u001b[0;32m4\u001b[0m\n\u001b[0;31m    print \"Data set 3:\\n\", treatment3\u001b[0m\n\u001b[0m                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "treatment3 = experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"]\n",
    "treatment4 = experimentDF[experimentDF[\"Virulence\"] == 0.9][\"ShannonDiversity\"]\n",
    "\n",
    "print \"Data set 3:\\n\", treatment3\n",
    "print \"Data set 4:\\n\", treatment4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-18-22a4911b6d53>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-18-22a4911b6d53>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"MWW RankSum P for treatments 3 and 4 =\", p_val\u001b[0m\n\u001b[0m                                                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "z_stat, p_val = stats.ranksums(treatment3, treatment4)\n",
    "\n",
    "print \"MWW RankSum P for treatments 3 and 4 =\", p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With P > 0.05, we must say that the distributions do not significantly differ.\n",
    "Thus changing the parasite virulence between 0.8 and 0.9 does not result in a\n",
    "significant change in Shannon Diversity.\n",
    "\n",
    "### One-way analysis of variance (ANOVA)\n",
    "\n",
    "If you need to compare more than two data sets at a time, an ANOVA is your best bet. For\n",
    "example, we have the results from three experiments with overlapping 95% confidence\n",
    "intervals, and we want to confirm that the results for all three experiments are not\n",
    "significantly different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-19-65db3e2a7d2d>, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-19-65db3e2a7d2d>\"\u001b[0;36m, line \u001b[0;32m5\u001b[0m\n\u001b[0;31m    print \"Data set 1:\\n\", treatment1\u001b[0m\n\u001b[0m                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "treatment1 = experimentDF[experimentDF[\"Virulence\"] == 0.7][\"ShannonDiversity\"]\n",
    "treatment2 = experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"]\n",
    "treatment3 = experimentDF[experimentDF[\"Virulence\"] == 0.9][\"ShannonDiversity\"]\n",
    "\n",
    "print \"Data set 1:\\n\", treatment1\n",
    "print \"Data set 2:\\n\", treatment2\n",
    "print \"Data set 3:\\n\", treatment3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-20-8e96dae18e99>, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-20-8e96dae18e99>\"\u001b[0;36m, line \u001b[0;32m5\u001b[0m\n\u001b[0;31m    print \"One-way ANOVA P =\", p_val\u001b[0m\n\u001b[0m                            ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\t\n",
    "f_val, p_val = stats.f_oneway(treatment1, treatment2, treatment3)\n",
    "\n",
    "print \"One-way ANOVA P =\", p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If P > 0.05, we can claim with high confidence that the means of the results of all three\n",
    "experiments are not significantly different.\n",
    "\n",
    "### Bootstrapped 95% confidence intervals\n",
    "\n",
    "Oftentimes in wet lab research, it's difficult to perform the 20 replicate runs\n",
    "recommended for computing reliable confidence intervals with SEM.\n",
    "\n",
    "In this case, bootstrapping the confidence intervals is a much more accurate method of\n",
    "determining the 95% confidence interval around your experiment's mean performance.\n",
    "\n",
    "Unfortunately, SciPy doesn't have bootstrapping built into its standard library yet.\n",
    "However, there is already a scikit out there for bootstrapping. Enter the following\n",
    "command to install it:\n",
    "\n",
    "> sudo pip install -e git+http://github.org/cgevans/scikits-bootstrap.git#egg=Package\n",
    "\n",
    "You may need to install `pip` first, e.g., for Mac:\n",
    "\n",
    "> sudo easy_install pip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bootstrapping 95% confidence intervals around the mean with this function is simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-21-876daa4f0608>, line 4)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-21-876daa4f0608>\"\u001b[0;36m, line \u001b[0;32m4\u001b[0m\n\u001b[0;31m    print \"Small data set:\\n\", treatment1\u001b[0m\n\u001b[0m                            ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# subset a list of 10 data points\n",
    "treatment1 = experimentDF[experimentDF[\"Virulence\"] == 0.8][\"ShannonDiversity\"][:10]\n",
    "\n",
    "print \"Small data set:\\n\", treatment1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-22-7b157f903014>, line 6)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-22-7b157f903014>\"\u001b[0;36m, line \u001b[0;32m6\u001b[0m\n\u001b[0;31m    print \"Bootstrapped 95% confidence intervals\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]\u001b[0m\n\u001b[0m                                                      ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "import scipy\n",
    "import scikits.bootstrap as bootstrap\n",
    "\n",
    "CIs = bootstrap.ci(data=treatment1, statfunction=scipy.mean)\n",
    "\n",
    "print \"Bootstrapped 95% confidence intervals\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you can change the range of the confidence interval by setting the alpha:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-23-79a0640a8f78>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-23-79a0640a8f78>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"Bootstrapped 80% confidence interval\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]\u001b[0m\n\u001b[0m                                                     ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# 80% confidence interval\n",
    "CIs = bootstrap.ci(treatment1, scipy.mean, alpha=0.2)\n",
    "print \"Bootstrapped 80% confidence interval\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And also modify the size of the bootstrapped sample pool that the confidence intervals\n",
    "are taken from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "Missing parentheses in call to 'print' (<ipython-input-24-0e4e15f64f31>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-24-0e4e15f64f31>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    print \"Bootstrapped 95% confidence interval w/ 20,000 samples\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]\u001b[0m\n\u001b[0m                                                                       ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m Missing parentheses in call to 'print'\n"
     ]
    }
   ],
   "source": [
    "# bootstrap 20,000 samples instead of only 10,000\n",
    "CIs = bootstrap.ci(treatment1, scipy.mean, n_samples=20000)\n",
    "print \"Bootstrapped 95% confidence interval w/ 20,000 samples\\nLow:\", CIs[0], \"\\nHigh:\", CIs[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, bootstrapped 95% confidence intervals provide more accurate confidence\n",
    "intervals than 95% confidence intervals estimated from the SEM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
