stats170ab-2018: clustering_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering Algorithms\n",
    "Stats 170A, Winter 2018\n",
    "\n",
    "Illustration of clustering algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First, we'll import pandas and numpy\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# import relevant parts of sklearn\n",
    "from sklearn import cluster, datasets\n",
    "\n",
    "# We'll also import seaborn, a Python graphing library\n",
    "# (you may need to run >conda install seaborn (if using Anaconda)) \n",
    "import seaborn as sns\n",
    "\n",
    "# and matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set(style=\"white\", color_codes=True)\n",
    "%matplotlib inline\n",
    "\n",
    "# and turn off annoying warnings...(if we were writing \"real code\" we shouldn't do this)\n",
    "import warnings \n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and Explore the Iris Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.read_csv('iris.csv')\n",
    "iris.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Kmeans to Cluster the Iris Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = iris.columns[0:4]   # pull out the real-valued columns for clustering, ignore species column\n",
    "X = iris[cols] \n",
    "kmeans = cluster.KMeans(n_clusters=3)\n",
    "clabels = kmeans.fit_predict(X) \n",
    "iris['kmeans_label'] = clabels \n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visually compare K-Means Clustering with True Species Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(rc={'axes.facecolor':'lightslategray'})\n",
    "sns.pairplot(x_vars=['sepal_length'], y_vars=['sepal_width'], data=iris, hue=\"kmeans_label\", size=4) \n",
    "sns.pairplot(x_vars=['sepal_length'], y_vars=['sepal_width'], data=iris, hue=\"species\", size=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3d Plot of Iris Data with KMeans Cluster Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "%matplotlib \n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(iris['sepal_length'], iris['sepal_width'], iris['petal_length'], c=iris.kmeans_label, s=60)\n",
    "ax.view_init(30, 185)\n",
    "ax.set_xlabel('Sepal Length')\n",
    "ax.set_ylabel('Sepal Width')\n",
    "ax.set_zlabel('Petal Length')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the Kmeans Centroids as 4-Dimensional Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "centroids = kmeans.cluster_centers_ \n",
    "sns.set_style(\"whitegrid\")\n",
    "plt.plot(centroids.transpose())\n",
    "ax.set_xlabel('Indices of Variables')\n",
    "ax.set_ylabel('Values of Variables')\n",
    "plt.show()\n",
    "centroids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Hierarchical (Agglomerative) Clustering to Cluster the Iris Data \n",
    "... and we will compare to the Kmeans clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hlabels = cluster.AgglomerativeClustering(n_clusters=3, linkage='ward').fit_predict(X) \n",
    "iris['hier_label'] = hlabels \n",
    "print(hlabels[::10])\n",
    "print(clabels[::10])\n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Housing Data\n",
    "Now we look at a different data set, a housing data set used in a Kaggle prediction competition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "houses = pd.read_csv('house_price_prediction/train.csv')\n",
    "houses.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis_vars = ['LotArea', 'LotFrontage', '1stFlrSF', 'GrLivArea', 'MoSold', 'YrSold',  'YearBuilt',   'SalePrice'  ]\n",
    "houses_real = houses[analysis_vars].dropna()\n",
    "houses_real.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans2 = cluster.KMeans(n_clusters=3)\n",
    "clabels2 = kmeans2.fit_predict(houses_real.dropna()) \n",
    "houses_real['kmeans_label'] = clabels2 \n",
    "sns.set(rc={'axes.facecolor':'lightslategray'})\n",
    "sns.pairplot(x_vars=['YearBuilt'], y_vars=['SalePrice'], data=houses_real, hue=\"kmeans_label\", size=6) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comment\n",
    "The plot abova actually gives us some useful information about what Kmeans is doing with this data. It is telling us that kmeans is splitting along the y-axis (Sale Price) and virtually ignoring the x-axis (YearBuilt). Why is this happening? And what might we do to the data to change this behavior?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centroids = kmeans2.cluster_centers_ \n",
    "np.set_printoptions(suppress=True)  # suppress scientific notation so that we can see the \"real\" values\n",
    "centroids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets normalize the columns to have mean 0 and standard deviation 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = houses_real.drop(['kmeans_label'],axis=1)\n",
    "tmp.head() \n",
    "np.set_printoptions(suppress=True)  # suppress scientific notation so that we can see the \"real\" values\n",
    "#Norm_houses_real = tmp.apply(lambda x: (x - np.mean(x)) / (np.max(x) - np.min(x)))\n",
    "Norm_houses_real = tmp.apply(lambda x: (x - np.mean(x)) / (np.std(x)) )\n",
    "Norm_houses_real.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Norm_houses_real.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clabels2 = kmeans2.fit_predict(Norm_houses_real.dropna()) \n",
    "Norm_houses_real['kmeans_label'] = clabels2 \n",
    "houses_real['kmeans_label'] = clabels2 \n",
    "sns.set(rc={'axes.facecolor':'lightslategray'})\n",
    "sns.pairplot(x_vars=['YearBuilt'], y_vars=['SalePrice'], data=Norm_houses_real, hue=\"kmeans_label\", size=6) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centroids = kmeans2.cluster_centers_ \n",
    "np.set_printoptions(suppress=True)  # suppress scientific notation so that we can see the \"real\" values\n",
    "centroids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style(\"whitegrid\")\n",
    "plt.plot(centroids.transpose())\n",
    "ax.set_xlabel('Indices of Variables')\n",
    "ax.set_ylabel('Values of Variables')\n",
    "plt.show() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(rc={'axes.facecolor':'lightslategray'})\n",
    "sns.pairplot(x_vars=['GrLivArea'], y_vars=['SalePrice'], data=Norm_houses_real, hue=\"kmeans_label\", size=6) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(rc={'axes.facecolor':'lightslategray'})\n",
    "sns.pairplot(x_vars=['YearBuilt'], y_vars=['GrLivArea'], data=houses_real, hue=\"kmeans_label\", size=6) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}