{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3: Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this lab session is to code an optimization algorithm that optimzes the penalized loss function of the logistic regression model.\n",
    "\n",
    "You have to send the filled notebook named **\"L3_familyname1_familyname2.ipynb\"** (groups of 2) by email to aml.centralesupelec.2019@gmail.com by October 17, 2019. Please put **\"AML-L3\"** in the subject. \n",
    "\n",
    "We begin with the standard imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns; sns.set()\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We import the dataset that we are going to use, an indian dataset including in the last column information about the diabetes status of patients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import model_selection\n",
    "\n",
    "diabetes_data = pd.read_csv(\"diabetes_data.csv\", sep=\",\")\n",
    "\n",
    "diabetes_train, diabetes_test = model_selection.train_test_split(diabetes_data)\n",
    "diabetes_train_x = diabetes_train.iloc[:, :-1].values\n",
    "diabetes_train_y = diabetes_train.iloc[:, -1].values\n",
    "diabetes_train_y[diabetes_train_y == 0] = -1\n",
    "\n",
    "diabetes_test_x = diabetes_test.iloc[:, :-1].values\n",
    "diabetes_test_y = diabetes_test.iloc[:, -1].values\n",
    "diabetes_test_y[diabetes_test_y == 0] = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we’ll be moving from linear regression to logistic regression, one of the simplest ways to deal with a classification problem. Instead of fitting a line, logistic regression models the probability that the outcome is 1 given the value of the predictor. In order to do this we need a function that transforms our predictor variable to a value between 0 and 1. Lots of functions can do that, but the logistic function is the most common choice:\n",
    "\n",
    "$$f(z) = \\frac{1}{1+\\exp{-z}}.$$\n",
    "\n",
    "To predict the class of our observations we'll have to minimize the corresponding loss function and as we are in a high-dimensional context we'll add an $l_2$ regularization to the model:\n",
    "\n",
    "$$L(\\textbf{w}) = \\sum_{i=1}^n log(1+\\exp(-y_i\\textbf{w}^Tx_i))+\\frac{\\lambda}{2} \\| \\textbf{w} \\|^2,$$\n",
    "\n",
    "where $x_i$ is the vector of features for the observation $i$ and $y_i \\in \\{-1, 1\\}$ is the class label.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first use the `sklearn` implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "model = LogisticRegression(penalty=\"l2\", C=2) \n",
    "model.fit(diabetes_train_x, diabetes_train_y)\n",
    "y_pred = model.predict(diabetes_test_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we compute the accuracy score to evaluate the model performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(diabetes_test_y, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assignment\n",
    "\n",
    "Implement from scratch your own logistic regression model with stochastic gradient descent optimization. \n",
    "\n",
    "- Fill in the class\n",
    "\n",
    "- Display the evolution of the cost function along iterations. Do this for several strategies for the setting of the learning rate\n",
    "\n",
    "- Try the different acceleration strategies\n",
    "\n",
    "- Train the model with the training set and evaluate its performance in the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class StochasticLogisticRegression():\n",
    "    \"\"\" Class for logistic regression:\n",
    "    \n",
    "    Attributes:\n",
    "    -----------\n",
    "    coef_: 1-dimensional np.array\n",
    "        coefficients \n",
    "    alpha_: float\n",
    "        regularization parameter\n",
    "    lr_: float\n",
    "        the learning rate\n",
    "    bsize: integer\n",
    "        the size of the mini-batch >=1\n",
    "    coef_history_: list\n",
    "        the list of all visited betas\n",
    "    f_history_: list \n",
    "        the list of all evaluations in visited betas\n",
    "    \"\"\"\n",
    "    def __init__(self, alpha):\n",
    "        self.coef_  = None\n",
    "        self.alpha_ = alpha\n",
    "        self.lr_     = None\n",
    "        self.bsize_  = None\n",
    "        self.coef_history_ = []\n",
    "        self.f_history_ = []\n",
    "\n",
    "    def logistic(self, z):\n",
    "        # logistic function\n",
    "        \n",
    "    def fit(self, X, y, start, lr=1e-1, bsize=50, max_iter=500):\n",
    "        \"\"\" Fit the data (X, y).\n",
    "    \n",
    "        Parameters:\n",
    "        -----------\n",
    "        X: (num_samples, num_features) np.array\n",
    "            Design matrix\n",
    "        y: (num_sampes, ) np.array\n",
    "            Output vector\n",
    "        \n",
    "        Note:\n",
    "        -----\n",
    "        Updates self.coef_\n",
    "        \"\"\"        \n",
    "        \n",
    "        def f_lr(beta):\n",
    "            '''evaluate the F=\\sum_{i=1}^n f_i in beta'''\n",
    "        \n",
    "         \n",
    "    def predict(self, X):\n",
    "        \"\"\" Make binary predictions for data X.\n",
    "    \n",
    "        Parameters:\n",
    "        -----------\n",
    "        X: (num_samples, num_features) np.array\n",
    "            Design matrix\n",
    "        \n",
    "        Returns:\n",
    "        -----\n",
    "        y_pred: (num_samples, ) np.array\n",
    "            Predictions (0 or 1)\n",
    "        \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply to the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comment the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement only one acceleration method and compare the results"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}