{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 1: Regression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Welcome to the advanced Machine Learning Course.\n", "\n", "The objective of this lab session is to code a few regression algorithms and to apply them to synthetic and real datasets.\n", "\n", "We begin with the standard imports:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# import libraries\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns; sns.set()\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Simple Linear Regression\n", "\n", "We will start with the most familiar linear regression, a straight-line fit to data.\n", "A straight-line fit is a model of the form\n", "$$\n", "y = ax + b\n", "$$\n", "where $a$ is commonly known as the *slope*, and $b$ is commonly known as the *intercept*.\n", "\n", "Consider the following data, which is scattered about a line with a slope of 2 and an intercept of -5:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEXCAYAAACDChKsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de1TUdf4/8OcM4KAr3nDAS2oiX2Hzst7LLqIrKsPVdDUL3czStdz06Gqi7LGkCGEttbNZWnhozaQ0lonfYcoDBh3T1XQ1QRH3wGpewhmBVJTLyHx+f7hMjsww8GHmMzN8no9zOjmfz2eG90tqnvN5v+f9fisEQRBARETURkpXN4CIiDwTA4SIiERhgBARkSgMECIiEoUBQkREojBAiIhIFG9XN4CoI7p8+TKmTZuGoUOHAgBMJhN8fHzwxz/+ETNnzmzxuX//+98RGhqK8PBwKZpKJBoDhMhJfH19odVqzY+vXLmChQsXonPnzpgxY4bN5x09ehTBwcFSNJGoXRggRBLp378/li9fjvT0dAwdOhRJSUm4c+cO9Ho9QkNDsXXrVuzfvx/FxcVIS0uDl5cXgoODrV6nUqlcXQ4Rx0CIpBQaGorz58/jiy++wMyZM/H555/jwIEDuHz5MgoKChAfH4/hw4fjtddew7Rp02xeR+QOeAdCJCGFQgFfX1+sWbMG33//PT766CNcuHABer0ed+7caXZ9a68jcgUGCJGEioqKMHToUKxatQqNjY3QaDSYPHkyfv75Z1hblq611xG5AruwiCTy3//+F9u3b8eiRYtw6NAhLFu2DJGRkVAoFPjxxx/R2NgIAPDy8sLdu3cBoMXriFyNdyBETlJXV4e4uDgAgFKphEqlwqpVqzB58mSsXLkSy5YtQ/fu3dG5c2eMHz8eP/30EwBgypQpSE1NhdFobPE6IldTcDl3IiISg11YREQkCgOEiIhEYYAQEZEoDBAiIhKFAUJERKIwQIiISBTZzQOprr4Nk6l931z29++KysoaB7XIM8ixZoB1y40c67ZXs1KpQM+ev7F6TnYBYjIJ7Q6QpteRGznWDLBuuZFj3WJrlrwL6/jx45g1axZiYmKwdOlS3Lhxo9k1DQ0NWLNmDTQaDZ5++mmUlZUBAARBQGpqKiIiIhAZGYkTJ05I3XwiIvofyQNk3bp1SEtLQ05ODoKDg5Gent7smt27d6Nz587Q6XRYv349EhISAADffPMNysrKkJubi/fffx8JCQnmNYOIiEhakgdIbm4ugoODYTQace3aNXTr1q3ZNQUFBYiNjQUAjB8/HtXV1bh69SoKCwsRGRkJpVKJwYMHo1+/fjh58qTUJRAREVwQID4+PigtLUVYWBiOHj2KqKioZtfo9Xqo1WrzY7VajYqKCuj1egQEBDQ7TkRE0nPaILpOp0NKSorFsaCgIGRkZCAkJASHDx9GZmYmVq5ciczMTLuvp1Qqre6DoFS2LQP9/bu26Xpb1Go/h7yOJ5FjzQDrlhs51i22ZqcFiEajgUajsThWX1+PvLw8hIeHAwBiY2ORmpra7LkBAQEwGAwYNGgQAMBgMCAgIACBgYEwGAzm65qOt0VlZU27v2WhVvvBYLjVrtfwNHKsGWDdctNR6j5ypgJZhWWovFkP/24qzAobgonD+li91l7NSqXC5gdvSbuwvL29sXHjRhQXFwO4d5cyZsyYZteFhYVBq9UCuPetLZVKhX79+mHSpEnIyclBY2MjLl68iAsXLmDEiBFSlkBE5NaOnKnAJ7pzqLxZDwCovFmPT3TncOSM47v7JZ0H4uXlhS1btmDDhg1obGxEYGAgkpOTAQB79+6FXq/HihUrsGDBAmzYsAFRUVHo1KkT0tLSAAARERE4ffq0eYA9OTkZvr6+UpZAROTWsgrL0HDXZHGs4a4JWYVlNu9CxJLdhlLswhJHjjUDrFtuOkLdizYdtHluV8Lvmx3zmC4sIiJyLv9uqjYdbw8GCBFRBzIrbAg6eVu+tXfyVmJW2BCH/yzZrYVFRNSRNY1ztPZbWO3BACEi6mAmDuvjlMB4ELuwiIhIFAYIERGJwgAhIiJRGCBERCQKA4SIiERhgBARkSgMECIiEoXzQIiInKwty6t7EgYIEZETNS2v3rRCbtPy6gA8PkTYhUVE5EQtLa/u6RggRERO1LSxU2uPexIGCBGRE0m5vLrUJB8DOX78ON5++20YjUb0798fqamp6N69u8U1er0e69atw/Xr16FUKvHaa69h4sSJMBqNePTRRzFgwADztVlZWfDy8pK6DCKiVpkVNsRiDARw3vLqUpM8QNatW4cPPvgAwcHB2Lx5M9LT07Fq1SqLa9LS0jBlyhTMnz8f5eXlWLBgAb777juUlpZi9OjRSE9Pl7rZRESiSLm8utQk78LKzc1FcHAwjEYjrl27hm7dujW7Zvr06YiJiQEADBo0CPX19bhz5w6KiopQVVWFuXPnYu7cuTh27JjUzSciapOO+hVewAV3ID4+PigtLcULL7wAb2/vZncfwL0AaZKeno7f/va38PPzg0KhwNSpU7Fs2TKUlJRg8eLFyMnJQa9evaQsgYioVTryV3gBQCEIguCMF9bpdEhJSbE4FhQUhIyMDPPjzMxMZGdnIzMz0+prZGRkYPfu3fj000/Rt2/fZudffvllzJ49G+Hh4Q5tOxGRIyx66wAM1bXNjqt7dsauv0638gzP4rQ7EI1GA41GY3Gsvr4eeXl55jf82NhYpKamWn1+WloaCgsLsWfPHvTpcy+ps7OzMWbMGAwcOBAAIAgCfHx82tSuysoamEzty0y12g8Gw612vYankWPNAOuWG0fXbS08mo67y9+vvZqVSgX8/btaPSdpF5a3tzc2btyIPn36YPjw4dDpdBgzZkyz6zIyMnD06FHs3bvXYoyktLQUp06dwhtvvIHy8nKUlJRg7NixUpZARNRq/t1UVud7+HdTdYixEad1YdnS9DXexsZGBAYGIikpCX369MHevXuh1+uxfPlyTJgwAV27drUIj507d+I3v/kN1q9fj/LycigUCiQmJuKxxx5r08/nHYg4cqwZYN1y4+i6HxwDafLbQT1QduVms6/2Pq8JlTxE2nMHInmAuBoDRBw51gywbrlxRt27vzmHb09ebdW1/t1U+NsrTzj059vTngDhTHQiIic6XVbZ6ms9bXkTBggRkRO1JRQ8bXkTBggRkRO1NhQ8cXkTBggRkRPNChuCTt6Wb7WdvJWYMrqfOVz8u6lcMoDeXtxQiojIiTryWlgMECIiJ5s4rE+HCIwHsQuLiIhEYYAQEZEoDBAiIhKFAUJERKIwQIiISBQGCBERicIAISIiURggREQkCgOEiIhEYYAQEZEoki9l0rQjodFoRP/+/ZGamoru3btbXHP16lVERUWZ9z7v3bs30tPT0dDQgMTERBQXF8PX1xebN2/GkCGetXolEVFHIfkdyLp165CWloacnBwEBwcjPT292TVFRUWIiYmBVquFVqs1X7N792507twZOp0O69evR0JCgtTNJyKi/5H8DiQ3Nxc+Pj4wGo24du0aQkJCml1TVFSE8+fPY9asWejatSsSExMREhKCgoICrFixAgAwfvx4VFdX4+rVq+jXr5/UZRBRB3LkTAWyCstQdbMevTrQarnOJvkdiI+PD0pLSxEWFoajR48iKiqq2TUqlQozZ85EVlYWXnzxRSxbtgwNDQ3Q6/VQq9Xm69RqNSoqKqRsPhF1MEfOVOAT3TlU3qyHgHs7CH6iO4cjZ/jeYo/T7kB0Oh1SUlIsjgUFBSEjIwMhISE4fPgwMjMzsXLlSmRmZlpc9+qrr5r/HBYWhnfeeQfl5eVWf45S2bYMtLU5fFup1X4OeR1PIseaAdbd0WUfOoKGuyaLYw13Tcg+9F/ETv4/F7VKWmJ/104LEI1GA41GY3Gsvr4eeXl5CA8PBwDExsYiNTW12XN3796N6Oho9OzZEwAgCAK8vb0REBAAg8GAQYMGAQAMBgMCAgLa1K7KyhqYTIKYkszUaj8YDLfa9RqeRo41A6zb2Zq6jly50ZKhutbmcTn87u39rpVKhc0P3pJ2YXl7e2Pjxo0oLi4GcO8uZcyYMc2u++GHH7B//34AwLFjx2AymRAUFISwsDBotVoA977NpVKpOP5B5KHu7zoCXNd1ZGvP8tbuZS5nkg6ie3l5YcuWLdiwYQMaGxsRGBiI5ORkAMDevXuh1+uxYsUKJCYmIiEhAVqtFiqVCu+88w6USiUWLFiADRs2ICoqCp06dUJaWpqUzSciB8oqLLPadZRVWCbpXcissCH4RHfOoi2dvJWYFcYpAvYoBEFoX3+Oh2EXljhyrBlg3c60aNNBm+d2JfzeqT/7QXL+FlZ7urC4JzoRuYR/N5W5++rB41Jr2rNcrh8YxOJSJkTkErPChqCTt+VbELuOPAvvQIjIJZq6iFz9LSwSjwFCRC7T1HVEnoldWEREJAoDhIiIRGEXFhG5NXeYrU7WMUCIyG01zVZvmuTXNFsdAEPEDbALi4jcVkuz1cn1GCBE5LasTTRs6ThJiwFCRG6LCx26N46BEJHTiR0I50KH7o0BQkRO1Z6BcM5Wd28MECJyqvYu287Z6u6LAUJETtXSQPiiTQd5V+HBOIhORE5lb8DbVTsRUvtJfgdy/PhxvP322zAajejfvz9SU1PRvXt3i2uWLl2Kn3/+GQBgMplw/vx57N+/H6GhoXj00UcxYMAA87VZWVnw8vKStAYiaj1rA+EPcsVOhNR+kgfIunXr8MEHHyA4OBibN29Geno6Vq1aZXHNhx9+aP7ztm3bMGrUKIwYMQLFxcUYPXo00tPTpW42EYn04EC4LZzb4XkkD5Dc3Fz4+PjAaDTi2rVrCAkJsXltWVkZsrOzkZOTAwAoKipCVVUV5s6dCwBYvXo1JkyYIEm7iUi8+wfC12z/3m12IqT2ccme6KWlpXjhhRfg7e2Nzz//HH379rV63erVqzFq1CjMnz8fAJCZmYnr169j2bJlKCkpweLFi5GTk4NevXpJ2XwiaoeCE5fw930/ot7YaD6m8vHCn+f8DpPHDmjhmeRunBYgOp0OKSkpFseCgoKQkZFhfpyZmYns7GxkZmY2e/6NGzcQERGBgoICqFTWP5m8/PLLmD17NsLDw1vdrsrKGphM7StZjvsmy7FmgHU7i7uusCvH37e9mpVKBfz9u1o957QuLI1GA41GY3Gsvr4eeXl55jf82NhYpKamWn1+YWEhJk2aZBEe2dnZGDNmDAYOHAgAEAQBPj4+TqqAiJyFczs6Bkm/xuvt7Y2NGzeiuLgYwL27lDFjxli99tSpUxg3bpzFsdLSUuzatQsAUF5ejpKSEowdO9a5jSYiIqskHUT38vLCli1bsGHDBjQ2NiIwMBDJyckAgL1790Kv12PFihUAgEuXLmHy5MkWz1+2bBnWr1+P6OhoKBQKpKamomtX67dWRETkXC4ZRHcljoGII8eaAdYtN3Ksuz1jIJyJTkREojBAiIhIFAYIERGJwgAhIiJRGCBERCQKA4SIiERhgBARkSgMECIiEoUBQkREojBAiIhIFAYIERGJwgAhIiJRGCBERCSK5HuiE5E03HXXP+o4GCBEHdCRMxX4RHcODXdNAIDKm/X4RHcOABgi5DAu68I6e/Yshg8fbvWcIAhITU1FREQEIiMjceLECfO5Xbt2ISIiAjNmzMCBAwekai6R2zpypgJrtn+PRZsOYs327813Hk3h0aThrglZhWUuaiV1RC65A6mtrUVSUhKMRqPV89988w3KysqQm5uLixcvYsmSJdDpdDh79iy++uoraLVa1NTU4JlnnsGECRPQo0cPiSsgcg+27jQeDI8mlTfrpWwedXAuuQPZtGkTFi5caPN8YWEhIiMjoVQqMXjwYPTr1w8nT57Ed999h2nTpkGlUsHf3x8TJkxAQUGBZO0mcje27jSUCuvX+3dTSdAqkgvJAyQ/Px91dXWIiIiweY1er0dAQID5sVqtRkVFhc3jRHJl647CJACdvC3/9+7krcSssCFSNItkwmldWDqdDikpKRbHgoKCUFNTg4yMjBafa22bdqVSafN4W9ja27et1Go/h7yOJ5FjzYB7163u2RmG6lqrx/+o+S3+oSvB9epa9P7f48ljB7T+td24bmeSY91ia7YbIK+++iqeffZZPP744216YY1GA41GY3Fs37592LFjB+Lj483H4uLisGfPHnTt+usbe2BgIAwGg/mxwWBAQECA1eODBw9uU7sqK2tgMjUPorawtwl9RyTHmgH3r3vmk4ObjXl08lZi5pODMWxgD6T+aaLF9a2txd3rdhY51m2vZqVSYfODt92P79OnT8f27dsxY8YMpKen45dffhHd0Dlz5iAvLw9arRZarRYAoNVqLcIDACZNmoScnBw0Njbi4sWLuHDhAkaMGIFJkybhwIEDqK2tRVVVFf71r39h4sSJ1n4UkSxMHNYHz2tCzWMb/t1UeF4Tyq/qkiTs3oHExMQgJiYGZWVl+PLLLzFnzhyMGjUKCxYswMiRIx3WkPz8fBw8eBDJycmIiIjA6dOnERsbCwBITk6Gr68vRo4cidjYWPzhD3/A3bt3sXz5cgQGBjqsDUSeaOKwPgwMcgmFYG1g4QEmkwkFBQXYv38/SkpKMGPGDBw7dgwTJ07EmjVrpGinw7ALSxw51gywbrmRY93t6cKyeweyZcsWZGVlYcCAAXjuueewbds2+Pj44M6dO5gyZYrHBQgRETmG3QCpqqrCRx99hNDQUIvjXbp0wTvvvOO0hhERkXuzGyBvvvmmzXNPPvmkQxtDRESeg4spEj3g/lVs1T07Y+aTgzlITWQFA4ToPg+uLWWoruUqtkQ2cEMpovtwFVui1mOAEN3H1tpSXMWWqDkGCNF9bK1Wy1VsiZpjgBDdZ1bYEK5iS9RKHEQnuk/TQLmtb2Fxn3GiXzFAiB5w/9pS9y/zsPubc/j25FXzddxnnOSOXVhErXDkTIVFeDThN7RIzngHQh7Dld1HLYUEv6FFcsUAIY/w4AQ/qbuPWgoJfkOL5IpdWOQRXD3Br6WQ4De0SK4YIOQRXD3Bb1bYEHgpmh+fMrofB9BJtlzWhXX27FnMnTsXxcXFzc7dvn0b69evR3l5OQBg6dKliIqKAgBMnTrVYgvcDz/8EH379pWm0eQy/t1UVsNCyu4jhVIBNP66GZm3lwLBD/WQ7OcTuRuXBEhtbS2SkpJgNBqtnt+5cyf69euHbdu2obKyEnFxcXj00Ufh5eUFHx8f837qJB+zwoZYjIEAtif4OWOwPauwDHcbLXeyvNsoIKuwjHcgJFsuCZBNmzZh4cKFOHnypNXzEyZMwODBgwEA/v7+6NGjB65fvw69Xg9BEBAfH487d+5gyZIl0Gg0UjadXOTBCX62gsFZg+2u7kIjckeSB0h+fj7q6uoQERFh85onnnjC/Ofc3Fw0NDQgODgYly9fxlNPPYW1a9fi2rVriI+Px9ChQzFkCAcx5eD+CX62tDTY3p4AcYcuNCJ347QA0el0SElJsTgWFBSEmpoaZGRktPo13n77bXz88cfw9vZGeHg4wsPDAQAPPfQQpk2bhkOHDrUpQGxtDt9WarWfQ17Hk3hCzVU27giqbtaLbr9a7YeF0cPw930/ot7YaD6u8vHCwuhhHvH3IkZHrcseOdYttmanBYhGo2nWvbRv3z7s2LED8fHx5mNxcXHYs2ePxcA4AOzevRvp6elIT09HSEgIAODbb79F7969MWLEiF8L8G5bCZWVNTCZBPsXtuD+5S3kwlNq7mXjTqFXN5Wo9jfVPWxgD/wxIqRZF9qwgT084u+lrTzl9+1ocqzbXs1KpcLmB29Ju7DmzJmDOXPmmB+HhIRYHRDPy8tDRkYG9u7da/ENqytXruDzzz/H9u3bUVVVhYMHD2L37t2StJ08Q1sG29uqNV1oRHLiNjPR8/PzcfDgQSQnJ+O9995DfX09li5daj7/1ltvYd68eSgtLUV0dDRMJhNWr16N/v37u7DV5G5aO9hORO2nEAShff05HoZdWOLIsWaAdcuNHOv2mC4sorbi/htE7osBQm7L1QsoElHLuBYWuS1XL6BIRC1jgJDb4uxvIvfGACG3ZWuWN2d/E7kHBgi5rVlhQ9DJ2/I/UUfN6SCi9uMgOrktzukgcm8MEHJrnP1N5L7YhUVERKIwQIiISBQGCBERicIAISIiURggREQkCgOEiIhEYYAQEZEoDBAiIhLFZQFy9uxZDB8+3Oo5o9GIMWPGIC4uzvxPY2MjBEFAamoqIiIiEBkZiRMnTkjcaiIiauKSmei1tbVISkqC0Wi0er60tBSjR49Genq6xfGvv/4aZWVlyM3NxcWLF7FkyRLodDp4e3NCPRGR1FxyB7Jp0yYsXLjQ5vmioiJUVVVh7ty5mDt3Lo4dOwYAKCwsRGRkJJRKJQYPHox+/frh5MmTErWaiIjuJ/lH9/z8fNTV1SEiIsLmNQqFAlOnTsWyZctQUlKCxYsXIycnB3q9HgEBAebr1Go1Kioq2vTzbe3t21ZqtZ9DXseTyLFmgHXLjRzrFluz0wJEp9MhJSXF4lhQUBBqamqQkZHR4nPnzZtn/vMjjzyCkSNH4t///jcEQWh2rVLZtpuoysoamEzNX6ct7G1C78ls7UHekWtuCeuWFznWba9mpVJh84O30wJEo9FAo9FYHNu3bx927NiB+Ph487G4uDjs2bMHXbv+2sDs7GyMGTMGAwcOBAAIggAfHx8EBgbCYDCYrzMYDBZ3JNQ+Le1BHjtZfp/KiKhlko6BzJkzB3l5edBqtdBqtQAArVZrER7AvUH0Xbt2AQDKy8tRUlKCsWPHYtKkScjJyUFjYyMuXryICxcuYMSIEVKW0KFxD3Iiagu3+fpSfn4+Dh48iOTkZCxbtgzr169HdHQ0FAoFUlNT0bVrV0REROD06dOIjY0FACQnJ8PX19fFLe84uAc5EbWFQrA2sNCBcQzEtjXbv7caFv7dVMh4PaJD1mxPR/1d28O65aM9YyCciU5m3IOciNrCbbqwyPW4BzkRtQUDhCw8uAf5kTMVWLP9e1TdrEcvBgoR3YddWGRT09d6K2/WQ8C9wfSPcs7i1a2FOHKmbRM4iajjYYCQTda+1gsAt+sa8YnuHEOESOYYIGRTS1/f5fwQImKAkE3+3VQtnuf8ECJ54yC6jNla96rJrLAhFkubPMhewBBRx8YAkamW1r1qCpGmf+/NO4+a2rsWz+f8ECJigMhUS+te3X8X0vS13jM//YKM/3eG80OIyIwBIlNtXfdq8tgBGDawhzObREQehgHixuyNUbSHfzeVzXWviIhag9/CclP3T+IDfh2jcNTcC657RUTtxQBxU87em2PisD54XhNqvuPw76bC85pQjmsQUauxC8tNSbE3x4PrXhERtYXLAuTs2bOYO3cuiouLm53bsGEDfvzxR/Pj8+fPY8uWLYiIiMDUqVMtdjD88MMP0bdvX0naLCWOURCRu3NJgNTW1iIpKQlGo9Hq+aSkJPOf9+/fD51OhxkzZqC6uho+Pj7m7XDdXXsGwa1N4uMYBRG5E5eMgWzatAkLFy60e111dTXee+89JCUlQaFQoKioCIIgID4+Hk8//TR0Op3zGytSewfBOUZBRO5O8juQ/Px81NXVISIiwu61GRkZiIqKQv/+/QEADQ0NeOqpp7B27Vpcu3YN8fHxGDp0KIYMcb9P5a2dqNcSjlEQkTtzWoDodDqkpKRYHAsKCkJNTQ0yMjLsPt9kMuHLL7/El19+aT4WHh6O8PBwAMBDDz2EadOm4dChQ20KEFt7+7aVWu3X4vkqG4PdVTfr7T7XXXlqu9uLdcuLHOsWW7PTAkSj0UCj0Vgc27dvH3bs2IH4+Hjzsbi4OOzZs8diYBwATp48iYcffhiBgYHmY99++y169+6NESNGmI95e7ethMrKGphMQpue8yB7m9ADQC8bg+C9uqnsPtcRHD0JsTU1d0SsW17kWLe9mpVKhc0P3pKOgcyZMwd5eXnQarXmgXCtVtssPADg1KlTGDt2rMWxK1eu4P3334fJZML169dx8OBBTJ48WYqmt5krJ+o5exIiERHgRhMJ8/PzkZiYaH586dIl9Olj+Yl53rx5UKvViI6Oxvz587F69Wrz+Ii7ceUguLMnIRIRAYBCEIT29ed4GKm6sFxp0aaDNs8tjnlEVIi5e83OwrrlRY51e0wXFkmjpcmG7MoiIkdhgHRA1sZfmrAri4gchQHSATWNv9jCvcyJyBEYIB3UxGF9bHZlcT0tInIEBkgHxj0/iMiZuJx7B9b0bStn7WpIRPLGAOnguJ4WETkLA8QNOXMvdCIiR2GAuJmmZUiaZpI3LUMCgCFCRG6Fg+huhsuQEJGnYIC4GSn2QicicgQGiJvh3A0i8hQMEDfDuRtE5Ck4iO5mOHeDiDwFA8QNce4GEXkCdmEREZEokt+BZGdnY/PmzfD39wcATJ48GStXrrS45ubNm1i9ejUuXbqEXr16YevWrVCr1WhoaEBiYiKKi4vh6+uLzZs3Y8gQ6cYGmib4Vd2sRy92LRGRzEkeIEVFRUhISEB0dLTNa7Zu3Ypx48Zh586dyM7ORnJyMrZu3Yrdu3ejc+fO0Ol0+OGHH5CQkIB9+/ZJ0m5O8CMisiR5F1ZRURGys7MRGxuL1atX48aNG82uKSgoQExMDAAgOjoa3333HYxGIwoKChAbGwsAGD9+PKqrq3H16lVJ2t2aCX5HzlRgzfbvsWjTQazZ/j13/iOiDk3yAFGr1Xj11Veh1WrRt29fJCUlNbtGr9dDrVYDALy9vdG1a1dUVVVZHG96rYoKad6k7U3wa7pDaXrcdIfCECGijsppXVg6nQ4pKSkWx4KCgpCRkWF+/NJLLyE8PLxVr6dUWs86W8dtsbU5vD3qnp1hqK61elyt9kP2oSNW71CyD/0XsZP/T9TPdDdqtZ+rm+ASrFte5Fi32JqdFiAajQYajcbi2K1bt5CRkYGFCxcCAARBgLd38yYEBATg+vXr6NOnD+7evYuamhr06NEDAQEBMBgMGDRoEADAYDAgICCgTe2qrKyBySS0uZ6ZTw62GAMB7k3wm16pppEAAAcfSURBVPnkYBgMt6yGCwAYqmthMNxq889zN2q1X4eoo61Yt7zIsW57NSuVCpsfvCXtwurSpQs+/vhj/PjjjwCATz/9FNOmTWt2XVhYGLKzswEAubm5GDduHHx8fBAWFgatVgsAOH78OFQqFfr16ydJ25v2GffvpoIC95YWeV4Tah5A5xIkRCQ3CkEQ2v5xvB2OHz+O5ORk1NXV4eGHH0ZaWhr8/Pywbds2BAQE4Nlnn8Uvv/yChIQEXLp0CX5+fti8eTMeeugh1NfXY8OGDSguLkanTp3w1ltvYdiwYW36+WLvQO5nLbEf/JYWcO8O5f6Q8WRy/GQGsG65kWPd7bkDkTxAXM1ZAQJ07I2g5Pg/FsC65UaOdbcnQLiUiQNxCRIikhMuZUJERKIwQIiISBQGCBERicIAISIiURggREQkiuy+haVUKtzqdTyJHGsGWLfcyLHulmpu6Zzs5oEQEZFjsAuLiIhEYYAQEZEoDBAiIhKFAUJERKIwQIiISBQGCBERicIAISIiURggREQkCgOEiIhEYYDYkJOTg8jISEybNg179uxpdr6kpASzZ8/GjBkzkJiYiLt377qglY5nr+68vDzExcUhNjYWr7zyCm7cuOGCVjqevbqbFBQU4Pe//72ELXMue3WXl5djwYIFiI2NxYsvvtghft/2aj5z5gxmz56N2NhY/OlPf8LNmzdd0ErnqKmpQXR0NC5fvtzsnKj3NIGaqaioEKZMmSJUV1cLt2/fFmJiYoT//Oc/FtdERUUJJ0+eFARBENatWyfs2bPHFU11KHt137p1S3jiiSeEiooKQRAEYevWrcKbb77pquY6TGt+34IgCAaDQYiIiBCmTJniglY6nr26TSaTMH36dKGwsFAQBEH429/+JqSlpbmquQ7Rmt/1s88+KxQUFAiCIAgpKSnCu+++64qmOtypU6eE6OhoYdiwYcKlS5eanRfznsY7ECsOHz6Mxx57DD169ECXLl0wY8YMfP311+bzV65cQV1dHUaNGgUAmDVrlsV5T2WvbqPRiDfeeAOBgYEAgJCQEPz888+uaq7D2Ku7yV//+lf8+c9/dkELncNe3WfOnEGXLl0wadIkAMDSpUsRHx/vquY6RGt+1yaTCbdv3wYA1NbWwtfX1xVNdbgvvvgCr7/+OgICApqdE/uexgCxQq/XQ61Wmx8HBATg2rVrNs+r1WqL857KXt09e/ZEeHg4AKCurg47d+40P/Zk9uoGgH/84x945JFH8Lvf/U7q5jmNvbp/+ukn9O7dG2vXrkVMTAxef/11dOnSxRVNdZjW/K4TEhKQmJiIJ598EocPH8a8efOkbqZTJCcnY9y4cVbPiX1PY4BYIVhZoFihULT6vKdqbV23bt3C4sWLERoaiqefflqKpjmVvbrPnz+PAwcO4JVXXpGyWU5nr+67d+/i2LFjmD9/PnJycjBgwABs2rRJyiY6nL2a6+rqkJiYiE8++QSHDh3Cc889h7Vr10rZRJcQ+57GALEiMDAQ169fNz/W6/UWt30PnjcYDFZvCz2Nvbqbjj333HMIDQ1FcnKy1E10Cnt1f/311zAYDJg9ezaWLFli/jvwdPbqVqvVGDRoEEaMGAEAiI6OxunTpyVvpyPZq/n8+fNQqVQYOXIkAOCZZ57BsWPHJG+n1MS+pzFArHj88cdx5MgRVFVVoba2FgcOHDD3AwNA//79oVKpcOLECQBAdna2xXlPZa/uxsZGLF26FBqNBomJiR3irguwX/fy5cvxzTffQKvVYufOnQgICMBnn33mwhY7hr26R48ejaqqKpw7dw4AcPDgQQwbNsxVzXUIezUPGjQIFRUVKC8vBwDk5+ebA7QjE/2e5rgx/o7lq6++EqKiooTp06cLO3fuFARBEF566SXh9OnTgiAIQklJiTB79mwhIiJCWLVqlVBfX+/K5jpMS3UfOHBACAkJEWJjY83/rF+/3sUtdgx7v+8mly5d6jDfwhIE+3WfOnVKmD17thAZGSksWrRIuH79uiub6xD2ai4oKBBiYmKE6Oho4fnnnxd++uknVzbX4aZMmWL+FlZ739O4IyEREYnCLiwiIhKFAUJERKIwQIiISBQGCBERicIAISIiURggREQkCgOEiIhEYYAQucg///lPTJ06Fbdv38adO3eg0WiQnZ3t6mYRtRonEhK50F/+8hf4+fmhoaEBXl5eePPNN13dJKJWY4AQuVBNTQ3i4uLg6+uLrKwsqFQqVzeJqNXYhUXkQpWVlaivr8fNmzeh1+td3RyiNuEdCJGLGI1GzJs3D/PmzYPJZML+/fvx2WefwcfHx9VNI2oV3oEQuci7774LtVqNOXPm4JlnnkGPHj2wZcsWVzeLqNV4B0JERKLwDoSIiERhgBARkSgMECIiEoUBQkREojBAiIhIFAYIERGJwgAhIiJRGCBERCTK/wf0ko9K07RC9wAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Simulate data\n", "rng = np.random.RandomState(1)\n", "x = rng.rand(30)\n", "slope, intercept, noise = 2, -5, 0.1 * rng.randn(30)\n", "y = slope * x + intercept + noise\n", "\n", "# Plot data\n", "fig, ax = plt.subplots()\n", "ax.scatter(x, y)\n", "ax.set(title='Data', \n", " xlabel='x', ylabel='y')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fill in the MultivariateLinearRegression class whose method 'fit' takes a matrix $X$ and vector $y$ as input and returns an vector of coefficients $\\beta$" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "class MultivariateLinearRegression():\n", " # Class for least-squares linear regression:\n", "\n", " def __init__(self,):\n", " self.coef_ = None\n", " \n", " def fit(self, X, y):\n", " \"\"\" Fit the data (X, y).\n", " \n", " Parameters:\n", " -----------\n", " X: (n_samples, n_samples) np.array\n", " Design matrix\n", " y: (n_samples, ) np.array\n", " Output vector\n", " \n", " Note:\n", " -----\n", " Updates self.coef_\n", " \"\"\"\n", " # Create a (num_samples, num_features+1) np.array X_aug whose first column \n", " # is a column of all ones (so as to fit an intercept).\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", " \n", " # Update self.coef_\n", " self.coef_ = np.linalg.pinv(X_aug.T @ X_aug) @ X_aug.T @ y\n", " \n", " \n", " def predict(self, X):\n", " \"\"\" Make predictions for data X.\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " \n", " Returns:\n", " -----\n", " y_pred: (num_samples, ) np.array\n", " Predictions\n", " \"\"\"\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", "\n", " y_pred = X_aug @ self.coef_\n", " return(y_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try your model on the data and plot the data points and the fitted line:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEXCAYAAACDChKsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de1yUZfr48c8MJyHxPOD5gCbkKc9mmYdEZRTRNM1DptXWurnVL9c2y/26ZUuGa2vtd7eDRYuZqXlYWL4rakJQpmXmERUw8Kw4I6CInIaZ5/cHQiCDMMMcgLner1evF/OcuG7G5prnvp/7ulWKoigIIYQQFlI7OwAhhBANkyQQIYQQVpEEIoQQwiqSQIQQQlhFEogQQgirSAIRQghhFXdnByBEY3Tx4kXGjRtHz549ATCZTHh4ePDkk08yderUu577j3/8g6CgIIKDgx0RqhBWkwQihJ00adKEmJiY8teXLl1iwYIFeHt7M2HChGrP+/HHH+nRo4cjQhSiTiSBCOEgHTp04MUXXyQyMpKePXuyYsUK8vPz0el0BAUF8d5777F161aSk5NZtWoVbm5u9OjRw+xxXl5ezm6OEDIGIoQjBQUFkZaWxldffcXUqVPZvHkzu3fv5uLFiyQmJjJ37lz69OnDH//4R8aNG1ftcULUB3IHIoQDqVQqmjRpwiuvvML333/PJ598wtmzZ9HpdOTn51c5vrbHCeEMkkCEcKDjx4/Ts2dPFi9ejNFoRKvVMnr0aK5cuYK5snS1PU4IZ5AuLCEc5MyZM3zwwQc8/fTT7N27l0WLFjFx4kRUKhVHjx7FaDQC4ObmRklJCcBdjxPC2eQORAg7KSwsZMqUKQCo1Wq8vLxYvHgxo0eP5uWXX2bRokU0b94cb29vhgwZwvnz5wEYM2YMERERGAyGux4nhLOppJy7EEIIa0gXlhBCCKtIAhFCCGEVSSBCCCGsIglECCGEVSSBCCGEsIokECGEEFZxuXkgOTm3MJnq9uRy69ZNycrKs1FE9Z+0t/FztTZLe2tPrVbRsuU9Zve5XAIxmZQ6J5Cy67gSaW/j52ptlvbWncO7sA4ePMi0adOYPHkyCxcu5MaNG1WOKS4u5pVXXkGr1fLoo4+Snp4OgKIoREREEBISwsSJE/n5558dHb4QQojbHJ5AXnvtNVatWkVsbCw9evQgMjKyyjHr16/H29ubuLg4Xn/9dZYuXQrArl27SE9PZ8eOHfzzn/9k6dKl5TWDhBBCOJbDu7B27NiBh4cHBoOBq1evEhgYWOWYxMREXnrpJQCGDBlCTk4Oly9fJikpiYkTJ6JWq+nWrRvt27fn8OHDDBkyxOp4FEUhJ0dPcXEhULtbPJ1Ojclksvp3NjSWtNfNzZ2mTVvg7W2+z1QI0Xg4PIF4eHiQmprKU089hbu7O4sXL65yjE6nQ6PRlL/WaDRkZmai0+nw8/Orsr0u8vJuoFKp8PfviEpVuxsyd3c1JSWuk0Bq215FUTAYirl+XQ8gSUSIRs5uCSQuLo6VK1dW2hYQEEBUVBSBgYHs27ePTZs28fLLL7Np06Yar6dWq82ug6BWW9YL17p100qvs7Iu07q1P+7ulv0p3N1d6wno2rbXw8MbNzc/cnOz6dy5rZ2jsh+NxtfZITicq7VZ2lt3dksgWq0WrVZbaVtRURF79uwhODgYgLCwMCIiIqqc6+fnh16vp0uXLgDo9Xr8/Pzw9/dHr9eXH1e23RJZWXmVnkYwGAwoimV3FHIHcndqtQdFRUXo9TftGJX9aDS+DTZ2a7lamxtDe/efyGR7UjpZuUW0bubFtFHdGd7b/Je2urRXrVZV+eJdvs+qK1rJ3d2dN998k+TkZKD0LmXgwIFVjhs1ahQxMTFA6VNbXl5etG/fnpEjRxIbG4vRaOTcuXOcPXuWvn371jkulUpV52uIX8nfUwj72n8ik3VxKWTlFgGQlVvEurgU9p+oW5e+pRw6BuLm5saaNWtYvnw5RqMRf39/wsPDAdi4cSM6nY6XXnqJefPmsXz5ciZNmoSnpyerVq0CICQkhGPHjhEWFgZAeHg4TZo0cWQT7OrQoYN89tla/vGPtZW2p6ScJDp6G0uX/o9D4oiM/JiYmO20atUaAJUK7r03kNdf/zMLFswhKupLTp5MJjExgeeff9EhMQkhfrU9KZ3iO3oFiktMbE9Kr/YuxB4cPog+ePBgtm/fXmX77Nmzy3/28vIy27WlUql49dVXefXVV+0aY30TFNSLpUt7OfR3TpkyjWee+S1QuQsrKupLAM6ePUNOTrZDYxJClCq786hpe16Bga+++YXsm0X8Yeb9Nu8dcLmZ6A1RxTuT3//+OXr16s3Ro0e4fj2H//f/XmH48IfIzs7ir399m6tXr6JWq/ntbxcxZMgw9HodK1e+RV7eTbKyrhEcPIHf/e4FduyIJS7u/7hx4zoPPTSS3/52Ua1iGTFiMHFx3/Dppx9RUFDAunWRzJ//jJ3/AkKIilo38zKbRFo38wJKn4j84eRVNsWf5lZBCXNDguzStSwJ5A7fH7/C3mNX7nqMSgXWLAQ8ol87HurbzsrIfmUwlPDxx/9i795v+eSTDxk+/CHef381kyaFMWLEKK5du8bzzz9DVNSXfP31LsaNm4BWG0peXh7Tpk1i9ux5AOj1Or74YovZJ9BiYrbz3XdJQGl733zzbTp37gqAr68vv/nNQg4f/lmShxBOMG1Ud9bFpVTqxvJ0VzNtVHd01wtYvyuVE2eyCWjfjCWzghjYu51dHhqQBNIADRs2HICAgO7cvJkLwMGDBzh37hyffvoxACUlJVy6dJE5c+Zx6NBBvvxyPWfOpFNSYqCwsACAnj2Dqn18ubouLCGE85WNc1R8CmvqwwFcv1nE8k9/RK1WMXdcT8YM6IBabb+HWiSB3OGhvjXfJTj7A9XT0xMoHRMqmxtjNJr4+98/pFmz5gBcu6anZctW/O//ruHy5UuMGxfCyJGjOXjwQPk5Xl5ezmmAEKLOhvduW55IzlzJJSouhQu6PAbc24a543rSqpn9HzByrdlwjdigQYPZvn0LAGfOZDB//iyKigo5ePBH5syZxyOPBKPTXUWv19mkDIubmxtGo7HO1xFCWK+gqIQvv07jL+sOcjO/mEWP9uWF6f0ckjxA7kDqnWPHjjBu3MPlr8eP1zJ27Pgaz3v55T+yalU48+fPQlEU/vSnFfj43MMTTyzgrbeW07SpL61atSIoqBeXL1+qc5z33debzz5by4cf/i+/+90Ldb6eEMIyh0/r+WJ3GtdvFjFmYAemjeyOTxPHfqSrFHP1QRqxO2eiZ2aeo23bLhZdw9ldWI5mTXut+bvWF41hlrKlXK3NDbm9OTeL+HJPGj+n6umguYf5IUH06ND8rufYaya63IEIIUQDYFIUkg5fYmtSOoYShemjApgwtDPubs4biZAEIoQQ9dwlfR7rdqbyy6Ub3NelJU+GBOLf0sfZYUkCEUKI+spQYiR231nifjiPt5c7z0y6jwf7tK039eYkgQghRD106lwOn+9M4WpOAQ/2acvjj/TA18fT2WFVIglECCFsxJIS69XJKzCwOeE03x/PxK+FN3+Y1Z/eXVvZKeK6kQQihBA2UFZivay8SFmJdaBWSURRFH44cZWN8acpKCph0vAuTH6wK54ebnaNuy4kgQghhA3UpcS6Lie/tH7V2RwC2jdjQUgQHf3MPzpbn0gCqWfefTeC48ePUlJi4OLFC3TtGgDAjBmzmDQprMbz9+79losXzzNr1hNERpbWxSqraSWEsJ/allivqMRoYvdPF4jZewY3B9WvsiVJIFayRV+nOX/4Q+laJ1euXOaFF35bvv5GbaWmnqpzDEIIy9VUYv1OGZdL61dd1OcxsKeGOcH3OqwEia04PIEcPHiQt99+G4PBQIcOHYiIiKB588qzKHU6Ha+99hrXrl1DrVbzxz/+keHDh2MwGBg2bBidOnUqP3b79u24uTm2j3Bf8pU69XVaIzLyY06cSEany2TatJkkJHzN008/x8CBg8uTzV//+j4xMaWLdbVtW1oQ8tSpEyxc+DR6vY6JEyfL3YgQdnK3EusVFRSVsP3bDBJ+vkjzpp4serQvgwI1jg7XJhyeQF577TU+/PBDevTowerVq4mMjGTx4sWVjlm1ahVjxozhiSeeICMjg3nz5vHtt9+SmprKgAEDiIyMdHTYlWz5xjnLSRYXF/HFF6UFExMSvq6yv1u3AKZMmQbApElhREZ+THZ2Nh999Bn5+fk89lgos2c/gY/PPXaLUQhXZa7E+p09E4fT9HzxdWn9qkcGdmTaqAC8vRpuR5DDI9+xYwceHh4YDAauXr1KYGBglWPGjx/PsGHDAOjSpQtFRUXk5+dz/PhxsrOzmTlzJgBLlixh6NChDo0fIOtGofntd+nrtIVevfpYfM4DDzyIp6cnnp6eNG/egtzcXEkgQtjB3bq1c24W8eXXafycVlq/6vmpfeheQ/2qhsDhCcTDw4PU1FSeeuop3N3dq9x9QGkCKRMZGcl9992Hr68vKpWKsWPHsmjRIk6dOsWzzz5LbGwsrVo59hnp1s2bmE0i1fV12krF9TsqzkQtKSmp9pyK3XsV1w8RQthOdY/wKopCYbGRbUnplBjrR/0qW7JbAomLi2PlypWVtgUEBBAVFUVgYCD79u1j06ZNvPzyy2zatMnsNaKioti8eTNffPEFALNmzSrf16tXL/r168ehQ4cIDg6udVx3VpXU6dS4u1v2Zs4Y053P/nuKYkOFvk4PNTMe6WHxtarjdvsfWNn1yp7KKHvdsmVLzp3LYOjQoXz/fVL5Pg8PD4qLi3B3V1c5p+y61sRo6TlqtRqNxtfi31NfNOTYreVqbbZle6P37jfbrR0Vl0qJ0UT/ezU8/9j9tGvjvLt/e7y/dksgWq0WrVZbaVtRURF79uwp/8APCwsjIiLC7PmrVq0iKSmJDRs20LZt6W1gdHQ0AwcOpHPnzkDpxBsPDw+L4rqznLvJZLK4VPmDfdphNCpVbleHBvnbrMy70Vh6nbLrlcVc9nr27HmEh79BbGwMDz88unxfv379CQ9/gxYtWlY5p+y6lsZoTTl3k8nUYMtlN+RS39ZytTbbur36nAKz20uMJn4Teh/De7dFpTjv/wl7lXN36HogRqOR0aNH8+GHH9KnTx+2bdvGjh07qgyKR0VFERsby7/+9S+aNWtWvj0iIoKCggLeeOMNMjIyeOqpp/jvf/9L06a1n3Aj64FYTtYDafxcrc22bu8rH3xvdgzUx0uNt5eHzR/3t1SjWA/Ezc2NNWvWsHz5coxGI/7+/oSHhwOwceNGdDodL774Iv/85z9p2rQp8+bNKz937dq1LFq0iNdff53Q0FBUKhUREREWJQ8hhLCHiQ90YcPXaZju+DpeUGwiv6g0sTjicX9HkxUJ5Q6kRnIH0vi5Wptt1V5FUdh/IpNN8b9wq8BAbT5MWzfz4q/PP1Tn322JRnEHUl8pilJv6us3Bi72nUS4KF1OPp/vSuXk2Ry6t2/GNTXcuGWo8Tx7P+7vSC6fQNRqN4zGEtzdLRuMF9UzGIpxc3P5f1qikSoxmth14Dz/+f4sbmoVT4zvyej+HfjNqm9qdb69H/d3JJf/v9zbuyk3b16nRYvWqFSN49lsZ1EUBYOhmOvX9fj6tnR2OELYXPrlG6yLS+Gi/haDemqYM64nLX1LE0J1tbAqMlfapCFz+QTStGlzcnL0XL16EWrVg1k6x8Fkcp0xEEva6+bmjq9vS7y9Zba7aDwKikrYnpRBwqGLtPD14vfT+jKwZ+X6VeZqYbm7qfDyUHOr0OjUp7DsxeUTiEqlolUrP4vOkQFHIVzHoTQ9G8rqVw3qyLSR5utX1aYWVmPj8glECCHMyblZxIav0ziUpqej5h6ef7QP3dvfvX7V8N5tG3XCuJMkECGEqMBkUvjm8CW2JaVjNCk8Nro744d0ajT1q2xJEogQQtx2UZfHup0ppF/OpXfXlsybEIhfSx9nh1VvSQIRQri8YoOR2H1n2fnjeby93Hk2tBcP9PaX+WE1kAQihHBpJ89m8/nOVHTXC3ioT1tmPtIDXx9PZ4fVIEgCEUK4pJv5xWxO+IV9yZn4tfRmyaz+9Orq2LWFGjpJIEIIl6IoCgkHz/NJdDIFRSWEPtiF0OFd8fRwq/lkUYkkECGEy7iak8/nO1M5dS6H7h2aMT8kiI4aqehtLUkgQohGr2L9Knc3Fb+b3o9BPVqjlkHyOpEEIoRo1NIv3WDdzsr1q3oGtJHqCjYgCUQI0SgVFJWwLSmdbw5dooWvFy9M68uAO+pXibpxeAI5ePAgb7/9NgaDgQ4dOhAREUHz5pXLA1y+fJlJkyaVr33epk0bIiMjKS4uZtmyZSQnJ9OkSRNWr15N9+6Np7KlEMI2KtavGjuoI49WU79K1I3D/6KvvfYaH374IT169GD16tVERkayePHiSsccP36cyZMns2LFikrb169fj7e3N3Fxcfz0008sXbqULVu2ODJ8IUQ9lp1byIav0zh8+hodNU1Z9GhfAto3c3ZYjZbDE8iOHTvw8PDAYDBw9epVAgMDqxxz/Phx0tLSmDZtGk2bNmXZsmUEBgaSmJjISy+9BMCQIUPIycnh8uXLtG/f3tHNEELUIxXrV5lMCjNGd2ecmfpV+09ksj0pnezcIlq5QLVce3N4AvHw8CA1NZWnnnoKd3f3KncfAF5eXkydOpVZs2aRlJTEokWL2LFjBzqdDo3m1z5MjUZDZmamJBAhXNiF2/WrMsrqV4UE4dfCu8px+09kVlqvIyu3iHVxKQCSRKxktwQSFxfHypUrK20LCAggKiqKwMBA9u3bx6ZNm3j55ZfZtGlTpeNeeOGF8p9HjRrFu+++S0ZGhtnfo1ZbViGzusXhLaXR+NrkOg2FtLfxa2htLjIY2bQ7lX8n/sI93h78Yc5ARg3sWG39qui9+yst9gRQXGIieu8Zwkbf64iQncoe76/dEohWq0Wr1VbaVlRUxJ49ewgODgYgLCyMiIiIKueuX7+e0NBQWrYsXRZVURTc3d3x8/NDr9fTpUsXAPR6PX5+li0GlZWVh8lUu5UHq+NqCyxJexs/R7S5rPvIFostnTibzfqy+lV92/L4I/fS1NuDa9fyqj1Hn1NQ7fbG/n7X5f1Vq1XVfvF2aIF7d3d33nzzTZKTk4HSu5SBAwdWOe6nn35i69atABw4cACTyURAQACjRo0iJiYGKH2ay8vLS7qvhGgAyrqPytYML+s+2n8i06Lr5OYX80nsSd7ddARU8Mqs/jwzqRdNvT1qPLd1My+LtouaOXQMxM3NjTVr1rB8+XKMRiP+/v6Eh4cDsHHjRnQ6HS+99BLLli1j6dKlxMTE4OXlxbvvvotarWbevHksX76cSZMm4enpyapVqxwZvhDCStuT0s12H21PSq/VXYiiKOxLzmRzwi9W168yt2a5p7uaaaNkKoC1VIqi1K0/p4GRLizLSXsbP3u3+el3Eqrd99nSR+56ri3rV7nqU1j26sKSmTVCCLtr3cyrvPvqzu3VubN+1bwJgYzq375O9avK1ix3xS8J9iAJRAhhd5Z2H/1yu37VJf0tBgVqmBPck5a+MlZR30gCEULYXVk3UU1PYeUXlrDt23QSy+pXTe/LgHulflV9JQlECOEQZd1H1fk5Vc+Gr1O5kVfM2MEdefRhqV9V38m7I4Rwqor1qzr5NeX30/pJ/aoGQhKIEMIpTCaFhEMX2f5tRmn9qjHdGTe4av0qUX9JAhFCONwFXR5RcSmcuZJL726tmDchEL8W3jadrS7sTxKIEMJhigxG/vP9GXYfuIBPE3eendyLB3r5o1KppNhhAyQJRAjhECfOZPP5rhT01wsZ0bcdMx/pUakESV1nqwvHkwQihLCr3PxiNsefZv+Jq/i39OaV2QO4r0vLKseZm2h4t+3C+SSBCCHsoqx+1ab40xQWG5n8YFdCH+yCh7v5+lXWzFYXziUJRAhhU/tPZLIl4TTXbxkA8G/pzdK5A+lQQ/0qKXbY8EgCEULYzN7jV1gXl4KxQsHS7NxCzuvyakwgtZ2tLuoPSSBCCJv45dKNKskDwGBUaj0QXtNsdVG/SAIRQtRJfmEJ25LSSTx8ieoWSsjKLeLpdxLkrqKRkQQihLCKoigcStPzxddp5N6uX3UoVUf2zeJqz5G5HY2LwxPIwYMHefvttzEYDHTo0IGIiAiaN29e6ZiFCxdy5coVAEwmE2lpaWzdupWgoCCGDRtGp06dyo/dvn07bm61X5VMCFF32bmFfLE7jSO/lNavenF6P7q1a0a3ds2qDITfSeZ2NB4OTyCvvfYaH374IT169GD16tVERkayePHiSsd89NFH5T+///779O/fn759+5KcnMyAAQOIjIx0dNhCCH6tX7Xt2wwUM/Wr7hwIr47M7WgcHJ5AduzYgYeHBwaDgatXrxIYGFjtsenp6URHRxMbGwvA8ePHyc7OZubMmQAsWbKEoUOHOiRuIVzd+as3WbczlTNXculzu36VpoV3leMqDoS/8sH3MrejEXPKmuipqak89dRTuLu7s3nzZtq1a2f2uCVLltC/f3+eeOIJADZt2sS1a9dYtGgRp06d4tlnnyU2NpZWrVo5MnwhXEphcQmbdqfy76R0fH08eHZKX0YO6ICqFkvLJv58gX9sOUqRwVi+zcvDjd/PuJ/Rgzrd5UzRENgtgcTFxbFy5cpK2wICAoiKiip/vWnTJqKjo9m0aVOV82/cuEFISAiJiYl4eZn/tvK73/2O6dOnExwcXOu4srLyMJnq1mRXW09Z2tv4Vdfm5DNZfL4zlWs3ChnRrx0zx1SuX1Ub9bHCrqu9x3Vpr1qtonVr83N47NaFpdVq0Wq1lbYVFRWxZ8+e8g/8sLAwIiIizJ6flJTEyJEjKyWP6OhoBg4cSOfOnYHSp0A8PCz7xyyEqFml+lWtfPjj7AEEmalfVRsyt6PxcugYiLu7O2+++SZt27alT58+xMXFMXDgQLPHHjlyhMGDB1falpqaypEjR3jjjTfIyMjg1KlTDBo0yBGhC+ESFEXh++OZbE6oXf0q4docmkDc3NxYs2YNy5cvx2g04u/vT3h4OAAbN25Ep9Px0ksvAXDhwgVGjx5d6fxFixbx+uuvExoaikqlIiIigqZN714eQQhRO5nZ+Xy+M4WU89fp0bE580OC6NDmHmeHJeoxpwyiO5OMgVhO2tu4lRhNfJucyabdaXi4q5kxujsj+7dHXYtB8obK1d7jBjcGIoSo/365eIN1O1O4dO0Wg4P8mBN8Ly2ayiO2onYkgQjhgsrqV31z+BKtmnnxP88Mo5tGuquEZSSBCOFCFEXh51Q9G/akkXurmHGDO/HoyG506tDSpbp0hG1IAhHCRVSsX9W5Qv0qIawlCUSIRs5kUoj/+SLbv8tAURRmjunBuCEdcVOrnR2aaOAkgQjRiJXWr0rhzJWb9Aloxbzx5utXCWENSSBCNEJFBiP/2XuGXQcu0NTbnefCejHsPv9a1a8SorYkgQjRyCRnZPH5rtL6VQ/3a8cMK+pXCVEbkkCEaCRybxWzKeE0P5y4SttWPrw6ZwCBna2rXyVEbUgCEaKBUxSFvcev8FXCLxQWGwl7qCuThkv9KmF/kkCEaMAq1q+6t2NznpT6VcKBJIEI0QCVGE3E/XCO2H3n8HBXMz8kkIfvb9z1q0T9IwlEiAbm9MXrrNuZyuVrtxgS5MdsqV8lnEQSiBANRH6hga1JGSQevkTrZl689Fg/7u/Rpnx/fVz5TzRukkCEqOfK61d9nUZufjHjh3Ri6sPdaOL56/+++09ksi4uheISEwBZuUWsi0sBkCQi7MZptQxOnjxJnz59zO5TFIWIiAhCQkKYOHEiP//8c/m+zz77jJCQECZMmMDu3bsdFa4QTpF1o5C/bz3GB9HJNG/qyf/MH0yXtr78z6c/8vQ7Cbzywffldx5lyaNMcYmJ7UnpTopcuAKn3IEUFBSwYsUKDAaD2f27du0iPT2dHTt2cO7cOZ577jni4uI4efIk//nPf4iJiSEvL4/HH3+coUOH0qJFCwe3QAj7Kq9f9W0GCr/WrzpwSmf2TuPO5FEmK7fIkWELF+OUBPLOO++wYMECDh8+bHZ/UlISEydORK1W061bN9q3b8/hw4f58ccfGTduHF5eXnh5eTF06FASExOZOnWqg1sghP2cv3qTqLgUzmbepG9Aa+aN70mb2/WrqrvTUKvA3EKbrZvJ4LqwH4cnkPj4eAoLCwkJCan2GJ1Oh5+fX/lrjUZDZmYmOp2Ovn37VtkuRGNQVGwk5vsz7L5dv2rhlN4MCfKrVL+qujsKkwKe7upKycXTXc20Ud3tHrdwXXZLIHFxcaxcubLStoCAAPLy8oiKirrrueaWaVer1dVut0R1a/taSqPxtcl1Ggppr30dStHxz21H0WXnM35YFxaE9sLXx7NqXC290ecUmN3+pPY+Po87xbWcAtrcfj16UKdaxyDvceNmj/bWmEBeeOEFZs+ezYMPPmjRhbVaLVqtttK2LVu28PHHHzN37tzybVOmTGHDhg00bfrrB7u/vz96vb78tV6vx8/Pz+z2bt26WRRXVlYeJnP3+haoywL1DZG0135ybxWzKf40P5ysXL+q8FYRhbeq3m1MHdGtypiHp7uaqSO60btzCyJ+O7zS8bVth7zHjVtd2qtWq6r94l3j1/fx48fzwQcfMGHCBCIjI7l+/bpVQQDMmDGDPXv2EBMTQ0xMDAAxMTGVkgfAyJEjiY2NxWg0cu7cOc6ePUvfvn0ZOXIku3fvpqCggOzsbH744QeGDx9u7lcJUa8pisJ3Ry+z7JMfOJiqI+yhrrz59NAaix8O792W+dqg8rGN1s28mK8Nkkd1hVPUeAcyefJkJk+eTHp6Otu2bWPGjBn079+fefPm0a9fP5sFEh8fT0JCAuHh4YSEhHDs2DHCwsIACA8Pp0mTJvTr14+wsDAee+wxSkpKePHFF/H397dZDEI4wpWsW6zflUrK+ev0vF2/qr0F9auG924rCUPUCyrF3MDCHUwmE4mJiWzdupVTp04xYcIEDhw4wPDhw3nllVccEafNSLtcDA0AABtvSURBVBeW5aS9tlFiNLHjh3P8376zeLi7MXNM93pTv0re48bNXl1YNd6BrFmzhu3bt9OpUyfmzJnD+++/j4eHB/n5+YwZM6bBJRAhnCHtwnXW7UzhSlY+Q+/zY/bYe2ku9atEA1djAsnOzuaTTz4hKCio0nYfHx/effdduwUmRGOQX2hga2I6iUcu07qZF/9vRj/6dW9T84lCNAA1JpC33nqr2n0jRoywaTBCNBaKonAwVc+Xd6lfJURDJ/+ahaByJVtNS2+mjuhm9UB11o1CvtidytH0LLr4+/LSjH50bdvMxhEL4XySQITLu7OSrT6nwKpKtiaTwp6fL/Lv2/WrHn+kB8GDO+Jm4WRXIRoKSSDC5d2tkm1tE8i5zJus22m+fpUQjZUkEOHyqqsvVZtKtkXFRmL2nmH3Txdo6uNhtn6VEI2VJBDh8lo38zKbLGqqZHs8I4v1u1K5dqOQkfe3Z8aY7tzTxMNeYQpR70gCES5v2qjuZutLVVfJ9sbt+lU/nrxKu9Y+LJ07kJ6dZE0a4XokgQiXVzbOUd1TWBWf0LqniTslRhNGk8KUEd2Y+EAXPNxlkFy4JkkgQlC5vlTFsg/rd6XwzeHL5cfdKixBBTw2ujvaB7o4I1Qh6g356iRENfafyKyUPMooQMKhi44PSIh6Ru5ARL1XsQupdTMvpo3q7pBqtJvjT1e7T9YaF0ISiKjn7pzkl5VbZNUkP0vk5RezbmcKufmGao+RtcaFkAQi6jlbTPKrLUVR+ClFx6aEX7iRV0QTTzcKi41mj5W1xoWQBCLqubpM8rPEtRsFfLE7jWPpWfTo2JyXpvfjctYtPvu/kxjvWD5mzID2sqCTEDgxgZw8eZKZM2eSnJxcZd+tW7d4/fXXycjIAGDhwoVMmjQJgLFjx1ZaAvejjz6iXbt2jglaOJy1k/xqy2gyEX/wIv/+7gwKCrMe6cGskPvIzr7F5axbqNQqKmYQdzcVPTrKnA8hwEkJpKCggBUrVmAwmO9jXrt2Le3bt+f9998nKyuLKVOmMGzYMNzc3PDw8ChfT100fpZM8rN0sP1c5k2idqZwLvMm/bq35onxPWnT3Bs3t9KHE7cnpVNyx+1HiVGxS/eZEA2RUxLIO++8w4IFCzh8+LDZ/UOHDqVbt24AtG7dmhYtWnDt2jV0Oh2KojB37lzy8/N57rnn0Gq1jgxdONidk/yqSwyWDLYXFRuJ3pvB7p8u4OvjWW39Kkd1nwnRUDk8gcTHx1NYWEhISEi1xzz00EPlP+/YsYPi4mJ69OjBxYsXefjhh3n11Ve5evUqc+fOpWfPnnTvLgOajVnFSX7Vqe1g+7H00vpVWbmFjOrfnsdGV1+/yt7dZ0I0dHZLIHFxcaxcubLStoCAAPLy8oiKiqr1Nd5++20+/fRT3N3dCQ4OJjg4GICOHTsybtw49u7da1ECqW5xeEtpNL42uU5DUd/bm13NXUF2bhEajS85Nwv5NDqZb49copN/U16ZN4LeAa2rvZ5G48uC0N78Y8tRigy/Ponl5eHGgtDe9f7vYY3G2Ka7kfbWnd0SiFarrdK9tGXLFj7++GPmzp1bvm3KlCls2LCh0sA4wPr164mMjCQyMpLAwEAAvvnmG9q0aUPfvn1/bYC7ZU3IysrDZFJqPvAuKpa6cAUNob2tqrlbaOnrybY9qXyV8AvFJUamjuiG9nb9quraVNbe3p1b8GRIYJXus96dW9T7v4elGsJ7bEvS3tpTq1XVfvF2aBfWjBkzmDFjRvnrwMBAswPie/bsISoqio0bN1Z6wurSpUts3ryZDz74gOzsbBISEli/fr1DYhf1m7nBdg83FZ4e7kTFpdCzUwvmhwTSrvU9Fl23Nt1nQriqejMPJD4+noSEBMLDw/n73/9OUVERCxcuLN//l7/8hVmzZpGamkpoaCgmk4klS5bQoUMHJ0Yt6os7B9u9vdwoMpjIvVXMAm0QI/q1Qy2LPAlhUypFUerWn9PASBeW5RpSe9MuXGfdzhSuZOUzrJc/s8beS/N7PC26RkNqr624WpulvbVXb7qwhKgtS+d03Co0sOWbdL49epk2zZvw8sz76XuXQXIhRN1JAhH1jiVzOsrqV3255zR5+QZChnZmyohueHm6OTxuIVyNJBBR79R2TkfF+lVd2vry8oz76dLWtR7NFMKZJIGIeqemGeBGk4k9By/y7+8yUKFi1th7GTuoA25qWR9NCEeSBCLqnbvNAD+XeZOouBTOXS2tXzVvfCCtmzdxQpRCCPnKJuqdaaO64+le+Z+mh5uK9m3uYcW6n8jJK+J3U/vw0mP9JHkI4URyByLqnTvndPh6u6Og4nhGNqNv16/yqaZ+lRDCcSSBiHppeO+29OrSko3xpzlwSke71j78flpfenaStTiEqC8kgYh6x6QofHf0Mlu+SS+tX/VwN7TDSutXCSHqD0kgol65fO0Wn+9MIe3iDQI7lRYztLR+lRDCMSSBiHrBUGLiv/vPsuOHc3h5uPHU7fpVdy7yJISoPySBCKdLPZ/D57tSuZKVzwO361c1s7B+lRDC8SSBCKcprV/1C98evSL1q4RogCSBCIdTFIUDp3RsjL9dv2pYZ6Y8JPWrhGhoJIEIh7p2vYD1u9M4npFF17a+LJ55P539pX6VEA2RJBDhEEaTia9/ukj03tL6VbPH3svYQR1Rq2WQXIiGymkJ5OTJk8ycOZPk5OQq+wwGA8OGDaNTp07l27Zv345arWbVqlV88803qNVq3nrrLQYNGuTIsIUVzmbmEhWXwvmredzfvTVPSP0qIRoFpySQgoICVqxYgcFgMLs/NTWVAQMGEBkZWWn7zp07SU9PZ8eOHZw7d47nnnuOuLg43N3lRqo+KiwuIfq7M3x98ALNfDx5fmofBgVq5NFcIRoJp0ztfeedd1iwYEG1+48fP052djYzZ85k5syZHDhwAICkpCQmTpyIWq2mW7dutG/fnsOHDzsoamGJo79c438+/ZHdP11gVP8OhD87jMFBfpI8hGhEHP7VPT4+nsLCQkJCQqo9RqVSMXbsWBYtWsSpU6d49tlniY2NRafT4efnV36cRqMhMzPTot9f3dq+ltJoXGvgt7btzcktZG30cfYevUwnf19emTeE3g3w0VxXe3/B9dos7a07uyWQuLg4Vq5cWWlbQEAAeXl5REVF3fXcWbNmlf/cq1cv+vXrx6FDh1AUpcqxagsXEcrKysNkqnodS9Rlgfr6rLp1yGvTXpOi8O3t+lWGEiOPPtwN7QNdcHdTN7i/VWN9f+/G1dos7a09tVpV7RdvuyUQrVaLVquttG3Lli18/PHHzJ07t3zblClT2LBhA02b/hpgdHQ0AwcOpHPnzkDpvAEPDw/8/f3R6/Xlx+n1+kp3JMJ6d1uHPGz03b+5XL52i3U7Uzh98QZBnVvwZEgQbVv52D1mIYRzObQLa8aMGcyYMaP8dWBgIDExMVWOS01N5ciRI7zxxhtkZGRw6tQpBg0aRH5+Ptu2bSM0NJSLFy9y9uxZ+vbt68gmNFp3W4c8bPS9Zs8xlBj57/5z/Hf/OZp4uvHUxCBG9JX6VUK4inrz+FJ8fDwJCQmEh4ezaNEiXn/9dUJDQ1GpVERERNC0aVNCQkI4duwYYWFhAISHh9OkiTwOags1rUN+p9TzOazbmUpmdj4P9PZn1iNSv0oIV6NSzA0sNGIyBmLeKx98X+065FF/Dilvb15Baf2q746V1q96ckIgfRrgIPndNMb3tyau1mZpb+05ZQxENCzTRnWvNAYC4OmuZtqo7kCF+lV70sgrKEE7rDNhI7rh5SH1q4RwVZJABFB1HfKKT2FlZt3i/S1HSc7Ipls7XxY/HiT1q4QQkkDEr4b3blueSPafyGRb4i98EnsSAHc3FbOD72XsQKlfJYQoJYtMiyr2n8jkXztOkX2zuHxbiVHhP3sz+PHUVSdGJoSoTySBiEoKi0tYvyuVEmPVBw1uFRpZF5fC/hOWzf4XQjROkkBEuSO361cVFhurPaZsbogQQsgYiOB6XhFf7jnNwRQd7dvcQzMfE7n55islQ/VzQ4QQrkUSiIsqGyTPvlmMitJnvR8dGYB2WGd+StFVeaS3otbNvBwbrBCiXpIE4oL2n8gkascpDLfHORRArYI2zZvg7qYufxKrbM5HRRXnhgghXJuMgbgYQ4mRL3allieP8u1GpdLYxvDebfn7SyN5dnIvNC29gdI7j/naoPIEI4RwbXIH4kJSzuWwblcqBdUMkpsb2xjeuy1ho+91qbIPQojakQRST1W3Noc18goMfPXNL+y9Xb/K18eDm2YGyWVsQwhhCUkg9dDd1uawJIkoisKPJ6+yKf50af2qBzoT9lA3DqXp71r3SgghakMSSD10t7U5aptA9NcLWL8rleQzVetX3a3ulRBC1JYkkHrI0rU5KjKaTOz+6QIx351Bpa6+flXFuldCCGENpyWQkydPMnPmTJKTk6vsW758OUePHi1/nZaWxpo1awgJCWHs2LGVlr/96KOPaNeunUNidpTWzbyqXZvjbs5cyWVdXArndXn079GGJ8b3pFUzWXBLCGEfTkkgBQUFrFixAoPB/GznFStWlP+8detW4uLimDBhAjk5OXh4eJhdBre+qcsgeE1rc9ypoKiEf3+XQfzPF2l2jyeLHu3DwJ4aWVpWCGFXTkkg77zzDgsWLODw4cN3PS4nJ4e///3vbNy4EZVKxfHjx1EUhblz55Kfn89zzz2HVqt1UNS1V9dBcEvGKI6cvsYXX6eSk1vE6IEdmD6yOz5NpGdSCGF/Dv+kiY+Pp7CwkJCQkBqPjYqKYtKkSXTo0AGA4uJiHn74YV599VWuXr3K3Llz6dmzJ92716+nh2wxCF7TGMX1vCK+/DqNg6l6OrS5h4VP9KFHx+Z1ilsIISxhtwQSFxfHypUrK20LCAggLy+PqKioGs83mUxs27aNbdu2lW8LDg4mODgYgI4dOzJu3Dj27t1rUQKpbm1fS2k01a/Il13NYHd2btFdz6sNk0lh1w9nWfffkxSXmJinvY9HR/fAw92+RQXqGndD42rtBddrs7S37uyWQLRabZXupS1btvDxxx8zd+7c8m1Tpkxhw4YNlQbGAQ4fPkzXrl3x9/cv3/bNN9/Qpk0b+vbtW77N3d2yJmRl5WEyVV3rwhI1LVDfqppB8FbNvOo0o/uSPo91O1P55dINgjq3YH5IEP6tfLiec6vScbachAg1t7excbX2guu1Wdpbe2q1qtov3g7twpoxYwYzZswofx0YGFjtgPiRI0cYNGhQpW2XLl1i8+bNfPDBB2RnZ5OQkMD69evtGrM1LB0Er4mhxEjsvnPE/XCOJp5uPDPpPh7s09bsILmtJiEKIURN6s1oa3x8PAkJCYSHhwNw4cIFAgMDKx0za9YsUlNTCQ0NxWQysWTJkvLxkfrElhP1Us7lsG5nCldzChje25/Hx95LMx/Pao+3xfiLEELUhkpRlLr15zQwjujCsoW8AgNfJfzC3uNX0LRowpMTgujdrVWN5z39TkK1+56d3MuqJCK3+42fq7VZ2lt79aYLS9SsrH7VxvjT3CooYeIDXZj8UFe8PNxqdX51kxAB6coSQtiUrAdSj+iuF/C3r46yNvYkbZp7s3zBYB4b3b3WyQNKx188q3kiS9YzF0LYktyB1AMlRhNf/3SBmL2l9avmjuvJmAEdqtSvqo2yu4tPYk+a3S/rmQshbEUSiJOduZJLVFwKF3R5DLi3DXPH1b1+1fDebcsH8O8ka34IIWxFEoiTFBSV8O9vM4g/dJHm93iy6NG+DArU2Oz6tn6UWAgh7iQJxAmOnL7G+t2pXL9pv/pVsuaHEMLeJIE4UM7NIr7ck8bPqXo6aO7hd1P70KOD/epXyZofQgh7kgTiACZFIenIZbYm/oKhRGH6qAAmDO2Mu1vVp6VsXYZECCHsRRKInVWsX3Vfl5Y8OSEQ/1Y+Zo+VMiRCiIZEEoidVKxf5e3lftf6VWWkDIkQoiGRBGIHp87l8Hl5/aq2PD62x13rV5Wpy1roQgjhaJJAbCivwMDmhNN8fzwTTYsm/GFWf3p3rbl+VRlr10IXQghnkARiA4qi8MOJ0vpVBUWW168qI3M3hBANiSSQOtJdL2D9zhROnM0hoH0z5ocE0cnPulUPZe6GEKIhkQRipYr1q9R1rF9VkczdEEI0FJJArJB2Poc1Xx7iot529auEEKKhcXgCiY6OZvXq1bRu3RqA0aNH8/LLL1c6Jjc3lyVLlnDhwgVatWrFe++9h0ajobi4mGXLlpGcnEyTJk1YvXo13bs7bnxgX/IVvtidRmGxEZUKggd3ZE5wT4f9fiGEqE8cnkCOHz/O0qVLCQ0NrfaY9957j8GDB7N27Vqio6MJDw/nvffeY/369Xh7exMXF8dPP/3E0qVL2bJli0PiLpvkZzCWrmaoKPDtkct0a9dMupyEEC7J4QtKHT9+nOjoaMLCwliyZAk3btyockxiYiKTJ08GIDQ0lG+//RaDwUBiYiJhYWEADBkyhJycHC5fvuyQuLcnpZcnjzJ3LtC0/0Qmr3zwPU+/k8ArH3zP/hOZDolNCCGcweEJRKPR8MILLxATE0O7du1YsWJFlWN0Oh0aTWlpc3d3d5o2bUp2dnal7WXXysx0zId0TZP8yu5Qyl6XlSGRJCKEaKzs1oUVFxfHypUrK20LCAggKiqq/PVvfvMbgoODa3U9tdp8rqtue3WqWxy+JpqW3uhzCsxu12h8id6732wZkui9Zwgbfa9Vv7M+0Wh8nR2CQ7lae8H12iztrTu7JRCtVotWq6207ebNm0RFRbFgwQKgdAKeu3vVEPz8/Lh27Rpt27alpKSEvLw8WrRogZ+fH3q9ni5dugCg1+vx8/OzKK6srDxMJqXmA+8wdUQ3s5P8po7ohl5/02xyAdDnFKDX37T499UnGo1vg2+DJVytveB6bZb21p5arar2i7dDu7B8fHz49NNPOXr0KABffPEF48aNq3LcqFGjiI6OBmDHjh0MHjwYDw8PRo0aRUxMDAAHDx7Ey8uL9u3bOyT24b3bMl8bROtmXqgoLS8yXxtUPoBeXbkRKUMihGisVIqiWP51vA4OHjxIeHg4hYWFdO3alVWrVuHr68v777+Pn58fs2fP5vr16yxdupQLFy7g6+vL6tWr6dixI0VFRSxfvpzk5GQ8PT35y1/+Qu/evS36/dbegVRkLpvfWYodSu9QKiaZhkq+rTV+rtZmaW/t3e0OxOEJxNnslUCg8S4GJf+zNX6u1mZpb+3dLYHITHQbkjIkQghX4vDHeIUQQjQOkkCEEEJYRRKIEEIIq0gCEUIIYRVJIEIIIazick9h1XXBJ1tfp6GQ9jZ+rtZmaW/dz3O5eSBCCCFsQ7qwhBBCWEUSiBBCCKtIAhFCCGEVSSBCCCGsIglECCGEVSSBCCGEsIokECGEEFaRBCKEEMIqkkCEEEJYRRJINWJjY5k4cSLjxo1jw4YNVfafOnWK6dOnM2HCBJYtW0ZJSYkTorStmtq8Z88epkyZQlhYGM8//zw3btxwQpS2U1N7yyQmJvLII484MDL7qKm9GRkZzJs3j7CwMJ555pkG//5CzW0+ceIE06dPJywsjN/+9rfk5uY6IUrbysvLIzQ0lIsXL1bZZ/PPLUVUkZmZqYwZM0bJyclRbt26pUyePFk5ffp0pWMmTZqkHD58WFEURXnttdeUDRs2OCNUm6mpzTdv3lQeeughJTMzU1EURXnvvfeUt956y1nh1llt3mNFURS9Xq+EhIQoY8aMcUKUtlNTe00mkzJ+/HglKSlJURRF+etf/6qsWrXKWeHaRG3e49mzZyuJiYmKoijKypUrlb/97W/OCNVmjhw5ooSGhiq9e/dWLly4UGW/rT+35A7EjH379vHAAw/QokULfHx8mDBhAjt37izff+nSJQoLC+nfvz8A06ZNq7S/IaqpzQaDgTfeeAN/f38AAgMDuXLlirPCrbOa2lvmT3/6E7///e+dEKFt1dTeEydO4OPjw8iRIwFYuHAhc+fOdVa4NlGb99hkMnHr1i0ACgoKaNKkiTNCtZmvvvqKP//5z/j5+VXZZ4/PLUkgZuh0OjQaTflrPz8/rl69Wu1+jUZTaX9DVFObW7ZsSXBwMACFhYWsXbu2/HVDVFN7AT7//HN69erF/fff7+jwbK6m9p4/f542bdrw6quvMnnyZP785z/j4+PjjFBtpjbv8dKlS1m2bBkjRoxg3759zJo1y9Fh2lR4eDiDBw82u88en1uSQMxQzBQoVqlUtd7fENW2TTdv3uTZZ58lKCiIRx991BGh2UVN7U1LS2P37t08//zzjgzLbmpqb0lJCQcOHOCJJ54gNjaWTp068c477zgyRJurqc2FhYUsW7aMdevWsXfvXubMmcOrr77qyBAdyh6fW5JAzPD39+fatWvlr3U6XaVbwjv36/V6s7eMDUlNbS7bNmfOHIKCgggPD3d0iDZVU3t37tyJXq9n+vTpPPfcc+Vtb6hqaq9Go6FLly707dsXgNDQUI4dO+bwOG2ppjanpaXh5eVFv379AHj88cc5cOCAw+N0FHt8bkkCMePBBx9k//79ZGdnU1BQwO7du8v7hgE6dOiAl5cXP//8MwDR0dGV9jdENbXZaDSycOFCtFoty5Yta/B3XDW198UXX2TXrl3ExMSwdu1a/Pz8+PLLL50Ycd3U1N4BAwaQnZ1NSkoKAAkJCfTu3dtZ4dpETW3u0qULmZmZZGRkABAfH1+eQBsju3xu1WkIvhH7z3/+o0yaNEkZP368snbtWkVRFOU3v/mNcuzYMUVRFOXUqVPK9OnTlZCQEGXx4sVKUVGRM8O1ibu1effu3UpgYKASFhZW/t/rr7/u5Ijrpqb3uMyFCxca/FNYilJze48cOaJMnz5dmThxovL0008r165dc2a4NlFTmxMTE5XJkycroaGhyvz585Xz5887M1ybGTNmTPlTWPb83JIVCYUQQlhFurCEEEJYRRKIEEIIq0gCEUIIYRVJIEIIIawiCUQIIYRVJIEIIYSwiiQQIYQQVpEEIoST/Pvf/2bs2LHcunWL/Px8tFot0dHRzg5LiFqTiYRCONEf/vAHfH19KS4uxs3NjbfeesvZIQlRa5JAhHCivLw8pkyZQpMmTdi+fTteXl7ODkmIWpMuLCGcKCsri6KiInJzc9HpdM4ORwiLyB2IEE5iMBiYNWsWs2bNwmQysXXrVr788ks8PDycHZoQtSJ3IEI4yd/+9jc0Gg0zZszg8ccfp0WLFqxZs8bZYQlRa3IHIoQQwipyByKEEMIqkkCEEEJYRRKIEEIIq0gCEUIIYRVJIEIIIawiCUQIIYRVJIEIIYSwiiQQIYQQVvn/7t3J72CNKzoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Train model\n", "model = MultivariateLinearRegression()\n", "model.fit(x[:, np.newaxis], y)\n", "\n", "# Visualize the linear fit\n", "xfit = np.linspace(0, 1, 1000)\n", "yfit = model.predict(xfit[:, np.newaxis])\n", "\n", "fig, ax = plt.subplots()\n", "ax.scatter(x, y, label='Truth')\n", "ax.plot(xfit, yfit, label='Linear Fit')\n", "ax.set(title='Data', \n", " xlabel='x', ylabel='y')\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Print the slope and the intercept:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True slope: 2\n", "True intercept: -5 \n", "\n", "Model slope: 1.9292055341290553\n", "Model intercept: -4.976046835178198\n" ] } ], "source": [ "print(f\"True slope: {slope}\")\n", "print(f\"True intercept: {intercept} \\n\")\n", "\n", "print(f\"Model slope: {model.coef_[1]}\")\n", "print(f\"Model intercept: {model.coef_[0]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the results are very close to the inputs, as we might hope.\n", "\n", "Of course our linear regression estimator is much more capable than this, however—in addition to simple straight-line fits, it can also handle multidimensional linear models of the form\n", "$$\n", "y = a_0 + a_1 x_1 + a_2 x_2 + \\cdots\n", "$$\n", "where there are multiple $x$ values.\n", "Geometrically, this is akin to fitting a plane to points in three dimensions, or fitting a hyper-plane to points in higher dimensions.\n", "\n", "The multidimensional nature of such regressions makes them more difficult to visualize, but we can see one of these fits in action by building a toy example:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model coefficients: [ 0.5 1.5 -2. 1. ]\n", "True coefficients: [ 0.5 1.5 -2. 1. ]\n" ] } ], "source": [ "# Simulate data\n", "rng = np.random.RandomState(1)\n", "X = 3 * rng.rand(100, 3)\n", "coef = np.array([0.5, 1.5, -2., 1.])\n", "y = coef[0] + np.dot(X, coef[1:])\n", "\n", "# Fit model\n", "model.fit(X, y)\n", "\n", "# Assess quality-of-fit\n", "print(\"Model coefficients: \", model.coef_) \n", "print(\"True coefficients: \", coef)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here the $y$ data is constructed from three random $x$ values, and the linear regression recovers the coefficients used to construct the data.\n", "\n", "In this way, we can our estimator to fit lines, planes, or hyperplanes to our data.\n", "It still appears that this approach would be limited to strictly linear relationships between variables, but it turns out we can relax this as well.\n", "\n", "## 2. Basis Function Regression\n", "\n", "One trick you can use to adapt linear regression to nonlinear relationships between variables is to transform the data according to *basis functions*.\n", "\n", "The idea is to take our multidimensional linear model:\n", "$$\n", "y = a_0 + a_1 x_1 + a_2 x_2 + a_3 x_3 + \\cdots \n", "$$\n", "and replace the variables $(x_1, x_2, x_3, ...)$ with *transformations* of them, for example $(x_1^2, x_1 x_2, x_3^2, ...)$.\n", "\n", "Notice that this is *still a linear model*—the linearity refers to the fact that the coefficients $a_n$ never multiply or divide each other.\n", "What we have effectively done is to create a *nonlinear* dictionary of terms, so that a linear model can fit more 'complicated' relationships between $x$ and $y$.\n", "\n", "### Polynomial basis functions\n", "\n", "This polynomial projection is useful enough that it is built into Scikit-Learn, using the ``PolynomialFeatures`` transformer:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 2., 4., 8., 16., 32.],\n", " [ 3., 9., 27., 81., 243.],\n", " [ 4., 16., 64., 256., 1024.]])" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.preprocessing import PolynomialFeatures\n", "x = np.array([2, 3, 4])\n", "poly = PolynomialFeatures(5, include_bias=False) # with or without intercept\n", "poly.fit_transform(x[:, None])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see here that the transformer has converted our one-dimensional array into a three-dimensional array by taking the exponent of each value.\n", "This new, higher-dimensional data representation can then be plugged into a linear regression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With this transform, we can use the linear model to fit much more complicated relationships between $x$ and $y$. \n", "For example, here is a sine wave with noise:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD7CAYAAACYLnSTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhTZb4H8O9J0qQ7LW3aYgsFCrS0yiKKFJGKKFAoRYsLiLIom151tuvojN7R5zqoM85yZ/TeO+JlBBxRUFFALLsgSJVFoEPpXqAtbelC6UabNMm5f9R2uiRdsp3k5Pt5Hp/H5mT5vUn59s173vc9giiKIoiIyK0ppC6AiIhsxzAnIpIBhjkRkQwwzImIZIBhTkQkAwxzIiIZYJgTEcmASqoXrq1tgslk2xT3kBB/1NQ02qki1+dp7QU8r81sr/xZ22aFQkBwsJ/F45KFuckk2hzm7c/jSTytvYDntZntlT9HtJnDLEREMsAwJyKSAcmGWYiIzBFFEbW1VdDrWwDIbwimslIBk8lk8bhSqYK/fxB8fCyPj5vDMCcil9LYWAdBEBAeHgVBkN/ggUqlgMFgPsxFUURrqx7Xr1cBwIACnWFOA5KRVYHtRwpRU69DSKAGaUkxSEyIkLoskpHm5kYMHhwuyyDviyAIUKs1CArSoq6ummFOjpGRVYFN6TnQ/9irqKnXYVN6DgAw0MluTCYjlErPjiYvLzWMRsOAHuPZ7xhZZK4Hvv1IYUeQt9MbTNh+pJBhTnYlCILUJUjKmvYzzKkHSz3w7kHerqZe58zyiJyqvLwMixenYfjwkV1uHzMmFtOn341p05Lw7LNr8Pbb70pUYRuGOQHo2hNXCED3NQ16g8ns7QAQEqhxTpFEEgkN1WLjxi0Wj585c9qJ1ZhnU5i/8847SE9PBwAkJSXhl7/8pV2KIuc6fr4cm9Jz0GpsS2pLi9NMIqBWKbr00NUqBdKSYpxRJpFLWbfuVUycOAl5eW3njVatWob33tskWT1Wh/nx48dx7NgxfP755xAEAStXrsT+/ftx33332bM+sjNdqxGXyutRWFaPorJ6XK29gStVTf16rAAgMSEc5y9e42wWcopv/1mOY5nlDnnuaeOG4M5bhvTrvtXVVVi+/NGOn2fNmtPx/z/96fP49NOtkgY5YEOYa7VavPjii1Cr1QCAmJgYlJWV2a0wsp8bLa04nVeFM3nVuHDpWkfPOjzYBzeF+vU7zEUAR86VI+amQCyaORrjR4VCpfS86WPkecwNs6xb96o0xVhgdZiPHj264/8vXbqEr776Ch9//HG/Hx8S4m/tS3eh1QbY5XncxUDam3v5GvZkXMY3Z69A32pEaJAPZk2JxqS4cIwZFoxAv7Y/xE/8dh+qapv7/byFZfX478/PQyEAk+Mj8PSD4xEc6D3gtvQXP2N5697eykoFVKq2TkLSxEgkTYyUoqwOyh87LO01tRMEAQqF0HF79+O96c99FQrFgH4XbD4Bmp+fjzVr1uCFF17A8OHD+/24mppGm3cO02oDUFXVYNNzuJP+tFcURWRfrsWuby8ht+Q6NGolEhPCMX38TRgeEdAx5Ul3Q4eqG22zUO6fNqLX2SqWmETgu6wKnMy+ipmTopB8xzAM8tfYdWERP2N5M9dek8lkcYWkFIzGtlq61ySKbTu/GgwmKJVKtLTooVL1Ham9rQDtzGQydXlvFAqh106wTWF++vRpPPfcc/j1r3+NefPm2fJU1Im1YXixvB4fH8xHfmkdgvzVWDRzNO4aNwQ+mt4/5vbn3vDlBYsnP3ujUgo4cKoUR86VYcKoUPyQW9lxMpULi8gTTJs2HcuXP4oNGz6ARiPN7C5BFEWrusfl5eV44IEH8Oc//xmJiYkDfjx75uZ1n+MNtM0YWZYch9S7R5tt7/VGHT47XIhvz1cg0NcLqdNG4K5xQ+ClUtr82v31+uop+PhgPjILa8weDwnU4K2n7xzw88rxM+4N2wtUVFxGRES0RBU5Xn975t3fB4f1zDds2ACdToc333yz47ZFixZh8eLF1j4lAb2usky9e3SX20VRxNHMcnx8MB8GownJdwxDytThffbELWnvOXf+VjAuJgSZhTUW558DbUEdMdgXP31oPJ5485DZ5+bCIiLHsjrMX375Zbz88sv2rIVgOfRq6nU4fLoEG7/MQk29DkH+avh5e+FKdRPihgVhWXIcwoN9bX79xIQIi8Mhlr41dJ5nHhKoMdsGLiwicizOK3MxlkLPz1uJdz451xGU1xv1uFLdhDtvicC/L55olyDvS2JCBJYlx3XUGBKowbLkuC7hn5YUA7WZM/W3x4U5vD4iT8bl/C4mLSnGbO9XEAToWnvuopZzuRYKJ25K1FvPvf048K+hmiB/NRQKAXtPlMDX2wvzEqM9fhMlIkdgmLsYS+PWX58xvyDLFceiuwe+rtWITek52P5NEWrqW/DYrDFQKvilkMieGOYuqHMYZmRV4P2vsi3e1x3GojVeSqyaH4/QIG98efwy6hr1WLMgARqvgc22ISLL2D1ycR8dyIPBaH4KpzttciUIAtKmx2DJfWNwrqAaf9x6Fs26gW2+T0SWsWfuokRRxM5vL6Gx2XLgdT/56A5mTopCoJ8a63dm4U9bz+Lnj0yweiolkTP88Y+/wz//eQ4GQytKS0s69jV/6KFFmDcvtc/HHzv2DUpLi7Fo0WPYsOFdKBQCVqxYbfc6+a/IBZlEER/uz8PXP1yBxksBXWvPBQYhgRq3C/J2t8eFQSEI+NuO8wx0sgtHXpv2F794AUDbRSqefXZNr/uam5Oba3mY1J44zOJiDEYT1u/Mwtc/XMGcO4bh8dmxPab6udPwiiWTYrVYu+BmXKpowJ+2nYVOb5S6JHJT7esf2icDtG8hkZFV4dDX3bDhXfz858/isccewvbtn+CZZ1bjhx9OAWgL/gcfnI+LF4uwY8d27NixHbt37wQAXLiQhbVrn8DChSnYsMF+VydimLsQXasRf/00EyeyK/HQjBg8PGMUpt48pGNutwDzc7vdVVugJ6CorB7//cU/YTC6zuZK5D56WzXtaHq9Dv/4xydIS3vI7PERI0ZiwYI0LFiQ1jEkc+1aDf76179hw4Z/4KOPPsCNG/3bgrov/G7rItqDPKe4FiuS43DX+Js6jrXPbpHjvh2TYsOwdHYsNu3JxftfZePJlHinzpsn99fbqmlHi4+/ecCPSUy8E2q1Gmq1GoMGBaG+vh6+vn4218IwdwGdg3zlvHgk3uz+ve6BSJoQifomPT4/ehGD/DR4+J5RUpdEbkTKLSQ675DYeTGcwWB54oJS+a8puYIgwMq9DnvgMIvE9K1GvP1ZJnIue2aQt0uZOhwzb43CnhPFOHCqROpyyI2Y20JCivNKgwYF4eLFtqGdo0cPd9yuVCphNDr+nBB75hJqNRjx2qZTuFLdNma2/ZtCQPDMfb8FQcDie0ejpr4FHx3Ix67jl9BwoxXaYB/cP22ER74n1D/mVk1LcW3aJUuWYt26V7F7907cddfdHbdPmHAr1q17FYMHD3bo61u9n7mtPH0/c6PJhHWbT+NSRdf62/cuN/eL6M7t7a8j565gc3ouOv9m9PaeyI0nfMadcT9zywa6nzmHWSQgiiI278ntEeSA887Cu6ovv72E7n/iPf09IeoPhrkEPj1SiKOZ5RaPu+LmWc4i5cwEInfGMXMHMrcqra5Rj/TvinH3xEhkFlThWoO+x+PcYfMsR+HFLQho+/bqyVslWzP6zZ65g5hblfb33dnY9nUBbo8Lw2P3jcHCu0e5xFl4V2Lp4ha38eIWHkOhUMJo9OxN2Fpb9VAqB9bXZs/cQcytSjOaRKiUAlamxEOhEFzmLLwr6f6ehAZ5QzQBxzLLERygwf6TJXyvZM7Hxx8NDdcRFBQCQfCs/qYoimht1eP69SoEBAQP6LEMcwexNMZrMIrw6tTz7OvKPZ6o83ui1QYgK78Sv9nwPbYeLOg4Odq+/0b7/Uk+/P0Hoba2ClevlgI9Toe7P4VCAZPJ8mwWpVKFgIBg+PgMbFUow9xBOPZrP2FBPtCoFNC3mt9/g2EuL4IgYPBg+Q6rOWr6qWd9h3GitKQYeCl7nsBp0RscvpubHDVY2Neds1yI2jDMHWRKfDiiIwJ73N7UYnTK9pxy09s3Gr6XRBxmsbvO0xEBwEsloNXQddyPwwMD19tFrfleEjHM7ap9OmLnWSzdg7wdhwcGJrOwxuIxvpdEHGaxK3PTES3hidCB6S2wFZ67toSoA8PcjvrbQ/T0hUHW6O2Pn437tRHJAodZ7EQURahVCrM9c38fFTReSi52sUFaUgze23XB7DFv9b82+3fkhX2JXBnD3E72nCiG3mCCUiHA2KmrqFYpsPjeMQwUGyUmRKCg9HqPk6AKAWjRG5FXch019S1dzllwYRF5Eg6z2EFeyXV8drgIt8VqsWJuXMeQgJwuvuwKHp8dh1Xz47u8v0vnxEEb5I0Nuy/gs8MFkl3Yl0hq7JnbqL5Jj7/tOA9tkDdWzB0LH40KU28eInVZsmVu+4OIwb743Yc/WFz4zdku5AnYM7eBySTivV1ZaGox4Kn7b4aPhn8bpTBmaBBmTR5q8ThnDpEnYJjb4MuMS8i6VIsl943BsPAAqcvxaGnTRyLIX93jds4cIk/BruQAdJ4pEejrhfobrUhMCMdd4zisIjUvlRLPLhyH324+BbVKAV2ribNZyKMwzPup++rO+hutAACVUvDoK6K4khFDAjHnjmFI/64Y/75oAuKHO/Zq6ESuhMMs/WRpdefRzApu9ORCFtw5AuGDfbExPQc6vVHqcoichmHeT73NiODUN9eh9lJiRXIcquta8PnRIqnLIXIam8O8sbERKSkpKC0ttUc9Likjq6LX/T849c21jBkahBm3RmL/yRIUXqmTuhwip7ApzM+dO4fFixfj0qVLdirH9bSPlfe2/wenvrmeB5NiEByowfvpOWjt5+ZnRO7MpjDftm0bXnnlFYSFyfcST33thMipb67JR6PC0tmxKKtuwu6MS1KXQ+RwNs1mWbdunb3qcEkZWRW9DqFw6ptrGxcTisSEcOzOuIzbYsMQFeYvdUlEDiOIomjzBqL33HMPNm/ejKioKHvU5BIOny7BO5+cg67V/IwIbbAP/v7yLCdXRQNV16jDv711CGHBvnjruelQcvNzkinJ5pnX1DTCZONG1I66yjUAbPwyy2KQq1UK3D9thMNe2xJHttdV2aPNi2eOxt92ZOGj9AuYPXmYnSpzDE/7jD2tvYD1bVYoBISEWP52yamJFvQ2vMKdEN3L7XFhmDAqFJ9/U4TK681Sl0PkEAxzC8zt8wG0jZMzyN2LIAh4bNYYKBQCPtiTAzuMLBK5HLuE+aFDh2Q1Xi6KIvy8vXrczpkr7mtwoDceujsGWZdqcfw8V+yS/LBnbsaRs2W4Ut2EabcM4YUmZCRpYiRGRQ3CxwfzUd+kl7ocIrviRlvdVF1vxtZDBYgfHowVc+O4iZaMKAQBy+fE4dX3T2DLgTysXXCz1CUR2Q175p2Iooj3v8qGQgGsSB7LIJehm0L9kDJ1OE5kV+JsQbXU5RDZDcO8k2OZ5cgpvo6HZ4xCyCBvqcshB5k7JRqRoX74YG8umnUGqcshsguG+Y/qmvTY9nUBxgwNwvTxN0ldDjmQSqnA8uQ4XG/QYfsR7qxI8sAw/9HHB/OhazVi2ZxYDq94gJjIQZg5KQqHfihFQSl3ViT3xzAHkFlYg+8vXEVK4nAMCfGTuhxykrSkkRgcqMH76dncWZHcnseHuU5vxAd7czEkxBfJU6KlLoecyFutwuOz41Bec4M7K5Lb8/gw/+JYEWrqW7BsThy8VB7/dniccTEhmPLjzopXqhqlLofIah6dXqVVjdh/shRJE27CmKFBUpdDElk0czR8NCpsTM+xefM3Iql4bJiLoogt+/Pgo1FiIZfoe7RAXzUWzxyNwrJ6fH3mitTlEFnFY8P8ZE4lcoqvI236SPj79NyHhTzLlIRw3DxiMD49UoiauhapyyEaMI8Mc53eiK2HCjAszB9JEyKlLodcgCAIWDo7FhCBD/blcmdFcjsesTdLRlYFth8pRE29DiGBGgwNC0Btgw5rFyRAwSvP0I9Cg3yQNn0kPjqYj++zr2JKPDdVI/ch+555RlYFNqXndFxsoqZeh7MF1RgVOQijo3jSk7qaOSkKI4YEYsv+fDTc4M6K5D5kH+bbjxRCb2ZBSE0drzhDPSkUAlYkx6FZZ8DWQwVSl0PUb7IPc0uXf6ttZK+LzIsK80fylGgcP1+B8xdrpC6HqF9kH+btF5fo7+1EADB/ajQiBvti855ctOi5syK5PtmHeVpSDNTdVnby8m/UFy+VEsuT41Bd14Ivjl6UuhyiPsk+zBMTIvD47NiOWSuDA9S8/Bv1y5ihQbh7YiT2nypBUVm91OUQ9Ur2YQ4ALXojTCYRP31oHP7wb9MY5NRvDybFYJCfGhvTs2EwcmdFcl2yD3Od3ohd315E3LAg3DIyROpyyM34eqvw+OxYlFY1Yc/3xVKXQ2SR7MP80JlS1N9oRdr0GF50gqwycbQWt8WFYee3l1Be0yR1OURmyXIFaOcVn4IARGn9MCpqkNRlkRtbcu9oXLh4DZvSc/DLJbdCwY4BuRjZ9cy7r/gURaDi2g1kZFVIXBm5s0H+Gjxyzyjkldbhm7NlUpdD1IPswtzcik+DUcT2I4USVURyMW3cEIyNDsYnhwtQ22B+MRqRVGQ3zGJpxael24n6SxAELJ0Ti99sOIG/fpqJhhs6XGvQIyRQg7SkGM6SIknJrmfOFZ/kSOHBvpg4OhSXrzbgWkPblhA19TpsSs/hUB5JSnZhnpYUA2W3bW254pPsqaD0eo/b9AYTh/JIUrIL89vjwuCjUUGlbAv0kEANV3ySXbX3yLvjUB5JSXZj5ieyr6KxuRU/eXAcxo8KlbockqGQQI3Z4OZQHklJVj1zURSR/n0xIrV+GBfD1Z7kGOY2b/NSChzKI0nJKswzC2twpaoJyXcM42pPcpjEhAgsS47r0hOPjgjkUB5JSlZhvu9kCYIDNJg8NlzqUkjmEhMi8NbTd2LV/Hj4qJUouFKH5/7yDWe0kGRkE+Zl1U3IvlyLe26NhEopm2aRC2tfbdysNwIAGpsN2PhVNgOdJCGbE6AHfyiFUiHg0OlSfHakiAs5yOHMrTZuNYr47HABf+/I6WTRhb3RYsDRc2UQRbHj2p5cyEGOZmkqoqWpi0SOZFPPfNeuXfjf//1ftLa2Yvny5ViyZIm96hqQb8+Xw2AUe9zevpCDvSRyBEtTFAUA+0+WYN/JYtTU6/gtkZzC6p751atX8ec//xlbtmzBjh07sHXrVhQUFNiztn4xiSIOnS61eJwLOchRLE1RVCgEfHwov+N3j98SyRmsDvPjx49jypQpCAoKgq+vL2bPno09e/bYs7Z+yb5Ui6u1zfD3Mf8lgws5yFG6T1EMCdRg+dyx8FYrIXb7osjl/uRoVg+zVFZWQqvVdvwcFhaGzMxMuxQ1EEczy+DnrcLDM0bhH/vyupyQ4p4s5GiJCRE9hk/e23XB7H35LZEcyeowF7t3PYABLdQJCfG39qVx+HQJNqdno6q2GQAwcYwWD8yMRXCQLzanZ6O6thmhwT5YmjwWd08aavXruCKtNkDqEpzO3dqsDfbp+N3sfnt/2uJu7bWVp7UXcEybrQ7z8PBwnDp1quPnyspKhIWF9fvxNTWNMJl6/kHoS/vc3s498H8WVmPn4XwkJkTgd2sSu9y/qqphwK/hqrTaAFm1pz/csc33TxvR43dUrVLg/mkj+myLO7bXFp7WXsD6NisUQq+dYKvHzKdOnYqMjAxcu3YNzc3N2LdvH6ZPn27t0/UbryRErs7ccv97bo3kbBZyKJt65j/72c+wdOlStLa24sEHH8S4cePsWZtZvJIQuYP2sfRWgxGvbTqF4+crMOeOaAT6qaUujWTKpkVD8+fPx5dffom9e/di1apV9qqpV7ySELkTL5USq1MTcENnxN+/yjZ7ronIHtxuBai5ub2ctUKuLErrj4dnxCCzsAaHfrgidTkkU24X5p3HIwXwSkLkHmZOisK4mBBs+7oAV6oapS6HZMgtN9pqH4/0xDPh5J4EQcCKuWPxmw3f492dF/AfyybBS6WUuiySEbfrmRO5q0F+ajwxdyxKqxrxyWHOviL7YpgTOdH4UaGYOSkKB06VIrOwRupySEYY5kRO9vCMGERq/fD33RdQ18Ttcsk+GOZETualUmJNagKa9UZs2H0BJk5XJDtgmBNJIErrj0fuGYXzRddw4JTlLZyJ+othTiSRGRMjMWFUKD49XIDiq5yVRbZhmBNJpG26Yhz8fLzw7s4s6FqNUpdEboxhTiShAF81VqXEo6LmBrYezJe6HHJjDHMiicUPH4w5dwzD4bNlOJ5ZJnU55KYY5kQu4IHpIzE8IgBvbzuLa/UtUpdDbohhTuQCVEoF1qQmwGA04f++vGDVhVvIszHMiVxE+GBfrHlgHHKKryP9+8tSl0NuhmFO5EJm3j4Uk8eG4fNvLqKwrE7qcsiNMMyJXIggCFg6OxbBARqs35mFZp1B6pLITTDMiVyMr7cXVqfGo7quBf/Ylyd1OeQmGOZELmh0VBBS7xyBjKwKZGRVSF0OuQGGOZGLSpkajVFRg/DB3lxUXm+WuhxycQxzIhelVCiwen48BEHA+p1ZMBhNUpdELoxhTuTCQgf5YNmcWBSV1WPnt5ekLodcGMOcyMVNHhuOabcMwe7jl5BbXCt1OeSiGOZEbuDR+0YjLNgH63ddQGNzq9TlkAtimBO5AW+1CqtTE1DfpMem9ByIvDoRdcMwJ3ITI4YEIi1pJE7nVeFoZrnU5ZCLYZgTuZHZk4dhbHQwthzIQ3lNk9TlkAthmBO5EYUgYGVKPNQqJd7dkYVWA6crUhuGOZGbCQ7QYMXcOBRXNuKzI4VSl0MugmFO5IYmjtZixq2R2HeyBOeLaqQuh1wAw5zITT0yYxQiQ/3wf7uzUd+kl7ockhjDnMhNqb2UWJOagBstrfhgXy6nK3o4hjmRG4sK88cDd43E6dwqnMiulLockhDDnMjNzZ48DCNvCsQ/9uWirlEndTkkEYY5kZtTKAQ8OW8sdK0mbN7L4RZPxTAnkoEhIX5Imz4SZ/Kr8V3WVanLIQkwzIlkYtbtQzEqchA+3J+H2gYOt3gam8P8L3/5C95++2171EJENlAoBDwxbywMRhM27eFmXJ5GZe0DGxoa8MYbb2D37t1YuXKlPWsiIitFDPbFwqQYfHQwH5v35uJ8UQ1q6nUICdQgLSkGiQkRUpdIDmJ1z/zgwYMYPnw4VqxYYc96iMhGM2+LQsRgXxw5W4aa+rbhlpp6HTal5/Di0DJmdZjff//9WL16NZRKpT3rISIbKQQBLXpDj9v1BhO2cy8X2epzmCU9PR1vvPFGl9tGjhyJjRs32vTCISH+Nj2+nVYbYJfncRee1l7A89psj/bWNZpf3n+tXudy76er1eMMjmhzn2GenJyM5ORku79wTU0jTCbbTtBotQGoqmqwU0Wuz9PaC3hem+3V3sGBmo4hlu63u9L76WmfL2B9mxUKoddOMKcmEslQWlIM1Kqu/7zVKgXSkmIkqogczerZLETkutpnrWw/UtjRQ7/3tigAwPP/8y1nuMiQzWH+7LPP2qMOIrKzxIQIJCZEQN9qxG/+fgJHM8vRojOg1dg2vNk+w6X9vuTeOMxCJHNqLyWWz4lDw43WjiBvxxkuzlXXqIPR6JhL/THMiTxAXHSwxWPmTpSS/VXXNeOFdzOw57vLDnl+hjmRhxAs3K6wdIDsatuhAkAEJsc7ZkiLYU7kISxNBLZxhjD1Q/blWpzKrcLcxGhog30c8hqczULkIUIszD0PCdRIUI38ZWRVdMwmUioE+Pt4Yc7kYQ57PfbMiTyEubnnAKBrNXLPFjvLyKrApvScjj+eRpOIFp0Bp/OqHPaaDHMiD5GYEIFlyXHw8+66n1Jjs4GbcNnZ9iOF0Bu6zloxmESHzhximBN5kMSECHire46ucoqifVmaIeTImUMMcyIPI0XQeBpL5yEceX6CYU7kYfx9zM97sHQ7DdzcKdE9bnP03jgMcyIPY+lyciaTY1YmehpRFHHhci0UAjDIzwtAW498WXKcQ7dN4J9iIg/T1GI0e/sNHcPcHjKyKnA6twoP3h1jtofuKOyZE3mY3sZtL5bXO7ES+blS1YjNe3MxOmqQQ+eUm8MwJ/Iw5uabeykF+GpU2JSeA4ODNoKSu2adAe98fh7eahXWLrgZCifvk8BhFiIP032v8/Z9zb2UCvzPF+ex/2QJkp04POCOOq/u9PdRwWQydQxTpUyNRnCA81fVMsyJPFD7XuediaKIiaNDsePYRUyK1SIs2Fei6lxb++rO9kVBjc1dL56970QJhoT4OX2PeA6zEBEAQBAELLlvDBQKAZv35lqc9eLpzK3u7EyqBVgMcyLqMDjQGwuTYnDhUi2On+fyfnP6s7hKigVYDHMi6mLGrZGIiQzE1kMFqL+hl7ocl9OfVZxS7ETJMCeiLhSCgOVz4tCsM2DrwXypy3E5KVOHQ+hlooqjV3pawjAnoh4itf5InhKNjKyrOF9UI3U5LuNKVSP2nSwBAPh5t80f8fdRdexE6YyVnpZwNgsRmTV/ajRO5VRi895cvPbkHdColX0/SKaMJhP2fF+MHccuwkejwvOLJvZ6XVUpsGdORGZ5qZRYNicW1XUt+OJYkdTlSKasugmvf/ADPjtShPGjQvHak3e4XJAD7JkTUS9ihwVj+vibsO9kCabERyA6IkDqkpzGZBKx92QxPv/mIrzVSqxdkIDb48Ig9DZgLiGGORH10HmFY3CAGt5qJd5Pz8Z/LLsNSoWix33aV5FKMVbsCOU1Tfj77mwUltXj1jFaPD47FoP81FKX1SuGORF10X2FY22DHiqFgOKrjdh/shRz7hjW4z419TpsSs8BALcOdJNJxP5TJdj+TRHUKgVWz4/HHfHhLtsb74xhTkRdWLp+pZdKgc+PFmFcTIjZ+7SvfHTXMK+svYH/252NgtI6TBgViqVzYl79B7YAAAp4SURBVBHk7/z54tZimBNRF5ZWL7YaTPD38cL6XVmyuvScKIo4crYMWw8VQKEQ8OS8sZh6c4Rb9MY7Y5gTURchgRqzoRwSqMHie8fgne3/hLdaiRZ9z4tcSLHysb/MjfGPjQ7GxvQcZBbWYGx0MJ6cNxaDA72lLtUqDHMi6iItKabLeDjwr1WNt47RYvr4IfjmXDlUSgEGo9jjPq7I3Bj/hl0X0N5CX40Sd94S4bZBDjDMiagbS/udt9++aOZo5Fy+jhs6A7xUAmob9C43m6V7L7xFb+gxxt/5pxs6IzbvyYUgCC7ThoFimBNRD+b2O2/nrVZhVWo8Xv/gNLyUrrfu0FwvvD/c/QSu630SROTyKmuboRCEHlMTM7Kk3zb3owN5ve433ht3PIHbjmFORAO2/UghjKauF6+Q6qIMnWVkVfS48s9AuPIJ3L4wzIlowFx1amJvf0wEARjk5wUA8PNWQqXsOvXQlU/g9gfHzIlowCxNXxwcIO2S997+mDw+KxZ3T4zs+Flu2xEwzIlowMxNXwSAoABvmEQRCokW3Fj6I+OrUXQJcqD3k7zuiMMsRDRgiQkRWJYc1zHGHBKoweSxYSgqq8f2I9Jtl5uWFGN2+GTJrDiJKnIeq3vmp0+fxuuvvw6DwYCgoCC8/vrriIyM7PuBRCQL3Xu2oijCV6PCV99dRoCvF2ZPHiZJXaLYNj4u/nh+1kvlXsvyrWV1z/z555/HunXrsGPHDsyfPx+//e1v7VkXEbkZQRCwZNYY3BarxdZDBdj/4+XVnMFgNOHD/Xl4b9cFaIN8oFL8K8CbWowuM23SkawKc71ej5/85CeIi2v76hIbG4vy8nK7FkZE7kepUGB1agImjdHio4P52Hei2OGvWdugw+8/OoODp0sx6/ah0Lca0Gp0vWmTjiaIoij2fTfLTCYTnnrqKdxyyy145pln7FUXEbmxVoMJf/jwFI5nliN1+kg8Mf9mKBX2H+44lX0Vf9l6Bi06A557eCLumhiJ1F/sgLlQEwDs/OMCu9fgKvocM09PT8cbb7zR5baRI0di48aN0Ov1ePHFF2EwGLBmzZoBvXBNTSNMJpv+jkCrDUBVVYNNz+FOPK29gOe1WU7tfWJOHHy9lNj5TREuX6nDkynx8Pfx6jiekVWBL45dRFVt84CnBrboDdh2qACHz5YhUuuHXzw8HpFaf1RVNWCwpWmTgRqXeG+t/YwVCgEhIf4Wj1vdM29qasJTTz2FoKAg/OEPf4BaPbD5pQzzgfO09gKe12Y5tvfg6VJ8fDAfgX5qrEyJx9jo4B77pwBts06WJcf1GuiiKOKHvGpsPZSPmroWzJ48DA9MHwEvlbLjPtY+t7M4Ksytns3y/PPPIzo6Gv/5n//pdpu4E5HzzJwUhVGRg/C3nVl466MzmBIfjpzi2gFdqUgUReQUX8euby8ip/g6IrV+eGHJrRgzNKjHffva9VGurArzCxcu4ODBgxg1ahTuv/9+AEBYWBjee+89uxZHRPIQHRGA5DuGYevBfHx34arF+3UfHqm/occnXxfg+6yrMJhECAIw9eYIrJgb13FhaXPktiCoP6wK8/j4eOTm5tq7FiKSqYysCmzZ3/duht5qJbYcyENdox6lVY0or7nR5bgoAqdyKpEwYrDHhXVfuAKUiBzO3AWgzTEYTTh6rhzFVxugDfKBr0bZ4z6eMM3QGtybhYgcrq/dFC2Naz/x5iGrns8TMcyJyOEsbYClDfbB79YkDvhx7rzvuKNwmIWIHC4tKQZqVde4UasUWJo81qrHufO+447CnjkROZyl6YJ3Txra65xrT51maA2GORE5hbXTBT1xmqE1OMxCRCQDDHMiIhlgmBMRyQDDnIhIBiQ7Aaqw097G9noed+Fp7QU8r81sr/xZ0+a+HmPzxSmIiEh6HGYhIpIBhjkRkQwwzImIZIBhTkQkAwxzIiIZYJgTEckAw5yISAYY5kREMsAwJyKSAbcI8127dmHu3Lm477778OGHH/Y4np2djYULF2L27Nl46aWXYDAYJKjSfvpq74EDB7BgwQKkpqbi6aefRl1dnQRV2ldfbW53+PBh3HPPPU6szDH6am9RUREef/xxpKam4sknn3T7z7iv9mZlZWHhwoVITU3FmjVrUF9fL0GV9tXY2IiUlBSUlpb2OOaQzBJdXEVFhThjxgyxtrZWbGpqEufPny/m5+d3uc+8efPEM2fOiKIoir/61a/EDz/8UIpS7aKv9jY0NIh33nmnWFFRIYqiKP7Xf/2X+Nprr0lVrl305zMWRVGsqqoS58yZI86YMUOCKu2nr/aaTCZx1qxZ4pEjR0RRFMW33npL/P3vfy9VuTbrz+e7ePFi8fDhw6IoiuIbb7wh/ulPf5KiVLs5e/asmJKSIiYkJIglJSU9jjsis1y+Z378+HFMmTIFQUFB8PX1xezZs7Fnz56O41euXEFLSwsmTJgAAEhLS+ty3N301d7W1la8+uqrCA8PBwDExsaivLxcqnLtoq82t3v55ZfxzDPPSFChffXV3qysLPj6+mL69OkAgLVr12LJkiVSlWuz/ny+JpMJTU1NAIDm5mZ4e3tLUardbNu2Da+88grCwsJ6HHNUZrl8mFdWVkKr1Xb8HBYWhqtXr1o8rtVquxx3N321Nzg4GPfeey8AoKWlBevXr+/42V311WYA2Lx5M+Lj4zF+/Hhnl2d3fbW3uLgYoaGheOGFFzB//ny88sor8PX1laJUu+jP5/viiy/ipZdewrRp03D8+HEsWrTI2WXa1bp163DbbbeZPeaozHL5MBfNbOooCEK/j7ub/ranoaEBq1atQlxcHB544AFnlOYwfbU5Ly8P+/btw9NPP+3Mshymr/YaDAacOHECjz32GHbt2oWhQ4fizTffdGaJdtVXe1taWvDSSy9h06ZNOHbsGB599FG88MILzizRqRyVWS4f5uHh4aiuru74ubKysstXl+7Hq6qqzH61cRd9tbf9tkcffRRxcXFYt26ds0u0u77avGfPHlRVVWHhwoVYvXp1R/vdVV/t1Wq1iI6Oxi233AIASElJQWZmptPrtJe+2puXlweNRoNx48YBAB555BGcOHHC6XU6i6Myy+XDfOrUqcjIyMC1a9fQ3NyMffv2dYwlAkBkZCQ0Gg1Onz4NAPjiiy+6HHc3fbXXaDRi7dq1SE5OxksvveTW30La9dXm5557Dnv37sWOHTuwfv16hIWFYcuWLRJWbJu+2jtx4kRcu3YNOTk5AIBDhw4hISFBqnJt1ld7o6OjUVFRgaKiIgDAwYMHO/6QyZHDMsvmU6hOsHPnTnHevHnirFmzxPXr14uiKIorV64UMzMzRVEUxezsbHHhwoXinDlzxJ///OeiTqeTslyb9dbeffv2ibGxsWJqamrHf7/+9a8lrth2fX3G7UpKStx+Noso9t3es2fPigsXLhTnzp0rPvHEE2J1dbWU5dqsr/YePnxYnD9/vpiSkiIuW7ZMLC4ulrJcu5kxY0bHbBZHZxavNEREJAMuP8xCRER9Y5gTEckAw5yISAYY5kREMsAwJyKSAYY5EZEMMMyJiGSAYU5EJAP/D+6Zn63TJNuUAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Simulate data\n", "rng = np.random.RandomState(1)\n", "x = rng.rand(50)\n", "noise = 0.1 * rng.randn(50)\n", "y = 2 * np.sin(1.8*np.pi*x) + noise\n", "\n", "# Fit Model\n", "poly = PolynomialFeatures(degree=25, include_bias=False)\n", "polyx = poly.fit_transform(x[:, None]) # apply a polynomial transform to 1 feature\n", "polyxfit = poly.fit_transform(xfit[:, None])\n", "model.fit(polyx, y)\n", "yfit = model.predict(polyxfit)\n", "\n", "# Visualize \n", "fig, ax = plt.subplots()\n", "ax.scatter(x, y, label=\"Truth\")\n", "ax.plot(xfit, yfit, label=\"Fit\")\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try with different maximum degrees. Our linear model can provide an excellent fit to this non-linear data!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Regularization\n", "\n", "The introduction of basis functions into our linear regression makes the model much more flexible, but it also can very quickly lead to over-fitting and numeric problems." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ridge regression ($L_2$ Regularization)\n", "\n", "Perhaps the most common form of regularization is known as *ridge regression* or $L_2$ *regularization*, sometimes also called *Tikhonov regularization*.\n", "This proceeds by penalizing the sum of squares (2-norms) of the model coefficients; in this case, the penalty on the model fit would be \n", "$$\n", "P = \\alpha\\sum_{n=1}^N \\theta_n^2\n", "$$\n", "where $\\alpha$ is a free parameter that controls the strength of the penalty." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Fill in the following class:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "class RidgeRegularization():\n", " # Class for least-squares linear regression:\n", "\n", " def __init__(self, alpha):\n", " self.coef_ = None\n", " self.alpha_ = alpha\n", " \n", " def fit(self, X, y):\n", " \"\"\" Fit the data (X, y).\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " y: (num_sampes, ) np.array\n", " Output vector\n", " \n", " Note:\n", " -----\n", " Updates self.coef_\n", " \"\"\"\n", " # Create a (num_samples, num_features+1) np.array X_aug whose first column \n", " # is a column of all ones (so as to fit an intercept).\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", " \n", " # Update self.coef_\n", " self.coef_ = np.linalg.inv(X_aug.T @ X_aug + self.alpha_ * np.eye(n_features+1)) @ X_aug.T @ y\n", " \n", " def predict(self, X):\n", " \"\"\" Make predictions for data X.\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " \n", " Returns:\n", " -----\n", " y_pred: (num_samples, ) np.array\n", " Predictions\n", " \"\"\"\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", "\n", " y_pred = X_aug @ self.coef_\n", " return(y_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try the model in our data:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD7CAYAAAB68m/qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU9b0/8PeZmcxk3yarAQIJkJDIKiqRyKYsAQJIagU3EKqord4ul2rV+9Nbi9ra1lqXtnpRoIKKlF2CIEuUxSIIBEIWQggQyJ6QfWYyM+f3R5pAkpkss51Z3q/n6fM0c3Imn3MY3/nmux1BFEURRETk8mRSF0BERLbBQCcichMMdCIiN8FAJyJyEwx0IiI3wUAnInITDHQiIjehkOoH19Y2wWjsPgVerfZHdXWjBBU5B16/Z18/wHvg6dcPmL4HMpmAkBC/Hs+TLNCNRtFkoLcf82S8fs++foD3wNOvH7DsHrDLhYjITTDQiYjchGRdLkREphiNRtTUVECn0wDwvK4XuVwBpTIalrS3GehE5FSqqqogCAIiIwdAEDyrE0EURbS26lBaWgZf3yD4+PQ8CNqVZ90tInJ6NTW1CAgI9rgwBwBBEKBUqhAcHIbGxuv9Pt/z7hgROTWDwQC53LM7D5RKFQwGfb/PY6ATkdMRBEHqEiRl6fV79q9BIqJelJZew+LFCzF4cFyn14cPT8CkSVOQmjoZzzyzAu+88w+JKryBgU5E1IuwsHCsWbPB7PGTJ084sBrzGOhERBZYteoVjB17GwoK8gAAjz++BB9+uFbSmhjoROS0Dp8pxaHsUru8d+qoaEwcGd2n762qqsTSpQ92fD1jxqyO///zn6/Epk2fSx7mAAOdenA0pwybsy6gul4LdaAKCyfHIyU5SuqyiBzOVJfLqlWvSFNMDxjoZNLRnDKszcyDTm8EAFTXa7E2s+1PS4Y6OcrEkX1vRRMD3eOZa4VvzrrQEebtdHojNmddYKATdSGXy6HX66FQSBupDHQPc3OA+3nLoW01Qm9o2y/j5lZ4db3W5PnmXifyZKmpk7B06YNYvfqfUKlUktUhiKJo8e437777LjIzMwEAkydPxq9//es+n1td3Whyv9/w8ABUVjZYWpLLs+f1d+1GMUcd2PaBNBXe6kAV3nx6ol3qA/jvD/AeVFRcQUTEQKnLkJRCIUNJyUVERcV2vCaTCVCr/Xs8z+KVokeOHMGhQ4ewZcsWbN26FTk5Odi7d6+lb0cOYKobxZTqei00Oj0U8s6r1ZQKGRZOjrdXeURkJYu7XMLDw/H8889DqVQCAOLj43Ht2jWbFUa215/ukiaNAXIB8PdRoLFFz1kuRC7A4kAfNmxYx/8vLi7Grl278Nlnn9mkKLIPdaCqX6FuEIFmTf83CCIiaVg9KHr+/HmsWLECzz33HAYPHtzn83rqCwoPD7C2LJdmzfXrWg0or2lGRW0zKmqacb1Bi2atHi1aPSLVfqht0MHYj2GT9mGO6not1u3OR2CAN6bc1rl/8+CJK1iXmYuq2haEhfjg0bQR3b6nPzz93x/w7HtQUdHWh+zpZDJZvz8HVgX6iRMn8Oyzz+KFF17AnDlz+nWupw+Kmpsu2N/rL69txrniWlwsrUdxaQOuVTV1C2yllwzeSgW85DJ4K+Vo1lrW6ta2GrBmZw6SBwV3uo6bB1ora1vwzsZTqG/QWNQ94yn//j3hPQD0fRjrcWcKhQxGo7HT56Avg6IWB3ppaSl++tOf4q233kJKSoqlb+ORelq0M29Kz7+RjUYR+ZdrcTy/EmcvVqPyugYA4O/jhcHRARgzTI1b1H5QB3kjLMgHgX5ekMs6t3ZWvn/Y4umHXc/jfHUi52FxoK9evRparRZvvPFGx2uLFi3C4sWLbVKYO+spBOdNGWbynKuVjfg2uxTHcstxvVEHpZcMIwaFYMbtg3DrkFBEhPj0eQ/lhZPju01fVCpkmDgyCtkXqlFdr4UA009z9FXJUd+kQ6Bf22A456sTOQ+LA/2ll17CSy+9ZMtaPEZPIbjsd3tQWdsCdaAKCybFwVelwNfHS5B7qRZymYCRcWpMSI7E6KFhUHnJLfr57S3nnvZpMTVnXQDQrDXgV+8dxthhYbh3/ECzA63tc9mJXN2f/vR7nDlzGnp9K0pKrnTsi37//YswZ868Xs8/dOgblJRcxqJFD2P16rY905cvX2GXWrlSVAI9zTaprG0B0Bbuq3fmAgBCAlTImByHSaNvQYCv0iY1pCRH9dglYi70B0UG4NvT13D4TCmO51ciMsQHCpkA/U3jIZyvTo5mz43kfvWr5wC0PejimWdW9Lgvuin5+bk2qaMvGOgSMNXlYY6/jwJ/eCqlWz+4I5gL/UX3DMN9k+JwKLsUXx27DL1RhFwmwGAUERqgRMaUoew/J4eRaiO51av/gZycs6ioKMPChT/G/v17sWzZExg3bnxH+L/55tvYtm0zACAqqm2TsdzcHDz55DJUVlZg9ux0m7bWOTdIAinJUViSltjRLeHnbb7rpLFFL0mY90blJcc9tw3A6ysm4PH0JIT+51rCg30REeIjcXXkSXoak7I3nU6LTz75AgsX3m/y+JAhcZg/fyHmz1/Y0T1TU1ODv/7171i9+hN8+uk/0dzcZLN62EKXSHvr98DJq1i/J9/s9zl7X7RcJkNKchRuT4xA1qlr2HH4IlatO4HUkdH40dR4BNqoi4jIHCkH5pOSbu33ORMm3AWlUgmlUomgoGDU19fD19fPJvUw0CV0sqASn+zJh7l1Pq7UF62Qy3DPbQMwcWQUdh65hK+OXcbJ85XImBKPSaNvgczDn+JO9iPlwPzNOyvePMtMrze/1kMuv/EXuSAIsGJ/xG6c7295D9DY0ooPtufgnc1nzIY5ACxJS3S5vmhvpQI/mhKPV5bdgQHh/li3Ox+/X/8DKmqbpS6N3NTCyfFQdllZKkVjKCgoGBcvtnXzfPvtwY7X5XI5DAaDQ2pgoDtY/uVa/M/qf+P7vAosSB2C0ADTXRLqQJXLhfnNYsL88OsHx2L5nBEoqWzCyx99jwMnr9q0NUIEdB+TUgeqJGkMPfTQo9iyZROWLXsIWu2NvxjGjBmHvXt3Y9Mm++91ZdV+6NbwtKX/RqOIL48WY+uhi4gI8cVT85MxKDLA5HxvpULmkq1zc2rqNfh4Vy5yimtx65BQLJ8zAkH+pv8cdtd///7w9HvA/dAt3w+dfegOUN+swwfbc3CuuBYTkiPxyIwE+Kjabn3X+d7hIT5YkDrEbcIcAEIDvfHLB8bgwMmr2Li/EK98/D1WzEtGYmyI1KURuRUGup2VVDTi7U3ZqG/WYWlaIu4eFd1tif7N873dtXUmCAKmjRuA4QOC8f7Ws3jzs5O47+44zE6J5YApkY2wD92OTp6vxKpPTkBvNOL5h8Zh0uhb+rzfirsaEOGP/1kyHrcnRmDzN0V4+4tsNGlapS6LnIynj7VYev3sQ7cDURSx+9hlbDpwAeogb+gNRlxv1PVpSbI7XH9fiKKIgyevYsPX5+Hv4wUBIq43tbpll1N/ecpnwJzq6msICgqHQuEldSmSMRh0qKmpRHh4TMdr7EOXgFEU8dm+8/j6eAnibgnElfIGtBrafnE5akmyKxAEAVPHDUBtoxY7j1zqeL2ytoX3yMOFhobg+vXrCA5WQxA8qxNBFEW0turQ0FANf//g3k/ogoFuQ3qDER99mYvvzpVj+viBOJFf3hHm7bhXeGdHz5Z1e433yLOFhYWhtrYB5eUlML2Js3uTyxWIiYmGTtf/X2YMdBvR6gx4b8sZnL1Yg4zJcZg9IRZ7j18x+b3cK/wG7qdOXclkMoSGRkhdhqSCgizrdmOgW+HmLTsVcgEGg4ilaYmYNPoWANIuSXYV5u6RyksGg9HolBuTETkr/tdiofYFQe1hpDeIkMkEeN20BNlZliQ7M1P3SCYA2lYj3t9yFrpWxyyZJnIHbKFbyNSWnQaj2Knvty9PBvJ05hZWNWv02LC3AK98/D10rXrUNPRtlhCRJ2OgW6ivfb+9PRmIzC+sKq1uwv4frnZ8H2cJEfWMXS4W0BuM8JKbXiAkE9q6Y8h6pwurur3mqAcXELkiBno/GY0i/m/nObQa2h671u24CKzNzGOo2wBnwBD1DwO9H46cLcVP38rCsdwK+KrkmDQ6GiYyna1IG/H3Md8jyF+YRN0x0PvoaE4ZPvoyF9rWtoHQZq0Bh8+UwcTuBQDYirSFnnal2LDX/GP7iDwVA72PNuwt6BbeOr3RZAsd4FxzW2jSmJ+y2KQxQMspjUSdMND74FxxDZo0pp8RaBTBueZ20tsvxfe2nEFrl6mjRJ6Mgd6L0uomvLflrMkBUODG466kfvyVO+rpl6JSIeBsUQ3+sT0HBqMRR3PKsPL9w1j2xn6sfP8w+9jJI3Eeeg8amnX4yxen4SUXMD81Hpuziro9Kq59oQsD3PZSkqNQWHIdB05e6/S6XACWpI1AY3MrPt13Hm9+ehIXr9VzV0vyeAx0M/QGI97dfAbXG3X49eKxiI8JQoCvkqs+HeyRmYkYOiDY7H1v0rRi++Hibudxx0byRAx0Mz7ddx7nS+rw5PxkxMcEAeCqT6n0dN/npw4xGegAZxqR52EfugmHsktx4IerSLtzEO4YESl1OdQDQRAQGqA0eYwzjcjTsIWOztvgBvl5obFFjxGxIVg4OU7q0qgPMqYMxZpduZ0eJsKZRuSJPD7Q27fBbR/srGtqe2DxbQnh3IvbRbR3x2w6UIjaRh0EAbhvUhy7x8jjeHximdoGF2hbSMSpb64jJTkKf/pZKl5/YgJ8VQp8m12KZjNrB4jclccHurmBM26y5ZoiQ33x9H0jUV7TjL9vPwuDkQuPyHNYHeiNjY2YO3cuSkpKbFGPQx3NKTO7dB/gJluuakRsCB6eMRxni2qwcT///chzWBXop0+fxuLFi1FcXGyjchynve/c3OZa7Tj1zTVNHhOD6eMHYu/xKzh46mrvJxC5AasCfePGjXj55ZcREeF6T+g213feFae+ua4Hpg3FqHg11u8pQG5xjdTlENmdVYG+atUqjB8/3la1OMzRnLI+tbw59c21yWQCVsxLRlSoL97fehaV11ukLonIrgSxp02n+2jatGlYt24dBgwYYIua7OrgiSt494vTZrdelckEiEYRYSE+eDRtBKbcNtDBFZKtlVU34edvZSEixAd/eOZueCs9frYuuSnJPtnV1Y0wmujAvvkhwfawZmeO2TBXKmTddkq0Zy2m2Pv6nZ09rl8O4In0JPxl42n86ZPjeHxuEgShh9FwifEz4NnXD5i+BzKZALXav8fzPG7aYk9dLdz21n2NjFNjwaQ4fJdTjq+Pu96MLKK+8LhAD/T1Mvm6OlDFMHdzc1JiMXZYGD7fX4j8y7VSl0NkczYJ9P3797tE/3mzRm9ymiIHPz2DTBDwk7lJiAjxwd+2nkVNvUbqkohsyqNa6J/uK0CzRo/5qYP5hCEP5aNS4GcLR0KrN+K9LWf5CDtyKx4z3H+yoBKHz5Qh/a7BmJ8ah/mp3EnRU90S5oefzBmB97acxRcHCvHg9OFSl0RkEx7RQm/StGLdV/kYFOGP9ImDpS6HnMBtCRGYPn4gvj5RguN5FVKXQ2QTHhHon+8vRENzKx6bPQIKuUdcMvXB/VPjMSQ6EB9n5qKitlnqcois5vbpdq64BoeySzHrzkGIjQqQuhxyIgq5DE8tSIZMEPC3rTlo1Zten0DkKtw60LU6A9Zk5iEy1Bfz2NVCJoQF+WD5nCRcKm/AZ/sLpS6HyCpuHehbvi1CVZ0GS2clQOkll7occlJjhoVh5h0DceCHqziWWy51OUQWc9tAv3CtDnuPX8HUsTFIGBQidTnk5DImxyM+JhBrMvNQXsP+dHJNbhnoeoMRazLzEOyvwo+mcMEQ9U4hl+Gp+bdCLhPwt62cn06uyS0Dfe/3V3C1sgmPzEiAj8pjptqTlUIDvbF8bhIuVzTiX3xSFbkgtwv06joNth2+iLHDwjBmWJjU5ZCLGTM0DPeMG4A931/BmaJqqcsh6he3C/RP950HRGDxvcOkLoVc1P1T4xET7ofVO8+hrkkndTlEfeZWgX66sAo/FFQifeJghAX5SF0OuSillxwr5iWjRWfA6i/PwWj9M2CIHMJlO5iP5pRhc9YFVNdroQ5UYV7qEOw4XIxotS9m3jFI6vLIxQ0I98cD04bikz0F+Pp4CWbczidXkfNzyRb60ZwyrM3M63hYRXW9Fmsz81BVp8HDMxK4vJ9sYurYGIwZGoZNBwtxudyzn6BDrsElk29z1gXoukwrM4pt+5qPiOWcc7INQRDw2OxE+Pl44R/bzT+6kMhZuGSgm3uMXNeQJ7JWgK8Sj89NQll1Mz7bd17qcoh65JKB3v5wir6+TmSNpMGhmHXnIGSduoYfCiqlLofILJcM9IWT46FUdC7dSy7wMXJkN/dNisOgSH+syczjVEZyWi4Z6CnJUViSlghfVduGWwE+CiydPYKPkSO7UchleDw9GRqdAWsz8yByKiM5IZcMdAC4dUgoAAHJg0Pwl2fvZpiT3cWE+eFHU+JxqrAK32aXSl0OUTcuMw+967zzyFBfaHQGLLpnGARBkLo88hD3jh+A04VV+PTr80gcFIyIEF+pSyLq4BItdFPzzs8V1yIxNhgx4f4SV0eeRCYIWDZ7BGQyAf/3ZS6MRna9kPNwiUA3Ne8cAEqrmiSohjydOsgbD08fjsKSOmT++5LU5RB1cIlANzfvvLaRsw1IGhOSIzE+MQJbv73IVaTkNFwi0DnvnJzJ0Zwy/PpvR3A8rwJGUcTbm7L5gGlyCi4R6KbmnSsVMs47J4frOp4jikBtgxbvbj4jcWVELhLoKclReGj6cLRPZgkNUGJJWiKnKpLDmRvPOVNUg9xLtRJURHSDy0xbrGvSQRSB5x8ah+EDg6UuhzyUufEcAPjoy1z8dvkdfOwhScYlWuj1zTrs+u4SxgwNY5iTpMyN2wT6eqGmQYONBwodXBHRDS4R6DsPF0PbakDGFPaZk7TMjec8cM8wzLyjbQOvsxf5LFKShtMHesX1Fhw4eRV3j4pGTJif1OWQh2vfR6i9pS4T2rZt3px1AdFqX0SrffHxrjw0a/QSV0qeyOkDfXPWBchlAuanxkldChGAtlBvb6m3LxStrtdi/Z4C3JkUibpGHfdOJ0k4daBfKmvAsdwKyGUCfvXeYax8/zCO5pRJXRaRydkuOr0R356+hrQJg3DoTClOF1ZJVB15Kqcejv9oVy4AoEXXtmij/dmhADhlkSRlbrZLdb0WkSE+kMsEvL0pGyEBSvxoylB+XskhrGqh79ixA7Nnz8b06dOxfv16W9UEACgsqcOVisZur7f3VxJJydxsFz9vOT7ZUwDDf/piaht0WJuZx78sySEsDvTy8nK89dZb2LBhA7Zt24bPP/8chYW2m7K15dsis8d6mgtM5AjmZrsIgmCyK4aNEHIEiwP9yJEjmDBhAoKDg+Hr64uZM2di9+7dNikqt7ht1V37E4m64h4uJLWus13UgSosSUtEY4vp2S1shJAjWNyHXlFRgfDw8I6vIyIikJ2dbXEhNz/AQiEX4OetwI+nDcX6PQWdWjzcw4WcRUpyVLe+8fbPcFddW/NE9mBxoJt6pmJ/nhykVt94MMXBE1ewbnc+tK1tg596gwiIBoQG++KZH4/BusxcVNW2ICzEB4+mjcCU2wZaWrZLCA8PkLoESbny9S+dm4x3vzjd8VkGALmsrRsm/1o9UkfH9Ol9XPke2IKnXz9g2T2wONAjIyNx/Pjxjq8rKioQERHR5/Orqxs7nvayZmdOp/8AAEBvFLFmZw7efHoifr8ipdOxykr33X86PDzAra+vN65+/cmDgvHorIROj0tccHcc9v9Qgve+OI3oIG8E+il7fA9XvwfW8vTrB0zfA5lM6NQQNsXiQL/rrrvwzjvvoKamBj4+PtizZw9effVVi96rpylgRK7GVFfM4OhA/O/H32PdV/n46X238jm4ZBcWd+xFRkbiF7/4BR599FEsWLAAc+fOxahRoyx6Lz7AgtxdTJgf7ps0BD8UVOLf58qlLofclFULi9LT05Genm51EQsnx2PNrly0Gm70y3Pwk9zNzNsH4YeCSqzfW4DE2BAE+7PBQrblFEPvKclRWDp7BNSBKgi4MQWMq+vInchkApbPSYJOb8S63fkmJxYQWcNplv639ztyQITcWVSoLzImx+Ozfedx5GwZJo6MlrokciNO0UIn8iT3jh+A4QOCsOHr86ht4MA/2Q4DncjBZIKAx+aMgMFoxJrMPHa9kM0w0IkkEBnii/unDMWZomocyi6VuhxyEwx0IolMHReDxEHB+Gz/eVTXaaQuh9wAA51IIjJBwGOzR8BoBNZk5rLrhazGQCeSUHiwD348bShyimuRdfqa1OWQi2OgE0lsyphbkDQ4BJ/vL0TV9RapyyEXxkAnkpggCHgsbQQEtD12sX3TOqL+YqATOQF1kDcW3TMMeZevI/PIRanLIRfFQCdyEnePisatcaH4+MtzqKhtlrocckEMdCInIQgCls5KhEIm4KMvc2HkrBfqJwY6kRMJDfTG4wtGoqCkDvuOl0hdDrkYBjqRk5k2fiBGx6vxr6wLKKth1wv1HQOdyMkIgoBHZyXCSyFr63rhrBfqIwY6kRMKCVDhwenDUXi1Dnu+vyJ1OeQiGOhETmpCUiTGDQ/H5m+KcK2qSepyyAUw0ImclCAIeGRmAryVcqz+MhcGo1HqksjJMdCJnFiQnxIPzxiOi6X12P3vy1KXQw7Q2NJq8bkMdCInd8eISIxPjMC2QxdRUtkodTlkR1XXW/Dzvx5C3qUai85noBO5gIdnDIePSoHVX+ZCb2DXi7uqadDCKIpo0egtOp+BTuQCAn2VeGRGAi6VNSDzu0tSl0N2om01AAB8VAqLzmegE7mI8YkRuDMpEtsPF+NyeYPU5ZAdaHVtge7NQCdyfw9NHw4/Hy98xK4Xt9TeQvdWyi06n4FO5EL8fbywZFYCLlc0YvthbrPrbjQ6drkQeZSxw8KROjIaXx69hMKSOqnLIRvqaKEz0Ik8x+J7h0Ed6I0Pd+agRWvZjAhyPhqdHoIAKBWWRTMDncgF+agUeDw9CVV1Gny677zU5ZCNNLXo4eftBUEQLDqfgU7kooYNCMaclFgcyi7FifwKqcshG2jStMLPx8vi8xnoRC5s3sQhiI0KwNrd+bjeqJW6HLJSU0sr/L0t6z8HGOhELk0hl+GJ9CToWg34aFcuRD62zqU1tujZQifyZNFqP9w/dSjOFtVg/w9XpS6HrNCkaYWfNwOdyKNNGxeDW+NCsfFAIfdOd2GNLa3wl7KF/vbbb+Odd96x9m2IyAqCIGDZ7BFQecnx4Y5zXEXqgvQGIzQ6A/x8JOhDb2howAsvvICPPvrI4h9ORLYT7K/CklmJuFTegG2HbqwiPZpThpXvH8ayN/Zj5fuHcTSnTMIqyZym/+ywaE0L3eJfBfv27cPgwYPx2GOPWfzDici2bksIR+qoaOw6eglJsSG43qTD2sw86PRtLfbqei3WZuYBAFKSo6QslbpobNYBkCjQFyxYAADsbiFyMg/dOxyFJXX4YOc5yICOMG+n0xuxOesCA93JVNdrAAChgd4Wv0evgZ6ZmYnXX3+902txcXFYs2aNxT8UANRqf7PHwsMDrHpvV8fr9+zrB6y/B79Zegf++6/foFVvui+9pl7r1PfZmWuzF01BFQAgIS4MgGX3oNdAT0tLQ1paWr/fuDfV1Y0wGrvPmQ0PD0Blpefu9czr9+zrB2xzDwKUMvx46lCs31tg8nhooMpp77OnfgYuXb0OhVyGVo0OCPTudg9kMqHHhjDAaYtEbmvauBjERnVv5SkVMiycHC9BRdSTqjoN1EHekFm4jwtgRR86ETk3QRDwqwfG4Df/OIoWrR5GEVAHqrBwcjxSkqNwNKcMm7MuoLpe2+l1kkbl9RaEBaqseg+rA/2ZZ56x9i2IyE78fbzwTMYo/H7DD5iQHInH5yZBEAT886s8HDh5reP7OPtFWnqDESWVTbjnthir3oddLkRubvjAYMxPHYLvcspx5GwZjuaUdQrzdu2zX8jxSquboTcYERtp3WAwu1yIPMDclMHIu1SLf+7Jh08Pz6usrueOjVIoLqsHAJNjHv3BFjqRB5DJBDwxLxneSgXqmlrNfp/ayj5cskz+5evw9/FCZKivVe/DQCfyEMH+Kjw1P7nH7+HsF8c6mlOG/37vEI6cLYOu1YB/nyu36v0Y6EQeJGFQCO4YEWHymMrL8uly1H9Hc8qwNjMPNQ1tS/51eiPWZuZZtdcOA53Iw6yYl2yyr1bbKlodKNR3m7MumN2WwVIMdCIPIwgCVi4aC5mse4ucM10cx9wAtDUD0wx0Ig/k660wufUGwJkujmJuANqagWkGOpGH6ik42O1ifwsnx6PrKn9rt2VgoBN5qJ6Cg90u9jcowh+iCPiq2tYFqANVWJKWaNVKXS4sIvJQKclR+HDHOZPH2O1if9sPF0PlJcfrK1IQ4Ku0yXuyhU7kwezRj0u9u1haj+/zKjD99oE2C3OAgU7k0RZOjodS0TkGBADzUodIU5AH0BuM+HhXHoL8lZh1xyCbvje7XIg8WHt/bfs2ugE+CjS06JFbXIvUkdEQrNib25PdvDWxv48CoiiiSWNAaIASYcE+KKlsxDMZI+HrbdsIZqATebiU5KhOA3E7jxRj8zdFiAz1xXy21PutfQVo+6KhxhZ9x7GaBh1qGnQYNzwMY4eF2/xnM9CJqJM5KbEor2nGtkMXoQ70RuqoaKlLcgktWj2KrtVj/Z78bitAuyourbdLDQx0IupEEAQsSUtEbaMWa3fnISRAheQhoVKX5bQultZjz/dXcCK/EnpDz0Hern3/FlvjoCgRdaOQy/D0gpGIVvvivS1ncLnc8x7a3JvGllZ8uCMHr649jjMXqjFpdDR++cBohAT0PmvFXrOIGOhEZJKvtwI/v380fFQKvL0pGzX1GqlLchr5l2vx8kfHcCy3AnNSYvHm03fh4RkJuHWIGj+aMrTbzKGb2fMh3Qx0IjIrNPNrZJsAAAvKSURBVNAbv7h/NDQ6Pd7aeBqNLeYfjuEpDp66ij98ehJKLzleenQ8MibHw0d1o/c6JTkKS9ISO1rh/j4K+HnbbjVoT9iHTkQ9GhDhj2cWjsKfN57Gnz8/hZWLx3YKME8hiiK2fHsRO48UY1S8Gk/Ob3sClCldZw45ClvoRNSrxNgQPH3frbhS0Yi3N2VD22qQuiSHEkURn359HjuPFOPuUdF4JmOk2TCXkvNVREROp32hjMEoouDKdaxadxz/b+ntUMhlnY5X12uhDlRh4eR4SVqo9iCKIjYeKMTXJ0owffxALLpnqNMuuGILnYh61L5Q5uYNu0oqm/DaP09AbzB2O15dr3WbJx+Jooh/ZRXhq2NXcM+4AU4d5gADnYh6YepRaQBQXNaAv209i38dLLT5o9ScxfbDxdj13SVMGXMLHpw+zKnDHGCXCxH1oqetdE+er7LoPFew59hlbDt0ERNHRuHhmQlOH+YAA52IeqEOVJkMZ3WgCnPuGox1u/PNnucquo4BJA8JxTenS3FbQjiWpiVC5gJhDjDQiagXCyfHd9psCrixOCYlOQqXyhqQdepap3PsuXjGGqYGbwF0ur7qei2+OV0KuUzAifxKPP/3oy4zyMtAJ6Iedd1it+ssliWzEiGXCdj/w9WOc7wUztei7boLYvvgrZdCMDlGYPjPQ7Tbvw+A04c6A52IetXbQpn4mCB8c+oa9P8JwSaNwelC0NTgrk5vhE5v5oQu37c564LTXIs5nOVCRFbbnHWhI8zbOdtMF2sHaV1hkJeBTkRWMxd21fVaiKJo8pijyazsBXKFQV4GOhFZraew+9vWs9D0pV/Dzow9/F4J9lcixL9t21s/bzkU8s7p76yDvF2xD52IrGZqJoyXXMC4hHAcy63AlcomrJiXhMFRgZLVaG76pVwm4OXH7kCQ3419zF11KwMGOhFZraeZMJNH1+LDneewat0JLLh7CNLujIXM2v4PC5j6pSMAeGj68E5hDki3W6K1LA70EydO4LXXXoNer0dwcDBee+01xMTE2LI2InIh5kIwMTYEv11+B9buzse/sopw+kI1lsxMQEy4v8Pra9UbsWFvAXR6I5QKGR6eMRypo25xaB32ZHEf+sqVK7Fq1Sps27YN6enp+N3vfmfLuojIjfh5e+Gp+clYPmcESqua8MrH3+NfWRccug3vlYpGbP22qKOF7u+jgFzuXsOIFrXQdTod/uu//guJiYkAgISEBHzyySc2LYyI3IsgCJg4Mhoj49X44kAhvjx6CUfOliF94mCkjozu2IrX1lr1Ruw8UoydR4tx84Sbmgad082Vt5ZFga5UKjF//nwAgNFoxLvvvot7773XpoURkXsK9FVi+Zwk3D3qFmw6eAHrdudj93eXMTslFhOSIm32c4yiiON5Fdj8TREqalug9JJB12p6V0h3CXRB7GWSaGZmJl5//fVOr8XFxWHNmjXQ6XR4/vnnUVdXh7///e/w8vKya7FE5F5EUcTx3HJ8kpmHomt1EARAFIGQABUem5uEqeMH9fs9NTo9sn64ip2HilBcWo/YqAA8lp6M//3wO5gKOwHA9j/Nt/panEGvgW5OU1MTnnrqKQQHB+OPf/wjlEpl7yfdpLq6EUYTE0PDwwNQWdlgSUlugdfv2dcPeOY9OHK2FGt25XVbbToqXo3Jo29BXEwQAn29TG5haxRFVF1vQeHVOmRfqMaZomq0aA0YEO6HWXcOwoSkKMhkAla+f9jsrpFvPj3RbtdmCVOfAZlMgFrd80CyxbNcVq5cidjYWPz2t791iX2Cich5bfmmqFuYA0D2hWpkX6gGAHgr5QgP9oGPUg4vLzl0rQY0a/SoqtN0DK4G+ilxW0IEUkdGY9iAoE7Z1NOuke7CokA/d+4c9u3bh6FDh2LBggUAgIiICHz44Yc2LY6I3NvNC3jMef6hcbhU1oCK6y2out4CbasBlbUtqGnQQG8Q4a2U4+7R0Zg2dgAGRvqb3bu8t10j3YFFgZ6UlIT8fNOb2hMR9UXX7WxNUQeqMHxgMIYPDO52nt7Q1qLX6Az4d045EgeFIDYqoMef6aoLhvrKvSZhEpHLMPes0nbmukPMbYPrTDs7SoVL/4lIEj11s/TUHdLTzo6ejoFORJIwt1lWeIgPfr8ipd/nucL2tvbGLhciksTCyfFQKjpHkFIhw6NpIyw6z51mq1iKLXQikoS5WSdTbhvY4zx8T5itYikGOhFJxtJZJ+4+W8VS7HIhInITDHQiIjfBQCcichMMdCIiN8FAJyJyEwx0IiI3wUAnInITDHQiIjfBQCcichOSrRSVycw/5ainY56A1+/Z1w/wHnj69QPd70Ff7onFzxQlIiLnwi4XIiI3wUAnInITDHQiIjfBQCcichMMdCIiN8FAJyJyEwx0IiI3wUAnInITDHQiIjchSaDv2LEDs2fPxvTp07F+/fpux3Nzc5GRkYGZM2fixRdfhF6vl6BK++rtHnz99deYP38+5s2bh6effhp1dXUSVGk/vV1/u4MHD2LatGkOrMxxersHRUVFeOSRRzBv3jwsX77c4z4DOTk5yMjIwLx587BixQrU19dLUKV9NTY2Yu7cuSgpKel2zKIcFB2srKxMnDp1qlhbWys2NTWJ6enp4vnz5zt9z5w5c8STJ0+KoiiKv/nNb8T169c7uky76u0eNDQ0iBMnThTLyspEURTFv/zlL+Krr74qVbk215fPgCiKYmVlpThr1ixx6tSpElRpX73dA6PRKM6YMUPMysoSRVEU33zzTfEPf/iDVOXaXF8+A4sXLxYPHjwoiqIovv766+Kf//xnKUq1m1OnTolz584Vk5OTxStXrnQ7bkkOOryFfuTIEUyYMAHBwcHw9fXFzJkzsXv37o7jV69ehUajwZgxYwAACxcu7HTcHfR2D1pbW/HKK68gMjISAJCQkIDS0lKpyrW53q6/3UsvvYSf/exnElRof73dg5ycHPj6+mLSpEkAgCeffBIPPfSQVOXaXF8+A0ajEU1NTQCAlpYWeHt7S1Gq3WzcuBEvv/wyIiIiuh2zNAcdHugVFRUIDw/v+DoiIgLl5eVmj4eHh3c67g56uwchISG49957AQAajQYffPBBx9fuoLfrB4B169YhKSkJo0ePdnR5DtHbPbh8+TLCwsLw3HPPIT09HS+//DJ8fX2lKNUu+vIZeP755/Hiiy8iNTUVR44cwaJFixxdpl2tWrUK48ePN3nM0hx0eKCLJjZ3FAShz8fdQV+vsaGhAY8//jgSExNx3333OaI0h+jt+gsKCrBnzx48/fTTjizLoXq7B3q9HseOHcPDDz+MHTt2YODAgXjjjTccWaJd9Xb9Go0GL774ItauXYtDhw7hwQcfxHPPPefIEiVlaQ46PNAjIyNRVVXV8XVFRUWnPzm6Hq+srDT5J4kr6+0etL/24IMPIjExEatWrXJ0iXbV2/Xv3r0blZWVyMjIwBNPPNFxL9xJb/cgPDwcsbGxGDlyJABg7ty5yM7Odnid9tLb9RcUFEClUmHUqFEAgAceeADHjh1zeJ1SsTQHHR7od911F44ePYqamhq0tLRgz549Hf2EABATEwOVSoUTJ04AALZu3drpuDvo7R4YDAY8+eSTSEtLw4svvuh2f6H0dv3PPvssvvrqK2zbtg0ffPABIiIisGHDBgkrtr3e7sHYsWNRU1ODvLw8AMD+/fuRnJwsVbk219v1x8bGoqysDEVFRQCAffv2dfxy8wQW56Dtxmz7bvv27eKcOXPEGTNmiB988IEoiqL4k5/8RMzOzhZFURRzc3PFjIwMcdasWeIvf/lLUavVSlGmXfV0D/bs2SMmJCSI8+bN6/jfCy+8IHHFttXbZ6DdlStX3HKWiyj2fg9OnTolZmRkiLNnzxaXLVsmVlVVSVmuzfV2/QcPHhTT09PFuXPnikuWLBEvX74sZbl2M3Xq1I5ZLtbmIJ9YRETkJrhSlIjITTDQiYjcBAOdiMhNMNCJiNwEA52IyE0w0ImI3AQDnYjITTDQiYjcxP8Hh8c0SVnlMmoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "alpha = 1e-4\n", "\n", "# Fit polynomial-basis regression with L2 regularization\n", "poly = PolynomialFeatures(300, include_bias=False) \n", "holu = poly.fit_transform(x[:, None]) \n", "model = RidgeRegularization(alpha) \n", "model.fit(holu, y)\n", "\n", "pred = model.predict(poly.fit_transform(xfit[:, None]))\n", "\n", "# Visualize\n", "fig, ax = plt.subplots()\n", "ax.scatter(x[:, np.newaxis], y, label='Truth')\n", "ax.plot(xfit, pred, label='Fit')\n", "ax.set(xlim=(-0.02, 1.02), \n", " ylim=(-2.5, 2.5))\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The $\\alpha$ parameter is essentially a knob controlling the complexity of the resulting model.\n", "In the limit $\\alpha \\to 0$, we recover the standard linear regression result; in the limit $\\alpha \\to \\infty$, all model responses will be suppressed." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Lasso regression ($L_1$ Regularization)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import LinearRegression\n", "\n", "class LassoRegression(): \n", " # Class for least-squares linear regression:\n", " # with coordinate descent algorithm\n", " \n", " def __init__(self, alpha):\n", " self.coef_ = None\n", " self.alpha_ = alpha\n", " \n", " def soft_threshold(self, alpha, x):\n", " if x > alpha:\n", " return x - alpha\n", " elif x < -alpha:\n", " return x + alpha\n", " else:\n", " return 0\n", " \n", " def fit(self, X, y):\n", " \"\"\" Fit the data (X, y).\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " y: (num_sampes, ) np.array\n", " Output vector\n", " \n", " Note:\n", " -----\n", " Updates self.coef_\n", " \"\"\"\n", " # Create a (num_samples, num_features+1) np.array X_aug whose first column \n", " # is a column of all ones (so as to fit an intercept).\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features+1))\n", " X_aug[:, 1:] = X\n", " X = X_aug\n", " \n", " self.coef_ = np.zeros((n_features+1, ))\n", " \n", " self.coef_[0] = np.mean(y)\n", " \n", " convergence = False\n", " \n", " while not(convergence):\n", " newcoef = np.copy(self.coef_)\n", " # coordinate descent\n", " for j in range(1, n_features + 1):\n", " exclude_j = np.array(range(n_features + 1)) != j\n", " partial_residuals = y - X[:, exclude_j] @ newcoef[exclude_j] \n", " beta = self.soft_threshold(self.alpha_, X[:, j] @ partial_residuals)/(X[:, j]**2).sum()\n", " newcoef[j] = beta\n", " newcoef[0] = np.sum(y - (X[:, 1:] @ newcoef[1:])) / (X.shape[0])\n", " convergence = np.inner(self.coef_ - newcoef, self.coef_ - newcoef) < 10**(-5) \n", " self.coef_ = newcoef\n", "\n", " def predict(self, X):\n", " \"\"\" Make predictions for data X.\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " \n", " Returns:\n", " -----\n", " y_pred: (num_samples, ) np.array\n", " Predictions\n", " \"\"\"\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", "\n", " y_pred = X_aug @ self.coef_\n", " return(y_pred)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD7CAYAAAB68m/qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3hUdb4/8PeZmt4mDRASEiAhoYqFJgEunQQwrCvFBhZcXN27uz+urvhcvbuLete7er16vYqrAoqrqAiihCJNgSgSqSEFQksgdZKQTMpMZub8/sgmS5KZlGlnyvv1PD6PzHAmn3MI73z5VkEURRFEROTxZFIXQEREjsFAJyLyEgx0IiIvwUAnIvISDHQiIi/BQCci8hIMdCIiL6GQ6gvX1DTAbO46BV6jCYJWq5OgIvfA+/ft+wf4DHz9/gHLz0AmExAeHtjtdZIFutksWgz0tvd8Ge/ft+8f4DPw9fsHbHsG7HIhIvISDHQiIi8hWZcLEZElZrMZ1dUVMBiaAfhe14tcroBK1Q+2tLcZ6ETkVqqqqiAIAmJiboEg+FYngiiKaGkxoLS0DAEBofD3734QtDPfelpE5Paqq2sQHBzmc2EOAIIgQKVSIywsEjpdbZ+v970nRkRuzWQyQS737c4DlUoNk8nY5+sY6ETkdgRBkLoESdl6/779Y5CIqAelpdexdGkm4uMTOrw+bFgSpkyZismT0/Dkk6vwxhvvSFThPzHQiYh6EBkZhQ0bPrb6/okTOS6sxjoGOhGRDdatewFjx45DYWE+AODRRx/Eu+9ulLQmBjoRua0jZ0px+HSpUz578qh+mDSyX69+b1VVJR56aFn7r2fNmtP+///6r2vw+eefSh7mAAOdupGdW4ath4qgrdNDE6JGZloiJqTGSl0WkctZ6nJZt+4FaYrpBgOdLMrOLcPGrHwYjGYAgLZOj41Zrf+0ZKiTq0wa2ftWNDHQfZ61VvjWQ0XtYd7GYDRj66EiBjpRJ3K5HEajEQqFtJHKQPcxNwd4oJ8c+hYzjKbW/TJuboVr6/QWr7f2OpEvmzx5Ch56aBnee+9DqNVqyeoQRFG0efebN998E1lZWQCAtLQ0/Nu//Vuvr9VqdRb3+42KCkZlZb2tJXk8Z95/524UazQhrd+QlsJbE6LGK6snOaU+gH/+AJ9BRUUxoqMHSl2GpBQKGUpKLiE2Nq79NZlMgEYT1O11Nq8UPXr0KA4fPowvv/wS27ZtQ25uLvbu3Wvrx5ELWOpGsURbp0ezwQiFvONqNZVChsy0RGeVR0R2srnLJSoqCs888wxUKhUAIDExEdevX3dYYeR4fekuaWg2QS4AQf4K6JqMnOVC5AFsDvShQ4e2///ly5exc+dOfPLJJw4pipxDE6LuU6ibRKCxue8bBBGRNOweFD1//jxWrVqFp59+GvHx8b2+rru+oKioYHvL8mjOuv+H0lPx5menoG8x9fqatmEObZ0em3YVICTYD1PHdezfPJhTjE1ZeaiqaUJkuD8emDu8y+/pC1//8wd8+xlUVLT2Ifs6mUzW5+8DuwZFc3Jy8NRTT+HZZ5/F/Pnz+3Strw+KWpsu6Oz7z84tw7s7ztl8fedBUUsDrSqFDA/OTbape8ZX/vy74+vPgIOitg+K2txCLy0txRNPPIHXXnsNEyZMsPVjfFJ3i3YWTHVMy6zFaEZ1fTO0N5rR0GxEk771P7NZRIBajkZ971vpN+vcZcP56kTuw+ZAf++996DX6/Hyyy+3v7ZkyRIsXbrUIYV5s+5CcMHUoVausu6GTo/zJTdwpbweV8t1KKnUobZeb9NpjDLhn90slrRNaWzD+epE7sPmQH/uuefw3HPPObIWn9FdCK788x5U1jR1O6vEaDIj70oNThdpkXelBterGgAAMkFA/8hAJA8KR0y4PzShfogM9UOQvxL+agX8VHLI5TKYzSJ+zCvHjsOXUKMzICRAiVuTohASoEJFTROuaxtQXKFD5844hUzAgkmDO7xmbaC1c/ATeaq//vU/cebMKRiNLSgpKW7fF/2ee5Zg/vwFPV5/+PB3KCm5iiVL7sN777Xumf7ww6ucUitXikqgu9kmlTVNALrunWI2i8i9XI1j58px4nwVGvVGqBQyDB0YhkkjYpE0KBwDowOhVMh7VcPUMQMwdcwAq++3GE34JvsKvj1e3N49YzSL2PxtIfKu1ODOlBikDo5AZlqixT50zlcnV3LmRnK///3TAFoPunjyyVXd7otuSUFBnkPq6A0GugQshaAlBqMZnx24gMqaJnx3+jqq6/TwVytw69BIjEuORmp8BJROmg2gVMix6K4ELLqrtTXSYjShoLgWOQWVOJ5fgR/OlSPIX4mJI2Jx95QEfHu8mLsykiSk2kjuvffeQW7uWVRUlCEz85fYv38vVq58DLfeelt7+L/yyuvYvn0rACA2tnWTsby8XDz++EpUVlZg3rwMh7bWGegSaPsmu3lPlYZmy4OUtToDth2+hNT4cCyZPhRjhkZCIXf9lC6lQo4RgzUYMViD5TOH4eylahw9W4Z9OSUwm0WMTNTgwTnJSB0c4fPnQZJrSTkwbzDo8dFHnwEA9u/vulJ+8OAELFyYCQCYP38B3nvvHVRXV+Ptt99HY2MjfvGLdCxdeh8CAgIdUg8DXSITUmMxITW2vXVhjZ9KjhdW3oHoMH8XVtc9hVyGMUMiMWZIJGrq9Th08hoOnbyOV7ecwuB+wciYNBijEzUMdnIJKQfmU1JG9Pma8eMnQqVSQaVSITQ0DHV1dQx0b9Hd/ipKuYD7Zye5VZh3Fh6sxqK7EpA+MR5Hz5bh66OX8T+fn8agmCAsTkvEyASN1CWSl5NyYP7mnRVvbsAYjdZXWMvl/xznEgQBdiwF6oLLsSTWXSvioXnDPaYvWiGXYcro/njxsfFYOW84mvRGvLblFF799CRKKnVSl0deLDMtEapOY0lSDMyHhobh0qUiAMD33x9sf10ul8Nksm3dR18x0CUiiiJ+yC2DtV4JTYjaY8L8Zgq5DJNH9cO6R8djyfQhuHi9Ds+/fwybdhegoblF6vLIC01IjcWDc5PbW+SaELXNK5XtsXz5A/jyy8+xcuVy6PX/bKiNGXMr9u7dhc8/d/5eV3Yt/beHLy/9L69pxIe7C3Ducg2iwvxQU69vP2QCsG/pvLvRNbVg++FLOPDzNQQFKLFsxlDcnhxttX/dF/78e+Lrz4BL/yVY+k99J4oiDp68jk/3nYdcLmD5zGGYNnYAfswrb5/xEhXuj0WTB3tFmANAkL8Sy2cOw+SR/bBhVz7e3p6Lo2fLcP+sJGhC/aQuj8irsIXuIjd0enyQlY/TRVqkxodj5fwUhAd3HbTx1vsHAJPZjH051/Dldxchkwm4f9YwjO/0g8ub77+3fP0ZsIXOFrpbO12kxd++Pgd9iwnLZgzF9HG3QOaDU/rkMhlm3T4QY4ZG4m87zmH9jnM4VaTFfbOGIdBPKXV55EZEUfTpaa+2trMZ6E5kNot4a9sZ/FxYBQAIDVQi0F/pk2F+s+gwfzy9fCx2/nAVXx2+hLMXtZDLBNQ1tnhdlxP1XeusECMUCt/9IW8w6CGX9z2eGehOUt9owF/+fgLXKhvaX7vR0OKSJcmeQC6TIWNiPERRxLbvL7W/XlnTxGfk4yIiwlFbW4uwMA0Ewbcm4omiiJYWA+rrtQgKCuvz9Qx0Jyip1OH1z05DW9fc5T3uFd7R96e6nkPLZ+TbIiMjUVNTj/LyEsCmTaA9m1yuwIAB/WAw9P2HGQPdwc5e1OKtbWehVlnf9ZB7hf8T91OnzmQyGSIioqUuQ1KhobYNjDPQ7dB5y87kuHBkny3HgKhA/OYXo/DSRzncK7wH1pZtCwDyLldjeHyE64si8lC+1UHlQG2barWFkbZOjyNnynBLdCD+cN+tiAjxc5slye7M0jNSygWEBqnx6pZT+M5ClwwRWcYWuo2sbaqlazTAT9X6WDtvk8u9wrvq/IzaZrmMTozE29vPYkNWPj7Zdx7NBhOfH1EPGOg2stbHW11v6PDrtm1yybqbn9HNi2ruTInBucvVaDa0bmzkqoMLiDwVu1xsFBGssvi6TGjtjiH7bfv+YpcDq9tmwBBRVwx0GzTpjVbP7jSLwMasfIa6A3AGDFHfsMulD7Jzy/DFwQvt3Sqp8eHIu1JjtRXJbgH7BPkroGuyfFDAt8eLMeM2397vg6gzttB7KTu3DBt25nXoIz9fcqNLmLdhK9J+3e1n8fd951FtYeEWkS9joPfSFwcvoMXUMWAMRjNk3RxQQfaxdnA2AIgi8JePTzDUiW7CQO8Fo8ncZfZKG7MIzjV3kp5+KNY3GfDy5p9RdaPJRRURuTcGeg/MoogN/5gqZ0nbcVdSH3/ljbr7oRjoJ8fv7x2LhmYj/vLxCez56SrWvHUEK1/ejzVvHeGgNPkkDor24PMDRTh6tgzjkqJwpkjbYTFRW0ucc82dY0JqLC6U1OLAiY6rReUCsGxmEhL6h2DN0jF4efPP+GTfhfb3OV+dfBVb6N3I+vEKdh27ium3DsDqRSPYEpfA/bOT8WhGSofnvjI9pf25x8eGwE/ZdQop56uTL2IL3Yrs3DJ8dqAItydHY9mMYRAEgS1xifT03OsaWyy+zplG5GvYQregsLgWH+zMQ9LAMDySngKZtaks5BasDZ5yphH5GrbQ0XEb3LAg1T82gvLDE5kjoVTwZ567y0xLxMas/A7jG4IALLorQcKqiFzP59Oq8za4tToDmg0mpI3pjyB/3z3T0JNMSI3tML4R6KeAKAIFxbU2H7ZL5Il8voVubRvczw8WITRIzT5zD9G5n33b9xfx1ZHL0IT4YeHkwRJWRuQ6Ph/o1gbO2jbZAjj1zRMtnDwY2rpmbD98CRHBatw1ur/UJRE5nd1dLjqdDunp6SgpKXFEPS6VnVtmdek+wKlvnkwQBDw4JxmpgyOwcVcBzlzUSl0SkdPZFeinTp3C0qVLcfnyZQeV4zptfefWNtdqw6lvnkshl2H1ohG4JSoQb315FlfK+n7oLpEnsSvQt2zZgueffx7R0Z53Qre1vvPOOPXNs/mrFfjXX45GkL8C//PFadTq+AOavJddgb5u3TrcdtttjqrFZbJzy3rV8uYmW94hLEiNp34xGo3NRrzxxWkYWqzv4kjkyQTRAfO6pk+fjk2bNuGWW25xRE1OdTCnGG9+dgp6K3+pZTIBollEZLg/Hpg7HFPH8RAFb/HD2VK8uOEY7ho9AP/vvnEQBC4YI+8i2SwXrVYHs4UO7JsPCXaGDV/nWg1zlULWZX8WZ9ZiibPv39058/4TY4KQOSUBXxy6CE2wChmT3HM6I78HfPv+AcvPQCYToNEEdXudz01b7K6rhZtteb954+NwvaoRX35/Cf00gbgt2fPGf4is8amVomazCIXc8j+zNSFcROQLBEHAQ3OTkNg/BH/75hyulvt2S5C8i0MCff/+/R7Rf/5N9mUYTSLknSafc/DTtygVcvx68SgE+inx5tYz0DVZ3q2RyNP4TAu9sLgW2w5fwviUGKyYx33NfV1ooAqr7x6BWp0e63fkWhzPIfI0PtGH3tjcgne+ykVUmD/un50Ef7UCE0f0k7osklhi/1AsmzkMm3YVYPvhS7h7CndnJM/mEy30j/YWoq7BgFULUuGv9omfYdRLaaP7Y/Kofthx9DJOnK+Uuhwiu3h9oB/LK8cPueXImBSPwf1CpC6H3IwgCLh/1jDExQbjb1+fQ3l1o9QlEdnMqwO9pl6PD3cXIKF/COZPiJO6HHJTSoUcT9w9AnKZDG9uPYNmg1Hqkohs4rWBLooiPtiZhxajGY+kp0Au89pbJQeIDPXHqoWpuK5twKZdBTwYgzyS16bcgRPXcPZSNe6dPgSxEQFSl0MeIDU+AosmD8YP58rx/elSqcsh6jOvDPSy6kZs2X8BIxIiMHXsAKnLIQ8yf0I8UuLDsXlvIUoqdFKXQ9QnXhfoZlHE+zvzoFTIsGLucG7ARH0ikwl4NCMVAWoF/m/7Wfank0fxukDfn1OCCyU3sORfhiI8mHuZU9+FBqrwWEYKyrSN+HB3IfvTyWN4VaBX1Tbhi0MXMSIhAhNHcOUn2W54fAQWTB6M7NwyHD7D/nTyDF4T6KIoYsOufEAAHpydzK4WslvGxHgMjwvH5j2FuFbJ/nRyf14T6IdPl+Lc5RrcMzURmlA/qcshLyCTCXgsIwV+Kjn+b7v1ffSJ3IXHroPPzi3D1kNF0NbpER6kQkOzEcMGhnFWCzlUaJAaj2ak4q+fnsSWAxdw/6wkqUsissojW+jZuWXYmJXfflhFjc4Ag9GMMUM0kLGrhRwsdXAEZt8xEAd+voaT56ukLofIKo8M9K2HimAwmru8vi+nRIJqyBdkTknEwOggvL8zDzd0PR8wTiQFjwx0a8fIdXe8HJE9lAoZHluQCn2LCe99kwczpzKSG/LIQG87nKK3rxM5woDIQCyZPgRnL1Vj33H+a5Dcj0cGemZaYpezQXmMHLnC1LEDMGZIJD47eAHF3BqA3IxHBvqdKTHQhPqhbfyTx8iRqwiCgIfmJSPAT4n1X+XCwKmM5EY8ctrikdOlKK9uwop5ybhrVH+pyyEfExKgwiPzh+PVLafw2cEiLJ85TOqSiAB4UKDfPO9cEICYCH9MGslzQUkaIxI0mHnbQOw9XozRiRqMSNBIXRKRZ3S5dJ53LoqA9kYzfjxXLnFl5Mt+MTUB/SMD8f7OPDQ0t0hdDpFnBLqleedGk4ith4okqoio9ei6R9KHo76xBZv3FkpdDpFnBDrnnZO7io8NQfrEePyQW47j+RVSl0M+ziP60DUhaovhzXnnJIWbx3M0IWosuisBcbHB2LS7AEMHhiE0UCV1ieSjPKKFnjFpcJfXOO+cpNB5PEdbp8eHuwtwe3I0mg0mbNqVzwMxSDIeEejaG80AgJAAJQDOOyfpWBrPMRjNOPBzCRanJeDE+SocPVsmUXXk69y+y6Wqtgm7jl3FnSkxWLUgVepyyMd1N54z8/aBOHG+Ch9/W4jkQeHcl59czu1b6J8euABBAO6Zyu4Vkl53+wjJBAEPzx8Oswi8v5MbeJHruXWg51+pQU5BJeaPj0NECFs7JL3MtESoFB3/2tw8nhMV5o8l04cg70oNDvx8TYoSyYe5baCbRRGfHriAiBA1Zt8xSOpyiAAAE1Jj8eDc5PaWukxo7UPfeqgI2bmtfedTRvfHyAQNPjtwARU1jVKWSz7GbQP92LlyXCmrR+aUBKiUcqnLIWo3ITW2vaVu/kevirZOj41Z+cjOLWvdwGtuMuRyGT7Ymc+uF3IZtwz0FqMJXxy6iEExQQCANW8dwcqX92PNW0faW0FEUrI226Vt9XJ4sBpLpg9BQXEtDp5g1wu5hlvOctmXcw3aumbcmTIIm3YVtP/FaWsFAeCURZJUd7Ndbl54pJQL+GTfeYxK0CAyzN/FVZKvsauFvmPHDsybNw8zZ87E5s2bHVJQXYMBO45exsgEDX48V95tK4hIKtZmuwT6yTssPGoxiTCaRLz++WkuOCKnsznQy8vL8dprr+Hjjz/G9u3b8emnn+LChQt2F/TptwVoNhhxz7RE7uFCbsvabBdBECweYH6tqgHfnbruqvLIR9kc6EePHsX48eMRFhaGgIAAzJ49G7t27bKrmIqaRuw8cgmTR/bDLVFBPDuU3Fbn2S5tq5d1TUar13y6/wKq65pdVSL5IJv70CsqKhAVFdX+6+joaJw+fdrmQtr2yDCaRJy9qEV2bhky0xKxMSu/Q4uHe7iQu5iQGttlLKet77yzsCAVmvQmbNiVj9/eMxqCIHT5PUT2sjnQLfUH9uWbVKMJav//gznFHYK7RmfApl0F+PU9o/HkL8dgU1YeqmqaEBnujwfmDsfUcQNtLdsjREUFS12CpDz5/h9KT8Wbn52C/qazRtVKOR5eMAK6pha88+UZnL5cixk9rK3w5GfgCL5+/4Btz8DmQI+JicHx48fbf11RUYHo6OheX6/V6mD+xyTeDV/ndul31LeYsOHrXLyyehL+c9WEDu9VVtbbWrbbi4oK9ur764mn33/qoDA8MCepw/a6mWmJSB0UBrMo4sDAMLy77QwGRQYgPNhy16GnPwN7+fr9A5afgUwmdGgIW2JzH/rEiRORnZ2N6upqNDU1Yc+ePZgyZYpNn8XBT/ImE1Jj8crqSXj/mel4ZfWk9m4ZmSBgxbxkGE1mfLi7gLNeyOFsDvSYmBj89re/xQMPPIBFixYhPT0do0aNsumzOPhJviImPACZUxJw8kIVfuCZuORgdi0sysjIQEZGht1FcPCTfMmM2wbip4IKfLy3ECnxETzhiBzGLZb+3zwFTAAPsCDvJpMJWDlvOPQtZmzeUyB1OeRF3Gbpf9sUMA6IkC/opwnEwsnx+OLQRRzPr8Btyb2fUEBkjVu00Il80ew7BiEuJhgf7S2ErqlF6nLICzDQiSSikMuwYl4yGppa8Mm+81KXQ16AgU4koUExwZg3Pg5Hz5bhdFGV1OWQh2OgE0ksfWI8+kcGYuOuAjTpre8FQ9QTBjqRxJSK1q6XWp0enx2wf8dS8l0MdCI3kNg/FLNuH4iDJ6/jzAV2vZBtGOhEbmLRXQmIDvfH/2w5Ab3B1PMFRJ0w0InchFopx4q5ySjTNuLL7y9KXQ55IAY6kRtJGhSOuRPjsfenYhRduyF1OeRhGOhEbuah+SmICFHj/Z15aLFwnB2RNQx0IjcT4KfEg3OSUaptxI6jl6QuhzwIA53IDY1I0GDSyFjszL6KK2Xc24h6h4FO5KaW/MtQBAco8cHOPBhN7HqhnjHQidxUoJ8S981KwtUKHbJ+vCp1OeQBGOhEbmxcUhRuT47GjiOXcK2qQepyyM0x0Inc3PKZw+CnUuCDnXntB6sTWcJAJ3JzIYEqLJsxFBev1+Hb48VSl0NujIFO5AHuTInB6EQNtn53ERU1jVKXQ26KgU7kAQRBwANzkiGXC9iQlQ+zyK4X6oqBTuQhwoPVuHf6UORfrcV3J69LXQ65IQY6kQe5a1Q/DI8Lx5YDF1Bd1yx1OeRmGOhEHkQQBDw0NxlmUcTGXQUQ2fVCN2GgE3mYqDB/LE5LxJmLWhw9WyZ1OeRgJRU6m6enMtCJPNC/jLsFQ24JxSf7zuOGTi91OeQg16oa8O/vH0Pe5WqbrmegE3kgmSBgxdxk6FvM+GhPodTlkINU1jQBaD1n1hYMdCIP1U8TiIWT45FTWInj+RVSl0MOUPuPf21FhPjZdD0DnciDzblzEOJig/HRngLomlqkLofsVKvTQwAQFqy26XoGOpEHk8tkWDE3GQ3NRvz9W3a9eLpanQHBgSoo5OxyIfJJg2KCMX9CHLJzy3HqQpXU5ZAdanV6hAWqbL6egU7kBdInxmNAZCA27S5AY7NR6nLIRrU6PUKDbOtuARjoRF5BIZdhxbzhqNXp8dnBC1KXQzbS3miGJtS2AVGAgU7kNRL6h2D27YNw6OR1m+cxk3Sa9EY0NBsRKWWgv/7663jjjTfs/RgicoCFdw1GdLg/PsjKh95gkroc6oOqG61780gS6PX19Xj22Wfx/vvv2/zFicix1Eo5VsxNRtWNZnzxXREAIDu3DGveOoKVL+/HmreOIDuX2wW4o6ra1kVFUWH+Nn+GwtYL9+3bh/j4eKxYscLmL05Ejpc0KBzTbh2AfcdLEKBWYNePV2EwmgEA2jo9NmblAwAmpMZKWSZ1UillC33RokV47LHHIJfLbf7iROQcv0hLRESIGt9kX2kP8zYGoxlbDxVJVBlZU3WjCWqlHEH+Sps/o8cWelZWFl566aUOryUkJGDDhg02f1EA0GiCrL4XFRVs12d7Ot6/b98/4Jhn8NSSW/H8+myL71XX6d36Obtzbc5S32RErCYA0dEhAGx7Bj0G+ty5czF37ty+V9cDrdbyFpFRUcGorKx3+NfzFLx/375/wHHPYGCEP9RKGfQt5i7vRYSo3fY5++r3QEl5PaLC/FFZWW/xGchkQrcNYYDTFom82r3Th3Z5TaWQITMtUYJqyBqjyYyy6kYMiAq063NsHhQlIvc3dewAlGobsPd4CQBAE6JGZloiJqTGIju3DFsPFUFbp+/wOrleeXUjTGYR/SMlDvQnn3zS3o8gIidaOmMY6hpbcDy/AqvvHonB/ULw4e58HDjxz4OmOftFWteqGgAAA+wMdHa5EPmA+2YNQ2iQCuu/ysV3p653CPM2nP0inWuVDZAJAvppAuz6HAY6kQ8I9FPi4fkpqKhp6nabXW0dj7OTwrWqBsRE+EOpsG8aOAOdyEcMjwvH7DsHWZz10kYTYvtOf2QbURRxqbQOA6O7n8HSGwx0Ih9y910JkMsEq+9z9otrZeeW4ff/ewQ19XrkXqq2e1sGBjqRD1EqZLh7SoLF99RK60FPjpedW4aNWfmo1RkAAA3NRmzMyrcr1BnoRD5m3vg4TBjRdSaLvkW0O1Co97YeKnL4tgwMdCIf9Mj84VDKu7bIOdPFdawNQNszMM1AJ/JBgiCgxdR16w2AM11cxdoAtD0D0wx0Ih/VXXCw28X5MtMS0Xl82t5tGRjoRD6qu+Bgt4vz3TE8GgF+Sij+0fWlCVHjwbnJdq3U5V4uRD5qQmos3t1xzuJ77HZxvpyCSuiaWvDk4pEYOzTKIZ/JFjqRD3NGPy71rMVoxvbDlxAT7o/RQyId9rkMdCIflpmWCJWiYwzIBC4wcrZt319EqbYRy2YOg0xw3Px/drkQ+bC2/tq2bXT9VHI0G0wQRcszYKh3bt6aOMhfAVEU0dBsQkSwCokDwvBTfgXSxvTHyASNQ78uA53Ix01IjW0PdqPJjL9+chIbdxVgQGQQ4mJ97yg4e7WtAG1bNKRrMra/V11vQHV+BRL6h+C+WcMc/rXZ5UJE7RRyGX61aASC/JX43y/PQNfUInVJHsfSCtDOauubIZc5Pn4Z6ETUQUigCk/cPRK1Oj3e2X7W4tm/1JUoiii6fqNXM4Sq6w1OqYGBTkRdtHYJJCH3cg2++I5z0rsjiiJOXajCnzYex7pNOb26xlmziNiHTkQWTRndH5dL65D1w1XcEhXEo+ksKK7Q4e/fFiL/ai2iw9EWodMAAAvsSURBVPxx36xhkMsF/H3veavdLs48pJuBTkRWLZs5DKXaRnywMw+RoX4YekuY1CW5BaPJjJ0/XMGOI5fhr1Zg+cxhSBvTHwp5a6eHSiG3OMvF2YdxM9CJyCqFXIYnMkdi3abjeOOLM3juwdsQHeYvdVmSqq5rxlvbzuLi9TrcMTway2cOQ3CAqsPvuXnmkCuxD52IuhXkr8Rv7hkNURTx+men0NjsuzNf8q7U4D82/IRrVQ14fGEqHl84okuYS4ktdCLq0aXSOshlAkq1jfjtG4dx/5xkTB7Zr/39mxfSOLtbQSp7jxfjk33nERsRgF9njkQ/TaDUJXXBQCeibnVeKNNiErFhZx4EAZg0ol+X97V1emzMygcArwh1syhiy/4L2PNTMcYOjcQj6SnwV7tndLLLhYi6ZWmhjFkEPtpdAFEUnXKUmrtoMZrwzvZc7PmpGDPG3YIn7h7ptmEOsIVORD2wtlBG39K6Y6AzjlJzBw3NLXjjizMoLK7FL6cNwew7BkJw4EZazsBAJ6JuaULUFsNZrZThqyOXEaCWo1Fvsnidp+g8BjDrjkE4dPI6yqsb8diCFIxP8YyuIwY6EXUrMy2xQx850Lo45r5ZSTh5oQo5BZWQywSYbtoiwJmLZ+xhafAWQJcxgL9/e779mi8OFkEUPWM8gIFORN3qvMXuzbNY7kyJwfqvcnG8oBIqhaw9FJUK9+uasDZ4q1QI3W6m5UmDvAx0IuqRtYUyCrkMjy1IRfXmn3Hxel376w3NJrcLQWuDtwajlQs6/b6th4rc5l6s4SwXIrKLQi5DbX1zl9fdbaaLvYO0njDIy0AnIrtZ2w7WnUJQZmcvkCcM8jLQichu1sIuPMh9lsV3t637iIQIRAS31hroJ4dC3jH93XWQtzP2oROR3SzNhAFaV5VeLa/HoBjpj7KzNv0y0E+B3/1yTIfXPHUrAwY6EdnN0kyYyaP647tT1/Hihzl4YE4SJo7o18OnOJelHzoKuYBlM7ue7SnVbon2sjnQc3Jy8OKLL8JoNCIsLAwvvvgiBgwY4MjaiMiDWArBqWMH4P+2ncXfvs7D6SIt7p+dhEA/pST13To0CodOXkdhcS0AICxIhXumDfHI4LbG5j70NWvWYN26ddi+fTsyMjLw5z//2ZF1EZEXCA1UYc3SMbj7rsHIKajEv793DGcual1eR2FxLf5jw08oLK6Fv0oOAJDbO0rqhmwKdIPBgN/85jdITk4GACQlJaG0tNShhRGRd5DLZMiYNBhrHxgHP5Ucr205hf/degZVtU1O/9q6phZsyMrDy5t/hq6pBQqZgCZD6zYFbQuGsnPLnF6Hq9gU6CqVCgsXLgQAmM1mvPnmm5gxY4ZDCyMi7xIfG4IXVtyOzCkJOHNJi7V/+xGf7j+PGzrHT23UG0z4Jvsynnk7G4dPl2HOnYOgUggwdprq4m5z5e0liKLYzWQeICsrCy+99FKH1xISErBhwwYYDAY888wzuHHjBt5++20oldL0jRGRZ6msacKHWedw6OcSKOQyzLhjECLD/JGVfRlVNU2IDPfHA3OHY+q4gX363Oq6ZuzOvoydRy+jVqfHHSmxuH/ecMT3C8GC32+HpbATAHz114UOuCvp9Rjo1jQ0NOBXv/oVwsLC8F//9V9Qqfo231Sr1cFsYWJoVFQwKivrbSnJK/D+ffv+Ad96BuU1jfjm6BUczS3rkgdKuYCH5g3vcdCypl6PMxe1+LmwErmXqmE2ixiRoEHGpHgMGRDa/vvWvHXE4rRFTYgar6ye5JgbchBL3wMymQCNJqjb62ye5bJmzRrExcXhj3/8o9vvEUxE7ikmPAAr5w9H7iUtanQdV5u2mER8sDMP54trERKoQqCfEiIAo8mM2no9tHXNuFpe3x7SkaF+mHn7QKSN6Y+Y8IAuX8varpGesGCot2wK9HPnzmHfvn0YMmQIFi1aBACIjo7Gu+++69DiiMi7tS3g6RzmbYwmETmFlahv7HgwtVIhg9kswmQWEaCWY86dcZg/Ia7bxmV3u0Z6C5sCPSUlBQUFBY6uhYh8SOftbC1p6w4xmsxoNpggE4DjBZX4eG9h+/7rjXoTvj56GZpQvx7D2VMXDPUW93IhIklY2s72Zjd3hyjkMgT5KxHgp8SOI5e89gxTe3HpPxFJorudGLvrDvHWM0wdgYFORJKwtllWVLg//nPVhD5f5wnb2zobu1yISBKZaYlQKTpGkEohwwNzh9t0nTfNVrEVW+hEJAlrs06mjhvY7Tx8X5itYisGOhFJxtZZJ94+W8VW7HIhIvISDHQiIi/BQCci8hIMdCIiL8FAJyLyEgx0IiIvwUAnIvISDHQiIi/BQCci8hKSrRSVyaxvRN/de76A9+/b9w/wGfj6/QNdn0FvnonNZ4oSEZF7YZcLEZGXYKATEXkJBjoRkZdgoBMReQkGOhGRl2CgExF5CQY6EZGXYKATEXkJBjoRkZeQJNB37NiBefPmYebMmdi8eXOX9/Py8rB48WLMnj0ba9euhdFolKBK5+rpGXz77bdYuHAhFixYgNWrV+PGjRsSVOk8Pd1/m4MHD2L69OkurMx1enoGFy9exP33348FCxbg4Ycf9rnvgdzcXCxevBgLFizAqlWrUFdXJ0GVzqXT6ZCeno6SkpIu79mUg6KLlZWVidOmTRNramrEhoYGMSMjQzx//nyH3zN//nzxxIkToiiK4h/+8Adx8+bNri7TqXp6BvX19eKkSZPEsrIyURRF8b//+7/FP/3pT1KV63C9+R4QRVGsrKwU58yZI06bNk2CKp2rp2dgNpvFWbNmiYcOHRJFURRfeeUV8S9/+YtU5Tpcb74Hli5dKh48eFAURVF86aWXxFdffVWKUp3m5MmTYnp6upiamioWFxd3ed+WHHR5C/3o0aMYP348wsLCEBAQgNmzZ2PXrl3t71+7dg3Nzc0YM2YMACAzM7PD+96gp2fQ0tKCF154ATExMQCApKQklJaWSlWuw/V0/22ee+45/PrXv5agQufr6Rnk5uYiICAAU6ZMAQA8/vjjWL58uVTlOlxvvgfMZjMaGhoAAE1NTfDz85OiVKfZsmULnn/+eURHR3d5z9YcdHmgV1RUICoqqv3X0dHRKC8vt/p+VFRUh/e9QU/PIDw8HDNmzAAANDc3Y/369e2/9gY93T8AbNq0CSkpKRg9erSry3OJnp7B1atXERkZiaeffhoZGRl4/vnnERAQIEWpTtGb74FnnnkGa9euxeTJk3H06FEsWbLE1WU61bp163DbbbdZfM/WHHR5oIsWNncUBKHX73uD3t5jfX09Hn30USQnJ+Puu+92RWku0dP9FxYWYs+ePVi9erUry3Kpnp6B0WjEsWPHcN9992HHjh0YOHAgXn75ZVeW6FQ93X9zczPWrl2LjRs34vDhw1i2bBmefvppV5YoKVtz0OWBHhMTg6qqqvZfV1RUdPgnR+f3KysrLf6TxJP19AzaXlu2bBmSk5Oxbt06V5foVD3d/65du1BZWYnFixfjsccea38W3qSnZxAVFYW4uDiMHDkSAJCeno7Tp0+7vE5n6en+CwsLoVarMWrUKADAvffei2PHjrm8TqnYmoMuD/SJEyciOzsb1dXVaGpqwp49e9r7CQFgwIABUKvVyMnJAQBs27atw/veoKdnYDKZ8Pjjj2Pu3LlYu3at1/0Lpaf7f+qpp7B7925s374d69evR3R0ND7++GMJK3a8np7B2LFjUV1djfz8fADA/v37kZqaKlW5DtfT/cfFxaGsrAwXL14EAOzbt6/9h5svsDkHHTdm23tfffWVOH/+fHHWrFni+vXrRVEUxUceeUQ8ffq0KIqimJeXJy5evFicM2eO+Lvf/U7U6/VSlOlU3T2DPXv2iElJSeKCBQva/3v22WclrtixevoeaFNcXOyVs1xEsedncPLkSXHx4sXivHnzxJUrV4pVVVVSlutwPd3/wYMHxYyMDDE9PV188MEHxatXr0pZrtNMmzatfZaLvTnIE4uIiLwEV4oSEXkJBjoRkZdgoBMReQkGOhGRl2CgExF5CQY6EZGXYKATEXkJBjoRkZf4/0Ebb+sp5ZiCAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "alpha = 1e-4\n", "\n", "# Fit polynomial-basis regression with L1 regularization\n", "model = LassoRegression(alpha)\n", "poly = PolynomialFeatures(degree=300, include_bias=False) \n", "holu = poly.fit_transform(x[:, None]) \n", "\n", "model.fit(holu, y)\n", "\n", "pred = model.predict(poly.fit_transform(xfit[:, None]))\n", "\n", "# Visualize\n", "fig, ax = plt.subplots()\n", "ax.scatter(x[:, np.newaxis], y, label='Truth')\n", "ax.plot(xfit, pred, label='Fit')\n", "ax.set(xlim=(-0.02, 1.02),\n", " ylim=(-2.5, 2.5))\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Robust regression" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "class RobustRegression():\n", " # Class for least-squares linear regression:\n", "\n", " def __init__(self, potential, k):\n", " self.coef_ = None\n", " self.potential_ = potential\n", " self.k_ = k\n", " \n", " def mad(self, x):\n", " '''Computes the Median absolute deviation. \n", " It is the L1 equivalent of Variance.'''\n", " return(np.median(np.abs(x-np.median(x))))\n", " \n", " def weight_function(self, x, potential, k):\n", " if potential == \"huber\":\n", " if np.abs(x)<=k:\n", " return(1)\n", " else:\n", " return(k/np.abs(x))\n", " if potential == \"bisquare\":\n", " if np.abs(x)<=k:\n", " return((1-(x/k)**2)**2)\n", " else:\n", " return(0)\n", " \n", " def fit(self, X, y):\n", " \"\"\" Fit the data (X, y).\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " y: (num_sampes, ) np.array\n", " Output vector\n", " \n", " Note:\n", " -----\n", " Updates self.coef_\n", " \"\"\"\n", " # Create a (num_samples, num_features+1) np.array X_aug whose first column \n", " # is a column of all ones (so as to fit an intercept).\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", " \n", " self.coef_ = np.linalg.inv(X_aug.T @ X_aug ) @ X_aug.T @ y\n", " \n", " convergence = False\n", " \n", " while not(convergence):\n", " \n", " residuals = y - X_aug @ self.coef_\n", " norm_residuals = residuals/self.mad(residuals) # normalize with MAD\n", " weight_matrix = np.diag(np.array([self.weight_function(ri, self.potential_, self.k_) for ri in norm_residuals]))\n", " newcoef = np.linalg.inv(X_aug.T @ weight_matrix @ X_aug) @ X_aug.T @ weight_matrix @ y\n", " \n", " convergence = np.inner(self.coef_-newcoef, self.coef_-newcoef) < 10**(-5) \n", " self.coef_ = np.copy(newcoef)\n", " \n", " def predict(self, X):\n", " \"\"\" Make predictions for data X.\n", " \n", " Parameters:\n", " -----------\n", " X: (num_samples, num_features) np.array\n", " Design matrix\n", " \n", " Returns:\n", " -----\n", " y_pred: (num_samples, ) np.array\n", " Predictions\n", " \"\"\"\n", " n_samples, n_features = X.shape\n", " X_aug = np.ones((n_samples, n_features + 1))\n", " X_aug[:, 1:] = X\n", "\n", " y_pred = X_aug @ self.coef_\n", " return(y_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try it in the following data and compare with the performance of the different models:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD7CAYAAACPDORaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAdv0lEQVR4nO3de3RT14Eu8O9IsiU/ZVsWGAgxMUnghiQNKTMNCYU4pYB4GGKXhEBy86CQ3KySZrW3JYWsBasZ1zRk2rBg2oSZUKBDZ1KyGEzaOKGB4E4Cd6VhyAMHQoak5TEYy5YfkpFkPfb9w0ixLNmSpSPL2vp+a7FA58jn7I3g0zn77IcihBAgIiIpaVJdACIiSh6GPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQS06W6AP21t3fD70+s677JlI+2NodKJUoPmVZn1ld+mVbneOur0SgoLs4bcP+IC3m/XyQc8oHjZJpMqzPrK79Mq3My6svmGiIiiTHkiYgkNuKaayIRQqC93YqeHheA6LczLS0a+P3+5BdsWCjIzjaguNgMRVFSXRgiSjNpEfIORycURcHo0ddAUaLffOh0Gni9coS8EH50dLTC4ehEQUFRqotDRGkmLULe6XSgpGR0TAEvG0XRoKCgGDbbZYY8URo51tSMfY1n0dblhqlQj+pZEzF9StmwlyMtQt7v90GrTYuiJoVWq4Pf70t1MYjS3nAF77GmZuxqOI2eqy0KbV1u7Go4DQDDHvRpc2mcye3RmVx3IrUEgretyw3gq+A91tSs+rn2NZ4NBnxAj9ePfY1nQ7YJIdDZ3YMvL3XB1eNVvRxAmlzJjzSXLv0PHnigGhMmVIRsv/HGSZg5827MmDELa9Y8jq1bX05RCYmov8GCV+2r68AXSaTtew6egbXTidZOF1o7nejx9JbpO/fcgPl/P17VcgAM+biVlpqxc+fvBtx/4sTxYSwNEUUzWPAmwuP1obXTBWtHb2i3driQrdOEfaEEHG1qhtloQFlJLm6+rgSlRgNKi3Lwza+Ph73TmVBZImHIq6i2diOmTv06zpzpbXtbteph/PM/70pxqYgIADQKEGlAqSZKa6jP70d7lxvWThdaO5y9v18Nc2unE52OnpD3Z+k0yDPo4OnuQd/FVXVaBctn34i7p46LeB5Dtg72oVYqBmkX8u99cgnvfnxp0PcoChDPyrUzbh2Du24ZE9N7W1uteOSR5cHXc+bMC/756ad/hNdee5UBTzSCDDRjgF8AHQ53MLRbOwJNKS5YO5ywdbnh7xMoGkVBSaEepUYDbqkwwXz1StxszEFpkQGFednQKAp716S7SM01tbUbU1MYIhpUt8sDY14WOrs9Eff/YNt7Ia+NedkoLTLg+nFGlE4xoNSYEwzz4gI9dNrofVamTylLSaj3l3Yhf9ct0a+2ZRoMRTTSjJQr1L7cnt528cBVuDVwNX61ecXpjr3nSpZWwX33XJ/yOqkl7UI+XWi1Wni9Xuh0/Cum9BEtwFPV/9vr88Nmdw8Y4l3doe3i2ToNTEYDzEU5uP4aI0qNOXjj//0VDmf0sPf4RFJ63KQKEyhJZsyYiUceWY5XXvkt9Hp9qotDFFUsAZ6sboh+IdDp6Al5oOlw+XDhchesHS7Y7K6Q52xaTaBdPAe3XW9C6dX28ECzSmFedtj4kt+/898xlyfRHjcjCUM+DmPGjMVrr70etn39+o3BP9fWbh7GEhElLpYAj7cbohAC3S5vyBV4a2fgQWfvQ06vL7yJVadVMKGsEHfeXIbSIkPw4WZxgR5azdDGcpoK9TGHt6lQngszhjylvZHYRpyOYgnwgYLSVKiHu8cXDO2vwtsZ7D/u6gmdmiPPoENpUQ6uMefhthtKYTYaYO104dAH5+Hx9V62e30C5y7bUXn7uIQ/0+pZE0PuVIDeLxHhF/D1uUvI1mlQPWtiQucaSRjylNZG0hwh6W6wAAd628W/NW089h05C2+f/ogKgG6XF//nF40hP5edpem98jYaMOnaomDvlNKrbeU5+vD4+dGv3gsGfIBao1L7Njn1vSCItE2mfzsMeUprwzlUXXb3zqzArobTISGrUYAsnRb/91fvod3uDht/olGAMaY8TBxnhPlqm3igWaUgN2vI8y4la1RqwEDdGmX+t8KQp7SW7FCQiRACDqcH75y4iLc/OA+H0wt9lgalRTmwX/GE9VABgBy9DgW5WSg1FgZDPPB7cYEemmjDRYco2t0EDR1DntIaQyGU0+0N6Vr41ejN3tfufu3ibo8fF63d6B/VWVoFD82dhBm3jh2+wiNyu7lsbeTDjSFPaS3TQsHj9aOtKzTE+4a5wxk6olOfre1tCzfmYHJ5MY5+cglX3OFrE/Qf8e/xCdS/++Wwh3zfdnNblxslEraRDzeGPKW1gR6mpWso+P0C7XZ3SK8Ua4cLnVd6cKm1Gx12d0gg67QKTIW9DzQnlBWEPNgsNRqQnxPaLv72BxdiLkuqmrwC7eZmcwGs1mRM2ZVZkhbyn376Ke677z6cPHkyWadIiX/8x5/jk08+gtfrwYUL54Nzyi9dugwLFlRF/fl33/0zLlw4h2XLHsQrr/TON79y5eNJLbPsRsocIX0N1K1TCAH7FU/ELoatHS60dbng69dzpbhQjzGl+bipvDgsxIsK9NAM4eFmpvYVz2RJCXmn04mf/vSn8HgiTwY0HJLVd/qHP1wLoHfhkDVrHh90TvlIPvvsVMJloJGt8cOL2POnM/Be7aXS1uXGv/zhU7x25Cy6XZ7gIhEBvQ82czBhTAGmTR4VMujHVGiATqtR7ao2U/uKZ7KkhPymTZvwyCOP4MSJE8k4fFRHT14a9r7Tr7zyMpqaTqKlpRnV1ffh8OE/4bHHVuP226cFvxA2b96C+vp9AICyst5J1k6dasITTzwGq7UF8+cv4lV9Goi0SETfK/NuV/j8KEIADqcHlVPH9c6pEhyGb4Ahe/haTTO1r3gmU/1f16FDh+ByuTBv3rzob06Sve+kpu90T48b//qvewEAhw//KWz/dddVYPHiagDAggVVeOWVl2Gz2fDSSztw5coVfOc7C/HAAw8iNzcvaWWUlZp3bkNdJEKn1Vxd3ceAirGFeOfExYjH9Xj9WPatG+Iqk5oysa94Jos75BsaGlBXVxeyraKiAg6HAzt37oy7QCZTfti2lhYNdLrY56lo63RF3t7lHtJxBqO9Op904HgajYKbb74l+FpRFGi1veXu+95Av+LAn++88y7k5hqQm2tAUVERurvtKCwsCDufRtN7yz6YaPtlE6jvr1/7EG8c+1twe1uXG7vf/AyFBQbc/fXwNTOFEOiwu3HZdgXNtiu4bOvG5bYruGzr/dXa4QxpF9coQGlRDkaX5OHvxhox2pSL0SVf/SouMIT0Fz/5Vxus7eHLuJmLcxL6jDLt8wUyr87JqG/cIW+xWGCxWEK27d27Fy+//DJWrFgR3LZ48WLs2bMH+fnh4R1JW5sD/n5LuPj9/iHND28yGiIGvalQr9o8876rkykFjuf3C2RlZYcc3+frLbfb3RN8b6BugT8rSt+57xV4vZHr6vf7B2yTPdbUjP3vfglruzNjbrUDbdTHmppDAj7A7fHhX+o/gcvZ81WzytUpats6XWF3eoV52TAbDZhQVoBpk8zBB5ulRTkoGWSRCH+PF21tjpBtS2ZcF7Fb55IZ18Xdrp6JPU0yrc7x1lejUSJeHAeo2lyzdOlSLF26NPh60qRJqK+vV/MUsZWjciJ2/OFUSvtOG41F+PLLs7j99mn4z/88Etyu1WrR0xM+sjBemTp3i9vjw/+0duPf3z4z4Hs6uz34p//o7d2Vo9fBbDRgjCmvd8m2PiFeajRAn6VVrWyydeuk9CZlP/k7bx4D39WJ/1P1n2zFiv+N2tqN+OMfD+Cb37w7uP22225Hbe1GlJSUqHKedJq7ZSjt5oMtEtHa6UJnhCH4/eXn6PDD+6eitMiAPEOW2tUZ1Ejs1kmZSREiniWvkydSc01z899QVlYe8zFkXP5voL+DxzYdHvBndjxzTzKLNCT97ziA3qHz8+4oR1lxbli/8XZ75MWTA1fg5eOMyNVp8O+HPkfXlchddVctukmaoM20pgsg8+qcFs01NPxSNXdLtKvywCIRgV4pew5+FnbH4fEJvP7eX4OvjfnZMBtzcMN4Y8jCyWajAcWFerx/qgX7Gs/i1N/acfp8B5bMuA73f+uGsC8PAKicOlaagCdKBEM+zaVi7pZIzwF2/PEUjjU1I0urCU6I5YwwR0oktau+AVOhAdmDtIv3P6e13YldDafxsGUyHrZMZvs30QDSJuSFEEOem1oWg7WoBcIsGb1rvD4/bF2ukAmwrB1O/NcZa3A0Z4DPL3DyCxvGluah1GjAjdcUfbXmZpEBW177CO328HZ0U6EeY0zRxwUM9uxh85N3MdSJBpAWIa/RaOHzeaHTDe/Ds5HC5/NCo9EO2EQyfUoZqu6+Ycjtef6r/cX7P9gMDP7pv0hEYPHk/gHf1z989xsRt3/n7usTuuPgvPFE8UmLkM/JyYfd3oGiIhMURZ3BTIlwOD1ot7vh8/mh1WpQXKBHfk5yvoCE8MNub8flDn94E8kfPsW/vX0GDqcX5uIcLJlxXVi7uMPpiRziHU60dblCAlsBUFSg712ubXzxVyv9XB3NWXJ10M+PfvXekJ8DJNqtkPPGE8UnLXrXCCHQ3m5FT48L4TNfh9NoNPD7k9O7xtXjQ7fTE1YKjaIg16CDIVud/tauHh+cbi+8PoHmdg/+8H4b7M7B27i1GgX/q7wYOq1mwEUi8nOygv3Dzf1+NxUakBXDiOBIPWWydRo8bJmctGaTVJxzJMm0niZA5tU5o3vXKIqCkpJRMb8/mf84BrqKBYYeOgM1v0QKtFj4/AInv7RhnDkPZmMOJl9bHBLipUZDxMWThyoVg336nzPSnQsRhUuLkFeLGpNYDdYGPJRBSAP1UDn+WQuavrQNOeD7em5l5HZxNaVisE/fc2baVR5RvDIm5NUa/h9t0YX++wZaJOLYyUvwROih8l9nWmMuy0DlIyIKyJiQV2v4f6R+6X3l6rXY86czfRZQdsHtCW0XL8jNCgv4vgb6IsnP0UGfpUVblxt5Bi3cHn/Ig9NULvSQrEVaiCgxGRPyiXTBCywS0drpgsvtxeTyYjR9aQuZjjbgituH9z65hFJjDkYV5+CmCSUhK/0EFokYrIfKQAOcHph9Y0hw9g3WVLZRZ+okaUTpIGNCfrAueH6/gM3uClvhJ9DVsCNskQgFpUU5yNIqsHY44fb4kZ+jw+xp43HP7dcgz6CLOnBrsJGqsT7YHClt1Ok0SRpRpsmIkBdCYN43yvHq4c9D+4UrvWH0+AtHQhdPVoCSAj1KjTmYcl1Jn6vwHJiLcmDMzx7S4smRRAvydJrFkAOViEYuqUL+WFMzXjvy32i390CfpcGo4lx4ff6Ii0QoCmC+unhy37nFzUYDSq4unpxs6RTkg+FAJaKRS5qQ798u7Pb4caHFgWvLCnD31HGhA4CMOdCrNGiJUjNJGhHFRpqQj9QuLAA4rvSMiMWTZcaVkIhGLmlCnu3CqSVL0xORbFI/25dKBmr/ZbswEWUyaUK+etZEZPebXIvtwkSU6aRprunbLmzrcqOE7cJERPKEPPBVuzAnryIi6iVNcw0REYVjyBMRSYwhT0QkMYY8EZHEGPJERBJjyBMRSYwhT0QkMYY8EZHEVA/5lpYWrF69GkuWLMGyZctw4cIFtU9BREQxUj3kf/zjH6OyshL79+/H4sWL8cILL6h9CiIiipGq0xrYbDacPn0av/nNbwAANTU1mD59upqnICKiIVD1Sv78+fMYO3Ysfvazn6GqqgpPPfUUsrKy1DwFERENgSKEENHfFq6hoQF1dXUh28rLy/GXv/wFv/71r1FZWYm9e/fiwIED+O1vf6tKYYmIaGjiDvlIzp07h3vvvRfHjx8HADidTtxxxx346KOPYj5GW5sDfn9iRcrEWSgzrc6sr/wyrc7x1lejUWAy5Q+8P5FC9Xfttddi9OjRaGxsBAC88847mDJlipqnICKiIVB9Pvlt27Zhw4YN2Lx5M/Lz87Fp0ya1T0FERDFSPeQrKirYBk9ENEJwxCsRkcQY8kREEmPIExFJjCFPRCQxhjwRkcQY8kREEmPIExFJjCFPRCQxhjwRkcQY8kREEmPIExFJTPW5a9LJsaZm7Gs8i7YuN0yFelTPmojpU8pSXSwiItVkbMgfa2rGrobT6PH6AQBtXW7sajgNAAx6IpJGxob8vsazwYAP6PH6sa/xLEOekoZ3jzTcMjbk27rcQ9pOlCjePVIqZOyDV1OhfkjbiRI12N0jUbJkbMhXz5qIbF1o9bN1GlTPmpiiEpHsePdIqZCxzTWB22O2j9JwMRXqIwY67x4pmTI25IHeoGeo03CpnjUxpE0e4N0jJV9GhzzRcOLdI6UCQ55oGPHukYZbxj54JSLKBAx5IiKJMeSJiCTGkCcikhhDnohIYgx5IiKJMeSJiCTGkCcikhhDnohIYqqH/IULF7BixQosXrwYDz30EC5evKj2KYiIKEaqh/yWLVuwYMEC1NfXY86cOfjlL3+p9imIiChGqoe83++Hw+EAADidThgMBrVPQUREMVKEEELNA547dw7Lli2DVquFx+PBq6++ivLycjVPQUREMYo75BsaGlBXVxeyraKiAm63GytXrsTs2bPx1ltvYdu2bThw4AAURYnpuG1tDvj9iX3vmM0FsFrtCR0j3WRanVlf+WVaneOtr0ajwGTKH3B/3FMNWywWWCyWkG02mw0WiwWzZ88GAMydOxcbNmxAe3s7SkpK4j0VERHFSdU2+eLiYuj1enzwwQcAgOPHjyMvL48BT0SUIqouGqIoCrZt24bnnnsOLpcLeXl52Lp1q5qnICKiIVB9Zahbb70Ve/fuVfuwREQUB454JSKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSGEOeiEhiDHkiIokx5ImIJMaQJyKSWMIhv2XLFmzdujX4uqurC6tXr4bFYsGKFStgtVoTPQUREcUp7pC32+1Yt24dduzYEbL9xRdfxLRp09DQ0IClS5eitrY24UISEVF84g75Q4cOYcKECXj00UdDth85cgSLFi0CACxcuBB//vOf4fF4EislERHFJe6QX7JkCVavXg2tVhuyvaWlBWazGQCg0+mQn58Pm82WWCmJiCguumhvaGhoQF1dXci2iooK7Ny5M+aTaDSxf5eYTPkxv3cwZnOBKsdJJ5lWZ9ZXfplW52TUN2rIWywWWCyWmA84atQotLa2oqysDF6vFw6HA0VFRTH/fFubA36/iPn9kZjNBbBa7QkdI91kWp1ZX/llWp3jra9Gowx6cax6F8pZs2Zh//79AIA33ngD06ZNQ1ZWltqnISKiGES9kh+q73//+3jmmWewYMECFBQU4IUXXlD7FEREFKOEQ37NmjUhr4uKivDSSy8lelgiIlIBR7wSEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJDGGPBGRxBjyREQSY8gTEUmMIU9EJLGEQ37Lli3YunVr8PXZs2exfPlyLF68GPfffz9OnTqV6CmIiChOcYe83W7HunXrsGPHjpDtzz77LFatWoX6+no8/fTTWLt2bcKFJCKi+MQd8ocOHcKECRPw6KOPhmxfunQpZs6cCQCYNGkSLl26lFgJiYgobooQQiRygEBTzZo1a8L2bdy4EW63G3V1dYmcgoiI4qSL9oaGhoawkK6oqMDOnTsH/BkhBJ5//nl89NFH2L1795AK1NbmgN+f0PcOzOYCWK32hI6RbjKtzqyv/DKtzvHWV6NRYDLlD7g/ashbLBZYLJaYT+j1erF27VpcvnwZu3fvRkFBQcw/S0RE6ooa8kP185//HA6HAzt27EB2drbahycioiFQNeRtNhv27NmDa665BkuXLg1ur6+vV/M0REQUo4RDvu8D15KSEnz66aeJHpKIiFTCEa9ERBJjyBMRSYwhT0QkMYY8EZHEGPJERBJjyBMRSYwhT0QkMYY8EZHEGPJERBJjyBMRSYwhT0QkMYY8EZHEGPJERBJjyBMRSYwhT0QkMdVXhqLYHGtqxr7Gs2jrcsNUqEf1rImYPqUs1cUiIskw5FPgWFMzdjWcRo/XDwBo63JjV8NpAGDQE5Gq2FyTAvsazwYDPqDH68e+xrMpKhERyYohnwJtXe4hbSciihdDPgVMhfohbSciihdDPgWqZ01Eti70rz5bp0H1rIkpKhERyYoPXlMg8HCVvWuIKNkY8ikyfUoZQ52Iko7NNUREEmPIExFJjCFPRCQxhjwRkcRG3INXjUYZUcdJJ5lWZ9ZXfplW53jqG+1nFCGEiLdAREQ0srG5hohIYgx5IiKJMeSJiCTGkCcikhhDnohIYgx5IiKJMeSJiCTGkCcikhhDnohIYmkd8q+//jrmz5+Pb3/729izZ0/Y/lOnTqGmpgZz587F+vXr4fV6U1BK9USr79tvv43FixejqqoKTz75JDo7O1NQSnVFq3PAkSNHcM899wxjyZIjWn2/+OILPPTQQ6iqqsLKlSvT/jOOVt+mpibU1NSgqqoKjz/+OLq6ulJQSnU5HA4sXLgQFy5cCNuXlMwSaaq5uVlUVlaK9vZ20d3dLRYtWiQ+//zzkPcsWLBAnDhxQgghxE9+8hOxZ8+eVBRVFdHqa7fbxV133SWam5uFEEK8+OKL4rnnnktVcVURy2cshBBWq1XMmzdPVFZWpqCU6olWX7/fL+bMmSMaGxuFEEJs3rxZPP/886kqbsJi+XwfeOABceTIESGEEHV1deIXv/hFKoqqmg8//FAsXLhQTJkyRZw/fz5sfzIyK22v5I8ePYo77rgDRUVFyM3Nxdy5c/Hmm28G91+8eBEulwu33XYbAKC6ujpkf7qJVl+Px4ONGzdi9OjRAIBJkybh0qVLqSquKqLVOeDZZ5/F9773vRSUUF3R6tvU1ITc3FzMnDkTAPDEE09gxYoVqSpuwmL5fP1+P7q7uwEATqcTBoMhFUVVze9//3ts2LABo0aNCtuXrMxK25BvaWmB2WwOvh41ahQuX7484H6z2RyyP91Eq29xcTFmz54NAHC5XNi+fXvwdbqKVmcA2L17N2666SZ87WtfG+7iqS5afc+dO4fS0lKsXbsWixYtwoYNG5Cbm5uKoqoils/3mWeewfr16zFjxgwcPXoUy5YtG+5iqqq2thbTpk2LuC9ZmZW2IS8iTJ6pKErM+9NNrPWx2+1YtWoVJk+ejHvvvXc4ipY00ep85swZHDx4EE8++eRwFitpotXX6/Xi/fffx4MPPojXX38d48ePx6ZNm4aziKqKVl+Xy4X169dj165dePfdd7F8+XKsXbt2OIs4rJKVWWkb8qNHj0Zra2vwdUtLS8gtUP/9Vqs14i1SuohW38C25cuXY/LkyaitrR3uIqouWp3ffPNNWK1W1NTUYPXq1cH6p6to9TWbzSgvL8ctt9wCAFi4cCE+/vjjYS+nWqLV98yZM9Dr9bj11lsBAPfffz/ef//9YS/ncElWZqVtyN955504duwYbDYbnE4nDh48GGyrBIBx48ZBr9fj+PHjAID9+/eH7E830err8/nwxBNPwGKxYP369Wl91xIQrc5PPfUU3nrrLdTX12P79u0YNWoUfve736WwxImJVt+pU6fCZrPh9OnTAIDDhw9jypQpqSpuwqLVt7y8HM3Nzfjiiy8AAIcOHQp+wckoaZmV8KPbFDpw4IBYsGCBmDNnjti+fbsQQojvfve74uOPPxZCCHHq1ClRU1Mj5s2bJ37wgx8It9udyuImbLD6Hjx4UEyaNElUVVUFf61bty7FJU5ctM844Pz582nfu0aI6PX98MMPRU1NjZg/f7547LHHRGtrayqLm7Bo9T1y5IhYtGiRWLhwoXj44YfFuXPnUllc1VRWVgZ71yQ7s7gyFBGRxNK2uYaIiKJjyBMRSYwhT0QkMYY8EZHEGPJERBJjyBMRSYwhT0QkMYY8EZHE/j8DG7o+F7OF4QAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Simulate data\n", "np.random.seed(300)\n", "rng = np.random.RandomState(1)\n", "x = rng.rand(30)\n", "noise = 0.1 * np.random.standard_cauchy(30)\n", "y = 2 * x - 5 + noise\n", "\n", "# Fit model\n", "model = RobustRegression(\"huber\", 1.0)\n", "model.fit(x[:, np.newaxis], y)\n", "\n", "xfit = np.linspace(0, 1.0, 1000)\n", "yfit = model.predict(xfit[:, np.newaxis])\n", "\n", "# Visualize\n", "fig, ax = plt.subplots()\n", "ax.scatter(x, y, label='Truth')\n", "ax.plot(xfit, yfit, label='Fit')\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Application: Predicting Bicycle Traffic" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As an example, let's take a look at whether we can predict the number of bicycle trips across Seattle's Fremont Bridge based on weather, season, and other factors.\n", "\n", "In this section, we joinned the bike data with another dataset, and try to determine the extent to which weather and seasonal factors—temperature, precipitation, and daylight hours—affect the volume of bicycle traffic through this corridor.\n", "\n", "We will perform a simple linear regression to relate weather and other information to bicycle counts, in order to estimate how a change in any one of these parameters affects the number of riders on a given day.\n", "\n", "Let's start by loading the dataset:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "daily = pd.read_csv('data.csv', index_col='Date', parse_dates=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With this in place, we can choose the columns to use, and fit a linear regression model to our data:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "# Drop any rows with null values\n", "daily.dropna(axis=0, how='any', inplace=True)\n", "\n", "column_names = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun', 'holiday',\n", " 'daylight_hrs', 'PRCP', 'dry day', 'Temp (C)']\n", "# PRCP precipitations\n", "X = daily[column_names].values.astype(float) # converts to numpy array\n", "y = daily['Total'].values.astype(float)\n", "\n", "from sklearn import preprocessing\n", "\n", "\n", "# xx = X.values.astype(float) #returns a numpy array\n", "scaler = preprocessing.MinMaxScaler()\n", "X_scaled = scaler.fit_transform(X)\n", "\n", "model = RidgeRegularization(0.1)\n", "model1 = MultivariateLinearRegression()\n", "model.fit(X, y)\n", "y_pred = model.predict(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we can compare the total and predicted bicycle traffic visually:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEXCAYAAABRWhj0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOydeXwb5Z3/PzM6bfm+cp8cCRCaBAIlHEnTkjgkuKEhLSVAytKFkrak8GuhQKCBvshSKCVdyqbHtqVbtt2FUggQgjmXEEggF4Tcl6/4lGzLuqW5nt8foxlJoxlpZEu2bM/79UosjeZ45nq+z/M9KUIIgYGBgYGBgQ7ooW6AgYGBgcHwwRAaBgYGBga6MYSGgYGBgYFuDKFhYGBgYKAbQ2gYGBgYGOjGEBoGBgYGBroxhIbBqKC1tRXnnXceVqxYgRUrVqCurg7f/OY3sW/fPgDAwYMHsW7duqwe8/7778ef/vSnjLfbv38/vvvd72LFihW49tprcccdd+DEiRNZbVs8t912G3p7e3O2f4ORhXmoG2BgMFjY7Xa8+uqr8vdt27bhgQcewNtvv40LL7wQzzzzzBC2TmTPnj2499578eyzz2LWrFkAgNdeew233HIL3nzzTVRUVGT9mB9//HHW92kwcjFmGgajlr6+PlRXVwMAPv30U1x77bUAgEAggAceeAC1tbVYtmwZnn76aQSDQVxyySVobGyUt/+Xf/kXvPvuu6rrK2NmT58+jdtuuw0rV67EihUr8NJLL6m26ZlnnsH3v/99WWAAwNe//nX8/Oc/B8/zAIAXXngB1157Lb7+9a/jtttuk9uknNnEf//qV7+K3/zmN1i9ejUWLVqEJ598EgDwwAMPAAC+853voKOjo/8X02DUYMw0DEYN4XAYK1asAAB4vV64XC78x3/8R9J6zzzzDCKRCLZt2wae53Hbbbfh4MGDuO666/CPf/wD9913H1paWtDY2Ch3wMr1d+/eLe+P4zisW7cOTz75JC644AL4fD7ccMMNOPvsszFnzpyEYx86dAgbNmxIalNtbS0AYNeuXfjjH/+IF154ARUVFXj55Zfxgx/8AG+88Uba8w8Gg/j73/+Orq4uLF68GDfeeCMef/xxvPzyy/iv//qvnMxiDEYehtAwGDUo1VP79+/H7bffji1btiSst3PnTjzwwAMwmUwwmUz47//+bwBATU0Nbr75Ztxzzz144YUXsGrVKphMJs31X3nlFQBAU1MTWlpa8OCDD8rHCIfDOHLkSJLQoGkagiBonsOOHTuwbNkyuYNfuXIlNm7ciNbW1rTn/7WvfQ0AMGbMGFRWVsLj8WDSpElptzMwiMcQGgajlosuugjTpk3DwYMHUVlZKS83m82gKEr+3tHRAbvdjmnTpmHGjBl477338Prrr+Mf//hHyvUleJ5HSUlJgsDq7u5GcXFxUpvmzJmDAwcO4Nxzz01Y/uijj2Lx4sVJai8AIISA4zhQFJXwO8uyCevZbDb5s3JdAwO9GDYNg1FLY2MjmpqacN555yUsnz9/Pl555RUIggCGYbBu3Trs2bMHALB69Wo8+eSTmD17NsaMGZN2fQCYNm0abDabLDQ6Ojpw7bXX4tChQ0ltWrt2LZ599tmE315++WW89dZbOPfcc3HllVdi27ZtsrfTP//5T5SVlWHKlCkoLy+Xt+vt7cXevXt1XQeTyQSO4/ReNoNRjjHTMBg1xNs0AEAQBPz85z/HtGnT4HQ65eU//OEPsXHjRqxYsQI8z2PZsmVYsmQJAGDRokV46KGH8O1vfzvt+u+//z4AwGq1YvPmzdi4cSP++Mc/guM4/OhHP8LFF1+c1MZ58+bhsccew8aNGxEMBsGyLCZPnoy//vWvqKqqQlVVFW699VZ85zvfgSAIqKiowO9//3vQNI1bbrkFP/nJT1BbW4uJEyfi0ksv1XVdFi9ejNWrV2Pz5s1JMxwDAyWUkRrdwEA/+/fvx8MPP4ytW7cmqKQMDEYLxkzDwEAnP/3pT7F792488cQThsAwGLUYMw0DAwMDA90YhnADAwMDA90YQsPAwMDAQDeG0DAwMDAw0I0hNAwMDAwMdDPivafc7gAEoX+2/srKIvT0+LPcovzCOMeRwWg4R2B0nOdQnyNNUygvd2j+PuKFhiCQfgsNafuRjnGOI4PRcI7A6DjPfD5HQz1lYGBgYKAbQ2gYGBgYGOhmxKun1CCEwO12gWHCALSngU5n6jTVwxMKVqsd5eXVRlSzgYFBxoxKoeH3e0BRFMaMmQiK0p5smc00OG5kCQ1CBPT1dcPv96C4uGyom2NgYDDMGJXqqVDIj+LispQCY6RCUTSKi8sRCo1sDxQDA4PcMPp6TQCCwMNkGpWTLACAyWSGIPBD3QwDA4NhSE6Fxvvvv4+VK1di6dKleOyxxwCIpTTr6uqwZMkSbNq0SV736NGjuP7661FbW4v169fLRWHa29tx0003YenSpVi7di0CgUBW2jaa9fmj+dxHAmGGw5YdDWh1GbNFg8EnZ0LjzJkz2LBhAzZv3ozXX38dR44cwfbt2/Hggw9i8+bN2LZtGw4dOoTt27cDAO699148/PDDeOutt0AIwYsvvghALHO5evVq1NfXY9asWdi8eXOumjwk/OpXT+DWW1fj5pu/ia985TLceutq3Hrrarzxxmu6tv/oow/xv/8r1qT+059+jz/96fe5bK5BHuAPimVcGzu8Q9wSg9FIzoTGO++8g2XLlmHs2LGwWCzYtGkTCgoKMGXKFEyaNAlmsxl1dXWor69HW1sbwuEw5syZAwBYuXIl6uvrwbIs9uzZg9ra2oTlI4kf//in+Mtf/o5f/vLfUVVVjb/85e/4y1/+juXLv65r++PHj2Zt9mUwPDCZxNc2nwPADEYuOVPsNzc3w2Kx4Lvf/S5cLhcWLVqEc845B9XV1fI6NTU16OrqgtPpTFheXV2Nrq4uuN1uFBUVwWw2JyzPhMrKoqRlTicNs1mfvNS73kCROgLpeP/5n7/D4cMH0dnZiVWrbsB7772Df/3X7+Hii+ehvb0d3//+7di06Td49dWXAQATJowHTVM4cuQw1q69DS6XE8uXfx23336n6vFomkZ1dTEAyH9HMiPpHM02CxwOGwoKbQnnNZLOMRWj4Tzz+RxzJjR4nsfevXvx/PPPo7CwEN///vdRUFCQtB5FUVCrA5VqeSb09PiTRmSCIMiutC1dPrR0qeuGTSYKPN//0dzkMUWYPEbfzed5sT1SuwSBIByO4L//+x8AgHfffRs8L7ZbWnfSpKlYsWIlAGDp0jr86U+/R09PD373uz8jGAxi1aprccMNN6GwMDmPjCAIcLl8qK4uhsvl6/c5DgdGwjn6Qyx8QQbjKh3o80cQCESA6D0ERsY56mE0nOdQnyNNU6qDbfn3XB24qqoK8+fPR0VFBex2O772ta/h448/Rnd3t7yO0+lETU0NxowZk7Dc5XKhpqYGFRUV8Pv94Hk+Yflo4fzzZ2W8zWWXXQ6r1YqysjKUlpbB6zX03iOBd/eewadHxFm2NJYy1FMGQ0HOZhqLFi3CT3/6U3i9XjgcDuzYsQNLly7FH/7wBzQ3N2PixInYunUrrr/+ekyYMAE2mw379u3DxRdfjC1btmDBggWwWCyYN28etm3bhrq6Onl5Npk8plhzNjDUwX02m03+HD/DkjzL1DCZTAnbGNV8Rx7SPeUNoWEwBORMaMyePRv/+q//itWrV4NlWVxxxRW48cYbMX36dNx1112IRCJYuHAhli5dCgB46qmn8NBDDyEQCOD888/HmjVrAAAbNmzA/fffj9/+9rcYN24cnn766Vw1Oa8pLS1DY+NpXHTRPOzY8YG83GQygWGYoWuYwaBjjAMMhpKcRritWrUKq1atSlg2f/58vPZasjvpzJkz8dJLLyUtnzBhAp5//vmctXG4cNNNa7Bx4yN4443XcNVVX5GXz5lzETZufAQVFRVD1ziDQcWYPRoMJRQZ4U+gmiG8s7MZY8dOSbvtUKuncol0DYba6DYYjIRz3LKjAQBw3VXT4ewLYefBDlgtJiy7THyOR8I56mE0nOdQn+OQGcINDAxygzTOY1geHD8yBzUG+YshNAwMhhnxuoHmzpE96jbIPwyhYWAwzIjXKB9s6IE3aDhCGAwehtAwMBhmKD1tvzjVMzQNMRiVGELDwGCYofRdGeG+LAZ5hiE0DAyGEYSQpDgNQ2QYDCaG0DAwGGYoZxbCIM00OF7Alh0NhvF9lDN6y9flER0d7bjxxpWYOnV6wvJzz52BBQu+giuvXIi77voefvMbo1bGaIcgOSJ8sLRTYUbMAXeitQ9TxuZvFlaD3GIIjTxBqqWhxWef7RvE1hjkK4JAkmYWhk3DYDAZ9UKD72mB0N2s/puJhjCA4Cm6agpMlZP7vf3GjY9g7tyLceLEMQDA7bd/B//5n//V7/0ZDH+27mxKWmbIDIPBZNQLjXyhu9uFW29dLX9fsmSp/Pnuu+/FSy+9YAgMA1UGy6ZhYAAYQgOmysmas4HBzD2lpp7auPGRQTm2wTBnkGSGoQYzAAzvKQODYc9gdeaGzDAADKExbDCZTCmLLxmMXgarFpMx0zAADKExbLjyygW49dbViEQiQ90UgzyDDJJ+yigUaAAYNo28YNy48XjppdeTlq9f/4j8eePGXw5iiwyGE9maAHC8gG27mjFvZg3GVzlUjmNIDQNjpmFgMOxR68y9AQbBsLo6k+MFvPpRI9q7AwnLgxEOAiE42uxW3c7w0jIADKFhYDDsUVMbvb+/FW/vaVFdP8zwIITgUENidlyaosT9aeihDJlhAIxioTGap9r9PXdCCFiOz3JrDAaK3vvJcgJe39kEV18IQCwtiBJeU2iM3ndmoATDLD450jkiKi2OSqFB0ybw/Oj1ROJ5DjRtSloeZjgca3Zrdg5nnH68sasZ3oBR9Cef0NuX+4IMeF7Asaj6SalukmYYWvffkBn951BjLzp7gujqDQ51UwZMToXGmjVrsHz5cqxYsQIrVqzAgQMH8Prrr2PZsmVYvHgx/va3v8nr7ty5E3V1dViyZAk2bdokLz969Ciuv/561NbWYv369VlxOy0oKILP1wdChr/UzxRCBPh8bhQUJBeO33PMiWMtbng0hEKPN5zw1yA/yHQGoLW2JES0bBeGTSMzeEFIvjdRFeBwJmfeU4QQNDQ04IMPPoDZLB6mq6sL99xzD15++WVYrVZ8+9vfxpe//GVMnDgRDz74IJ5//nmMGzcO3/ve97B9+3YsXLgQ9957Lx577DHMmTMHDz74IF588UWsXr06zdFTU1RUCrfbha6uVqQKp6VpGoIw0gQLBavVjqKi0qRf/CEWADSn0AVW8T6GI6N3lpavNLZ7YKMAizn9ODDdTMKwaQwcjhewdWcTZkwqw3lTK+RrN/xFRg6FRkNDAyiKwu23346enh5861vfgsPhwGWXXYaysjIAQG1tLerr63HppZdiypQpmDRpEgCgrq4O9fX1OPvssxEOhzFnzhwAwMqVK/HMM88MWGhQFIWKipq061VXF8PlGvm1A1hOwBu7muTvDKsuNCwWsUNieaP3yDd2HexAeaEF82ZqP9dSxxXf+R9tdqPQZsaUscWysNCKxzBsGvpho+mHWpx+nDe1Ql6uNdHo7gvBbjOjqMAyGM0bEDkTGl6vF/Pnz8cjjzyCcDiMNWvW4JprrkF1dbW8Tk1NDb744gs4nc6k5V1dXUnLq6ur0dXVlVE7KiuT1TCZUF098usGlJYVwuGwyd8dxXbV83aHODgcfpSUqP+ezwy39sZDCEm4P1pYbGb5PKX148+bmExwOGwwmSjwUcHf2iPq2OddOB48TatuJ+FnBTgcNhQVWob0eg6HexkIsXA4bCi0i/ekuNgDb5hDebkD1dXFOHnGjT5fBJecPxYA8M7+NgDA6tqZAPL7HHMmNObOnYu5c+cCAAoLC7Fq1So8/vjjuPPOOxPWoyhKdQSTankm9PT4Nafb6RgNM43q6mL09voRCMQizV3dfpTakg3lfX1BBAIR9PWFhtV1Ge73URBIwv1Rw+GwwesNy+cprR9/3q6+EAKBCEw0leQh5XL50O0Oqm4HAI0dXvB8tB28gBMN3Wjr9uOCqRUZv5MDYbjcy2CYRSAQAeF4uFw+eL3itXe7A7DTwAdRd+ip1WIQZfx1H+pzpGkq5WA7Z4bwvXv3YteuXfJ3QggmTJiA7u5ueZnT6URNTQ3GjBmja7nL5UJNTXq1kkFmKGVzOiFrqCkGFy0XWCU93jC27GiQVSNKZO8oje21bqs/xOLAqW4cauyJrkfw4YF2nGr1GPXJFXT0BLBlRwOCkt1vBBi+leRMaPh8Pjz55JOIRCLw+/145ZVX8Mtf/hK7du1Cb28vQqEQ3n77bSxYsACzZ89GY2MjmpubwfM8tm7digULFmDChAmw2WzYt0+sWrdlyxYsWLAgV002iHLG6VddLnnPGB3F4JKp15KWWyfPCygOtYPmWfXjaAgnXuEYEb+aMYBIpKVLfHf6/KIHYjDMwtkXkt8Z5axsOF6/nKmnFi1ahAMHDuC6666DIAhYvXo1Lr74Ytxzzz1Ys2YNWJbFqlWr8KUvfQkA8Itf/AJ33XUXIpEIFi5ciKVLxSJETz31FB566CEEAgGcf/75WLNmTa6abBDFF9SIw5ANqcPvQR/OZKpe3Xvcqf5DyIsq/wmE2B50llyYfJy4+8rxAkw0BYqiUiYqNB4FdeLfkT1Hu1BRbE9aDojX3BQnSEgkAM4TBmAflHb2h5wmLLz77rtx9913Jyyrq6tDXV1d0rrz58/Ha6+9lrR85syZeOmll3LWRgP1F58QojIq0l4/ni53ELsOdeJrc2pQ0NcI04QLQNGjMo40K2QrPoKPuo+bePVBQfxhtu5swnlTyjFjcnlSRxf/3RAaiUivjNZ1UVMFm+JeDfbER/CaeZAZtaBoE9y+CGwWEwrt+ZNb1niTDaCmcDrS7MaWHQ2JHQTS6MTZMISQF20uMRGe//QB8F2nIPS1Zb3Fo4lsdcyxFOrJO3zz0+Yktdaxlj4AKpHjCULDkBpqxF8XlhM0Ayc5niS+YxHxHoS8fXD7Itj+eZtmDrGhIn/El8GQofbanzzTl7xeGvUUe+R9EDYCuvQqcT2B1z6AgW6y1zFrD4MjDI9WV6ItSzqucvX478atTUTLk0zK90VIouDgBfVqKB8daEOQTvZgcvsi8AYYTBk7dC65htAw0I1WJyLBM2EEQizoMlENIuviKaDHE0aJwwKLOdmV1yA12eqYpftHQUCV7zg8BZPAmgs116fpaNZbpXpKZZ8GIpLISBUgGX/NjjW7Y55WALjohhxPVPVA2z9vAwgxhIbB0JLqvSeIvQjpZhrdnjB6vWGEqCZMcx0CNXYcYBWNqjsOt2NMeSHmzxqb1baPBgbaL0v2KSqaa83Ch6L/gvAWTEDYXALelGx4pTRSpcd3fIbMUEf5jkixMYQA8ZmJEmZ3RECE4cDzAiiNoUJhpBtjvIcQ9lXCXFgCs2nwLQyGTcMgJa991Ch/ljXiGh0FQ8RZRKFX3IYKiSquCMOjMNINX9BIdNgvBtAzv/5xI3Z80aG6H5PAoMZ7BOM9n8vLKCLI65GosFDGiSSopwyhkYBsCFcsl2ZtYYbDgVPdKA2eQQHTi7JAE2hBnGlY+SDOOP042tiLat9xTHN9gAKmBxYugAOnuvHqR40oCYn2we2fnsT7+4fGVmjMNAx0qxik9bS8eXizA0AAFj4MJrp+IMxBcHdhjPcoWMvZAKaCYXmcbPVgxuSyIRkpDTcG0i/zAkGvnJU4cU90dOZh5sXfaYHFlJ6P0eM4C377WAi0mAdJbaYRa5shNdRI9jgT/x5tdgNEwLTAafk3sxBBxFyMKv8JeZmFF51JxnoOAgAazV+R1wUAUBSCYfV4m1xjvLEGukk30yAmK4DYaKvbG8YZpw8ejzgFt3Lii9DY4cXJ1j40deR/Ooh8IGveU4qMzRRJLMIkdUiVgdOY0vMxzHxo0No2XPEGGYWAEB/+ZOeB2AILnzjjNgksSkP6Zg2SilH6y574GFz70QxbPTAMoWGge6xIBAGV/pMwMV4A4oxD8gpJBceKIyKG4bDzUAdMEFAWbIbPJwoRQgjcvtS5lQwGRlenE36PN2EZTRJT3JtJ4sjVwqUvGDSaDeFOdxDv72uVo8ABgGFFQZzK44yCMs0LSboXWkjbSn/9znYEmw/jyP4D+LR+W2Yn0E8M9ZSBbqlBMwGUhNpg6Q0AmIFjzW6cONOHBbPHo6IkZkhNqjsTfSE4nofTHYLZ147yQCOsbjOA8Wjq9OHAqW7Mv2AsxlRoe/OMVgbcMRMBnTu3pl3NpBAaNISo0TvFrgfWsmGNL1p/xhOIAChGrzeMLrcoaFPFtigvKEWEpFmfEkrgQGizvJ4002hxSrP1fYMWQ27MNEY5Wn7iqnDibIBQosFbqvDHKBLkJRkBhcQHnWLD0WPzONjQA19QfPm8WulLRjkDlRlmPp0DgqhSoUiy+irh2ISgynccVs4fvwhA5qlORiLxFS9JBteDgpB07ZVM7fkIJaE2UNELXu07hqJwcpmIY81utHTlVu2bsdBg2aExvhhkH4bl8cI7x3GqNTmQTw2KF4WGEBUaAs+BEjjQyngmpRFQTpAXjRPgxE6szxtE98mD6OwWH3KOVxoPCZgj74PvPaP3lEYkamK90ncCxaEOXdsrZxBqR1CDIkLCCNkkRFAc7pCNs4B4j1pdfrz2caNc+XG0QGnYL4DUqV+U7rQUEVRUVsmUBRMjwyvijOkSx1rc2H/ClXZfAyGt0Ni7dy82b94MhmHwjW98A/PmzcO2bYOjOzPILZGo/lUrq208R5t60d4V1YlHX4jyrj2Y2vMRaGWOKsW2Aid2JtJoSjLIFoc7URFoQCEjpr9XlpklgoCTJ1rQfXCn/pMaiSSp+3iUhNtR5T+ua3OToLMzV3R0xeEOeZGdccv7oeP2RwjgdIt2rR6P4VItoSUz1CLGbZy+mYFJiCi+D83MPK3Q+OUvf4k5c+bg3XffRVVVFd544w38+c9/Hoy2GeQY5QNs4QKwM27VdY+f6QMFUcgEwwwaO7wwM+LDTtOp0z2Hw+LDbhYiqPSdAOETjX40TcPOuCEwiS8Fz3HgBAGdvemN7SMZZf9j4nU4DcTFW9BpZxrq2DifGBgY6sU4zwFUBBoAJBpyCSGwRssAS4MQg+SZBkV4FIU7QQ/QCqRnRpJr0goNnudx+eWXY+fOnbj66qsxceJECMLQN9wg+0x078E4zwHN32PufgQHTnXLoymKokDYMKxBMSW31uNhEhiUhNthDydOn02MH+M8B2BzJ063JVUoRY0O05tAiGrHqxTCtI6OY1r3hxgXDdqjBmAUcXnCoCLi4MAsJAsrQgCbRVRXMtzoEhpagXxq1HiPoNp3DFYhqHOLgaGctWeTtG+jIAj44osv8MEHH+CKK67AiRMnDLvGCCHTlNuyeomSdLnRyGGeB3sypkJKZ1pXHtbEiq63JiZxms5LQmOUpFU/3NiLNz9pBqvofJVXM53RVLrAdtaj/+CEgCj2y9MW7DnaBV9QbI/ajIWAyAGaSpvUaEb5bhUyYtVDPQJ/IDgiLtR4j8Djz53qKq3L7Z133okf//jHWLVqFSZOnIivfvWrWL9+fc4aZDB4ZDoAlX3ElXmH2g+ChNPbRWIHTnxxaDYY3V9ig+TByQgsmalGe7coPFmOwBL/ZibZNFJ3PMk+/+lv9CT3p/Daxyv2Ix6H4XhQiLeNxBUNInG3x5AZMoSI/5WGziReV0E7r1Q2qPEeFj/w+uI++kNaobFkyRIsWbJE/v7OO+/AZDIylY4EMvX/l9VTUl0NKWuqzwlYMjmwwrVT8q4SEkfY3CibaWiJRgLAzvTBJDAI2Gu09dqEgCYs7Gw0+DLq5aZH5Jr5MGxcouCXYgIIISnbFvMiGt1Soy8uQJUQAjvnQUWgQbYFAQAlsDDxuVfjEcYPIDm1ejZI+za6XC7ccccdqK2tRXd3N+644w44nRrlJA2GFZq+9RovPx3tRKycD2P74m0fFAYyzCQKryq5fdHR0mixacRIdj0e5/kcNb4jABJnGhPcezGuT7RdlIbOYErPTtg4UWhwpgLYWC8sfPrIbuV+lcfXap+YQVf8PJpDNSIMj+a4+AhCAKLSvVZ5j8n3MacIuRNMad/GRx99FFdffTVsNhtKS0sxc+ZMPPTQQzlrkMHgoekWqBGdGm9QLWDdsl6CRGVGv7VIRKq/odDlc6LQ0B7njjC0LmBS0F3su5Xzw872wcr0yTOMeFvG+L79cmbU9CQ/EBQRQKXogAhJnzJ/xEMAVukurnEppIy2OSeH9yKt0Ghra8O3vvUt0DQNi8WCe++9Fx0d+oKKDPIbLUO4trFOfX0i/9+/zl0ZvyG3jxONeYQ2geMFsNzo8NpLinNhE72W1NRTNa59cryLLDQyVT+q3N8pPR+jyK0vHmS0EmF5vLs3MQB1JAvQtEKDoqgEF1u/32+43I4QtB5sivAoDzSgNJi6NrEsIggBw/LRHDz9akn0j2K0FlVPEcqEt3a34I1dTf3c//CGUhg103pPSetleJwCpjd5H4RPqXgkhABEgJXzZ+yNN1Lo6AkkLdO6EoOVSj6XQiut0FiyZAl+8pOfwOfz4X//93/xne98B9dcc43uAzzxxBO4//77AQBHjx7F9ddfj9raWqxfvx5cVP3Q3t6Om266CUuXLsXatWsRCIg3wev14o477sA111yDm266CS5XbsPjRxvK50oynFr4EMqCLQkGvNQ7As4MIN+NZCtJSj8SnWkIlGlUzDKUFRIllB2N/gCv7HQc6fofU28jJrj3whROTEfj8Ufw9u6WERv0l0odKwrQoRSiQyg07rzzTixYsAAXXnghdu7ciRtuuAE/+MEPdO18165deOWVV+Tv9957Lx5++GG89dZbIITgxRdfBCDaTVavXo36+nrMmjULmzdvBgD8+te/xrx58/Dmm2/im9/8JjZu3NifczTQQDkyZMyit4Wd1ZeLSoIgubpbv1DYUqScVbl0URwWxM/sh2A0H2ZU9PDRdrR0+dJMwWkAACAASURBVNHULKqrTUyi99WpNg+CEU5OMzKqGOpHVudstD/ocku57rrr8Otf/xrPPPMMvvWtb6nmT1HS19eHTZs24c477wQg2kbC4TDmzJkDAFi5ciXq6+vBsiz27NmD2trahOUA8MEHH6Curg4AcO211+LDDz80AguziLKfFyjRA1vpeimh7LzlCOBsPaBJiQ4TXXxHC92eED480C57t/Vf1ZC76ybNdjq7vfJzQylyXJloKehv5M0SQxEOB071aP4+1DaNXB5eM05j7ty5qsJBKlK/f//+lDv+2c9+hnvuuUc2mjudTlRXV8u/V1dXo6urC263G0VFRTCbzQnLlduYzWYUFRWht7cXY8aM0X2ClZUD81Wuri4e0Pb5TIAjQEsfHA4bAMAeNsNKzKBMHCzR6LIZfR+huno2HA4bbBEzrELyI1NQYIXVah5wemyrhU643q5CK4IWM8w2s9zG/t6P4XAfi4ptgInGyXZR1ecoKUBRgQU+tx2+6P1wOCwogAXWSPJ9sFoSl9FmM8xcbkrmlFo5EIpgrPMTEIoCZTHDbKMSrnNFTxAuXwQlJQVZvf75cC//b98ZFBZaNX93OKygIxZYA/27/sp7mSmlpdm95vFotmzr1vRFW7T4xz/+gXHjxmH+/Pl4+eWXAahLXoqiNJdrQWcY6NXT4+93Z1ZdXQyXa+SWJO3pEWcUgagBuzjCgmY5gPUiPgmBy+VDIBCBI8LBzCarKoJBBgzDDdwQSgicTo8cl+H1h8CwHLhwBAE6IrclU4bLffT7IwjEpRfv7vYhZLegzx0AE73uwUAE5hAjf5ewWsxJyziBlWNdsk15+y4ASHhOWH8w4Tr7fWEEAhF09/jhcmQS/alNvtxLjyckvzdqUIIALph8n/Sgdi8zJfDFdnTyHEwVEzLelqaplINtTaExYYJ4sGPHjuHpp5/GH/7wBxw/fhz33XcfNm3alPKg27Ztg8vlwooVK+DxeBAMBkFRFLq7u+V1XC4XampqUFFRAb/fD57nYTKZ5OUAUFNTg+7ubowdOxYcx8Hv96OsrCyjC2CgTSZ9/MTe3dpBYlmaCxNC0O7yY3x1sThwEOQfsrL/fEezLEmc+s/ER9JWeYvtICvN0k38ADAY5tDmEh1asmLvyjPSDZCG+pElAARXQ7+ERjrSDtsfeeQRfPOb3wQAzJgxA3fddRc2bNiQcpvnnnsOW7duxauvvop169bhq1/9Kh5//HHYbDbs27cPALBlyxYsWLAAFosloUaHtBwAFi5ciC1btgAQBdG8efNgsWRnxGIgvszWiDvWCaV40lNFFZMsxd6xvIDPjpxBZ+MpAIAgdZaCgPHufXCER2cmAhLnLTXJvRvlwSbdWw4m8UJjxxftCITFWdOITGQoZXjWEOAkfqURRlqhEQqFsHjxYvn71VdfDb8/g+R0cTz11FN4/PHHcc011yAUCmHNmjUAgA0bNuDFF1/EsmXLsHfvXtx9990AgB/96Ef4/PPPsXz5cvz973/Hz372s34d10AdwoRQ070f1b6BBW8JjpostQgY4z0Mqnkfms84EWGi2VX5MGycb3DSL+QhmZQOjWew4+jjMwaEIjH1ijJr70iAALCxXkzt3oHCSEyDYubDsLGeITeEA7H0PNkmrbWFoigcP34cM2bMAACcPn06I7vCypUrsXLlSgDAzJkz8dJLLyWtM2HCBDz//PNJy8vKyvC73/1O97EMMkOKuC5gelEabBFTg/QHPnsebWL96UIcPN2NsmAExUBcHp2Rm06koyeQHJ8hJYTMQ+8pNQRC4AsyKI4aiE0Cg7JgM8KlFwxqOwYDgRC54l4B04ugrQoAUOU7jgLWjS77VSN0nqFDaPzoRz/CzTffjHPPPRcUReH06dN46qmnBqNtBjlGCp6jCa8/kE+NLAoNQOws41VesT5zZL6Gvd4wPj3SBRABZoEBZ7IDiJ2t0O8R4+Ber2CIwXv7WrFwjqhHLws0oSTcjkBfOYCJg9qWXCMmJIxh5kMwCYwsSKrcX6DHln17QkbkaLaTVmgsWrQI9fX12L9/P0wmE2bPno3KysqcNMZgcJFqdxOKGlB1t6wLDcWn2FR/ZM40uKj6qcp/EsXhDjRXXgGBtsQuRD+FxoDu6QCQbBkkmmGAZ0ZgcB9JzLU2wb03ltkAgJX1oYJrHIKGiUj1PHJBWqHxjW98A6tXr8a1116LgoKCnDTCYGggfHJRnX7BZ7dKmPisU6BI/PeRizlaY70gWp+dJjyE+AIl/bwAg11PWq6zwvOY5vpAnjHRkeTcTMOdeDMTBZIgMESI7hxhuSM3L05a48TDDz+MvXv3YvHixfj5z3+OkydP5qQhBoMPic40dLtwaiFkOUo/5msa/V/6O/xnGk2dXvR4wgnL5DiiWAZI8X8yMKk5+DMN8XhCtMiQmRfP08725YVhOBt4gww+Ptih4kac+GyKv4+Mc1aSVmhcdNFFeOKJJ7Bt2zacddZZWLt2LVavXo0333xzMNpnkCWIwCW/uFx2OntlFtaBQgBU+44ljd4oCOA6TyTV3RhOfH6yGzu+aAcg2jL2HHPGdUBix1MU7sI01wdy59v/zmdwOy1HxAULF0hwEQbEmZM/xKKp0zuo7ckGgteFyP7XZPvfwdM9cPWFEAyzkO6XhQ+BqAQkm4Tc1elOD8lZ/ildblBerxevvvoqXnzxRRQXF+Oaa67Bq6++ivvuuy8njTLILoQJgtn/OoTu5sTlWZohZHsQSYhY5EmqDxHPyd278OH7n8jfvQFm2GbA3X3UiTaXH8GIVGxKRI7D4KIzkmFUiqAk1AYhKS5DwMcHO/H5ye5hd6/4zuOAwIMExSSehe4TGO8WUyhJzhp2tk+1St9QQoChs2n8+Mc/xvbt27Fo0SI88sgjmDt3LgDgxhtvxOWXX56TRhlkF8ErBsUJPhdM1VPl5XqriEmuuYNFqpoDDMfLdTYA4P39ragosWPB7PGD0bScIGgY+mNFqYaRmkNRfwcAaCLEZcodRueigsPXjBCX/N4Qihrup6abtELjnHPOwfr161FRUZG4odmM//mf/8lZw4YSQgh6vGE4fQzc7gBmTC4f6ib1C0IIwLMQPGICSKqwTF7Otx2GKeKFHkUP33o4h61MRu8ASVK39XrDadbMD7RyoGlpEeSgvmFkDyCgwCUF88W3f3jbpRJTu8TOJd9mGsih91TaM/3KV74iF1E6fvw4VqxYgYYG0af/rLPOykmjhppgMIRTH9bj0NEzONrsHrZGPL7rJJjP3wAJ+RBhedmQTMI+8J0nYWb06ZhJGpfarF8dvUIj28fNMbyGmonnWNR4D8uGY4lYRO9wOlNK8zyHIywnoMcblvsA6W8R14dq3zF5PbPQ36qVOWSohEZ/ck8Nd+wUh0KuD1ZG1GMO14RrQm8rAIAJh9DY4UVLQxOYg29lHleRrn5Ktm0aeteLeykONfTkfd0GrRxMVNgDR8SV5CIryB1VzpuWNQhFgVc5TyvnR4X/dM5SW+SKM04/XH0huH2iUJDuRY3nkLwOlbezpyESGtnMPTVcoK12mGgKFi4IK+sbtkJDSr/BCuJDLbjbQSJBkKAnwx2lfkyyX/dY3/7iO9NTbR6cceb3c6n1HGkOzFN0sPnbUVHgVYT3uL7PURo6A6JiD8gniMBD8Log9HWCdzbIsyZZUxhdL98Lg+UyYWLOc08NS0wWgDahxNcIO8uBY2fCZjENdasyJ2owlkasUp0SruWA+LveZ0pHpcZsPp76bRqJ33U0c0jRrOui4ZDg9Udw5rgL5cNoqkFAQSBC0jBDmkVJHkdd7iBAgDEVhYPbwDTwZ74A72qSv1OUmEcrFjIzfO7FkHlPxeeeAoCGhoYRn3uKoihRcEQjnXk2AsIIoKzDKyKeRGcakjcLTfevV91zzImU9Q+z/Gx6g6m9tSoCpwEsTHqBJXdOQgj6/AzKiqy6ShMPFsr2UgIHO9MHELvq+qda3QhbCArt+T06V0JUpk7KQMNdhzoBANddNX1Q2qQXIZRo55M8DGOBlgl/op/zU5DkSr4Zuae0oM2y0BAa94LhPLBetALUcJplyUJDfHqUMkPvw56voytlsyKseL4nWz040tSLKy8ch6qy/BH0yqtY5TkMc9AFqmKOxgYplAyU1g9DDEVppHKP9bYMO3yCM+loLNPnp7pxedW4uHcmX29AFCL/l3U0e8Bdu8Ryjm+//Tb27dsHQgg4jsO+ffvw9ttv56Qx+QShY+oo4o8WkM9yYr7BQnqJ+zvqTpe+Y6heHaXQk+pvdPaKBaP4PBN2chK5aLssnGiDoTTiYORcTiqnkT/zp0QKIz0pjd2EEBxp6mcK/qEgeqELmV54dr4k3wshb+9AHIOtnnrjjTcwf/581ToXFEVhyZIlOWlQvkDRsUsjj7R5FrDYhqhF/SAacMRFjY+mfqqneFOacx6CzjnC8AiGOTjCTkQsxeBpGyLRADI6KhzzTGYAPIepPTvQVzAZwFmQeyTNIEui+Jv/2DgvhHCv5u+EkCSHAIEQdHvCqMmjWaGENKouDZ0Biu1xcRr5f08oszUn+9UUGo899hgAqAqNUYEpdmnCDI8zLj/GTfChZlxK7X5eEgmL/v+mHKnWcjgT1uT9z1oRYXhMi6vmF8FMkHCJbBDXNDwPEYQJgCICSkOiK7Q8g9MYmVMKPfqwIWUGAZLksHAqqk6cf8HYoTeMJwkDovgWnbXn+U0hIDBNzE3xK81eRBAEbN26FZ9++ilYlsWGDRuwfPlyPPDAA/D5fDlpTD5BxQkNf0hUS7m6h1nCNYpCjzcst7/fvU8ejqoiDJ/UrhLPSTCH3oE9LOasEvKt3UQ5c5BmRBrxGyCwM25wbYMbkT9Q1Azhsd+SlwXD4kwrxOSfwV97bp5nz5YKFJ0bj0/Nmca//du/Yf/+/WAYBjU1NSgtLcV9992H9957D4888gh+9atf5aRB+QJFxeSp1Pkk58zPb0IRHq6+WAGc/j7m+T6qkuB5AldfCOYCDwBHwkxDCPaBshbmbMquRWT/q6DLJ8Iy7WJZZig7Iq26C2XBFkTMw29mmzq7arI1IDYzzG4zBIGgsdOLaeNKZJWlBN/dDMpRDrqgJHEjpYdbdps0ItAUGjt37sSWLVsQCASwaNEifPrpp7DZbLjqqquwfPnywWzjkEDRsa6Slw3JQ9eeVBBCADac5BLc2OHLUkac1EKjZ4hyPymFGSeIKR/6uDDgSJxpsEf+D5S9GNZZVw9uIwUBQk8LMO1iOWU4RXhE9r8GMx8EB+2ZhpXzwcKrFzDKt0lUPCkN4SrLKCr1jKu/nG734HBjLyhQmD4+UThwTWKmWtu8b8jLBH+vnM021rbY5zy+5BrkpsPS7FMsFgusVivKy8sxadIk2GyiMZSmadjt6n7lSv793/8dy5Ytw/Lly/Hcc88BEIVRXV0dlixZgk2bNsnrHj16FNdffz1qa2uxfv162Xjb3t6Om266CUuXLsXatWsRCAxWFbDYBZdGrPkahcu3HwXzRT1ItKwmc2w7IvtfTaizrYrOt2DoK5BpoaHWibZXOXIl4SFWq8a3J74mSKrIb43f8jU2AAAcgTbtHwlR6ctyky5FitthkxIoqiP4k1Px54Oo6Cy5EM6SzOwTuRxUaAqN+KhvszltOEcSu3fvxieffILXXnsN//znP/H888/j2LFjePDBB7F582Zs27YNhw4dwvbt2wEA9957Lx5++GG89dZbIITgxRdfBAA8+uijWL16Nerr6zFr1ixs3rw547b0h3j3VFk9BSEhLXe+ILjFoj5SYkHi7432mEmBGQDEGhTHWty6czXpUU8JlBneggm625wNtCrTOZgejPEchD/EoL1bHGR09gYRCA2ty7RmR5/P04Z+kOp0CCEJgy9fkEFDu2grzLYNSjrKiVaP5izmcGMvuqMqXLVB4VAME4MFNQhaq+TvjKUYAVt15jvKkWpEU2j09PTgueeew3PPPZfw+bnnnkNvr7ZLncSll16Kv/71rzCbzejp6QHP8/B6vZgyZQomTZoEs9mMuro61NfXo62tDeFwGHPmiEFOK1euRH19PViWxZ49e1BbW5uwfFBQueD27qNgPns9paFvSIiOWinaBKKjRoY3IHq3hPUGWel4mQlFoafoHPQ6BjPCV71dZj6EQqYHTa292H20KxohHsEZ11DnphotQkP/+Xj8MU+rrAeRRt9hnhcQiBrb+Z4WcJ0nxOMBONnah48OdiSsn7CLhG+Dc588JWejq3TWIB9VP5pTiCuuuAInTpxI+gxAd/Eli8WCZ555Bn/+85+xdOlSOJ1OVFfHJGZNTQ26urqSlldXV6OrqwtutxtFRUXyTEdaPhiojTpongFQEFUn5E9keLyg4E7uii3XeOSpDOM1SsLtadehojsPW0oz2vfASP06mfkQGLo4rmzq0EF4DpTrlPpveav+6x8p74ogaA7fsy4z4j9Hv3CN++KOFzsg72wAd+agWqtin0bWbeo3mkLj8ccfz8oB1q1bh9tvvx133nknmpqakn6nKEp1hJFqeSZUVvbP+6S1yAZfN2C1xC6R1WZBocOGkgIO5tL8KczkLjCDCBRKKx3wNPgAh2h/slpZmOM6TLvdjEKHDQV+BhE29gbEn2N/EWgTHA4brIwV1sDA95eOGrhQ7j0OpGh7UQGNiM0Gnhfkc6yuLs552+JpEACb1QSH7zRCrFtuR6HDBqvVDJoTUGAzg2ThHmTjPmYLq0ZMUGtvCKDtcESf0fIKh/y5pLRA1/3Rew+7Ojoxo+8jdNVcioqKIpQ4rOh1xAJVOU7A2c5PwVpLYHYBlkIrCAEa2zwYV+1Aod0Ce4TAGhSvK22mB+kaU+K71Ccey1Fog2Cyyt/1YLdbUFHhgKU8+897zq7A6dOnwTAMzjvvPBQUFGDJkiWor6+HyRTzHXY6naipqcGYMWPQ3R0zQrlcLtTU1KCiogJ+vx88z8NkMsnLM6Gnx9+vIK9gVIXDsLFRfCjMIhiIILjzzQSvi6Em4hPTZnDdfjCBWDEYhuUgxNlgwmEOwUAELMPJ52W1mBPO0URT/UoFL1BAIBABx7IJ+8sV1t5GMGnsS6FgBCEugmCIkdvkconGcObQuzDVTIepJnfqNEIIGlrFlBnnF49FKK4dgUAEDMOB4wVQofCAr5nyPuYrZ1p6wZljAXzu3gAC0WfW7Q7K90eL6uritOsAQFOnF8Gm02BZDujrgMs1HZGgFZHosXxBFhGWBx8JwcyE8flxguJCK0odVnj8YXAcj0k1RYjE3bNgCINzjaPvkvysBBkINMno2KEwi97eIGguc+cPmqZSDrZzpmNpbW3FQw89BIZhwDAM3nvvPXz7299GY2MjmpubwfM8tm7digULFmDChAmw2WzYt0+cOm7ZsgULFiyAxWLBvHnzsG3btoTlg4HahEY58xHCPkT2vgKhr2NQ2iTBdzcjsvcVEEXkbbKaI/EkwgyHVqc/pfqg0GbpZ6typXlVn1kqCxaprhO9Hu/taUlYTogAEvbFUsTrhHAM2Kb9IKxeF+O4a5JKt9EvvUx+evKlQ9nqhGyxWdRPfX6yGy6PeJ/KA41JRva2bj+6PaIBXMr+7AsyYKIzcKtUCiEvLnM/GpFDO5nmTOOdd97B4sWLwTAMrNbMA6IWLlyIAwcO4LrrroPJZMKSJUuwfPlyVFRU4K677kIkEsHChQuxdOlSAMBTTz2Fhx56CIFAAOeffz7WrFkDANiwYQPuv/9+/Pa3v8W4cePw9NNP9/NUM4RKL09JQBxF8u520GXjctYUvusUCMfAPOF88bvztHh8JpgyWE1p05Aiw+3W5NtOgcKUscXgBQJfKHVqcjWkI6VLbhh/PD1uoyS6ZjwCZdLpBixuxypHaFLiyQxVnbyzAUJ3M3hrIczjZ+o4fFy7BYVdJaG3zFxZrnZdMuVMxZdBCxwm9O1Lv3KWsHFecCYbCJUcraxngsv5ekE4kvTcM4fegWnsOTBVTZWXxT+LqfrQ+KfAH3323b4wAiEWRYVxg6gsdcR9hVPAmuyo9h1X/T3dO+QpmCTmwkqBsy8EX5sH55b0w+sqDZpC45lnnsHixYtxww034JVXXunXztetW4d169YlLJs/fz5ee+21pHVnzpyJl156KWn5hAkThib/ldp9UzwzhMge5jltimSgk4SGJoqHmqetsPBBfQehALvVJHuZZEyuRjbRpIsJi0B0dbQxl9zYDnYf7cLEYoJKADBlOBgiQjTPls5OPu6aCBybKEPivmi5DqdkgJm5CWhwpgJgkGuLVfuOwRFxyXEHhBBQhEdpqBWCcAE+OdKJUocN501JthkSgYf3s3fAmUphOfeK2HJCEPT0QejbjYorp6rOWFKpqON/CUZizz/D8QBiQkO3t2EavPbx4E02TaGR1D7FANZvH4PeorMwzfVBVtqTKZpCw+FwoLa2Fl1dXairq0v6/fXXX89pw4YaNe8p5WP30cFOVHs9OLdi4uA0SkZKdCe2KMzw4AWCEq8zcS2NAYvackrxN/MWSTkydO5BZ6dH1O4EASgq/cZmIYwCpheULdYBtXcH0NPhwVdLAMqcmSouxPBoaHFjvN2Hal0hKbE2+nyJQaknW2ORx0Q5C9G154GFmupR7+UKG+fDlJ6PAABnnLWoCDSgJNQGyluBzmAxOnuCqkID0eDV5CBNguYucVlF/OL4WKuom/wZpx9hJvF6p1KL5YN2Sik09M7mC/qtak6NptD44x//iKNHj2L9+vV4+OGHc3LwvEbHfWE5QXeAXFZRdMxNnWJw1MyiE4rV1E8il2EBepOk6R8oq/nOE10nURFoAACwY76WsFyqxgZTZi9VOJpQr83lha5Jf1wbTYQDP0hxGpNrihFhebGkap4iqRddfSFURT8HQxFYOBO0ngzCRvOoWRQZKeJW/+ykCxOqHCiMuGDiY04hkmAOhJMDPFOpxXLxrpBBykeUq8NoCo2ioiJccskl+P3vf4+amhocPnwYHMfhS1/6EoqKhmEStYxJtmkQQsAJJFYBj4otHxIUxz3Y4seF42MqFy2dt7or84AbAwAw27Ob2pqoqKcy1cuUFFoRivtuImLHQWUoNKRLxHM8iCCAomkQNgzCMcmJ74CE+0MTDlqKP0rIPFI9VfLMQrsZfJoAVM6kLxUQR9thFrKdW0z9YQsEWUz07wYACGRWUpJBRN3H4zNQi8TOtbnTh+ZOH6Z5EzMDCykKqKV6f3OTKTnxvPoKJ8PMR1AUcUJ8tsUhlat45sBS+ORIaqR1ufX5fLjllltQVVUFnufR1dWF3/3ud7joooty0qC8QeN6n2rtQ4HVjHPjljV3enHOWbHvRBAALpK7muLyw6AwECsU1FoxfNl4D9QM2aVFNlx6Xg1OxaUeYk2FqnaViKUEFsaj60gDhVdEyUslPJGhekrSi5sjHnRtfxG2c+ejsOtzQBDUXbDjX/iYASwJuh9CIxNoikro/DpKZ4M1qQv3kLUCBYyY8aGl4jIItBVTuz/ManuUl8HMJwslnhdAm5Wz1qgFUeDBO0+Drp4mZqOOO7fyQCN8tjFJ+xP6qRHIRU0WpXrJHc2iUORKVC/77WM19qDX2SQ3pBUaTzzxBJ566ilcdtllAMQysL/4xS/k3FAjlVQaYznvf1wup3i4ls8gdLfAelFdQgXAbEAEXrPIjZX3g6A0ruVa6imdriSpiOqXwpZS2Fmx87/onCo47Ba0l10EnragJNQGn30cJrr3JGzYUTYbZhqo7v087WH06m9T4fEEEC++TUS8fxSdKDQIFwF36lOYp1+iKvDl+BUmgC4G4I6dwKzSVKVN9XVUuZmnxq7bWRNKEWY4nHGKaVTCVvXA1KbKK0EoGtOiQoLXORsZSNtsrBcFbHL515T5q/y94Py9MNNmmKqmJNQkLws2w6oySOmP3QjI0b3RUhtn6BHHmgpg4UPaKwx27ikJv98vCwxA9H4KhVI0dISgGqehXEdKdQ0C3t0GIZpWWeiJusPlYGrLnfoEJKyeQ4kiQoJAsLF6RvLy1tE/+kJ3BJMVLRWXocdxtrzMZBK3jVhKwJkK0Ft0tsqIliBsKZOFAU+nKSWbhQe/tH23/LmA6YWZD6veGqG7BYK/R3RxZpKfcaWwNXtSuz0q19d8GnKs3aRpSlfVRkKbNe+/1sykX8TdUysX/yzHeZsFvfL7FPtVcaGiM0iiMOqrzVxIP2cMuVA9aw6EJHV3mmdealFb+Ty0VOpL6ZRN0j5JNE2jrS2mb2htbU2I6h6xqLw8ygdIcpXkBYLTO9+D/8C70oqJf7OI4HVCIEQ06Km5FsYt0qoJnrJVOvX8FKIj0bgHXHm8BbPH49LzxwKgErJ2JrZF3IbRKDaUjZlGPGM9X6Ao0oXjZ9yaMwHB3wPmi3rw7sQU36lUFc6+UFyFRJGkxJZadvD0zc4Iy8wF4MZ/Sf6eyRU8Z2JZ0rLmysvRWj4vCy0TiT9ftVkGAPDH3gd75P8ghOKEh/JCEUldlXidKRV7Tz7kH5PREMyZPuuEMkFQiXeRDzPYhnCJH/zgB7jhhhswf/58AMDHH3+MDRs25KY1eYR6RLhyifiwchyPEMPB1RdCSXQ9lhdgzZGBvLMnCG+QgSXMobgYiPdFEgWb2HhaS2ikapbO2AXJEBv/oJtN4udzJpWhvMiGihI7PP4IdlcvBEV4TO3eEbeH2HaNVQth5kOY5N4NJZnmGssEhhMQLyLlqI5o0CYJuIHymG+tmlFUIAQ0RWFnNFPqdVfF0pLEd2a8IMDlye0MfVylAyaaAl1UCVIS65joigkAXQJ07ky7j/OmluPgZ4nLBDrb1Q5j99QRccmfq/wx7z/pSrOH3wOAlGl7lLMImqi5HPR3ptGvzXTRUToHie3S666eiQ45+6QVGldffTWmT5+OTz75BIQQ3HnnnTjrrLPSbTYCSL7gypFmUuU4F5df3wAAIABJREFUXvze2RuAJ8Dg/NkcLJY06pd+wHAxYaVE+ZCrR16reE9JH3SWQ6VllUD8TEPsqC6YGuctL1VlU0xq5XZSVOxfFM5kB0fbYWf7clpiN8JwcGSwvpoxlRcIaFN64ex0h7Tds7PUMxXazLCYpesca5Nl+qWAR18OosFwBlXv1BWoXhKt66TUAKg8MwrVrV5y6Rk54/yzcaSpF+AG4ravfcdyNeDSZaWdPn06pk8fzDoJeYDKBY8vfM/xgqw7lYSHZCj1BaPFkDJ0lxP8PeDbj8F89nxQKjpoaX9S0/xBBo0nXAk+U0kPuUpARKqEhKRyKtDQlLatkWgK9Hj9q9rMRl6iuJ7SlYnFbMd+P1NxGQojLtjZPpgIkzOVv3IQ0O0Jwdflx+QxUVWZMqhKpSGtTj+mjlNxt0XiTCNlLIC+5qaFGjsDJku0zYq2p+o+ukpmwSxEUqyRXUxC+jQ1mciM5JmGymBKEJJTueggm89eW/k8sHTMuWBSTRGmjSvBlh0N0SXiXcokQ0D8mp0lF4ImHGp8R7PQWm3yJ5dy3pFaSn94oB0OKYOk0uAZ/ZpprSauaT9I2A8S8YNS8fvn248ltOxwUy/CVoJpKscWv4julryOh1CgTbDN+wYC3akLbFEUhebyy2Cz2wAdnqI2q7rONSbc1GPRuejLRYMgV3MNSXgSgQPfegTHG50oi7Cyp7wStVxZEZZHOMKhgOlByFqZuL7S5TbHUGNnwGwXFW5Jg44Uo86grSputXyIgdZCK+5IZx6y/giNLN42nrKIzgZRlIMsMsDUMCFb9PmLCo0hs2mMWtJccG+AkVUbypGN1LkImT6kUjS1loFWThMSHZFAzNsT76qnnGmIqc71N4FK4z0lGsBtAG0GwKU13lnNsf2JrriS+ktSW0leW4n7YaPpsxlLMUwRr+72Z4Lo798AgIB3nkZ5qA8E4jWkFCqzVPACwVjPQZyp+LJYEIvnQVlsCZ1Zqr4gWx1TfIevdBlXm7lmSkfZHFi4oGx7aCufB4oIGN+3f8D7VhIIsbAU22KJMAkB33EMbn8YFCHicyUbwnVcQCIA/VF1ZlFqSLPyKy8cB7vNnBS82Os4G1X+ExBoM1KPyCiNz4ND2ifpvvvuG4x25B2pRlx2iynhYdLS0aZ7mAkTiqZW74zuKCo02EiSu6HYJjr6V/xeGmrF1O4dCXmECBFtKywnJtczaejbNUmzOiXbKPRtQFEUJlQXYcaksqgrbnQGQSUKi6Qqg5QJzZWXo6c0TZLGAUB7OsC1HADfdSraJnG5PENUClCNDkQS1DbOB++erejZ97a4PAeBYbrJwjBz5uRyXHFhLHtzUdU4oCzmGMCaChCxqKvmBkqXO1bTXSAEzSdPgQ960eb040xXon1GT7ZkCAQkTf0VNbJ7B8XnqajQgqKCZC9Fv30MmqquSuv2rrdNQ2bTOHbsWGzkZQBAjEeY1r1d/q4pNNJMm0lUMPDdTaDLxoKiaBAAbOMegGNh/dLShCAzISh69Ui3opBxJz1ALCegxSm+VHarWdPtVm6D4r6mu8/Kn/Xk0blkZg1CEQ7Hz8QJQmUuFjXHA9oK0LkreiN5QxFW1OeL505kDzShpwUYN0PHfsS/FBHQ1Caq96qQOOtLNWBlVBwa9NBXOAUUEdTTZCsuZ3/e35lxSQMnVhfh4hnVCAbDONWa8a76heQ40OuN4KSnB1ZavE+x+xYGe3wHSOXZmvuQIBDAc/0QGinum91qRnGhBa4+fV5x0rsyeH3pEAmN6upqLF++HLNnz4bDEfM1eeihh3LSoPxB+4IrVUCS0FBukeSnr73H6I6iMw0uakjnWVDxscxKf3Qq+aHu6I3Lphrt/KpLC3S7eyY/0ImK1hSm7pQop+J0NEstkdVUmi3Stf/+IPu4R9WI8aoQACpBlKlnGvFqSkKIYtCQ/VmH2yFasyShEX+J1e5if1lx5bTo/pUqu9jnkKVcM+aiv8RsgwTV/uNgC6NlV6Nt4LubAJ6DwKc/u/aTJ4B+CQ3t+1ZkN6OoIFloRMwlsHExlarPPhY21id7EKYZx+lAZYBFDZ6lIe2R5s6di7lz5w5GW4YNSZHh8bprNuaFEmE47Nrdgi+fPwalRWqut1IIqDRUVVeHhBkeAiEotJnhigsiE9LkiuUEAjOAylI7+gIMWNURbdKQVP7os49D0FqJMd5DcT/3rztSbkanOG7C4izo4rXgeAFhhoc9aqyXzk2pVTrV6sGhxh5cqJFKLBKtsxDv6sl89joIp2+mMSgMYHSbYCuJ2410Sk1VV4GAktOPZAsCsZqeL8iAIgL6AjysFrNsQKashSAhLxBRz5AQjyPihOeYM+16am3QgqIo1VlDZ+mFMPNhubhVd9EMaQNpy4zbEdtO6R1JwVU8E+GoN2MCOXp10gqNH/7whwiHw2hubsY555wDhmFgt+cqJ00ekeolS6VqOLBN/tzT40UwYsHJVg/mzdRR2zxJlyke6EQ3gZXzYebkcvR441IkpFN/kVjVBa2zkafMUhPi1uwungELl5jHh4qTc5ddMBYFJoKm91I2I7qdcqaRePx4m0ZpkQ0evyh8M7bJZIDbE4TbQzBjUnlCqIhydHmkWVQ5Cbz6jXf7xLYmqCkFXs7KOiToUDtefuE4MAyPvccTO9OeorMRMZdgtvqOkz5LVfgEypTduBpC0Natnt691xtBRbHoVEEiAdV1skGq0gcUpe5mLlAmMJbi6HUhKvciu23USmyYqzcnrSw6cOAArr76anzve9+D0+nEwoULsX9/9r0l8g/tSx7RWcGroEO7jCYJ+8F3nVQsVDygggCu8wQoPcFQasfQ8h1NQDkFUBqkE7/LtZMBjK0oRJFDX/Bi/ITBYbdg+vjihOPT0RUoUFg0dwLGV4mq0Nzqf0UhwPJS/Iv6TEOyC6XLeGqKy1bLC0QOwowdKbfEXyvli612HWvKCjCxJjl9i7dgYgoDt/b9OFMxH2cqvqynqbpwatgKIiwPZ18QvZ5sp2zPDIrSSmsqLm2uvBzNlbEKg46oO7RWpoZ0ZF6HIzfvTlqh8cQTT+Avf/kLysrKMHbsWDz55JPYuHFjThozXNCbY18asaq9sMyhdyD4uqUVo38ThQYJ9IJvPSxnsowwuRi5KkZBKbThE6uLUFGsEBI6X4D4a7D4kkkoidZelpIczpicmHm1skSczXIao/tswkY7dykNCqeI0JUi3bk0Nqr4mUZDuwetrji1ySBIjYQ5QIYODrqPEbebQnuiB1BJqUOOrxkMpGp8Q6X7U5p4rGZTgu2OUKaEWugzJpfhuqumJ9cJ0X/E2IH1NjAHpBUa4XAYZ58d805YuHAh+HxK/pUzBn7BpWf5jNOHxg4dsQbKIEE2MUq3sTOzeIX43Wk9P0mvm3KmEZ8nyG6JudxKArG/uf2j21ssJlx31XSMrUjMohpTS+Xe00QqWCSl4FB6M9H9nGnEo8stdIDE3+Ok0Wy2OpC43SycOwHXXj4V5dGBxMLZ43MXUaaC7LAwaEdMRPnsTx5ThLMnlmZ8DcqUAzENMj3PIVNPmc1meDweubNoaGhIs8XIgGThisfrxg+c6s58eyY4oBeC6MibmdYbSvECSJ25bPzV+YIkryedmZT2QmnzSD+qMptoFNoG7jUiCAQnWz2ybYKNm2mwTfthjtbc6/OlVoeYslRMaeZk9XoX6VE3WCf+MjDiO0oTTcFsonHVl8bj2sungqIozJ+lVTgo+wyeeNJuQPxzbT7/a7DPWaa9usazPP+Csfjy+cmFo1QPiAziNHSulylphcbatWtx8803o6OjA//v//0/3HjjjVi7dq2unT/77LNYvnw5li9fjieffBIAsHPnTtTV1WHJkiXYtGmTvO7Ro0dx/fXXo7a2FuvXrwcXdY9rb2/HTTfdhKVLl2Lt2rUIBHJn9IonVX4mvRACjO07ADOnt1azMkkUizZXes+QAaEwXCof7ASxQ4kddar19R9X8hhTGOqjH6SRspbwpkDh7AmlKNVpU0kFL5CE8qjxKjGhuxmlwdR1MyRoktsKfOlImGkohXCGenTt0W+c3YSm5L/SczGmPLvlflMhD8qyNNXQU3MkHpoSPbhkLAWglPXLdWCzmDCu0oHiwnTJQoeJTWPRokV49tlnsW7dOlx00UX429/+htra2rQ73rlzJz766CO88sor2LJlCw4fPoytW7fiwQcfxObNm7Ft2zYcOnQI27dvBwDce++9ePjhh/HWW2+BECJXBnz00UexevVq1NfXY9asWdi8efMAT1kf2dClE0JQwLox0b0X4937UwT7icfyBhg0dnhjNhOBT6rRkG1okhz7kYjS5iEyrjKT/LDJSPpoSurMkiY4OtVTWXgveMW9TqqvrbPGiJXKXSCiHhLiNNLcx1QsuWQyroyLBNfazVCP9E1hMVg0W+qpKWPUa7powZ79FVhnLZa/p6tflu56XT5rLC46t1q7Dk5+yAx9nrwcx0EQBJjNZlgs+l6g6upq3H///bBarbBYLDjrrLPQ1NSEKVOmYNKkSTCbzairq0N9fT3a2toQDocxZ84cAMDKlStRX18PlmWxZ88eWUhJyweD/ggN5RZypDAE2DgvmJC6ekPwusC7mtDq9CPC8rJ6pD9pD7RI//wo1EXSUpWZRN0VU3HJeTpciFMdrUBUwQSLp4jfo4eXclXpfd6z8V4o3SrjbUEsJyDi6YGN9cCcJjtr7iJK9EGpzALk3zK4UIV2c9KMMrYfSvXzUJBtW4bJRMseTnqgaZMijig6a+7ndSmwmTF5THFstq0RE6WVBdeUpAXIzROZdq///Oc/ccstt+DgwYPYt28fbrrpJrz11ltpd3zOOefIQqCpqQnbtm0DRVGorq6W16mpqUFXVxecTmfC8urqanR1dcHtdqOoqAhmszlh+WAgBXxZNF4eVaTIYMl1UzFi/fBzbTUH1/yZ/GLLBlfh/7d33uFRlfke/546PX1SCKEHkN6kd6UTXRUVdcW9ruLlci14VwFF2VWRslhYEa9efcB1ve5iKApyc2VVXCmKy4KAV1TcBBBCEkhImUmmvvePmTmZdmbOtEwmvJ/n0XDOnPK28/7e8ivKhUbYDWn/ZSe/iF+eX/07GzZIw+NYNgYNEBdOXoVy42Q41C7hYdAKyEpTSyqgrY9PfMdkC6GLf7HWDJ2lGp2uHIXWGnpfKh5LmjHhMwuIXmgofEVYzmaNxrms0eEvjJF4xrwoCqKCLAcjJ1hlb1D2XM+3NXV4Z4y8Ji9ojBTA9a16C6hr/PbCkubldsuWLdi5cydyc10jywsXLuCBBx5QtEQFAD/++CMeeOABLF26FDzPo7y83Od3hmGCVnqo85GQnR3ZlNOD/pp8XLKlo7a+RTI0CwfPs9DqVBBFHk4nAcMxEL0qmucYGI0u+4TaIGvx2mYWNgsPmxOorGxEj+5aiIKyjV6tmoe5JVDIqNUCtDoV1KpmH/sDByeCc7hGzqLAgxdYGI0GmFtsqHC/U6dTgWcB8UrrMQBkGQ0+7/Ck0eh33h+dToWcDA2MRgNsYKDTqZCWppHuEwozpOe3OF3X67WCbBlodSrYCCA2hK+fUOXIcqzP76LgqkcA4IVmiApnncHcukSDVqdSVO+e+vDUT67RIH0f5hYbznjVi8NJ0JJVjBZ1FnRiaz14P6df96ywdWixOcLWt+d3dVo6WIcVYmN8XVxIz1e52jZYVvF3EgqdTgWWZRQ/KyfbAKPR4FUeaRBFATq9ykfTztOXZWXqwpav6/qL0OlUKMhPQ3eRx5kaE5qabSCWbIjN1dDq1HByIn4xqSdUIo+/7P0eADB6SCFGDynEF5v3AwAys5S9L1LClo4gCJLAAIBOnTopXqI6cuQIHnroITzxxBOYM2cODh8+jEuXWkdr1dXVyM3NRV5ens/5mpoa5ObmIisrC01NTXA4HOA4TjofCZcvN4VVlQyGo8EKnUZAZXUjrDZlI36nk4XZZIHd7nAtefhtRzSbmlFT43ImaDEFdnRWCw+bzY4L7mu++/Fi8ChkQeBZJmg6LRZOSpP373YHB95phyjysFrtIE4WNTWNaLHapetMJgtUPCsdm5vt4IzdpDxI6bbZ0aTKCzjvz5TBBWAZBjU1jbh02QSTyQKTigsoE0dNI+rqXL/D6YQ2SL4YMDCbLGhptoWtH1HgQ15js/lGNyTEVWYA0NJig1WhMWe8sGb1xlmNEfkNJ0JeZ3Kn0WqzQxR4XLrUqjRhsbXWd01NI5yEoJIrBGzA9CF5EAVWKnfPcwozNWHr0Gb3fW4wynWDwTtbYDZZIDIOxd+PErzrsqXFBrPJArPFHpd3mM0WsEzw7ygYV640g2Vb+4eaS00QBR4Du2WiorIBFy65lHZYhoGTENTVmaDlww96WcZVJzU1jVCLruedr2nCqaaeELhOsLYQABbUXjZBJXJS/Xnq35OeuloTGE5ZxEaf97NMyMG27NrLt99+i2+//RZ9+vTBM888g++//x6nT5/GunXrMGzYsLAvrqysxOLFi7F+/XrMmTMHADB48GCUl5fjzJkzcDgc2L17NyZOnIjCwkKoVCocOeKyoN65cycmTpwIQRAwYsQI7Nmzx+d8W8Aa3IFpWPnA7bL3ysyGpLjafsPRBrMV9U1W+K/SKhUYgPxU1CHqwWbkw673V4X001qSzvpNgb2m4KphN4AvGhTwjnLjZNSkXRM2jTzHtvoNcidYydp5ogmwofCqn2gGHLHC5fWS4omE4rrhnTFlaGHQ3wLMNLz+HWrPIh5YBQPMKiOmDu+MycM6J+w9JOAfsaHU5shD4B6C629uhgYjr1GiQhucwcWupXrPspReI6BPl0wQhoOVb+3MwxkZJ2pPQ3am8eCDD/oc79u3zysxTFgvt2+99RYsFgvWrFkjnZs/fz7WrFmDBx98EBaLBZMmTcLMmTMBAOvXr8eKFStgMpnQr18/LFiwAACwcuVKLFu2DK+99hoKCgrw4osvRpzJaGAEFTLHz4Plo71A0/fKbgrTeAvqj8H2kx2M2leKe0YkOrVVSTC8oMg1dycrQOg1Brh8QNkNfuejdXkQjtxMDfp0yUTPTq3uKvgeIyQVRoPbYrwgWwvHWfnnJFq2JHubIhShVDQTZREeya5GmlaEwxY+tGusxK2KIiwiJkDZQKaTZuBKpMI66Nk5A2kq+cFqr87pOP1zvddeR9siKzQ+/fTTmB68YsUKWcHy4YcfBpzr27cvSktLA84XFhbinXfeiSkt0cKwXNx7JWfdeenfDifBjz+3xphQ6p4kKLLJdD3Tf2TZGjEv9GP4BAkNlmFwTVffjTsuq0j6t04tYO7YbmixOvBTQlIgj78tdyrCMgwcrKgoHndCaZMZY2LryN/VuYdEemAORc9O6RjQPTvsdQmaaITf06ipqcGOHTtw5YpvJLmrJqJfBI0+XNNVi77F7e/jqNkS/bqs3NTabpf3fxXsPv/L8rN1sFZEnayY4DkWDBN6iS7ZtgKJI7acMQxczgMJkfFWG/1zw9EtPw1VdWbF10dNnH1OySW1xtAXdk6Fbpe+8DnPMsqWrj1LnB2lrYYVGosWLUJ+fj6KiorCXdpBiUBoEAKHU97LUIA2WDxbkcyzJCd7iuNgMLiYNhA2XosbxnVHvcmKswAa1J3ilNDICFtEHeVLjDMMw7jUqv3Kpyg3/to0/gwpzkn4O4D472nIvsdTln6wXOgBlz82e2iHl/Em0j0apYQVGjabDRs3bkzIy1OCCMrdSVzLTXKbjIlcH5dLpkOyQVBmx8EwQLPKNfX17GeUGyfHnsAokZshtYrmBHwY7kdb7c6k2V4YM7VAbXyfWTKuW8z2NZGSqM1Yb+JRQ/6rAEqI1O1ITMvPXiiuwmR5ue3fvz9++OGHhLw8FYhGWstp3NjsDpw6W9faeNqgP2p18SRjXRome4naCI8XiUidp1rOXoxcXTFeDOgu77QwU69yWQ5HCMeySbDibt/tBwC65BnQRYELkW75acjN1OJ85gicyxwZaHUfJq9do6izWEhUyYcVr8OGDcMvfvELGI1GyTIbAD75REG4tg5ANN9YuAGF0+nS5G2LMSyRGZD7R+zzEOAdNcnffFuPjF24Cs2R1Bit8uO5vKy2cwroTzuJAyQRKrKeUsJ7SnZlQhRYqEUVrM0uARPpgCpeAzB/4TR1WOe4zWKUEFZobNy4EevXr0eXLl3aIj3tDhKFR6GwsRMS8CHJf8yePY3Q+WCkv/5uRJQlNistQcF3IvOOEvdXp6buVHsicRXUaLbhckMLMg3RtT2NyCtSW/W0AaF4HFieB47WAwiSM78TvQrTwTCMj4ZkIkjTyaheJ+jjCCs00tPTMXu2vI/4jk5ClmSVxWGN7JEyz2sdgMhthIfWuVUyOJozpmvClrGSuTrGxMsvSDTvbqfLgonaXI0GT7CsaLUOu+YrXS5y5ZlNy3W3c7fQCNMpD+jh2huMu9BQWAVJW56aPHky1q5di+nTp0MUWyVa//79E5Sk9kUiPhJPNxRPR2uykfn8nCj6pyGcnYaSNXCBj9xqXjlt30n5hfqgeBOpAVwbFGK8rPYZUQ1iD2HXIpMVjmXgcMoHPFOLfHLaUoLsSMIKjV27dgGAj2dbhmGumj2N8OEcmPDLUf7EdfAqmZsGf5Xk8VxmI9z/rFwEvSQR7vWJ7JTaq8xgtOng8oql44vpg6BTJ9sxe9vjcQQYLw03od91gNMBVLzn/yLXH5n7uhWkodlil22LM0bG11wh2e0yrNCI1TI85UnA+lQ8ZQZxi63g2lHEazYTXVNLktGrRDJlVjLjRYR6t9hvqs9xs5gFVhN9BMPrRxQptiFIdoflDcswcBASP/9gHA+GD9wfCPd0gWMhRODSJdUJKzQ2b94c9Py//Mu/xD0x7RF5E7goZhh+xKWpS7u1vinlOBYOh8P3Ou93SzG4AbOYg6a04LGdk93gGcZlbOhgRRReOeL9i9f/44skZhOc9W75abDYHKi83DYhjOXQa5QHHgJcMxsbp4mrpXk0eOon1u9Q+ftSq/NPWjwNbxsNq9WKI0eOYNSoUYlJTTtEdk/De1Uo0tWpuNppeDp/33SqRB7m5lahIWdkxQCoSh8AnUzHkfTlKbQaG8peEGcIIWhqtiV8VK0WOVmV0fbaQTEMg2YxK9nJABC9jkJRrkEmpKr8ELE9wXHK0pM0i/DVq1f7HNfW1l49fqcQPNwpEJPMaH12VHf5vtGThgD7CnfsbzvnUkeUM+4L90Eku+9S2nkShovIlXw4fq5pCmsl3ClbB4eTSH6WIoXRGODUZgA1x4P8GNUjrzKi+4J06sisvzMMKtSa2tYFSCgUW6Ina6bhT1ZWFs6fPx/+wg6CbKcVgzpm621xnFb7pdOqyoBVpUFeb/ciQtgNfbnHpkbv5WB4MAwPC28IG5ZVMWHqN00nwmp3AnXRPV7sfz3IhQsAgggNrxrp0SkdxElQfjHQ02qqoVMLMLVEGwDAl3hrQ8u19RF988BwkS3hJYKu+Qa0WCIYGCl0qBgpEe1pEEJw8uRJZGeHd8vbUZDtM5PUmXo2vr0S4k6On0oty+PaKde3XsXIuEZPdfM1d/JZlsWZzFFgnTZ0vRwfoaGkZGJvBfIzWQ8iz6Z6LQEAehdlgGEYfH82SinrR1uVCccyUIW1Gk88Q93BmZSSKNftEe1pAEBBQcFVtTzVuuHqFxLUey8hwiFPTHsasuth/nYYcstRniMSdRLaE84Ao4r4CfO2sOuT/7BTY4YXCXHfH2ujxpsqs+0AkmUR7r+ncdXhpWXkO8CPrULMLXZcbmiJIjksQLzXV93p8Ot8/DsjWW+xsg4NUwNPyE2D1qVyKrcHFQ2K+qQYX9fOJrIpRaxaU4w2HcRcH/66MNb54qAZaI9Cvs3DvS5fvjxEYhg8//zzCUlQ+6NVtZMEOx/F132mKnrvqYRXgbF6u01wR+bjfasywAJc3mQ84NS1fXNhkPNn084QeRY9OqUDah1ONsq7U4kGJRb7Mb8t1J5ZjFw3vDPsjlSfSyYOoc8EEGszbP/3md9AzB/fuhg/sMDHbY4nRHF7o81VbouLiwPO1dXV4e2330ZhYfBg9h2RsMK6jYeETkELztqq16/TCMjUqJCWrkV1Vet1bJiZhjSB4lybZd4GUoXG8G6i2wN8z5GAzQKc/Qa2BCxPKfOgGtv7/OtJqVabN9cN74zMTB3sFt8N5lAxxCkAwwlgNALE/tfBaZIPXuL/iedkaBKcsvjQ5iq39957r8/xwYMHsXTpUpSUlMjG/u6IyBd8GFXVOBj/BYPleFxMHwS17QoyzGcBQpCmE8FwvlUZGPTe3z0IUJU2AJ26FqIXr0ZuZvscLclBwIDLLITT7HYGJ0nBthXisb9OZiPcX4U6xBMMWhGZaWrU1MRHK6m90r0gDXq9Cid+rInrcxm1HpxafqDEROHpuj2QKG8OYfc07HY7XnjhBezYsQO//e1vMXPmzMSkJNWQtjrkP/pEbKR6jKsEh8s2gGU8McB9W0jATCOIy3OzKgcOXoPB3duHsVZUkNAx0OO1+arXCGhqjn+n7C/cWxUs4v6qlEclcBBkomIGoyBLB5XIoSJGVeU2CD6YEBK1TxmyOM6cOYPbbrsNJ06cwI4dO6ISGE1NTZg7dy5+/vlnAK4ZS0lJCaZPn46XXnpJuu67777DLbfcghkzZuDJJ5+E3e5at79w4QLuuusuzJw5E4sWLYLJ1LYuF8IWvJzxX5zqy//9mQbXhq/IszBoRWR7YgmwXMj7/B0Weg7jEMMmubjXoj2edv0j2gVz2V7cOUMy3FM6hc9JD74kEdyyWDn+9eRsB/YA8aZzl0L0GHJtfB4WwYel0/BQi/GwVUhNCZ6oSbes0CgtLcWtt96KadOm4U9/+hMKCgoifvg333yDO+64AxUVFQCAlpYWPPHEE9i0aRP27Nkv+T/QAAAgAElEQVSDkydP4vPPPwcAPPbYY3jqqafwv//7vyCEYOvWrQCA3/3ud7jzzjtRVlaGAQMGYNOmTVFkMwbCBS+SrZj41JinT1IJHPp2yYTWbc0qcCwKc3TQqt0fhZ/QEHj/SWTw5aq2jPgVKyZVbqsLC8nMpHWmdcP47hhanCNzd2v+OZaJq/pnkVGPgixdVPd66oGAxSV9MS5mDPU5DwB80UBwBX1iT2iSyB4xA4YeAwPOJ9alPhC3bzDZXjujJFEugGRLY8WKFWhqasIbb7yBYcOGSf8NHToUw4YNU/TwrVu3YuXKlcjNzQUAHD9+HF27dkVRURF4nkdJSQnKyspw/vx5tLS0YMiQIQCAm2++GWVlZbDZbPj6668xY8YMn/NtCePujLmAaXHoDcvW+orPSLR1n9qjteU+JJJ1GwDXTCQvU4v8bN9OLGBPw50dZwpNNarT+uGK1jeCJKPLAKvPBtdlEFiGCdzwd/91yvneisN3pdMIkjCPFO9lxUZNIboU5QVcw+X1AlfQO+r0tVe65RvQs1N64l4QY90WGfXISVREyjYgUctTsi09HvEyVq1a5XNcXV0No7HVqjE3NxdVVVUB541GI6qqqlBXVwe9Xi+pk3rOtyXE3bsKHAObl6YrcX/sHONEsJXuVntrfwvuyPCMFqRgSqzvcesLXTMR6T7/0ZFf+xEEV5maWqKLetaWDO6VA62Kx6FvLwao1DIsD6HvRAVPYQG0umAwuzWN4jbTivb79Nq/v2Fcdy+zILlBSseBYxmf5T2eY+MS89tDrCWmUvGyjjyvZmSFRiLUaoPpvXsCqURyPhKys2NTH01L18Ms8EjTq5FmAGx2JxpNVlgENVTEijSdGNSfv1rFgbE4QFgOjDN6R3oqkQMYBhzHQKtTQTDooLOqIFo4aHWtMRRUGTpYWlqP1RlaaI2t6/uWBj3qhNbqTk/XQwcVDFoBRmPosJc693vCXZcoPO89XlEH0SJCNPFwMqxsekSvfAoCC2OGBrxai7PnXFo3Wp3K5xolaLViwD3e5W+3OyN+ptFogMA7cV7gQVgWeXlp0m8Ou116ntFoAHE6fI7lntcWRNseanUqCBm5ECtddkqe8vPk65ruWXA4CX44E9zNSM6g0XA01kE8rcxNjFYrguNY2XrLCtN+jNdeD97RAlUS2n0sdRmuncRKmzpUycvLw6VLrRVeXV2N3NzcgPM1NTXIzc1FVlYWmpqa4HA4wHGcdD4SLl9uijpIi9FoQEOTFVabHS0WG/KztKi50gyrzQ5thhbE1gAVz8LqPQVxmwHyLAOrzQ7CMmCc0Y/mWRaw2uxg7QzMJgvsnA0mkwVqiwVmwSJdZ2mywmHyOm5ogamm1YiwvqHZJ53NFidGDMhBl8IM1NSENjY0uZ8b7rpEYzJZYLe56oMwnGx6fOqDcMhK16DZxqBTthYsA9h7TMLZC98hvz6Yo8DgNDdb/eoZMHuVt8NJAn4PR01NIxob3fXilx+H0yE9r6amEcTp9Dn2x2g0tFn9RNseSJ+ZsDOA1eYqd0/5efJlcTsylCvHRrEAxr69Yf38S0XvM5ut4Nzfoc9593sdYdpPnZmBSpcLtHG7j7UuTxtGgAHBNVE+g2WZkIPtNt3hGTx4MMrLy3HmzBk4HA7s3r0bEydORGFhIVQqFY4ccQXZ2blzJyZOnAhBEDBixAjs2bPH53xbQgIc/blQ6Q3o3rML+HxfI0gn45LDsWrVePCseXsmXRzvOna6LX3Z7C5gNAawOd1CP8ffNxXLIdOggrodOGKLjOjXgdQiB1HgwKh0cLDKDd+yE7iuLeeiwnO2IxnoMSwbtWuLaAzVYl3ST1XXOg5ODTuXOAPENu0xVCoV1qxZgwcffBAWiwWTJk2S1HjXr1+PFStWwGQyoV+/fliwYAEAYOXKlVi2bBlee+01FBQU4MUXX2zLJAdoJXmkBgMW4jWTYam+4POzgxXBOmytQiPGhufZm1Cr3BvyrMfXkltlVNRA6D4cxH824/9x+h+zqSYsoqPVUFzGHkIBxgwNLLbWJcZmIRN2Tg2h31gQqwn2019FXc3yvqdY9BowAGJe1+ge3IHoUZAWVHU6HFy34RDSjcDZP/ue73QNGDF1N7iTTZv0HN5xxseMGYMPP/ww4Jq+ffuitLQ04HxhYSHeeeedhKYvJKIWl3U9ocrUAvg5MBSov549w+F85ghoDc2A6duYN+MIw6JbfhoE9wyDYVlMGdYZqtpmoKrGOyGhH+T3c2bn7jGmrP1SndYfrNOGnCYvD80xCm/vuy9muGKUsNp0EP9BRdjn+HoKkEbe/hbgDANd75HBE5BkhhTnIF0XfUxyOdiMAhCrGf4BSkQhOtVcVpMW1C8U36mvwie0o0JvR1wdw80YadAWIY83uZVvfF1xB3MMaOX1AG/3uS56GF8DJYZBuk6EvZ516wIFn9EwGl9VRs/0nldp0H/OHTGmKfmE2qUyqYzg7S6LebnSj9ixoVf5DinOCYiepngpw7Xl5XucYnTLTwt/UQgupg+CaDcB8PX3xPd0h5E+1RrDJxb/STF7eU3BumkLqNAIg6fREvjuLXg6CXn3Fe6ZAROd23+VwMFicwR6p/V8CAGaZa3XCf2vA6vx/bBZj51JSG+eweE4Fo52YM8xdmABLFdE1P4j9HU8x0LwbNfJzcQUdPLnskYBYHCN6hzAOgG43Gj7dJoRDgr8ZQbrnqk0aMJpK3acHqxZzHIbafoKDc+3VJijx/lLTVE922cmF+ueRmy3d1io0AiDyj3Kb7E7fVuR7J6FewbiHolGaqWhUfHQqwUIPIsLl00IbLrBOz9v4eUvMABXRwogKk2y6SOKYGsHQiM3Q4N6m+juauQ/6blju6HuUi3Oevu1U9C552dp0dRsk3xMeTYTxYEzYKmrBSp2yt6rtIPhWEZSYgBcy43lOZPipjiRKmToVRD7zwCxNqOg6a8+IWANWgHGdA1q6psjembPTulgWQY//uxyYhn7TOPqqhOlUKERBoPWZdzD8bxrecprIzwUnpG90nanVfEwW+xgGQbZ6Wo0mt0fUcBMwy2UsovgqDoNLquzz89cbs+Q6VESI8IflchBhUS7fFCGYkt7xu8qGeHuTYZeBY5lgzomjEfY37xMLXQaAf+80Br4h/E8I8xzGLkZUwoyZ0xXsCzj0qYStUjXi0jX+2mJRZFNnmeD3mbQimg0W6NKKyUQKjTCoFMLGN0/H5msGTh9qnXW4JlJyGjlMBFoT/XolA6rzQFzTZPUqct2jh6hoTZANewGn59UI26SfQfvXgYhUdqstDfCRegT3JunGhn3Hp5S0KkFn1GuvyF9H7eVvfzblPduHmeTPndHIHTqtN1gVuVgsOI72ieJ9znlwhO9slOODoRo8cO5K5Hdn6Ly+foRRVHbpimBCg0F5Gdp4TTb3O5C/DbCZZYVWEbZh5GZpobIs1IlO9x/nbwaQFN41VmFcJxnbybFhYbCEbdOxaN7QVqr5k1Aubnuz8/W4qfzXiN/v57imq4e1yxRljvLSHUajEg6piu6blGlob0j9B4Hp4Kwq+EIKErP0i2itblITamhT7DrEyo0FOJxXCj5B3R3QoLAoyBbBxCgstYkNVTB7Xo7nMRnJYeIrvtaWB3YtFy0ONKAyktuYz6v/YQohz+BDhdTE0/uw2k/EeKESuCkqQOblguHKdA9hf9T5DyDRiOrjekaGLQC/lmpIJ6DgnodUpwDuz3FhX4Q2LRcsGm+nh7iZBrrc9Ql1yB9Z8ruTk2hkWio0FCKvzGQ10eerhNh9nP859lAd5LQ3ZtnCs15wq7yKgi9x8Hy7SkAgMBx8BYabEbkLupdyXV7wdXHX7++TZGxawi4TNAAvApC9+HIKOqES/U2OCq/l37vWZgBlVkAK6rh0Ypi9dlgWywAIgjaE6Kzz04Pb0DmEVIFWeEjJ8aq6nrV4Vc3EXsipjIjKFRoKIRheahG3ATn4b8B5tOSl9hWbSnP8o/rr1pwTRFZhg29IOSeabAsi/wsLbQ5ro4hK10NE4B0gxqw2wCWhWrYjTFkgHF5wU3R2ACBhNk45niohswGALAqLRjG1w9Pv+ICOLKngFHrgfLtAACh70TY6q8A/zwX+DxpqSP8hnoksCyDGSO7uGZFFC/8NRmCI/IcrHYZh6Ax+xGJ7faOChUaEdLZqENjix6ZHn9Eksqr+9BtByGKPAqydBC1Opz5WV510KWaS8CIWmToHWCIy5lahl4NXZdMMGoBpAmxx471JLCDrG5EbJwHQOg3BYyX+xQuqzOIxTcSpCDIxN1AZNpwkaBJOf9fbUDrOmRIOufqfbTRgj2E0WWCBFmapERHRxl2thksiEsNl/E1HvOMRFnikM6n60WIfp0Q77W3oBI4ZBpctgCMyrU8wWZ2ct/veYycMV+kKPwK2zkqgQUDJiDIlBJYbYZrZhEEj0aPv6W3B48CQaDiQ2B5GtM1yM0Iv9xEkScWsxWtRwh79hd7j4M4YFrEz6F7GsGhQ5xIIb7aU56G1TrT8AiN4CNT70AznbJ1YDkOxOa6XhxWAkhaV743MkKcHKylUHjXYPAci95dMsBq4tQpMwx6F2W0HsoIDaekAOFXoVJ7aL3Pey+D0abDx5dSqupxtjGScGZckQuZIAarOrUAIYiCR+dcvVsBxf1tcgLQAWOvJwsqNCJF6nSD+x7yeIRq9U3FIiddA1FgceGSCYR4YvkRV5v26qS8l048jZw4bOB7jACry4ot3bFax7YbSKtBXFzwjxce/LkaFY+cdA3SdH5GaLwKjEoLLr8PcOZTn5/EYSUAGHSv93aXQYWGEjgvdVm+KEh8cY5FUW7wWSPLMGC58AaT4UhV1+iJhgqNCOEL+8Fut4BNd6sIuhsWz3HIz9KivClwryMnXS3NLhxO4vJHJbnL9XTmvjMARnBrOdkt4LKKYk94R2n/QUb28aZHp3T880K974yBYZATRBuKYVmIA2eA2C1BfnN9Xj6b3IzvEiUlOKFcoffqnKFw+aqjNPr2BRUaEcKo9RD6TAg8zwswDh8L1qqC0UIAxqMm6+rkOI6FQSsiU69yOWMj7pGM3GjGsxwVt7gXHeQDivvymvt5vHv5gmEg8ix6FWZAPSA/8PKYtc8Y9CxMD3/ZVY5n9hfMLTrvJzH0agFNLYGuX+hSYGKgQiNW3A2TEIA1GGEEYATgNPu6LGAAFOboAhoy41HJ9esMGZYD3+NasLpMxIUO8wH57inFDK8Go00H33mA69i9LMgLvO+meFhL9EiMxiih4DsPAK5cQI7ZivQgrlf8KTTq4SQEfJdBYEQtbKdd4WBjXl7qMN9MfKFCI2Yi7ERYzv2be21epQMaL7mDz/ji74wwNjrIB+BWaY7Zg6kbhmUh9pvqdcxB6DM+yMapu/w6jJ1L+4XLLwaXX4ycph2KrmcY1x6InLNOSnyhQiNRSKMUdyfDsoDTCYbjvTyvutRAkWkDZ+yR4PQk9vFtRhsof7EGo+yLoxFWjD4LcDpAzPV09JpEOGN31yAtDGlaEQ3UK64sVGjEipyvIi87DnHwLDjrq2Cv+Idrj4IBHKwAJrsb2Jwu4HITLDAAdBSTHMaQDc7YDVx+77Z9sSd4lZzQYFh0ytaBaNIB+Bpzin0ngditsB77CB1HeqcefNchiq4ryNEhn2hB6yo4HaMnSSoyDUsSJsRlY+GJJc22uh9hiwb7qtkmkg7S/hmGBd91KBgFI8a44l6uYo3dg/7McDyyB0+EcdjUoL9L2nTxsrehJAwG8o4rKXSmkTgktx2Sbq3r/5zb+y3T1kVPP4JYYDgB4vBfhLwmlDNJhhPAdxsGNi0XxNYS7+RdlYgDp4M0NwJqHYg5AieTSqGfTFBSYqaxa9cuzJ49G9OmTcO7776b7OQoxE9oSFscHOwFg3AxfVBIXfS4p4aOnGKGYZiYypHL6QpG1IDVZcZPK+4qhNFnuazEVTqwGflg1QZwWeFirFPiRbufaVRVVeGll17C9u3bIYoi5s+fj1GjRqFXr17JTlpovJan3Cekn/oPH4Iik5V6NqVQokDsOynZSbiqafczjYMHD2L06NHIyMiAVqvFjBkzUFZWluxkhcfLfsN13PoTz7HISqNr2xRK+4bOzoPR7oVGdXU1jMZWFcjc3FxUVVUlMUV+eIzBOvX1+8HT4Nx2Bf7LVRQKhZKCtPvlKRKkk41kXTk7O7hTM6UYjYbwF81aEHCKECca8/Kh7tofYrYBNt6ExkoVBL0Ig5JnJoBanQrqwt7Q+r1fUR5TnLbMY63OZcWc1cbl2tHqUa4cw+Uz1vL33J9pNEhhntua9lyX7V5o5OXl4e9//7t0XF1djdzc3BB3+HL5clPYON1yGI0G1NQ0hr9QjqIxsDgB1DTC2dgMm8kChjGjJZZnxsI1s2ECYPJ6f8x5TAHaOo8Wk8t5oaMN39kR69FisgCc4FOOSvIZa/l77rfXNMm6yk8kya5LlmVCDrbb/fLU2LFjcejQIdTW1qK5uRkff/wxJk6cmOxkRY5nxEKXpzo84uBZEAfPSnYyUh5x4HSIAyMPngS0BjWjxJ+UmGksWbIECxYsgM1mw7x58zBo0KBkJytyPJbETpl4xpQOAzXgiw/RGnCKQ+fGx10L3QcPSrsXGgBQUlKCkpKSZCcjNjzTXI87CgqFkhAYGqUvobT75amOQmusbyo0KBRK6kKFRltBhQaFkmLQ9algUKHRVriXp4iTCg0KhZK6UKHRVjAu7SlG1CQ5IRQKRQnUX1twUmIjvCPAcDz4XqPA6rKSnRQKhUKJGio02hAuo1Oyk0ChUCgxQZenKBQKhaIYKjQoFAqFohgqNCgUCoWiGCo0KBQKhaIYKjQoFArFC87YLdlJaNdQ7SkKhULxgusyBFyXwclORruFCg0KhULxwmXURw375KDLUxQKhUJRDBUaFAqFQlEMFRoUCoVCUQwVGhQKhUJRDBUaFAqFQlEMFRoUCoVCUUyHV7ll2dhU52K9PxWgeewYXA15BK6OfCYzj+HezRBCSBulhUKhUCgpDl2eolAoFIpiqNCgUCgUimKo0KBQKBSKYqjQoFAoFIpiqNCgUCgUimKo0KBQKBSKYqjQoFAoFIpiqNCgUCgUimKo0KBQKBSKYqjQCMKuXbswe/ZsTJs2De+++26ykxMTGzduxJw5czBnzhysW7cOAHDw4EGUlJRg+vTpeOmll6Rrv/vuO9xyyy2YMWMGnnzySdjt9mQlOyrWrl2LZcuWAZDPy4ULF3DXXXdh5syZWLRoEUwmUzKTHBGffvopbr75ZsycORPPPfccgI5Xlx988IHUXteuXQug49RlU1MT5s6di59//hlA5HXXbvJLKD5cvHiRTJkyhdTV1RGTyURKSkrIjz/+mOxkRcWBAwfI7bffTiwWC7FarWTBggVk165dZNKkSeTs2bPEZrORe++9l+zbt48QQsicOXPI0aNHCSGELF++nLz77rvJTH5EHDx4kIwaNYosXbqUECKfl4ULF5Ldu3cTQgjZuHEjWbduXXISHCFnz54l48ePJ5WVlcRqtZI77riD7Nu3r0PVpdlsJtdeey25fPkysdlsZN68eeTAgQMdoi6PHTtG5s6dS/r370/OnTtHmpubI6679pJfOtPw4+DBgxg9ejQyMjKg1WoxY8YMlJWVJTtZUWE0GrFs2TKIoghBENCzZ09UVFSga9euKCoqAs/zKCkpQVlZGc6fP4+WlhYMGTIEAHDzzTenTL6vXLmCl156Cf/6r/8KALJ5sdls+PrrrzFjxgyf86nA3r17MXv2bOTn50MQBLz00kvQaDQdqi4dDgecTieam5tht9tht9vB83yHqMutW7di5cqVyM3NBQAcP348orprT/nt8F5uI6W6uhpGo1E6zs3NxfHjx5OYougpLi6W/l1RUYE9e/bg7rvvDshfVVVVQL6NRiOqqqraNL3R8vTTT2PJkiWorKwEEFiHnrzU1dVBr9eD53mf86nAmTNnIAgCfv3rX6OmpgZTpkxBcXFxh6pLvV6Phx9+GLNmzYJarcbIkSMhCEKHqMtVq1b5HAfrZ0LVXXvKL51p+EGCOP1lmNR2xfzjjz/i3nvvxdKlS9GlS5eA3xmGSdl8v//++ygoKMCYMWOkc3J5SdU8Aq5R+KFDh/D73/8eW7duxYkTJ6S1cW9SOZ+nTp3Ctm3b8Nlnn2H//v1gWRYHDhwIuC6V8+gh0jbanvJLZxp+5OXl4e9//7t0XF1dLU0pU5EjR47goYcewhNPPIE5c+bg8OHDuHTpkvS7J395eXk+52tqalIi33v27EFNTQ1uvPFG1NfXw2w2g2GYoHnJyspCU1MTHA4HOI5LmTwCQE5ODsaMGYOsrCwAwHXXXYeysjJwHCddk+p1uX//fowZMwbZ2dkAXEswb731VoerSwABdRSu7tpTfulMw4+xY8fi0KFDqK2tRXNzMz7++GNMnDgx2cmKisrKSixevBjr16/HnDlzAACDBw9GeXk5zpw5A4fDgd27d2PixIkoLCyESqXCkSNHAAA7d+5MiXxv3rwZu3fvxgcffICHHnoIU6dOxerVq4PmRRAEjBgxAnv27PE5nwpMmTIF+/fvR0NDAxwOB7744gvMnDmzQ9Vl3759cfDgQZjNZhBC8Omnn2LkyJEdri6ByL/D9pRfGoQpCLt27cLrr78Om82GefPm4f777092kqLiueeew7Zt23yWpObPn49u3bph9erVsFgsmDRpEpYvXw6GYXDq1CmsWLECJpMJ/fr1w+rVqyGKYhJzEBnbt2/H4cOHsWbNGtm8nD9/HsuWLcPly5dRUFCAF198Eenp6clOuiJKS0uxZcsW2Gw2jBs3DitWrMBXX33VoeryjTfewPbt2yEIAgYOHIiVK1eivLy8w9Tl1KlT8cc//hGdO3fGoUOHIqq79pJfKjQoFAqFohi6PEWhUCgUxVChQaFQKBTFUKFBoVAoFMVQoUGhUCgUxVChQaFQKBTFUKGRwkydOhUnTpyI6t7GxkYsWLAgzimKH9u3b8fkyZPx61//OuR1ZWVluPvuuwEAGzZswM6dOwPur6ysxNy5c3HDDTfg6NGjCU/7ihUrcPLkyYDzDocDixYtwowZM/CnP/0p4ue2RZ19/vnnPh5XE8HGjRvx17/+FQCwbNkyvPXWWwl9nxzHjx/H008/HdE9a9aswVdffZWgFKUG1CL8KqW+vj5qgdMW7Ny5E0uWLMGNN96o+J6HH3446P07d+5ETk4OtmzZkoCUBnLw4EHcfvvtAeerqqqwf/9+HDt2zMeSWymJrrOmpiasX78eW7duTdg7AOCrr75Cr169EvoOJZw+fTpi/02LFy/GnXfeiffffx9qtTpBKWvfUKHRQRg4cCAWLlyIAwcOoLq6GgsWLMCvfvUr1NTUYOnSpairqwMATJo0CY888giWL1+OlpYW3Hjjjdi+fTt27NiBv/zlL7DZbKivr8f999+PO++8E9u3b8fevXvBsqzkNG/t2rXo3bs3ampqsHLlSvzzn/8Ey7KYP38+FixYgMbGRqxatQo//PADbDYbxowZg8cff1xytuahsbERv/vd73Dq1CkwDIMJEybg0Ucfxbp16yTfSnV1dfjVr37lc9+GDRuwa9cuZGRkoGvXrtL5ZcuWobi4GFVVVT73b9myBY2Njbj77rvxzjvv4NNPP8Vrr70Gm80GtVqNpUuXYujQoXjllVdw7NgxVFdXo0+fPli/fj1ee+01fPzxx3A6nSgsLMTKlSuRl5eHu+++G0OGDME//vEPVFZWYvjw4Vi7di02bNiA6upq/OY3v8G6deswePBgAK4O+b777oPdbsfNN9+MV155BYcPHw5a5gDw+uuvY8eOHeB5Hl27dsWaNWsC6uzo0aNYt24dmpubIQgCHnnkEUycOBHbt29HaWkpmpubodfr8eKLLwZtA/7893//N8aPHw+NRoPnn38eGo0GS5YsQU1NDSZMmIDNmzdjzJgx+PDDD/HJJ59gw4YNeP/99/Hee+/B6XQiIyMDTz31FHr27Iny8nI888wzMJvNqK6uRt++ffHyyy+jtLQUJ0+exLp16yTBefToUcyfPx+XLl1CcXExXnjhBWi1Wvz0009YtWoVrly5AofDgbvvvhvz5s3DV199hVWrVkGr1cJsNqO0tDSk4WKwNGq1WvzhD39AY2Mjli9fjtWrV6O0tBSbN28Gy7LIzMzE2rVrUVBQ4PMsg8GAoUOH4i9/+QvuueceJZ9mxyMZ/tgp8WHKlCnk+PHjhBBCevfuTd555x1CCCEnTpwgAwYMIC0tLWTjxo3kqaeeIoQQYjKZyCOPPEIaGhrIuXPnyJAhQwghhDQ1NZHbbruN1NbWEkIIOXr0qPTbtm3byPDhw0llZSUhhJBnnnmGPP7444QQQhYvXkzWrl1LCCGkoaGBzJkzh1RUVJBly5aRP/7xj4QQQux2O/nNb35D3njjjYD0P/744+TZZ58lTqeTWCwWcu+995LXX3+dEELIL3/5S/I///M/Affs3buXzJ49mzQ2NhKbzUYWLlxIfvnLXxJCCFm6dCl58803A+7ftm0bWbhwISGEkPLycjJ37lwprz/88AMZN24cMZlM5A9/+AOZMWMGsdlshBBCduzYQR555BHp+M9//jO57777pOc/9NBDxOFwkMbGRjJ+/Hhy6NChgHrxRmmZ//WvfyXTp08nV65cIYQQ8vzzz5NNmzb53F9bW0vGjBlDjh07JuVj5MiR5OzZs2Tbtm3k2muvJY2NjYQQItsG/LnpppvIl19+SQgh5PDhw+Smm26Sym/cuHHkhRdeIIQQ8tBDD5GPPvqIfPXVV+TOO+8kZrOZEELIF198QWbNmkUIIWTNmjVk586dhBBCrFYrmTt3LikrKwuom6VLl5J58+YRs9lM7HY7uemmm8iOHTuIzWYjs2fPJidPniSEuD5uSFkAAAb7SURBVNrXrFmzyNGjR8mXX35J+vbtS37++eeAPPgTKo3e7eK7774jo0aNIhcuXCCEELJ582apzPz57LPPyF133RX23R0VOtPoQFx33XUAgP79+8NqtcJsNmPChAlYuHAhKisrMXbsWPzHf/wHDAYD6uvrpft0Oh3+8z//E59//jkqKipw6tQpmM1m6ff+/fsjPz8fANCvXz/s3bsXgGsZ5rHHHgPgGoHt3r0bALBv3z6cOHECpaWlAICWlpag6f3b3/6G9957DwzDQBRFzJ8/H2+//TYWLlwom8dDhw5h2rRp0Ov1AIBbbrkF77zzjuIy8szEvGcvDMPg7NmzAIAhQ4ZIM6LPPvsMJ06cwC233AIAUqwHD1OmTAHLstDr9ejatatPmYYjVJkfOnQIM2fOlFxELF++HAB8vNoeP34cXbp0kWYyxcXFGDZsGA4fPgyGYdCnTx+pjOTagD/l5eXSzG348OGoqqrC5cuX8cUXX2DRokXYvn07/v3f/x1ff/01nn/+ebz66qs4c+YM5s+fLz2jvr4eV65cwWOPPYYDBw7gv/7rv1BRUYHq6mqfNuXN9ddfD41GI+WjtrYWFRUVOHv2LJ544gnpupaWFvzf//0fevbsiYKCAhQWFoYt53379smm0ZtDhw5h/Pjx0szCf3brTVFREcrLy8O+u6NChUYHQqVSAWh1mUwIwaBBg/DJJ5/g0KFD+PLLL3Hrrbfi1Vdf9fGQefHiRdx+++247bbbMHz4cMycOROfffaZ9Lv32q23m2ae533cM587dw6ZmZlwOp3YsGEDevbsCQBoaGgI6sbZ6XQGHIcLS+rvJjrSvQGn04kxY8bg5Zdfls5VVlYiNzcXe/fuhVar9bn2vvvuk5aMrFarj2CQKxclhCpzjuN8yquhoQENDQ0B+fCHEAK73Q5BEHzyIdcGhg0b5nM/wzBwOBwAAJZlMWXKFOzbtw/ffPMN1q5di9dffx1lZWUYMmQIdDodnE4nbrzxRmng4HQ6UV1djfT0dCxZsgQOhwOzZs3C5MmTUVlZKVs+3suWnnJ0OBxIS0vDBx98IP126dIlGAwGHDt2zCd/oQiVRm/8y7ylpQXnz5+X2rD/M1n26tUhunpzfpWwfv16bNq0Cddffz2efPJJ9OrVCxUVFeB5Hg6HA4QQnDx5EllZWfi3f/s3TJgwQeq8PB2IHGPGjMG2bdsAuPYn7rnnHlRUVGD8+PHYsmULCCGwWq1YtGhRUG2h8ePH491335Wu27p1K8aOHRvynRMmTEBZWRkaGhrgdDp9OhUljB49GgcOHMBPP/0EwKUtdMMNN8BisQRNX2lpKZqamgC49lIef/zxsO/gOC6s8AtV5mPHjsXevXul977yyivYsmWLT515vKR6AoT9+OOP+PrrrzFy5MiAd8m1AX+6deuGc+fOScfTpk3Dm2++id69e0MURYwePRovvviiFD1u3Lhx+Oijj1BdXQ0AeO+996R1/v3792Px4sWYPXs2GIbBN998I7UnJeXTvXt3qFQqqX49GnDBtNJCESqN3ukYNWoUDh06JF335z//Gb///e+DPvPcuXPo0aNHROnoSNCZRgfnnnvuwbJlyzB37lyIoog+ffpg7ty54DgO/fr1w6xZs/D2228jLy8PM2fOhEajwaBBg5CVlYUzZ86EfPbTTz+N3/72tygpKQEhBA888AAGDBiAJ598EqtWrUJJSQlsNhvGjh2L++67L+D+FStW4LnnnpOumzBhghSyVY5Jkybh+++/xy233IK0tDT07dtX2uBVQnFxMZ555hk8+uijIISA53m89tprQUeut956K6qqqnDbbbeBYRgUFBRgzZo1Yd9x/fXXY8mSJXjuuecwfvz4oNeMGzcOpaWlQct80qRJOH36NO644w4AQK9evfDss89Co9FIdfbee+9hw4YNePbZZ9HS0gKGYbB69Wp07949QK1Yrg34M3PmTHzxxRcYPXo0ANegoKqqSkrH+PHjsWfPHkydOhWAS4Dff//9uPfee8EwDPR6PTZu3AiGYbBkyRIsXrwY6enp0Gg0uPbaa6UlwClTpmDt2rWw2WyyZSiKIjZt2oRVq1bhzTffhN1ux8MPP4zhw4cHVXm9//77MX/+fGmJ1kOoNA4dOhQvv/wyFi9ejFdffRWPPfaY1E6NRiOef/75oM/2uKW/WqFebikUCgCXhtdtt92Gbdu2SXsMqcLWrVuRmZmJadOmJfQ9jY2NuOOOO7Bt2zZpOfhqgy5PUSgUAK4Y3Y8++ig2bdqU7KREDMdxmDx5csLfs3HjRjzxxBNXrcAA6EyDQqFQKBFAZxoUCoVCUQwVGhQKhUJRDBUaFAqFQlEMFRoUCoVCUQwVGhQKhUJRDBUaFAqFQlHM/wOPwgI72g9XiQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, ax = plt.subplots()\n", "ax.plot(y, alpha=0.5, label='Truth')\n", "ax.plot(y_pred, alpha=0.5, label='Fit')\n", "ax.legend()\n", "ax.set(title=\"Bicycle Count\", \n", " ylabel=\"Number of Bicycles\",\n", " xlabel=\"Instance of different factors (weather, etc.)\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is evident that we have missed some key features, especially during the summer time.\n", "Either our features are not complete (i.e., people decide whether to ride to work based on more than just these) or there are some nonlinear relationships that we have failed to take into account (e.g., perhaps people ride less at both high and low temperatures).\n", "Nevertheless, our rough approximation is enough to give us some insights, and we can take a look at the coefficients of the linear model to estimate how much each feature contributes to the daily bicycle count:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Intercept 34.314897\n", "Mon 490.296158\n", "Tue 596.243539\n", "Wed 578.405151\n", "Thu 468.195949\n", "Fri 164.217455\n", "Sat -1116.446711\n", "Sun -1146.596643\n", "holiday -1184.393535\n", "daylight_hrs 129.240672\n", "PRCP -663.374237\n", "dry day 551.976034\n", "Temp (C) 66.065583\n", "dtype: float64" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "params = pd.Series(model.coef_, index=pd.concat([pd.Series([\"Intercept\"]),pd.Series(column_names)]))\n", "params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We first see that there is a relatively stable trend in the weekly baseline: there are many more riders on weekdays than on weekends and holidays.\n", "We see that for each additional hour of daylight, 129 more people choose to ride; a temperature increase of one degree Celsius encourages 65 people to grab their bicycle; a dry day means an average of 548 more riders, and each inch of precipitation means 665 more people leave their bike at home." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }