{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import warnings\n", "# Ignore numpy dtype warnings. These warnings are caused by an interaction\n", "# between numpy and Cython and can be safely ignored.\n", "# Reference: https://stackoverflow.com/a/40846742\n", "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n", "warnings.filterwarnings(\"ignore\", message=\"numpy.ufunc size changed\")\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import seaborn as sns\n", "%matplotlib inline\n", "import ipywidgets as widgets\n", "from ipywidgets import interact, interactive, fixed, interact_manual\n", "import nbinteract as nbi\n", "\n", "sns.set()\n", "sns.set_context('talk')\n", "np.set_printoptions(threshold=20, precision=2, suppress=True)\n", "pd.options.display.max_rows = 7\n", "pd.options.display.max_columns = 8\n", "pd.set_option('precision', 2)\n", "# This option stops scientific notation for pandas\n", "# pd.set_option('display.float_format', '{:.2f}'.format)" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "from sklearn.feature_extraction import DictVectorizer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import confusion_matrix\n", "\n", "emails=pd.read_csv('selected_emails.csv', index_col=0)" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "def words_in_texts(words, texts):\n", " '''\n", " Args:\n", " words (list-like): words to find\n", " texts (Series): strings to search in\n", " \n", " Returns:\n", " NumPy array of 0s and 1s with shape (n, p) where n is the\n", " number of texts and p is the number of words.\n", " '''\n", " indicator_array = np.array([texts.str.contains(word) * 1 for word in words]).T\n", "\n", " # YOUR CODE HERE\n", " return indicator_array" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Evaluating Logistic Models\n", "\n", "Although we used the classification accuracy to evaluate our logistic model in previous sections, using the accuracy alone has some serious flaws that we explore in this section. To address these issues, we introduce a more useful metric to evaluate classifier performance: the **Area Under Curve (AUC)** metric." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Suppose we have a dataset of 1000 emails that are labeled as spam or ham (not spam) and our goal is to build a classifier that distinguishes future spam emails from ham emails. The data is contained in the `emails` DataFrame displayed below:" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
bodyspam
0\\n Hi Folks,\\n \\n I've been trying to set a bu...0
1Hah. I guess she doesn't want everyone to kno...0
2This article from NYTimes.com \\n has been sent...0
.........
997<html>\\n <head>\\n <meta http-equiv=\"Conten...1
998<html>\\n <head>\\n </head>\\n <body>\\n \\n <cente...1
999\\n <html>\\n \\n <head>\\n <meta http-equiv=3D\"Co...1
\n", "

1000 rows × 2 columns

\n", "
" ], "text/plain": [ " body spam\n", "0 \\n Hi Folks,\\n \\n I've been trying to set a bu... 0\n", "1 Hah. I guess she doesn't want everyone to kno... 0\n", "2 This article from NYTimes.com \\n has been sent... 0\n", ".. ... ...\n", "997 \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')\n", " plt.grid(False)\n", "\n", "ham_only_y_pred = np.zeros(len(y_train))\n", "spam_only_y_pred = np.ones(len(y_train))\n", "words_list_model_y_pred = words_list_model.predict(X_train)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAEeCAYAAADsNlzkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8zdf/wPHXzU5kGFGiJQkhVoYtRhEzRLR2rdpCUEqMtCq1FdWKXavE+lrhixoxqkajGqVWEDFTVDSRdTPv7w8/9+s2uBc3N8P72cd9PJLzOfdz3p9b3s75fM49R6FSqVQIIYR4KaO8DkAIIfI7SZRCCKGFJEohhNBCEqUQQmghiVIIIbSQRClEAScTV3KfJMpXcHV1ZePGjXkdRq7Zvn07rq6upKWlvdV5rl27xoQJE2jSpAkeHh60bduWxYsXk5qaqqdI/+fx48f07t0bNzc3OnbsqJdzent7M3fuXL2c62WefdYeHh4v/Vxat26Nq6srx44d0/m8V69eZfDgwa+sc/fu3dc+r9BkktcBiILt8OHDjB49mtq1azN+/HiKFSvGhQsXWL58OadOnWLlypWYmZnprb2tW7cSGRnJt99+ywcffKCXcy5cuJBixYrp5VzapKWlceLECVq0aKFRfvXqVW7evPna59u/fz+XLl16ZZ333nuPzZs3U6FChdc+v3hKEqV4Y3///TcTJkygbdu2zJw5U13u5eWFp6cnvXr1YvPmzfTu3VtvbSYmJvLee+/RunVrvZ2zatWqejuXNp6enoSHh+dIlAcOHKBSpUpcvXpV722amZnh6emp9/O+S2TorcWjR48YPnw4np6eNGrUiIULF2ocj4yMpF+/ftSsWRM3Nzf8/Pw4dOiQ+viECRMYO3YsixcvpmHDhtSoUYMvvviCpKQkvv76a2rVqsWHH37Ipk2bXiuu7Oxs1q9fT7t27XB3d8fHx4dt27apjz8bbv388898+umnuLu706xZs5e2s3r1atzc3EhKStIo7927NxMnTnzhe3bs2EFqaiqBgYE5jtWpU4fhw4dTunRpddmtW7cYPnw49erVo3bt2owePZoHDx7k+KyWL19O48aNcXd3Z8iQIeo6vXv3Zvny5cTGxuLq6sr27dsJCQmhYcOGGm0fO3YMV1dX7t69C8DDhw8ZMWIE9erVo0aNGvTr14/Lly+r6/976P22cb5Ky5YtOXLkCFlZWRrlBw4ceGHyP3LkCN27d8fT0xN3d3e6d+/O77//DkBISAgLFy7k0aNHuLq6EhERwfbt2/nwww9ZtGgRderUoX379ty5c0c99H7y5AmNGjVi4MCB6jZu376Nh4cH8+bN0xr/u0oSpRZLliyhbNmyLFmyhFatWhESEsLBgweBp8mob9++2Nvbs2jRIr777juKFCnC2LFjefLkifochw8f5tSpU8yaNQt/f3+2bt1Kx44dUSqVhISE4OnpydSpU9V/sXUxZ84cZs6cqb4f2KhRI4KCgli/fr1GvYkTJ+Ll5cWyZcuoWbMmkydP5vr16znO5+vrS1ZWlvraAB48eMCZM2fw8/N7YQynTp2iWrVqFC9e/IXHR4wYQcuWLQGIjY2la9euPHz4kOnTp/P1119z8eJFevXqpZGcjxw5wuHDh/n666+ZPn06Z8+eVfdWJ0+eTIcOHShZsiSbN2+madOmOn1WgYGB3Lt3j1mzZrFgwQJSU1MZPHgwmZmZOerqI85XadGiBQkJCZw5c0Zddvv2bW7cuIG3t7dG3bNnzzJs2DA8PT1ZunQps2fPJjExkbFjx5KVlUWXLl3o3LkzRYsWZfPmzVSrVg2AuLg4jhw5wnfffcfnn3+OQqFQn9PW1pagoCB++eUXwsPDAfjqq694//33GTFihE6f57tIht5atGjRgvHjxwNQr1499u/fz2+//UbLli2Jjo6mTp06zJ49GyOjp//mODg48PHHH3Pp0iXq168PQHp6OiEhIRQtWpTGjRuzefNmFAoF06dPx8jIiAoVKrB//34uX76s0323x48fs27dOgICAhg6dCgAjRo1Ijk5mQULFtCtWzd13Y4dO+Lv7w+Ah4cH+/bt49ixY7i4uGics2TJknh5ebFnzx4+/vhjAPbu3Yu9vT316tV7YRwPHz7E1dVVp89xzZo1AKxatQpra2sA3N3dadOmDVu3bqVv377qz2r58uXY2toCTx8UrVu3DgAXFxfee++91x5KRkZGMnz4cJo1awZA2bJl2bZtGykpKep29Bnnq5QuXRo3NzfCw8PVn+v+/fupV69ejliio6Px9fVlwoQJ6jITExOGDx9ObGwsZcuWpXTp0piYmGh8HpmZmYwePVrd0/73P8Bt27Zl165dzJ49m7i4OCIiIti0aZNe7yUXNtKj1KJGjRrqn42MjHBwcCAxMRGAJk2asHLlStLT07l06RJ79uxhw4YNAGRkZKjf5+joSNGiRdW/29vbU7lyZXVyfXbs2Xm1OX/+PBkZGbRp00ajvG3btsTHx3Pjxg11mYeHh/pnKysrbG1tSUlJeeF5/fz8OHXqFP/88w8Au3fvpm3btuo4/83IyCjHEPJlfv/9dxo1aqROPvA0Ybm5uREZGakuc3Z21kgYpUqVeuun57Vr1yYkJIQxY8awZ88e7O3tGTNmTI7EZKg4W7ZsqXF75uDBg+qe9/M6d+7MnDlzSEpK4vz584SFhbFr1y5A88/Xizg7O7/y+FdffcWjR48IDg6mb9++Gn9ORE6SKLWwsLDQ+N3IyEg9by0zM5MpU6ZQu3ZtOnfurDEl5vm5bUWKFMlxXktLyzeOKSEhAXiacJ9XokQJAI0horm5+Uvj/7eWLVtiZmbGgQMHuHXrFhcuXKB9+/YvjaNMmTLcv3//pccfPXqkHt4+efJEHd/zihcvrhHvqz7vNzV//ny6d+/OqVOn+Pzzz2nQoAHTp08nOzs7R11DxNmiRQvu3bvH5cuXuX//PhcuXMjxcAcgOTmZ0aNHU6dOHXr06MGaNWvUbWhr60XX8LwyZcpQs2ZNsrOzady4sU5xv8skUb6FpUuXsnPnThYsWEBkZCR79uxh2LBhud6unZ0d8DQRPe/Z78+Ovy4rKytatGjBwYMHOXDgAM7OzlSvXv2l9b28vLh48SLx8fEvPP7555+rbwPY2toSFxeXo05cXNwbxwugUChy9Gr/3WMuWrQoX375JSdOnGDLli18/PHHrF27lr179+Y4X27F+bzy5cvj4uLCoUOHCA8Pp2bNmjn+0QOYNm0av//+O2vXruXs2bOEhYXRpUsXvcTw888/c/z4cSpVqsS0adNIT0/Xy3kLK0mUb+GPP/6gZs2aeHt7q3sYJ06cAHL32xLu7u6Ympqyb98+jfKffvqJYsWK4eTk9Mbn9vPz4/Tp0+zbtw9fX1+tdS0sLF44WfvUqVP89ttv+Pj4AFCzZk2OHz+u0Su7c+cOFy5ceKupK1ZWVjx58kQjOT57KgxPb2d4e3tz8OBBFAoF7u7uBAcHY2Fh8cLecG7F+W8tWrTg8OHDhIeH06pVqxfW+eOPP/D29qZOnTqYmpoCcPLkSeB/f75edlvkVZKTkwkODqZdu3YsXryYO3fusHz58je8kneDPMx5C9WrV2fVqlVs3rwZJycnfvvtN/UfuJfdB9SH4sWL06NHDxYtWkR2djaenp4cO3aM7du3ExQUhLGx8Rufu0GDBtja2nLhwgW+/fZbrXEEBwczbtw4Hjx4QKdOnbC1teX3339n1apV1K1bl08//RSAvn37smPHDvr378/gwYNJT0/n+++/p1SpUnTq1OmN423UqBGzZ89m0qRJdOnShbNnz2r0FG1sbHB2dmbGjBmkpqby3nvvsXfvXrKysl741Dy34vy3Vq1asWzZMkxMTJg1a9YL61SvXp19+/ZRq1Yt7O3tOXz4sHpWw7M/X7a2tiQkJHD06FGN++mv8t133xEfH8/48eMpVaoUAwcOZNmyZfj4+Mik9JeQHuVbGDx4MO3atePbb78lICCAY8eO8f3331OuXDnOnTuXq21PmDCBgIAAtm7dir+/PydPnmTGjBn06dPnrc5rbGyMl5cXHh4eODo6aq3v6+vLjz/+iLGxMdOmTWPo0KHs378ff39/li1bpu4Jvf/++6xfvx4bGxvGjh3L119/TfXq1dm4caPGg5PXValSJaZMmcLZs2cZPHgwZ86cYc6cORp1vvnmG/XshEGDBnHhwgUWL16c48l/bsb5b9WqVaNMmTJUqVJFY67p8yZMmEDt2rUJDg5m1KhRXLlyhTVr1mBpaan+89W2bVtcXFwYPnw4x48f19ru+fPnCQ0NZfjw4ZQqVQqAIUOGULJkSSZPnizfG38JhWwFIZ6XmZmJt7c3/v7+9OjRI6/DESJfkKF3PpOVlaX1X3VjY2ONScT6kJmZyZIlS/jjjz9QKpUvnWQuxLtIEmU+07JlS+7du/fKOmvXrn3pJPA3ZWJiwq5du0hNTeWbb77R6zBTiIJOht75TFRUlNapGs7OzpLIhDAgSZRCCKGFPPUWQggt5B7lvyhzLigjDCxZ/ifkCyWs9ZseLGsM16le6tmF2isZmCRKIYRhKAruAFYSpRDCMIze/BtjeU0SpRDCMPQ899eQJFEKIQxDht5CCKGFDL2FEEILGXoLIYQWMvQWQggtZOgthBBayNBbCCG0KMBD74IbuRCiYDE21u2lg127dlGjRg2NV+XKlZk0aRIJCQkEBARQq1YtmjZtypYtW9TvS09PJygoiLp169KgQQOWLFmiU3vSoxRCGIYee5R+fn4ai0ufPHmScePGERAQwKRJk7CysuLkyZNERUUxaNAgKlasiKenJ/Pnzyc2NpZDhw4RFxdH//79cXR0pG3btq9sT3qUQgjDUCh0e72m5ORkJkyYQHBwMDY2NoSHhzNy5EjMzc1xd3fH19eXsLAwAHbu3MmQIUOwsbHBycmJXr16sWPHDq1tSKIUQhiGwki312tasWIFlSpVokWLFty6dQsTExPKli2rPu7s7MyNGzdISEggLi5OY1O5Z8e0kaG3EMIwcmF6UHJyMqGhofzwww/A0218LSwsNOpYWFigVCpJTU0FwNLSMscxbaRHKYQwjFwYeoeHh1OmTBk8PT2Bp0kwLS1No45SqcTKykqdQJ9PjM+OaSOJUghhGLkw9D5y5Ag+Pj7q3x0dHcnIyCA2NlZdFhMTg4uLC0WLFqVEiRLExMRoHKtQoYLWdiRRCiEMw8hYt9drOHfunLo3CWBtbU3z5s2ZN28eqampnD9/nt27d9O+fXvg6dPykJAQ4uPjuXnzJqGhoXTo0EF76K93pUII8Yb0PPTOysrir7/+omTJkhrlU6dOJTMzkyZNmjBy5EgCAwPx8PAAYNSoUTg5OeHj40OPHj3o2rWrRo/0paHLLoyaZLuWvCd75uQPet8zx1e3vXBSd+u2t44hyVNvIYRhyKIYQgihRQH+rrckSiGEYcjqQUIIoYUMvYUQ4tUU0qMUQohXk0QphBBaKIwkUQohxCtJj1IIIbSQRCmEEFrI0FsIIbSQHqUQQmghiVIIIbSQRCmEEFrIPUohhNCiIPcoC+5yHkKIAkWhUOj00tX9+/cZMmQINWvW5MMPP2Tt2rUAJCQkEBAQQK1atWjatClbtmxRvyc9PZ2goCDq1q1LgwYNWLJkiU5tSaIsRBaFLKByRWeK21rh06o5UVeu5HVI74z4f/6haoUPmD1jirrs4cMHjPAfQPVKjpT/wJ6PfVvx5/k/8jDKvKUwUuj00oVKpWLYsGGUL1+eiIgIVq5cycKFC4mMjGTSpElYWVlx8uRJFixYwNy5c/njj6ef+/z584mNjeXQoUNs2LCBLVu2sHfvXq3tSaIsJFavXMGEcWMYONifdRs2k5qaSts2LUhKSsrr0N4Jk78Yx8OHD9S/Z2Vl0avbx/x66gRfT5/N8lWhGBkZ0b6NN/fu3snDSPOOPnuU586d4+HDh4wdOxZTU1MqVqzIpk2bKFWqFOHh4YwcORJzc3Pc3d3x9fUlLCwMgJ07dzJkyBBsbGxwcnKiV69e7NixQ2t7kigLAZVKxcwZUxk+chRjA8fTzrc9O3f/RFJiIqFrf8zr8Aq9X44d5b+7dlDE2lpd9uupE0Se+Y2VazfSqUt3WrRqw7pN2zE3M2f9ujV5F2we0meivHjxIhUrVmTOnDk0bNiQ1q1bc+7cORISEjAxMaFs2bLqus7Ozty4cYOEhATi4uJwcXHJcUwbSZSFQPT169y5fZt27f3UZXZ2djT+sAmHwg/kYWSFn1Kp5PORQ/kyeBpFrIqoy83NzPi0/yDcPWqoy6ysrCjzwQfcuXUzDyLNe0ZGRjq9dJGQkEBERATFihXjyJEjzJw5k6lTp5KSkqLev/sZCwsLlEolqampwNO9v/99TGvsr3GdIp+6du0qABUquGiUOzo5Ex19PS9CemfMmTkV+xL29BswRKO8dt36zPt+sUbZndu3iLp8iYqVKhsyxPxDoeNLB2ZmZtjZ2TFkyBDMzMyoWbMmrVu3ZsGCBaSlpWnUVSqVWFlZqRPo84nx2TFtJFEWAk+ePAHAxsZGo9zGxobExMS8COmdcPHCeZYvCeHbkCVah4yZmZl8PnIollZW9Ojd1zAB5jP6HHo7OzuTlZVFVlaWuiwrK4uqVauSkZFBbGysujwmJgYXFxeKFi1KiRIliImJ0ThWoUIFre1JoiwEsrOzgRfPUyvIc9fys6ysLEYNH8KQYSOpUrX6K+tmZmYydFBfjh09zKJlq7H/1z7U7wp9Dr0bNmyIhYUFCxcuJDMzk8jISA4ePEibNm1o3rw58+bNIzU1lfPnz7N7927at28PgJ+fHyEhIcTHx3Pz5k1CQ0Pp0KGD9tjf6spfU0xMDEOHDqVOnTrUqFEDPz8/jTlO4s3Y2dkB5HjCnZiYiJ2tXV6EVOgtXxLC47g4PhsznszMTDIzn+5Fnp2drf4ZICUlhV7dPua/YdtYtHw1bdr65lXIeU+PQ28LCwvWrVvH+fPnadCgAWPHjuXLL7/E09OTqVOnkpmZSZMmTRg5ciSBgYF4eHgAMGrUKJycnPDx8aFHjx507doVHx8f7aGrVCrVG1zya8vOzqZly5Z07NiRAQMGYGZmxpkzZxg+fDhTp06ldevWhghDK2Wm9jr5zdWoKDyqV+bIsZPU9/JSl3f+2A9TMzM2bt6ah9G9vuQC8D/Bz6c5J48fe+nxR4kZJCYm0qWDD3+e/4Plq0Jp5/eRASN8eyWs9fvFvXIjdulU73aIn/ZKBmawrzD+888/3L17Fz8/P/VN1bp16xIYGEhGRgYhISHcuHGDuLg4zp8/T8WKFZkyZQpVqlQBYO3atWzZsoXY2FjMzc355JNPGDFiBACurq5MmTKFxYsXk5iYyIABA3BwcOC7775DqVQyZMgQBgwYYKhLNbiKlSpR5v332bN7lzpRJiQk8Muxn/l66ow8jq5wmvf9YpKSNO//dvvYl3Z+H9Gn30BUKhX9e3fj4oXzrP9PGE29W+RRpPmHrsPq/MhgibJEiRLUrVuXfv364efnR926dXF3d6dLly4AhISE8NNPP/Htt9+ycuVKli1bxrBhw9i/fz/nz59n6dKlbNiwAScnJ86cOUOvXr3w8/PD0dERgBMnTrBv3z4iIyMZMGAAPj4+HDhwgIiICIYOHUrXrl1zPOwoLBQKBZ+PGcf4wM+xtramups7c2bPxMbWlp69++R1eIVSxUquOcpMTEwo7VCGGjVrs23LJo4cOsjAIcOwtrbmzOlf1fWKl7Cn/L9mKLwLCvL9coMuirFixQo2btzIwYMHWb58OQCtWrVi0qRJAHh5edG2bVsAhg4dyrp164iMjMTDw4Pt27dTunRpHj16REZGBhYWFjx8+FCdKHv16oWlpSX169dHpVLRs2dPLCwsaNy4MVlZWTx48KDQJkqAgBEjSUlJZsnihTxJSKBuvfrs+elgob7m/OynPf8FYMWyxaxYpjlN6KNOXVmxZn1ehJW3Cm6eNGyiNDc3p2/fvvTt25e0tDR+//135s6dS1BQEFWrVqVcuXLqusbGxpQsWZJHjx5hZGTE4sWL2b9/PyVKlKB69adPGZ897YX/PdAwNn66ybqtrS3wv+7+83ULq8DxEwkcPzGvw3hnXYq+q/55xZr172YyfIWCPPQ2WOR79+6lefPmPHt2ZG5uToMGDRgxYgRX/n/xhocPH6rrZ2Zm8vDhQ0qXLs3q1au5evUq4eHh7N27l+nTp2s8WYSC3a0X4l2gUOj2yo8Mlii9vLxITk5m+vTpxMXFoVKpuHXrFuvWraNZs2YA/PLLL5w8eZKMjAwWLVpEsWLFqFGjBklJSZiammJqakpycjKzZ88mIyMjR7IUQuRf+l5mzZAMliiLFSvGhg0bePjwIb6+vnh6etKvXz/c3NyYMGECAO7u7vzwww/Uq1ePM2fOsGzZMoyNjenXrx8mJiZ4eXnRunVr0tPTqVmzJtHR0YYKXwjxlgpyj9Jg8yi1CQkJ4dq1ayxYsCBP4ygAU/gKvYIwj/JdoO95lJUn7Nep3pVZ+WNO9fNkKwghhEEYyZ45Qgjxavl1WK2LfJMon33LRghROEmPUgghtMivT7R1IYlSCGEQkiiFEEILGXoLIYQWBbhDKYlSCGEYBXnoXXC/pS6EKFCMjBQ6vXS1cuVKqlevTo0aNdSvM2fOkJCQQEBAALVq1aJp06Yauyikp6cTFBRE3bp1adCgAUuWLNGpLelRCiEMQt8dykuXLjF69Ogci3KPHDkSKysrTp48SVRUFIMGDaJixYp4enoyf/58YmNjOXToEHFxcfTv3x9HR0f18o4vIz1KIYRB6HtRjMuXL6t3QHgmOTmZ8PBwRo4cibm5Oe7u7vj6+hIWFgbAzp07GTJkCDY2Njg5OdGrVy927NihtS1JlEIIg9Dn0Ds1NZWYmBjWrl1Lw4YN8fHxYevWrdy6dQsTExPKli2rruvs7MyNGzdISEggLi4OFxeXHMe0kaG3EMIg9Dn0fvToEbVq1eKTTz5hwYIFnD9/Hn9/f/r166fek+sZCwsLlEolqampAFhaWuY4ps1LE+Xzm4Rr4+zsrHNdIcS7SZ9PvcuWLUtoaKj699q1a9OhQwfOnDlDWlqaRl2lUomVlZU6gSqVSqytrTWOafPSROnj44NCoeBlq7A9O6ZQKLh8+bL2KxNCvNP0OeH84sWLnDhxgsGDB6vL0tLScHBwICMjg9jYWMqUKQM87fS5uLhQtGhRSpQoQUxMDPb29upjFSpU0NreSxPloUOH3vZahBBCTZ89SisrKxYuXEi5cuVo1aoVERER7Nmzh9DQUBITE5k3bx7Tpk3j2rVr7N69W72ZoZ+fHyEhISxYsID4+HhCQ0MJDAzUHvvrLNwbGRlJTEwMrVu35q+//sLR0REzM7M3v9p8SNaMzXuycG/+oO+Fe5vMP6FTvZ9HN9Sp3uHDh5k/fz537tyhVKlSjB49mjZt2hAfH8/kyZM5deoUVlZWDB8+nM6dOwNPh9ozZszg4MGDKBQK+vTpg7+/v9a2dEqUjx8/xt/fn0uXLpGdnc3+/fuZPn060dHRrFq1SuMJU0Enf0fzniTK/EHfibLpdyd1qnd0VAO9tqsPOk0Pmj59Ovb29kRERGBubg7A7NmzKVeuHNOnT8/VAIUQhYO+v5ljSDolypMnTzJq1CiKFCmiLrOzs2PChAmcOXMm14ITQhQeBXlzMZ361llZWWRnZ+coT0xMxNjYWO9BCSEKH6P8mgV1oFOPskWLFsyZM4fHjx+rn1xdv36dqVOn0rx581wNUAhROBT6oXdQUBDW1tY0bNiQlJQU2rdvT/v27XFwcCAoKCi3YxRCFAJGCt1e+ZFOQ29ra2u+//577ty5Q3R0NJmZmVSoUEG+kSOE0FlBXo9S5+f/WVlZXL16lejoaMzMzLCwsJBEKYTQWUG+R6lTooyKimLo0KHEx8fj5OREdnY2t27dwsnJiYULF/L+++/ndpxCiAIuvw6rdaHTPcrJkyfj4eHBsWPH2L59O2FhYRw9epRSpUrx1Vdf5XaMQohCQN/rURqSTony0qVLjBgxQr3iBjydRzlmzBiZRymE0ImxkUKnV36kU6KsXLky586dy1EeFRUl9ymFEDoplBPON2/erP65Ro0aBAcHc/HiRdzd3TE2NubKlSuEhobm2K9CCCFeJL8Oq3Xx0kUxvL29dTuBQlGolmST9RjyniyKkT/oe1GMbj+e1ane5k9r6LVdfXjpJ3H48GFDxiGEKOQKbn/yNeZRPnjwgBs3bpCVlQWASqUiPT2dixcvMnLkyFwLUAhROBTkobdOiXL9+vXMmDGDrKwsje0hFAoFHh4ekiiFEFrl1yfautDpqffKlSsZOnQof/75JyVKlODo0aPs3r2bypUr07Jly9yOUQhRCOTWU+9Hjx7h5eXFkSNHALh79y6ffvopNWrUoHXr1upygISEBAICAqhVqxZNmzZly5YtOrWhU6J8+PAhHTp0wNTUlCpVqvDHH3/g4uLCxIkTdW5ICPFuy60J51988QXx8fHq3z/77DPc3d05ffo0QUFBjBkzhtjYWAAmTZqElZUVJ0+eZMGCBcydO5c//vhDaxs6JcqiRYuSmJgIPN2aNioqCoD333+f+/fvv/aFCSHePbmxetDGjRuxtLTEwcEBgOjoaK5evUpAQACmpqY0adKEunXrsmfPHpKTkwkPD2fkyJGYm5vj7u6Or68vYWFh2mPXJZhmzZrx1VdfceXKFerXr8/OnTuJjIxk3bp16gCFEOJVjBQKnV66iomJYfXq1QQHB6vLbty4wfvvv6/ewxuedu5u3LjBrVu3MDEx0djj69kxrbHrEtCECROoXLkyV65cwdvbmzp16tCjRw+2bNnChAkTdL4wIcS7S5+JMjMzk3HjxvHFF19QtGhRdXlKSgqWlpYadS0sLFAqlaSkpGgk0OePaaPTU+8iRYowbdo09e+zZ89m4sSJWFtbY2Ki30mpQojCSZ+zgxYvXkyVKlVo0qSJRrmlpWWOxKdUKrGyssLS0pK0tLQXHtNGp68watOtWzed6woh3k363OZh7969/P333+zduxeApKTK4Eg9AAAgAElEQVQkPv/8c/z9/bl37x7p6emYmZkBT4fo9erVw9HRkYyMDGJjYylTpoz6mIuLi9b2Xpooly1bplPACoVCEqUQQit9Lty7b98+jd+9vb2ZNGkSzZo1Y//+/Xz33XeMGjWKU6dOERERweTJk7G2tqZ58+bMmzePadOmce3aNXbv3s3y5cu1tidfYRT5zgeNR+V1CAJIPbtQr+cz1BdzQkJC+Oqrr/Dy8sLe3p5vv/1W/dB56tSpTJ48mSZNmmBlZUVgYCAeHh5az/nSRTHeVbIeQ94rVmd4Xocg0H+iHLHjsk71Qj6uotd29UGexAghDKLQf9dbCCHeVgH+qrckSiGEYRT6RTEA4uLiWLp0KePHjycuLo69e/dy9erV3IxNCFGI5MZXGA1F583FWrduzdGjR9mzZw8pKSmcPHmSLl26cOrUqdyOUQhRCBTkPXN0SpQzZ87k008/ZdOmTZiamgIwbdo0evfuzdy5c3M1QCFE4WCiUOj0yo90SpQXL17Ez88vR3m3bt2Ijo7We1BCiMKn0Pco7ezs1Ou5Pe/ixYsUL15c70EJIQoffa8eZEg6PfX+5JNP+Oqrrxg7dizwdD/vY8eOERISQr9+/XI1QCFE4WCs86Pj/EenRDl48GCKFCnCrFmzSE1NZfjw4djb2+Pv78+nn36a2zEKIQqB/Npb1IXO8yh79uxJz549SUlJISsrCxsbm9yMSwhRyBTgPKlbotS2VPpHH32kl2CEEIVXfp0jqQudEuW/pwBlZmby5MkTzMzMqFy5siRKIYRWxgW4S6lTojx+/HiOsoSEBCZNmkTNmjX1HpQQovApyD3KN34OZWdnx6hRo1ixYoU+4xFCFFK5tV2tIbzVA/u7d++Smpqqr1iEEIWYsZFuL13t3bsXHx8fatSoQbt27QgPDweejnYDAgKoVasWTZs2ZcuWLer3pKenExQURN26dWnQoAFLlizRqS2dht5jxozJUZaUlMTp06fx9fXVqSEhxLtNn9ODYmJiCAoKYtWqVdSsWZOTJ08yePBgjh07RnBwMFZWVpw8eZKoqCgGDRpExYoV8fT0ZP78+cTGxnLo0CHi4uLo378/jo6OtG3b9pXt6ZQon23S87xSpUoRFBREhw4d3uxKhRDvFH3eo3R2dubEiRMUKVKEzMxMHj16RJEiRTAzMyM8PJz9+/djbm6Ou7s7vr6+hIWF4enpyc6dO5k3bx42NjbY2NjQq1cvduzYoZ9E6erqio+PD6VKldLLRQoh3j36fupdpEgR7ty5Q+vWrcnOziY4OJjbt29jYmJC2bJl1fWcnZ05cOAACQkJxMXFaey66OzszPr167W2pdMdgUWLFpGenv4GlyKEEE/lxqIYDg4OnDt3jtWrVzN79mwOHz6MhYWFRh0LCwuUSqX6eYqlpWWOY9rolCg//PBDQkNDefLkyetcgxBCqOXGwr0mJiaYmpri5eVFq1atuHDhAmlpaRp1lEolVlZW6gT6fGJ8dkxrO7oEc+vWLfbs2cPatWuxtrbG3Nxc4/iL5lkKIcTz9LkVxM8//8zq1atZs2aNuiwjI4Ny5cpx7NgxYmNjKVOmDPD0wY+LiwtFixalRIkSxMTEYG9vrz5WoUIFre3plCh79uz5BpcihBD/o8+n3lWrVuXChQuEhYXh5+fHL7/8ws8//8x//vMf/vrrL+bNm8e0adO4du0au3fvZvny5QD4+fkREhLCggULiI+PJzQ0lMDAQK3tvXRf74ULFzJgwACN8fy7QPb1znuyr3f+oO99vVf9dlunev3rlNOp3pkzZ5gxYwY3b97EycmJcePGUb9+feLj45k8eTKnTp3CysqK4cOH07lzZ+DpUHvGjBkcPHgQhUJBnz598Pf319rWSxNllSpVOH78OCVKlNAp6MJCEmXek0SZP+g7Uf545o5O9T6tXVZ7JQN76dD7JflTCCHeSP78cqJuXnmPMiMjQ6dpQS+akC6EEM8rtAv3NmvWTKeTXL58WS/BCCEKr4K8etArE+WCBQuws7MzVCxCiEIsv64MpIuXJkqFQkHNmjXfuYc5QojcUYD3FpOHOUIIwyiUPcqPP/44xzdwhBDiTRXKhzkzZ840ZBxCiEKuUA69hRBCnwrl0FsIIfSp0E4PEkIIfTEqwN/NkURZiCwKWUDIgvk8fPCAevW9+G7BIlwrV87rsAqFxrUqcmDFZy89blljOJ1b1WTd7P45jvkFLOLgyadfyqjn7szM0R9RveL73H/0hMUbj7J087Fcizs/KcAjb0mUhcXqlSuYMG4Mk6dMo0qVqsyZPZO2bVpw7sIVrK2t8zq8Au+PK3do0meuRlkxOys2zhnI5n1nAKjmUoY/r95j+LSNGvWuxNwHoJxDMf67OIDjkdf5ZOwKalQtx9zAzqSlZ7J6x0nDXEgeKpRPvUXBoVKpmDljKsNHjmJs4HgAGjX+kErlyxG69kf8hwXkcYQFX2KyktN/3tQo+3FmX+49jOfz2U+3Q61e6X1+v3QrR71n2jf1wNjIiJ7jVpKqzODQr1eoVsGBfh83eDcSZQEeehfkJ/bi/0Vfv86d27dp195PXWZnZ0fjD5twKPxAHkZWeHl5lKdrm9qMn7edVGUGANVdHLh4Lfal77EwNyEjMwtl2v/W8vvnSQrF7LRvRVAY5MaeOYYiibIQuHbtKgAVKrholDs6ORMdfT0vQir0goe355ffr7H32AUArK3MKedQnHruzlzcNZknp7/n2Nqx1KnuqH7Plv2RZKtUTB7mS1EbSxrXqsgn7eqwdX9kXl2GQRkpFDq98iNJlIXAs03fbGxsNMptbGxITEzMi5AKteoVy/Bh7YrM//GQRpmRkRFlHYozevYWuo35gbSMTP67eDgOJZ8uLHP7r8dMnL+Dsf1a8texORxY8Rm/noth6tI9eXUpBqXvzcXOnDlDly5dqFWrFi1atGDTpk0AJCQkEBAQQK1atWjatClbtmxRvyc9PZ2goCDq1q1LgwYNWLJkiU5tyT3KQiA7Oxt48YTegjzJN7/q37EhN+78zU+/XFCXXY6+z8cjlvDL79dITn26huuxM1e5+N9gRvbyZuL8HfT2q8/SyT1ZuP4I/z16noqO7xEc0J7Fk3rg/7X2vaULOoUe71EmJCQwbNgwJk2aRLt27bh8+TL9+vWjXLlybNq0CSsrK06ePElUVBSDBg2iYsWKeHp6Mn/+fGJjYzl06BBxcXH0798fR0dH2rZt+8r2pEdZCDxbCi8pKUmjPDExETtbWSZP33ybuLE9/KxGWUJSKvuOX1QnSYDk1HQizsVQzeXpboDjB7Rm6/7fCZy7jWNnrrFy2wn8v17Ppx954VbpfYNeQ17Q59A7NjaWJk2a0L59e4yMjKhWrRr16tUjMjKS8PBwRo4cibm5Oe7u7vj6+hIWFgbAzp07GTJkCDY2Njg5OdGrVy927NihPfa3unI9cXV15erVqznK69WrR0RERB5EVLC4uFQEIObGDY3yWzdjcKlUKS9CKrQqly9NWYfi7Dx0TqPcvdL79OlQP0d9c3MTklOf7jP9QemiOZ6I/3ru6f+zys6lcyfgfESfQ+8qVaowZ84c9e8JCQmcOfN0mpaJiQlly/5v3x1nZ2du3LhBQkICcXFxuLi45DimNXYdr1HkYxUrVaLM+++zZ/cudVlCQgK/HPuZJk10W6Ve6KZm1XKkpWfwR5TmRlnurh+wLLgXrs6l1GUli1nj5VGeE5FPH6hdv/039T2cNd5X+/8f9tyKjcvlyPOeQsf/XldiYiL+/v7qXqWFhYXGcQsLC5RKJampqQAaO8s+O6ZNgblHeenSJb755huuX79OYmIitWrV4ptvvsHe3p4JEyZQvHhxIiMjuXLlCtWrV2fcuHHMmDGDqKgoPD09CQkJKbQTrxUKBZ+PGcf4wM+xtramups7c2bPxMbWlp69++R1eIVK1fIOxNyNIzMzW6N8R/hZJgxsw6a5g/h68W5UKhVBQ3yIi09m5fYTAMz6YR/rZvdn3rgn7Dx8jgplSzI5wJcjEVEvnXtZmOTG7fI7d+7g7+9P2bJl+e6774iOjiYtLU2jjlKpxMrKSp1AlUqlOhc8O6ZNvkmU3bt3x8hIs4P7/D23zz77jD59+rB69Wri4+MZPHgwoaGhjBo1CoDt27cTGhqKg4MDnTp1YtiwYfz444/Y29vTtWtXdu7cSc+ePQ16TYYUMGIkKSnJLFm8kCcJCdStV589Px3M8SRcvB37YtYkJKXmKE9OTaet/wJmjPqYBV90w9zUhMMRUYybu009z3LrgafTgMYNaM2ATg259yCe9f+NYNqyvQa9hrxirOdMefHiRQYOHIifnx/jx4/HyMgIR0dHMjIyiI2NpUyZp/eGY2JicHFxoWjRopQoUYKYmBjs7e3VxypUqKC1rXyTKDdt2kSlf91Pq1evnvrnlStX8sEHH5CamsqDBw8oVqwYDx48UB9v1qyZ+t6Dm5sbZmZm6g/Aw8ODe/fuGeAq8lbg+IkEjp+Y12EUaq96On37r3/oNX7VK9+/9UCkOmG+a/SZJh89esTAgQPp168fgwcPVpdbW1vTvHlz5s2bx7Rp07h27Rq7d+9m+fLlAPj5+RESEsKCBQuIj48nNDSUwMBAre3lm0Spzfnz5xk0aBDJycm4urqSkJBA8eLF1cef3wTN2NgYW1tb9e9GRkaytYUQeU2PmXLr1q08fvyYJUuWaMyF7NOnD1OnTmXy5Mk0adIEKysrAgMD8fDwAGDUqFHMmDEDHx8fFAoFffr0wcfHR2t7BSJR3r9/n/Hjx7Nhwwb1BU+cOFEj+cl8QSHyN31+68bf3x9/f/+XHv/+++9fWG5hYcGUKVOYMmXKa7VXIJ56Jycno1KpsLCwQKVS8fPPP7Nv3z4yMjLyOjQhhI4UOr7yowLRo6xQoQLDhg3j008/JTs7m/Lly9O9e3d+/fXXvA5NCKGr/JoFdaBQyc07DcpM7XVE7ipWZ3hehyCA1LML9Xq+328+0aleLSdb7ZUMrED0KIUQBV8B7lBKohRCGEgBzpSSKIUQBpFf15rUhSRKIYRBFNw0KYlSCGEoBThTSqIUQhiEDL2FEEKLgpsmJVEKIQylAGdKSZRCCIOQobcQQmhRcNOkJEohhKEU4EwpiVIIYRD63K7W0CRRCiEMQtcdFvOjArEepRCiEMilBSnPnz9Po0aN1L8nJCQQEBBArVq1aNq0KVu2bFEfS09PJygoiLp169KgQQON1dFfRXqUQgiD0PfQW6VSsW3bNmbNmoWxsbG6fNKkSVhZWXHy5EmioqIYNGgQFStWxNPTk/nz5xMbG8uhQ4eIi4ujf//+ODo60rZt21e2JT1KIYRBGCl0e+lq6dKlrF27VmNLiOTkZMLDwxk5ciTm5ua4u7vj6+tLWFgYADt37mTIkCHY2Njg5OREr1692LFjh/bYX/tqhRDiTeh56N2pUyd27tyJm5ubuuzWrVuYmJhQtmxZdZmzszM3btwgISGBuLg49W6tzx/TRobeQgiD0PfQ+7333stRlpKSgoWFhUaZhYUFSqWS1NSn+7FbWlrmOKaN9CiFEAah76H3i1haWpKWlqZRplQqsbKyUifQ5xPjs2NaY3+7sIQQQkcG2IbR0dGRjIwMYmNj1WUxMTG4uLhQtGhRSpQoQUxMjMaxChUqaD2vJEohhEEodPzvbVhbW9O8eXPmzZtHamoq58+fZ/fu3bRv3x4APz8/QkJCiI+P5+bNm4SGhtKhQwet55VEKYQwCEMMvQGmTp1KZmYmTZo0YeTIkQQGBuLh4QHAqFGjcHJywsfHhx49etC1a1d8fHy0nlO2q/0X2a4278l2tfmDvrervftPmvZKwAfFzPXarj7IU28hhIEU3O8wSqIUQhhEQf6utyRKIYRBFOB1eyVRCiEMQ5ZZE0IILaRHKYQQWkiiFEIILWToLYQQ2hTcPCmJUghhGDI9SAghtJChtxBCaCEPc4QQQgtJlEIIoYUMvYUQQgvpUQohhBYFOVHKwr1CCIPQ9wrnly5donPnznh6etKhQwf++OOPXItdEqUQwiAUCt1eukhLS8Pf35+OHTvy22+/0bt3b4YOHUpycnKuxC6JUghhEPpMlL/++itGRkb06NEDU1NTOnfujL29PT///HOuxC73KIUQBqHPp94v2j3R2dmZGzdu6K2N50mi/BcL+UTynL73ahH5g6Wp/s6VkpKCpaWlRpmFhYXGnt36JENvIUSBY2lpmSMpKpVKrKyscqU9SZRCiAKnfPnyxMTEaJTFxMTg4uKSK+1JohRCFDheXl6kp6ezbt06MjIy2Lp1K48ePaJRo0a50p7s6y2EKJCuXLlCcHAwUVFRODo6EhwcjKenZ660JYlSCCG0kKG3EEJoIYlSCCG0kEQphBBaSKIUQggtJFEKIYQWkiiFEEILSZTvKJkVZljZ2dnqn+WzL3gkUb4jsrKyAMjMzARAoVDIX1gDyczMxMjISP2zKHhkwvk7QKVSoVAouHr1KuvWrUOhUDBmzBjs7OzyOrR3RnZ2NkOGDEGpVNKwYUN8fHxwdHTM67CEjqRHWcg9S5IPHjygZ8+eKBQKzp49S+fOnXMsKiD061kvHmDo0KHY2trSoEEDfvvtN9avX8/NmzfzLjjxWqRHWYhlZWVhbGxMSkoKZ8+e5dy5cwwbNgyAwYMHExMTw4oVK6RnkwueffYqlYq7d++yZcsWPv/8cwD27t3Lrl27KFeuHD169MDJySlvgxVaSaIs5C5fvszMmTN59OgRJUuWZNmyZVhYWAAwaNAgbt++zeLFi3OsFi3e3LNefHZ2Nl27duXRo0fcv3+fqVOn0qVLFwD27NnDvn37sLW1ZciQIZQrVy6PoxavIkPvQujZkC8pKYkxY8bg4eFB27ZtiYiI4MCBA+p6P/zwAzY2NixYsCCvQi10niVJgODgYKpWrcrSpUtp06YNp0+fJjw8HIB27drRrFmzXF1sVuiP9CgLqVu3brFo0SIcHBwYPXo0ABs3bmTKlCnMnDmTjz76SF03Oztb/VRW6Mfq1as5fPgwX3zxBZUrV+bKlSusWbOG9PR0fH198fb2Bp7+Y2ZtbZ3H0QptjIODg4PzOgihH8/3ZlJTU5k8eTLXr1/H19eXIkWK4ObmRvHixfn6668pWrQobm5uAOphoqIg71Cfx7KystT/2GRkZHD06FF+/fVX0tPTqVOnDg4ODpQrV46LFy9y+vRpbG1tcXJywszMLI8jF7qQRFlIPPuLmpSUxJMnTyhVqhSdOnVi06ZN3Llzh8aNG2NqaoqbmxtmZmbs37+fjh07qpOjJMk3l5mZiYmJCSqViqioKCwtLWnWrBkKhYILFy7wzz//ULlyZUqXLk2ZMmW4c+cObdq0kZ5kASJD70Lg2dD52YrP8fHxdOjQgaFDh/LgwQM++ugjGjRowNSpU9X3w571Pp/vhYrX9+zpdlZWFt27dycxMZGSJUvyySef0LZtW3744QdOnz5Nw4YN6dKlC0WKFCE9PV16kgWM3JgqBIyMjIiOjqZ///54e3vTvXt3li9fzqJFiyhVqhRhYWFEREQwYsQI0tLS1O+TJPn2nk0B6tu3LxUrVmTRokUUK1aM9evX89///pdBgwZRv3599u3bx44dO1CpVJia6nHfVmEQsot1IZCZmcm6desYNGgQ/fr14969exw9epQVK1agVCoZM2YM27ZtY/Lkyeq/pJIg305iYiI2NjYAREREYGZmxowZMwAoUqQIycnJbN26FUtLSwYMGICFhYV6OC4KHulRFgImJiZcvXoVKysrsrKyGDNmDJ06dWL69On88MMPBAUFYWRkxNKlSzEyMtJYoEG8vjlz5hAYGMjDhw8BiI2NJT09HYDAwECUSiVTpkzh77//5ssvv2ThwoX07NmTMmXK5GXY4i1Ij7IAenZfDP53f3LUqFGUL1+eiRMnUqZMGdq3b8+FCxeoWrUqRYoUoUSJEur3y1Sgt9OxY0f8/f2ZN28eEydOpGPHjlStWpUTJ05w9epVtmzZgpmZGdWqVaNChQr4+vrmdcjiLUmiLGCys7MxNjbm2rVrrFu3jqSkJGrXrk2bNm0oXrw4AHXq1AFg7dq11K1bl3Hjxql7kpIk3056ejoVKlRg9+7deHl5YWZmxujRo6lcuTJ79uzBxMQEMzMz1q9fz4ULFxg9erT0JAsBeepdgDzrSd65c4dPPvmEVq1aYWNjw7Zt2/jwww/p168fW7du5ddff8XMzIzk5GR27tyJqampJEk9eP4zXL9+PQsWLMDCwoK6desSFBREZmYm7du3p0yZMvz111+sWLGCatWq5XHUQh8kURYwT548YdasWdjZ2TF+/HgArl+/zqhRo2jXrh3du3fnzJkz3L9/n08++QQTExP1PD/xZoKDg2nRogWNGjUCYNmyZYSGhvLTTz9hbGyMr68vNWvW5IsvviAzM5Pr16/j5ORE6dKl8zhyoS+SKPM5pVLJ8ePHefjwIVWqVCEyMpK1a9fi4uLCypUr1fV++eUXxo4dy549e7C3t1eXP38/U7yZgwcPMnHiRLZs2cLhw4dZs2YN69evVy9kcefOHQYOHIiLiwszZsyQdT4LIelm5GNJSUn06NEDR0dHrly5Qs+ePalTpw4ZGRn8/PPPhIWFqb+zXbFiRapUqZJjIrMkybfXsmVLzMzM8PHxwdbWlp07d+Lg4KBerbxs2bIsW7aM0aNHo1QqJVEWQpIo86nU1FS6du1KvXr1mDx5MvHx8djZ2aFQKChVqhSZmZlERkZy7949OnfuzMKFC1EqlfK1uFzSpEkT1qxZw5AhQ0hJSQFQ387IzMzEycmJLVu2yC2OQkqG3vlUWFgYBw4cYPHixeqy/fv3s3HjRiwtLYmPj6dhw4b85z//wc7ODnd3d6ZPnw7IakC56ejRo4wZM4Zly5ZRu3ZtjWPyTafCS/425VP//POPevOvEydOMG3aND777DOcnZ3x9PTE1taWhIQEunXrhoODA1WqVMnjiN8NTZs25dtvvyUgIIBTp05pHJMkWXhJjzKf+vPPP+nWrRuVK1cmIyMDOzs7AgIC8PLyAmDSpElUqVIFPz8/fvzxR86dO0fdunUZOHBgHkf+bggPD2f69On89NNP6hXjReElN1TyKTc3NzZv3syJEyeoUaMGZcqUoWzZsurjjx8/JjMzE2trazp37oxSqeT8+fMkJCTIwwQDaNGiBV5eXpIk3xHSoywgHj9+rP7mzbhx47hy5Qrbtm1TL3Lx999/Y2xsrK4jhNAfSZQFQGRkJAMGDMDV1RVLS0uSkpLYsGEDpqamMplcCAOQRFkApKSkEBERwc2bN/nggw/w9vbG2NhYkqQQBiKJsoCSb9wIYTiSKIUQQguZRymEEFpIohRCCC0kUQohhBaSKIUQQgtJlEIIoYUkykLO29sbV1dX9atatWo0bdqUOXPmkJGRobd2IiIicHV1Ve8b7u3tzcaNG7W+Lz09nQ0bNrxxu3fv3sXV1ZXo6OgXHtc1jtet+7pxiIJNZiu/A8aOHate4DcrK4sLFy4QGBiIlZUVAQEBudLm1q1bsbKy0lpvz549LFq0iB49euRKHELog/Qo3wHW1taULFmSkiVLUrp0aVq0aEH79u05cOBArrVZvHhxnRaMkGm8oiCQRPmOMjExUS+oMWHCBMaNG0enTp2oV68ev//+OxkZGcyePZsGDRpQu3ZthgwZwp07d9Tvj4uLY9iwYdSoUYM2bdpw4cIFjfM/P4zNzs5m0aJFNGnShBo1atC/f39u3bpFREQEEydO5NGjR7i6unL37l1UKhXLly+nadOm1KhRg169enHx4kX1eVNSUpg4cSK1atWiadOmHDt2TOdrTkpKYtKkSTRs2JBq1arh7e1NaGioRp2YmBi6dOmCm5sb3bt358qVKxrvDwoKok6dOtSvX58xY8YQFxen+4cuCixJlO+YrKwsTp06xc6dO2nevLm6fNeuXfTr149Vq1bh5ubG/PnziYiIICQkhM2bN1OyZEn69OmDUqkEYMSIETx58oSNGzcyceJEVq1a9dI2Fy5cSGhoKJMmTWLHjh1YW1urk2xQUBDFixfn+PHjODg4sGHDBjZv3sy0adPYvn07derUoXfv3vz9998AfPXVV/z555+sWrWKuXPn8uOPP+p87TNnzuTixYssXbqUn376iY8++ogZM2bw119/qets3LiRbt26ERYWRunSpRk2bJj6Xu4XX3zB/fv3WbNmDWvWrCE5ORl/f3/pFb8LVKJQa9asmap69eoqT09Plaenp6pKlSqq6tWrqyZOnKhKS0tTqVQq1fjx41W+vr7q96SmpqqqV6+uOnfunLosKytL1bhxY1VYWJjq6tWrqkqVKqliYmLUx0NDQ1WVKlVSKZVKdbsbNmxQZWdnq+rXr68KDQ1V1/37779Vs2bNUiUmJqq2bdumatCggfpYkyZNVHv37tW4hm7duqkWLVqkevLkiapKlSqq48ePq48dPXpUValSJdX169dfev0bNmxQqVQq1bZt21SXLl1SH0tPT1dVqlRJfb5mzZqpgoOD1ccTExNVnp6eqoMHD6pu3bqlcnV1VT169Eh9PCkpSVWtWjXVb7/9prpz584r4xAFmzzMeQf4+/vj6+sLgJmZGfb29uph9zMffPCB+ufbt2+Tnp5Onz59NLY3UCqVxMTEYGZmhpWVFU5OTupjbm5uL2z7n3/+4fHjxxrH7e3t1XuSPy85OZm//vqLCRMmEBQUpC5PT0+nbNmyxMTEkJWVpbHtxcvafZGPP/6Yw4cPs337dmJiYrh8+TLwtJf9jIeHh/pna2trnJyciI6OxtjYGJVKRYsWLTTOmZmZSUxMjOzhXchJonwHFC9eHEdHx1fWef7By7PEsXbt2hyrpdvY2HD69Gmys7M1yv+deLWVv8izdufMmYOrq6vGMSsrK9iwLLMAAALoSURBVO7fvw9oPgB6nWXmJkyYQEREBB06dKBjx454enrSrFkzjTr/XpEpOzsbMzMzsrKyMDc3JywsLMd5ixcvTkJCgs5xiIJH7lGKHMqVK4eJiQlxcXE4Ojri6OhImTJlmDdvHlFRUVSqVAmlUklUVJT6Pc8/cHmejY0NJUqUUPfeABISEqhfvz7R0dEaPVZbW1tKlizJw4cP1e06OjqyfPlyTp8+jbOzM6amppw7d05ru//2zz//EBYWxty5cxk9ejRt27ZVbzv7fOJ9/uFNQkICt27donz58pQvX560tDTS0tLUcdnZ2TFz5kxiY2N1ikEUXNKjFDkUKVKETz75hKlTp2JiYkLZsmVZsmQJv/76K19++SXvvfcejRs3JigoiClTpvDkyRMWLlz40vP17duXhQsX4uDgwAcffMB3331HqVKlKF++PNevXycpKYno6GgcHR0ZOHAgISEhlChRgmrVqvGf//yHXbt20bt3b6ytrenUqRMzZ87E1tYWU1NTZs6cqdM1WVtbU6RIEQ4cOEDp0qWJjY1lxowZABoT79evX0/lypWpXLky8+bNo1y5cjRu3BgjIyO8vb0ZN24ckyZNwtbWllmzZnHz5k2cnJzUD5tE4SSJUrzQuHHjMDIyYvz48aSkpFCtWjVWrlzJe++9B8C3335LcHAwvXr1olixYvTt2/elSat///4kJSUxceJEUlJSqFu3LosXL0ahUFC/fn1cXFz46KOPWL9+PX369CE1NZXZs2fz+PFjXFxcWLJkCZUrVwaePnmeNWsW/v7+WFhYEBAQQHBwsNbrMTU1Zd68ecyaNYtNmzbh4OBA586dMTc358KFC+p7jwMGDGDFihXcuHGD2rVrs2jRIvUe6bNnz2bmzJkMHTqUzMxMatWqxapVqzA3N9fDJy7yM1m4VwghtJB7lEIIoYUkSiGE0EISpRBCaCGJUgghtJBEKYQQWkiiFEIILSRRCiGEFpIohRBCi/8DJ7ohi1pWslYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from sklearn.metrics import confusion_matrix\n", "\n", "class_names = ['Spam', 'Ham']\n", "\n", "ham_only_cnf_matrix = confusion_matrix(y_train, ham_only_y_pred, labels=[1, 0])\n", "\n", "plot_confusion_matrix(ham_only_cnf_matrix, classes=class_names,\n", " title='ham_only Confusion Matrix')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Summing the quantities in a row indicates how many emails in the training dataset belong to the corresponding class:\n", "* True label = spam (first row): the sum of true positives (0) and false negatives (42) reveals there are 42 spam emails in the training dataset.\n", "* True label = ham (second row): the sum of false positives (0) and true negatives (758) reveals there are 758 ham emails in the training dataset.\n", "\n", "Summing the quantities in a column indicates how many emails the classifier predicted in the corresponding class:\n", "* Predicted label = spam (first column): the sum of true positives (0) and false positives (0) reveals `ham_only` predicted there are 0 spam emails in the training dataset.\n", "* Predicted label = ham (second column): the sum of false negatives (42) and true negatives (758) reveals `ham_only` predicted there are 800 ham emails in the training dataset.\n", "\n", "We can see that `ham_only` had a high accuracy of $\\left(\\frac{758}{800} \\approx .95\\right)$ because there are 758 ham emails in the training dataset out of 800 total emails." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAEeCAYAAADsNlzkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8jFf7+PHPZF9FJR6SiiSEWCOxxFYNoYhGtGqn2lDEllJiiaZSu9r6iL22Eltt0Qe1xNpaokQpaosgxJpKJJHINr8//MzXCGa0k2WS6+01r1dyzpn7vu6Iyzn3feYchVKpVCKEEOK1DAo7ACGEKOokUQohhAaSKIUQQgNJlEIIoYEkSiGE0EASpRDFkExm0S1JlEJNeHg4TZs2/dfHOX36NEFBQbz33nt4enri7+9PREQE2dnZOohS3fXr1+nYsSO1a9dmyJAhOjmmm5sb69at08mxXic8PBw3Nzd8fHxeWZ+ZmUndunVxc3MjNjZW6+OePHmS4ODgN7aJjo5+6+OWZEaFHYAofjZs2MCECRNo3bo148ePx9LSkujoaKZPn87p06eZOXMmCoVCZ+dbsWIFt2/fZv78+Tg4OOjkmBs2bMDR0VEnx3oThULB7du3+euvv6hevbpa3ZEjR0hLS3vrY27atImbN2++sU3NmjXZsGEDFSpUeOvjl0SSKIVOXbp0iYkTJzJw4EC13l2TJk1wcXFh9OjRfPjhh6/tRf0TKSkpVKlShffff19nx/Tw8NDZsd7E1tYWc3NzoqKi8iTKPXv2ULVqVS5fvqzz81pZWRXYNRYHMvR+yenTp+nWrRuenp40atSI0aNH8+jRIwBu3bqFm5sbe/fupWvXrri7u9OxY0eOHz+udowDBw7QrVs3PDw8cHd3p1u3bpw6dUpV/+mnn/Ldd98xefJkGjRogJeXF7Nnz+bx48d89dVXeHh40KpVK6Kiot4q9szMTObNm0fr1q1xd3fn448/5sCBA6r658OtmJgYOnXqRO3atWnTps1rzzNp0iSaNWtGbm6uWrmPjw/z5s175XvWrFlD6dKl6d+/f566Dh068Pnnn2NpaakqO3/+PAEBAdSvX59GjRoRGhpKSkqK2s9q5syZTJ8+nUaNGuHp6cmIESNITU1VxbJjxw5+//133NzciI6OZsyYMXTp0kXt3OvWrcPNzU31fWxsLH369KFu3brUr1+fIUOGEB8fr6p/eej9b+N8kw8++CDP30F2djb79++nTZs2edpv2bKFjz76CHd3dzw9PQkICODq1asAjBkzhq1bt3LmzBnc3Ny4desW4eHhdO/enfHjx+Pp6Um/fv3Uht7x8fF4eHgwfvx41TlOnTpFtWrVWL9+vcb4SwJJlC9ISUkhMDCQd999lwULFhAaGsqRI0f49ttv1dqNHTuWevXqER4ejp2dHf369ePatWvAs0Q7aNAgPDw8WLRoEdOnTyclJYWRI0eSk5OjOsb69eu5d+8ec+fOpUOHDixevJjOnTvj4ODAwoULKVeuHGPGjOHJkydaxz9y5EiWL19Or169mDdvHq6urgwcOFAtWT5v99FHH7F48WIcHBz46quvSEpKynM8f39/7t+/z8mTJ1Vlp0+f5vbt27Rv3/6VMRw/fpyGDRtiYmKSp06hUDB27FgaNmwIwLlz5+jWrRvGxsbMnDmTESNGsG/fPvr166f2s1q3bh1xcXF89913jBo1ij179rBw4UIA5s2bR5MmTahRowYbNmygZs2aGn9Oubm5DBo0CIVCwdy5c5k+fTqxsbF89dVXr2yvizjfpFWrVly8eJFbt26pyn7//XcMDQ2pV6+eWtudO3cybtw4fH19Wbp0KePHj+fatWt8/fXXAAwaNAhvb2+qVKnChg0b+M9//gPA2bNnSUhIYMGCBfTp00ftmI6OjgwePJiffvqJCxcukJmZSWhoKI0aNaJbt24a4y8JZOj9gtjYWJKSkujVqxeenp7AsyHK8yT4XKtWrRg1ahQAjRs3plWrVqxatYqwsDBiY2Px8/NjzJgxqvZGRkYMGTKEhIQE1X0vCwsLZs2ahbGxMfXr12fdunXY29szcuRIAExMTOjRowfXr1+nRo0aGmO/ePEiu3fvZsaMGfj7+wPw/vvvc//+fb7//ntatGihatuvXz+6d+8OgL29PW3btiU6OjpP78Xd3R0XFxd27NiBl5cXADt27KBOnTo4OTm9Mo779+9rfZ9w4cKFvPvuuyxcuBBDQ0MAXFxc6NmzJwcOHKBVq1YAWFpaMm/ePIyMnv26njx5ksOHDxMcHEyNGjUoXbo02dnZWg8lExMTuX79OkOHDuW9994Dng2BDx06RE5OjioWXcb5Jp6enpQtW5aoqCg+//xzAHbv3k2rVq0wMFDvy8THx/P5558zYMAAVVlSUhLTpk0jNzeXihUrUqZMGZKSktR+HtnZ2YwbNw5nZ2fg2ejiRQEBAezYsYPJkyfTqFEj7ty5w5IlS7T5cZYI0qN8gaurK6VLl2bgwIFMmjSJo0eP0rRpUwICAtTatWvXTvW1iYkJzZo1IyYmBoBOnToxY8YMUlNTOXv2LJGRkfz8888AZGVlqd5Xo0YNjI2NATA2NsbGxkYtIZYuXRpAbXj3JqdOnUKhUNC2bds8sV68eFFtCFinTh3V1+XLlwcgPT39lcf19/dnz549ZGdnk5OTwy+//IKfn99r4zAwMFDrZWmKuXXr1mqJqX79+pQtW1btVkXNmjVVyQegXLlyb9XTfpmtrS3Ozs6Ehoby9ddfs2/fPqpVq8aXX36ZJ0kWRJwGBgb4+Piwb98+4NnUnqioKD744IM8bQcMGMDo0aNJSkri1KlTbNy4kQMHDqBUKt84o8DIyOiND6eMjIyYMGECMTExLFiwgJEjR8qDnhdIonyBlZUVq1evxsvLi02bNhEQEMD777/P5s2b1do9H848V6ZMGR4/fgxAWloaw4cPp0GDBvTo0YOVK1eq5rS9OLftxft0z5mbm//j2JOTk7G2ts4z5LW1tVXF9Zypqanq6+c9lpfvQz7Xvn17Hj16xPHjx4mOjubRo0dq/1G8zMHBgbt37762/t69e6qfw+PHj1XxvRzzi4ndzMxMrd7AwOBfzRM0MDBgxYoVtG3bll27djFo0CCaNm3K4sWLX9m+IOL84IMPOHXqFI8ePSImJoanT5/SqFGjPO3u3btH3759adiwIX369GHjxo2q5Pymc5UuXfqV/wm8qFatWjg7O6NUKnUyRaw4kUT5kqpVqzJ37lxOnDjBkiVLcHV15euvv1b7x//y/bzExETKlCkDPHsAcurUKVatWsXp06eJjIykc+fO+R63jY0NKSkpZGZmqpU/fPhQVf9PODo6UrduXfbu3cuePXto3LgxdnZ2r23fuHFjTpw4odZ7flGPHj348ssvAShVqhSJiYl52iQmJv7jeOHZvdCXe7Uv9+wcHByYOnUqx48fZ/Xq1TRt2pTZs2dz+vTpPMfLrzhf1KhRIywtLTl48CBRUVH4+PioRhwvCg4O5v79+2zdupWYmBh++uknWrZsqZMY1q1bR3x8PE5OToSFhenkmMWFJMoXHDt2jMaNG/P3339jYmKCt7c3I0aMIDc3l/v376vaHTp0SPV1ZmYmhw8fpn79+gD88ccf+Pj40KBBA9Uv+tGjR4H8/bRE3bp1USqV7Nq1S638l19+oXr16nl6O2/D39+fQ4cOcejQodc+xHmuW7duPHr0iKVLl+ap27JlC7du3cLX11cV8549e9SS2smTJ3nw4MG/mrpiYWGh1nMF1IbI165d47333uP8+fMYGRnh5eXFN998A/DK3nB+xfkiY2NjvL29OXDgAFFRUbRu3fqV7f744w/8/f2pUaOGqof4/Pfr+ajg5fua2rh37x6zZ88mICCAyZMnc+zYMbZu3foPr6b4kYc5L6hVqxYKhYKgoCD69esHwJIlS6hQoQLVq1fn3r17AKxevZrSpUtTrVo1Vq9eTXp6uuo+Zq1atdi1axf16tXDzs6O/fv3s2bNGiBvr0aXqlevTqtWrQgLCyMpKQkXFxe2b99OdHT0a6fyaMvX15dJkyZhaGioenDxOpUrV2b48OHMmDGD2NhY2rVrh5GREb/99htr166lQ4cOqkQZGBhIjx49GDhwID179uTBgwd8//331K5dm+bNm//jeN977z0iIiKYNm0aLVq04ODBg2qJ0tnZmVKlSjFmzBiGDh2KlZUVa9euxdraWvVE/kX5FefLWrduzVdffYWxsbHqIdPLatWqxU8//YSzszPm5ub8/PPPqqlF6enpmJubU6pUKW7evMmxY8dUDyU1+fbbb7G2tmbgwIFYWFjQoUMHpk2bhre3t2q0VJJJj/IF1tbW/PDDDxgZGTFixAiGDRuGpaUly5YtUxsGjRw5kt27dxMUFERGRgarV6/G3t4eeDaPrX79+oSFhTFs2DAuXrzIypUrMTc358yZM/ka/6xZs+jSpQtLlixh8ODBxMbGsnDhQo3JTRMbGxvc3d3x8fHByspKY/svvviCBQsW8ODBA8aNG8eXX37J77//zrhx45gyZYqqnbu7OytWrODx48cMGTKE2bNn07JlS1asWKH2UORttWjRgqCgIHbs2EFgYCB37txRmyNoYGDAokWLcHR0JDQ0lIEDB5KUlMTy5ctfmRTyK86XNWvWDENDQ7y9vdXuI79o6tSpODg4MGrUKEaNGkVycjLLli0DnvU2Abp06YKVlRX9+/fnwoULGs+7e/du9u3bx5gxY7CwsACeDfFzcnKYOnWqjq5OvylkKwjt3bp1i5YtW/LDDz/o9FMgRV1ycjLNmjUjPDwcb2/vwg5HiAInQ289oM1CErrs2TyXmJjIunXrOHz4MPb29jRr1kzn5xBCH0ii1APafNpk3759Op/3ZmxsTEREBFZWVsyePfsfPSQQojiQobce+PPPPzW2cXNze+XHBoUQ/54kSiGE0EDGUkIIoYHco3xJYqruV+AWb8fSTH4tiwJd/zWYe2q3+nz66X837zc/yG+kEKJgKPR3ACuJUghRMAzevChHUSaJUghRMHS4T1JBk0QphCgYMvQWQggNZOgthBAayNBbCCE0kKG3EEJoIENvIYTQQIbeQgihgR4PvfU3ciGEfjE01O6lhZ9//hlPT0+1V7Vq1QgNDSU5OZnBgwdTr149mjdvzsaNG1Xvy8zMJCQkBC8vL5o0acLChQu1Op/0KIUQBUOHPUp/f3/8/f1V3x89epRRo0YxePBgQkNDsbCw4OjRo1y6dIl+/fpRpUoVPDw8mDNnDgkJCezbt4/ExET69OmDk5PTG7dgBulRCiEKikKh3estpaWlMWbMGMLCwrC2tiYqKoqgoCBMTU1xd3fHz8+PyMhIALZt28aAAQOwtrbG2dmZXr16abXbpCRKIUTBUBho93pLS5cupWrVqrRq1YobN25gZGSEo6Ojqt7FxYVr166RnJxMYmIirq6ueeo0kaG3EKJg5MP0oLS0NCIiIvjhhx+AZ1tCv7yHvZmZGRkZGaSnpwNgbm6ep04T6VEKIQpGPgy9o6KicHBwwMPDA3iWBJ8+farWJiMjAwsLC1UCfTExPq/TRBKlEKJg5MPQ+8CBA/j6+qq+d3JyIisri4SEBFVZXFwcrq6ulC5dGltbW+Li4tTqKleurPE8kiiFEAXDwFC711s4c+aMqjcJYGVlRcuWLZk1axbp6emcPXuW7du30759e+DZ0/Lw8HCSkpK4fv06ERERdOjQQXPob3elQgjxD+l46J2Tk8OdO3coW7asWvnEiRPJzs7G29uboKAggoODqVOnDgDDhg3D2dkZX19fevToQZcuXdR6pK8NXXZhVCd75hQ+2TOnaND5njl+2u2Fk75du711CpL8RgohCoYsiiGEEBro8We9JVEKIQqGrB4khBAayNBbCCHeTCE9SiGEeDNJlEIIoYHCQBKlEEK8kfQohRBCA0mUQgihgQy9hRBCA+lRCiGEBpIohRBCA0mUQgihgdyjFEIIDfS5R6m/y3kIIfSKQqHQ6qWtu3fvMmDAAOrWrcv777/PqlWrAEhOTmbw4MHUq1eP5s2bs3HjRtV7MjMzCQkJwcvLiyZNmrBw4UKtziWJUs8lPXpEjcoVmD5lgqrs/v17DA3sS62qTlSqYMfHfq358+wfhRhlyTI/fC7VqrhQppQFvq1bcunixcIOqUhQGCi0emlDqVQyaNAgKlWqRHR0NMuWLWPevHnExMQQGhqKhYUFR48eZe7cucycOZM//nj2+z9nzhwSEhLYt28fa9euZePGjezcuVPj+SRR6rnx40Zx//491fc5OTn06voxx48d4dvJ01myPAIDAwPat/Xh9q34Qoy0ZFixbCljRo3gi/6BrF67gfT0dNq1bUVqamphh1bodNmjPHPmDPfv32fkyJEYGxtTpUoV1q9fT7ly5YiKiiIoKAhTU1Pc3d3x8/MjMjISgG3btjFgwACsra1xdnamV69ebN26VeP5JFHqsV8PH+R/P2/F0spKVXb82BFiTv7OslXr+KRzN1q1bsvq9VswNTFlzeqVhRdsCaBUKpk6ZSJDgoYxMng0H/q1Z9v2X0hNSSFi1Y+FHV6h02WiPH/+PFWqVGHGjBk0bdqUNm3acObMGZKTkzEyMsLR0VHV1sXFhWvXrpGcnExiYiKurq556jSRRKmnMjIy+CpoIF+HTcLSwlJVbmpiwmd9+uFex1NVZmFhgUOFCsTfuF4IkZYcsVevEn/zJh+291eV2djY0Ox9b/ZF7SnEyIoGAwMDrV7aSE5OJjo6mnfeeYcDBw4wdepUJk6cyJMnT1T7dz9nZmZGRkYG6enpwLO9v1+u0xj7W1ynKEJmTJ2Ina0dAX0HqJXX92rErP8uUCuLv3mDS39doErVagUZYolz5cplACpXdlUrd3J2ITb2amGEVLQotHxpwcTEBBsbGwYMGICJiQl169alTZs2zJ07l6dPn6q1zcjIwMLCQpVAX0yMz+s0kUSph86fO8uSheHMDl+ocaiSnZ3NV0EDMbewoMennxdMgCXU48ePAbC2tlYrt7a2JiUlpTBCKlJ0OfR2cXEhJyeHnJwcVVlOTg41atQgKyuLhIQEVXlcXByurq6ULl0aW1tb4uLi1OoqV66s8XySKPVMTk4Ow4YMYMCgIKrXqPXGttnZ2Qzs9zmHD+5n/uIV2L20/7HQrdzcXODV8wX1eQ6hruhy6N20aVPMzMyYN28e2dnZxMTEsHfvXtq2bUvLli2ZNWsW6enpnD17lu3bt9O+fXsA/P39CQ8PJykpievXrxMREUGHDh00x/6vrvwtxcXFMXDgQBo0aICnpyf+/v5qc5yEZksWhvN3YiJfjhhNdnY22dnP9iHPzc1VfQ3w5MkTenX9mP9Fbmb+khW0bedXWCGXGDY2NgB5nnCnpKRgU8qmMEIqWnQ49DYzM2P16tWcPXuWJk2aMHLkSL7++ms8PDyYOHEi2dnZeHt7ExQURHBwMHXq1AFg2LBhODs74+vrS48ePejSpQu+vr6aQ1cqlcp/cMlvLTc3lw8++ICOHTvSt29fTExMOHnyJEOGDGHixIm0adOmIMLQKDE1W3OjQuTv25Kjvx1+bf3DlCxSUlLo3MGXP8/+wZLlEXzo/1EBRvjvWZrp5wfGLl+6RJ1a1Thw+CiNGjdWlXf62B9jExPWbdhUiNG9PV3/NVQc+rNW7W6G+2tuVMAK7Dfy0aNH3Lp1C39/f9VNVS8vL4KDg8nKyiI8PJxr166RmJjI2bNnqVKlChMmTKB69eoArFq1io0bN5KQkICpqSndu3dn6NChALi5uTFhwgQWLFhASkoKffv2xd7enu+//56MjAwGDBhA3759C+pS89Ws/y4gNVX9flfXj/340P8jegd8gVKppM+nXTl/7ixrfoqkuU+rQoq05KlStSoO777Lju0/qxJlcnIyvx4+xLcTpxRydIVP22F1UVRgidLW1hYvLy8CAgLw9/fHy8sLd3d3OnfuDEB4eDi//PILs2fPZtmyZSxevJhBgwaxe/duzp49y6JFi1i7di3Ozs6cPHmSXr164e/vj5OTEwBHjhxh165dxMTE0LdvX3x9fdmzZw/R0dEMHDiQLl265LnJro+qVHXLU2ZkZER5ewc869Zn88b1HNi3ly8GDMLKyoqTJ46r2pWxtaPSS09khe4oFAq+GjGK0cFfYWVlRa3a7syYPhXrUqXo+Wnvwg6v0OnzfdoCHeMsXbqUdevWsXfvXpYsWQJA69atCQ0NBaBx48a0a9cOgIEDB7J69WpiYmKoU6cOW7ZsoXz58jx8+JCsrCzMzMy4f/++KlH26tULc3NzGjVqhFKppGfPnpiZmdGsWTNycnK4d+9esUiUmvyy438ALF28gKWL1acJffRJF5auXFMYYZUYg4cG8eRJGgsXzONxcjJeDRux45e9JeJ3TyP9zZMFmyhNTU35/PPP+fzzz3n69CmnTp1i5syZhISEUKNGDSpWrKhqa2hoSNmyZXn48CEGBgYsWLCA3bt3Y2trS61az572Pn/KCP93I93Q8Nkm66VKlQL+r7v/Ytvi5kLsLdXXS1eukWRYyIJHjyV49NjCDqPI0eehd4FFvnPnTlq2bMnzZ0empqY0adKEoUOHcvH/Lxpw//59Vfvs7Gzu379P+fLlWbFiBZcvXyYqKoqdO3cyefJktSe8oN/deiFKAoVCu1dRVGCJsnHjxqSlpTF58mQSExNRKpXcuHGD1atX06JFCwB+/fVXjh49SlZWFvPnz+edd97B09OT1NRUjI2NMTY2Ji0tjenTp5OVlZUnWQohii5dL7NWkAosUb7zzjusXbuW+/fv4+fnh4eHBwEBAdSuXZsxY8YA4O7uzg8//EDDhg05efIkixcvxtDQkICAAIyMjGjcuDFt2rQhMzOTunXrEhsbW1DhCyH+JX3uURbYPEpNwsPDuXLlCnPnzi3UOIr6PMqSQF/nURY3uv5rqDZmt1btLk4rGnOqXyS/kUKIAmEge+YIIcSbFdVhtTaKTKJ8/ikbIUTxJD1KIYTQoKg+0daGJEohRIGQRCmEEBrI0FsIITTQ4w6lJEohRMHQ56G3/n5KXQihVwwMFFq9tLVs2TJq1aqFp6en6nXy5EmSk5MZPHgw9erVo3nz5mq7KGRmZhISEoKXlxdNmjRh4cKFWp1LepRCiAKh6w7lhQsXGD58eJ5FuYOCgrCwsODo0aNcunSJfv36UaVKFTw8PJgzZw4JCQns27ePxMRE+vTpg5OTk2p5x9eRHqUQokDoelGMv/76S7UDwnNpaWlERUURFBSEqakp7u7u+Pn5ERkZCcC2bdsYMGAA1tbWODs706tXL7Zu3arxXJIohRAFQpdD7/T0dOLi4li1ahVNmzbF19eXTZs2cePGDYyMjHB0dFS1dXFx4dq1ayQnJ5OYmIirq2ueOk1k6C2EKBC6HHo/fPiQevXq0b17d+bOncvZs2cJDAwkICBAtSfXc2ZmZmRkZJCeng6Aubl5njpNXpsoX9wkXBMXFxet2wohSiZdPvV2dHQkIiJC9X39+vXp0KEDJ0+e5OnTp2ptMzIysLCwUCXQjIwMrKys1Oo0eW2i9PX1RaFQ8LpV2J7XKRQK/vrrL81XJoQo0XQ54fz8+fMcOXKE/v37q8qePn2Kvb09WVlZJCQk4ODgADzr9Lm6ulK6dGlsbW2Ji4vDzs5OVVe5cmWN53ttoty3b9+/vRYhhFDRZY/SwsKCefPmUbFiRVq3bk10dDQ7duwgIiKClJQUZs2axaRJk7hy5Qrbt29XbWbo7+9PeHg4c+fOJSkpiYiICIKDgzXH/jYL98bExBAXF0ebNm24c+cOTk5OmJiY/POrLYJk4d7CJwv3Fg26/mvwnnNEq3aHhjfVqt3+/fuZM2cO8fHxlCtXjuHDh9O2bVuSkpIYP348x44dw8LCgiFDhtCpUyfg2VB7ypQp7N27F4VCQe/evQkMDNR4Lq0S5d9//01gYCAXLlwgNzeX3bt3M3nyZGJjY1m+fLnaEyZ9J4my8EmiLBp0/dfQ/PujWrU7OKyJbk+sA1pND5o8eTJ2dnZER0djamoKwPTp06lYsSKTJ0/O1wCFEMWDrj+ZU5C0SpRHjx5l2LBhWFpaqspsbGwYM2YMJ0+ezLfghBDFhz5vLqZV5zonJ4fc3Nw85SkpKRgaGuo8KCFE8WNQVLOgFrTqUbZq1YoZM2bw999/q55cXb16lYkTJ9KyZct8DVAIUTwU+6F3SEgIVlZWNG3alCdPntC+fXvat2+Pvb09ISEh+R2jEKIYMFBo9yqKtBp6W1lZ8d///pf4+HhiY2PJzs6mcuXK8okcIYTW9Hk9Sq0nAOTk5HD58mViY2MxMTHBzMxMEqUQQmv6fI9Sq0R56dIlBg4cSFJSEs7OzuTm5nLjxg2cnZ2ZN28e7777bn7HKYTQc0V1WK0Nre5Rjh8/njp16nD48GG2bNlCZGQkBw8epFy5cnzzzTf5HaMQohjQ9XqUBUmrRHnhwgWGDh2qWnEDns2jHDFihMyjFEJoxdBAodWrKNIqUVarVo0zZ87kKb906ZLcpxRCaKVYTjjfsGGD6mtPT0/CwsI4f/487u7uGBoacvHiRSIiIvLsVyGEEK9SVIfV2njtohg+Pj7aHUChKFZLssmiGIVPFsUoGnT919D1x9NatdvwmaduT6wDr/1R7N+/vyDjEEIUc/rbn3yLeZT37t3j2rVr5OTkAKBUKsnMzOT8+fMEBQXlW4BCiOJBn4feWiXKNWvWMGXKFHJyctS2h1AoFNSpU0cSpRBCo6L6RFsbWj31XrZsGQMHDuTPP//E1taWgwcPsn37dqpVq8YHH3yQ3zEKIYqB/Hrq/fDhQxo3bsyBAwcAuHXrFp999hmenp60adNGVQ6QnJzM4MGDqVevHs2bN2fjxo1anUOrRHn//n06dOiAsbEx1atX548//sDV1ZWxY8dqfSIhRMmWXxPOx40bR1JSkur7L7/8End3d06cOEFISAgjRowgISEBgNDQUCwsLDh69Chz585l5syZ/PHHHxrPoVWiLF26NCkpKcCzrWkvXboEwLvvvsvdu3ff+sKEECVPfqx9XnbuAAAgAElEQVQetG7dOszNzbG3twcgNjaWy5cvM3jwYIyNjfH29sbLy4sdO3aQlpZGVFQUQUFBmJqa4u7ujp+fH5GRkZpj1yaYFi1a8M0333Dx4kUaNWrEtm3biImJYfXq1aoAhRDiTQwUCq1e2oqLi2PFihWEhYWpyq5du8a7776r2sMbnnXurl27xo0bNzAyMlLb4+t5ncbYtQlozJgxVKtWjYsXL+Lj40ODBg3o0aMHGzduZMyYMVpfmBCi5NJloszOzmbUqFGMGzeO0qVLq8qfPHmCubm5WlszMzMyMjJ48uSJWgJ9sU4TrZ56W1paMmnSJNX306dPZ+zYsVhZWWFkJJODhRCa6XJ20IIFC6hevTre3t5q5ebm5nkSX0ZGBhYWFpibm/P06dNX1mmi1UcYNenatavWbYUQJZMut3nYuXMnDx48YOfOnQCkpqby1VdfERgYyO3bt8nMzMTExAR4NkRv2LAhTk5OZGVlkZCQgIODg6rO1dVV4/lemygXL16sVcAKhUISpRBCI10u3Ltr1y617318fAgNDaVFixbs3r2b77//nmHDhnHs2DGio6MZP348VlZWtGzZklmzZjFp0iSuXLnC9u3bWbJkicbzyUcYX1Kh2bDCDqHEe/T7vMIOQeSDgvpgTnh4ON988w2NGzfGzs6O2bNnqx46T5w4kfHjx+Pt7Y2FhQXBwcHUqVNH4zFfuyhGSWXuOaSwQyjxJFEWDbpeFGPo1r+0ahf+cXXdnlgH5EmMEKJAFPvPegshxL+lxx/1lkQphCgYxX5RDIDExEQWLVrE6NGjSUxMZOfOnVy+fDk/YxNCFCP58RHGgqL15mJt2rTh4MGD7NixgydPnnD06FE6d+7MsWPH8jtGIUQxoM975miVKKdOncpnn33G+vXrMTY2BmDSpEl8+umnzJw5M18DFEIUD0YKhVavokirRHn+/Hn8/f3zlHft2pXY2FidByWEKH6KfY/SxsZGtZ7bi86fP0+ZMmV0HpQQovjR9epBBUmrp97du3fnm2++YeTIkcCz/bwPHz5MeHg4AQEB+RqgEKJ4MNT60XHRo1Wi7N+/P5aWlkybNo309HSGDBmCnZ0dgYGBfPbZZ/kdoxCiGCiqvUVtaD2PsmfPnvTs2ZMnT56Qk5ODtbV1fsYlhChm9DhPapcoNS2V/tFHH+kkGCFE8VVU50hqQ6tE+fIUoOzsbB4/foyJiQnVqlWTRCmE0MhQj7uUWiXK3377LU9ZcnIyoaGh1K1bV+dBCSGKH33uUf7j51A2NjYMGzaMpUuX6jIeIUQxlV/b1RaEf/XA/tatW6Snp+sqFiFEMWZooN1LWzt37sTX1xdPT08+/PBDoqKigGej3cGDB1OvXj2aN2/Oxo0bVe/JzMwkJCQELy8vmjRpwsKFC7U6l1ZD7xEjRuQpS01N5cSJE/j5+Wl1IiFEyabL6UFxcXGEhISwfPly6taty9GjR+nfvz+HDx8mLCwMCwsLjh49yqVLl+jXrx9VqlTBw8ODOXPmkJCQwL59+0hMTKRPnz44OTnRrl27N55Pq0T5fJOeF5UrV46QkBA6dOjwz65UCFGi6PIepYuLC0eOHMHS0pLs7GwePnyIpaUlJiYmREVFsXv3bkxNTXF3d8fPz4/IyEg8PDzYtm0bs2bNwtraGmtra3r16sXWrVt1kyjd3Nzw9fWlXLlyOrlIIUTJo+un3paWlsTHx9OmTRtyc3MJCwvj5s2bGBkZ4ejoqGrn4uLCnj17SE5OJjExUW3XRRcXF9asWaPxXFrdEZg/fz6ZmZn/4FKEEOKZ/FgUw97enjNnzrBixQqmT5/O/v37MTMzU2tjZmZGRkaG6nmKubl5njpNtEqU77//PhERETx+/PhtrkEIIVTyY+FeIyMjjI2Nady4Ma1bt+bcuXM8ffpUrU1GRgYWFhaqBPpiYnxep/E82gRz48YNduzYwapVq7CyssLU1FSt/lXzLIUQ4kW63Ari0KFDrFixgpUrV6rKsrKyqFixIocPHyYhIQEHBwfg2YMfV1dXSpcuja2tLXFxcdjZ2anqKleurPF8WiXKnj17/oNLEUKI/6PLp941atTg3LlzREZG4u/vz6+//sqhQ4f46aefuHPnDrNmzWLSpElcuXKF7du3s2TJEgD8/f0JDw9n7ty5JCUlERERQXBwsMbzvXZf73nz5tG3b1+18XxJIPt6Fz7Z17to0PW+3st/v6lVuz4NKmrV7uTJk0yZMoXr16/j7OzMqFGjaNSoEUlJSYwfP55jx45hYWHBkCFD6NSpE/BsqD1lyhT27t2LQqGgd+/eBAYGajzXaxNl9erV+e2337C1tdUq6OJCEmXhk0RZNOg6Uf54Ml6rdp/Vd9TcqIC99kfxmvwphBD/SNH8cKJ23vh/RlZWllbTgl41IV0IIV5UbBfubdGihVYH+euvv3QSjBCi+NLn1YPemCjnzp2LjY1NQcUihCjGiurKQNp4baJUKBTUrVu3xD3MEULkDz3eW0we5gghCkax7FF+/PHHeT6BI4QQ/1SxfJgzderUgoxDCFHMFcuhtxBC6FKxHHoLIYQuFdvpQUIIoSsGevzZHEmUeqBZvSrsWfrla+vNPYfQqXVdVk/vk6fOf/B89h599oGAhu4uTB3+EbWqvMvdh49ZsO4gizYczre4S6r54XMJnzuH+/fu0bBRY76fOx+3atUKO6xCp8cjb0mU+uCPi/F4956pVvaOjQXrZnzBhl0nAajp6sCfl28zZNI6tXYX4+4CUNH+Hf63YDC/xVyl+8ileNaoyMzgTjzNzGbF1qMFcyElwIplSxkzagTjJ0yievUazJg+lXZtW3Hm3EWsrKwKO7xCVSyfeouiIyUtgxN/Xlcr+3Hq59y+n8RX059txVmr6rucunAjT7vn2jevg6GBAT1HLSM9I4t9xy9Ss7I9AR83kUSpI0qlkqlTJjIkaBgjg0cD8F6z96laqSIRq34kcNDgQo6wcOnz0Fufn9iXWI3rVKJL2/qMnrWF9IwsAGq52nP+SsJr32NmakRWdg4ZT7NVZY8eP+EdG83L4AvtxF69SvzNm3zY3l9VZmNjQ7P3vdkXtacQIysa8mPPnIIiiVIPhQ1pz6+nrrDz8DkArCxMqWhfhobuLpz/eTyPT/yXw6tG0qCWk+o9G3fHkKtUMn6QH6WtzWlWrwrdP2zApt0xhXUZxc6VK5cBqFzZVa3cydmF2NirhRFSkWKgUGj1KookUeqZWlUceL9+Feb8uE+tzMDAAEf7MgyfvpGuI37gaVY2/1swBPuyzxY1uXnnb8bO2crIgA+4c3gGe5Z+yfEzcUxctKOwLqXYeb75nrW1tVq5tbU1KSkphRFSkaLrzcVOnjxJ586dqVevHq1atWL9+vUAJCcnM3jwYOrVq0fz5s3ZuHGj6j2ZmZmEhITg5eVFkyZNWLhwoVbnknuUeqZPx6Zci3/AL7+eU5X9FXuXj4cu5NdTV0hLf7Z+6OGTlzn/vzCCevkwds5WPvVvxKLxPZm35gD/O3iWKk7/IWxwexaE9iDwW837GgvNcnNzgVdPrNbnyda6otDhPcrk5GQGDRpEaGgoH374IX/99RcBAQFUrFiR9evXY2FhwdGjR7l06RL9+vWjSpUqeHh4MGfOHBISEti3bx+JiYn06dMHJycn2rVr98bzSY9Sz/h512ZL1Gm1suTUdHb9dl6VJAHS0jOJPhNHTddnO9GN7tuGTbtPETxzM4dPXmHZ5iMEfruGzz5qTO2q7xboNRRXz5ckTE1NVStPSUnBppQsV6jLoXdCQgLe3t60b98eAwMDatasScOGDYmJiSEqKoqgoCBMTU1xd3fHz8+PyMhIALZt28aAAQOwtrbG2dmZXr16sXXrVs2x/6sr1xE3NzcuX76cp7xhw4ZER0cXQkRFU7VK5XG0L8O2fWfUyt2rvkvvDo3ytDc1NSIt/dkexxXKl87zRPz4mWvPjutSPn8CLmFcXasAEHftmlr5jetxuFatWhghFSm6HHpXr16dGTNmqL5PTk7m5MlnU+WMjIxwdPy/fXdcXFy4du0aycnJJCYm4urqmqdOY+xaXqMoAurWqMjTzCz+uKS+SZO7WwUWh/XCzaWcqqzsO1Y0rlOJIzHPHiJcvfmARnVc1N5X//8/7LmRkJjPkZcMVapWxeHdd9mx/WdVWXJyMr8ePoS3t3a7BRRnCi3/vK2UlBQCAwNVvUozMzO1ejMzMzIyMkhPTwdQ21n2eZ0menOP8sKFC3z33XdcvXqVlJQU6tWrx3fffYednR1jxoyhTJkyxMTEcPHiRWrVqsWoUaOYMmUKly5dwsPDg/DwcL2f8Fujkj1xtxLJzs5VK98adZoxX7Rl/cx+fLtgO0qlkpABviQmpbFsyxEApv2wi9XT+zBr1GO27T9DZceyjB/sx4HoS6+deynejkKh4KsRoxgd/BVWVlbUqu3OjOlTsS5Vip6f9i7s8ApdftymjY+PJzAwEEdHR77//ntiY2N5+vSpWpuMjAwsLCxUCTQjI0OVC57XaVJkEmW3bt0wMFDv4L54r+fLL7+kd+/erFixgqSkJPr3709ERATDhg0DYMuWLURERGBvb88nn3zCoEGD+PHHH7Gzs6NLly5s27aNnj17Fug16ZrdO1Ykp6bnKU9Lz6Rd4FymDPuYueO6YmpsxP7oS4yauVk1z3LTnmfTgEb1bUPfT5py+14Sa/4XzaTFOwv0Goq7wUODePIkjYUL5vE4ORmvho3Y8cvePE/CSyJDHWfK8+fP88UXX+Dv78/o0aMxMDDAycmJrKwsEhIScHB4dn8+Li4OV1dXSpcuja2tLXFxcdjZ2anqKleurPFcRSZRrl+/nqov3cdp2LCh6utly5ZRoUIF0tPTuXfvHu+88w737t1T1bdo0UJ176F27dqYmJiofgB16tTh9u3bBXAV+etNT6dv3nlEr9HL3/j+TXtiVAlT5J/g0WMJHj22sMMocnSZJh8+fMgXX3xBQEAA/fv3V5VbWVnRsmVLZs2axaRJk7hy5Qrbt29nyZIlAPj7+xMeHs7cuXNJSkoiIiKC4OBgjecrMolSk7Nnz9KvXz/S0tJwc3MjOTmZMmXKqOpf3ATN0NCQUqVKqb43MDCQrS2EKGw6zJSbNm3i77//ZuHChWpzIXv37s3EiRMZP3483t7eWFhYEBwcTJ06dQAYNmwYU6ZMwdfXF4VCQe/evfH19dV4Pr1IlHfv3mX06NGsXbtWdcFjx45VS34yT02Iok2Xn7oJDAwkMDDwtfX//e9/X1luZmbGhAkTmDBhwludTy+eeqelpaFUKjEzM0OpVHLo0CF27dpFVlZWYYcmhNCSQstXUaQXPcrKlSszaNAgPvvsM3Jzc6lUqRLdunXj+PHjhR2aEEJbRTULakGhlJt3asw9hxR2CCXeo9/nFXYIAjDTcTfq1PXHWrWr51xKc6MCphc9SiGE/tPjDqUkSiFEAdHjTCmJUghRIIrqWpPakEQphCgQ+psmJVEKIQqKHmdKSZRCiAIhQ28hhNBAf9OkJEohREHR40wpiVIIUSBk6C2EEBrob5qURCmEKCh6nCklUQohCoQut6staJIohRAFQtsdFosivViPUghRDOTTgpRnz57lvffeU32fnJzM4MGDqVevHs2bN2fjxo2quszMTEJCQvDy8qJJkyZqq6O/ifQohRAFQtdDb6VSyebNm5k2bRqGhoaq8tDQUCwsLDh69CiXLl2iX79+VKlSBQ8PD+bMmUNCQgL79u0jMTGRPn364OTkRLt27d54LulRCiEKhIFCu5e2Fi1axKpVq9S2hEhLSyMqKoqgoCBMTU1xd3fHz8+PyMhIALZt28aAAQOwtrbG2dmZXr16sXXrVs2xv/XVCiHEP6Hjofcnn3zCtm3bqF27tqrsxo0bGBkZ4ejoqCpzcXHh2rVrJCcnk5iYqNqt9cU6TWToLYQoELoeev/nP//JU/bkyRPMzMzUyszMzMjIyCA9PR0Ac3PzPHWaSI9SCFEgdD30fhVzc3OePn2qVpaRkYGFhYUqgb6YGJ/XaYz934UlhBBaKoBtGJ2cnMjKyiIhIUFVFhcXh6urK6VLl8bW1pa4uDi1usqVK2s8riRKIUSBUGj559+wsrKiZcuWzJo1i/T0dM6ePcv27dtp3749AP7+/oSHh5OUlMT169eJiIigQ4cOGo8riVIIUSAKYugNMHHiRLKzs/H29iYoKIjg4GDq1KkDwLBhw3B2dsbX15cePXrQpUsXfH19NR5Ttqt9iWxXW/hku9qiQdfb1d569FRzI6DCO6a6PbEOyFNvIUQB0d/PMEqiFEIUCH3+rLckSiFEgdDjdXslUQohCoYssyaEEBpIj1IIITSQRCmEEBrI0FsIITTR3zwpiVIIUTBkepAQQmggQ28hhNBAHuYIIYQGkiiFEEIDGXoLIYQG0qMUQggN9DlRysK9QogCoesVzi9cuECnTp3w8PCgQ4cO/PHHH/kWuyRKIUSBUCi0e2nj6dOnBAYG0rFjR37//Xc+/fRTBg4cSFpaWr7ELolSCFEgdJkojx8/joGBAT169MDY2JhOnTphZ2fHoUOH8iV2uUcphCgQunzq/ardE11cXLh27ZrOzvEiSZQvST8t+7UIkR/MjXV3rCdPnmBubq5WZmZmprZnty7J0FsIoXfMzc3zJMWMjAwsLCzy5XySKIUQeqdSpUrExcWplcXFxeHq6pov55NEKYTQO40bNyYzM5PVq1eTlZXFpk2bePjwIe+9916+nE/29RZC6KWLFy8SFhbGpUuXcHJyIiwsDA8Pj3w5lyRKIYTQQIbeQgihgSRKIYTQQBKlEEJoIIlSCCE0kEQphBAaSKIUQggNJFGWUDIrrGDl5uaqvpafvf6RRFlC5OTkAJCdnQ2AQqGQf7AFJDs7GwMDA9XXQv/IhPMSQKlUolAouHz5MqtXr0ahUDBixAhsbGwKO7QSIzc3lwEDBpCRkUHTpk3x9fXFycmpsMMSWpIeZTH3PEneu3ePnj17olAoOH36NJ06dcqzqIDQree9eICBAwdSqlQpmjRpwu+//86aNWu4fv164QUn3or0KIuxnJwcDA0NefLkCadPn+bMmTMMGjQIgP79+xMXF8fSpUulZ5MPnv/slUolt27dYuPGjXz11VcA7Ny5k59//pmKFSvSo0cPnJ2dCzdYoZEkymLur7/+YurUqTx8+JCyZcuyePFizMzMAOjXrx83b95kwYIFeVaLFv/c8158bm4uXbp04eHDh9y9e5eJEyfSuXNnAHbs2MGuXbsoVaoUAwYMoGLFioUctXgTGXoXQ8+HfKmpqYwYMYI6derQrl07oqOj2bNnj6rdDz/8gLW1NXPnzi2sUIud50kSICwsjBo1arBo0SLatm3LiRMniIqKAuDDDz+kRYsW+brYrNAd6VEWUzdu3GD+/PnY29szfPhwANatW8eECROYOnUqH330kaptbm6u6qms0I0VK1awf/9+xo0bR7Vq1bh48SIrV64kMzMTPz8/fHx8gGf/mVlZWRVytEITw7CwsLDCDkLoxou9mfT0dMaPH8/Vq1fx8/PD0tKS2rVrU6ZMGb799ltKly5N7dq1AVTDRIU+71BfyHJyclT/2WRlZXHw4EGOHz9OZmYmDRo0wN7enooVK3L+/HlOnDhBqVKlcHZ2xsTEpJAjF9qQRFlMPP+HmpqayuPHjylXrhyffPIJ69evJz4+nmbNmmFsbEzt2rUxMTFh9+7ddOzYUZUcJUn+c9nZ2RgZGaFUKrl06RLm5ua0aNEChULBuXPnePToEdWqVaN8+fI4ODgQHx9P27ZtpSepR2ToXQw8Hzo/X/E5KSmJDh06MHDgQO7du8dHH31EkyZNmDhxoup+2PPe54u9UPH2nj/dzsnJoVu3bqSkpFC2bFm6d+9Ou3bt+OGHHzhx4gRNmzalc+fOWFpakpmZKT1JPSM3pooBAwMDYmNj6dOnDz4+PnTr1o0lS5Ywf/58ypUrR2RkJNHR0QwdOpSnT5+q3idJ8t97PgXo888/p0qVKsyfP5933nmHNWvW8L///Y9+/frRqFEjdu3axdatW1EqlRgb63DfVlEgZF/vYiA7O5vVq1fTr18/AgICuH37NgcPHmTp0qVkZGQwYsQINm/ezPjx41X/SCVB/jspKSlYW1sDEB0djYmJCVOmTAHA0tKStLQ0Nm3ahLm5OX379sXMzEw1HBf6R3qUxYCRkRGXL1/GwsKCnJwcRowYwSeffMLkyZP54YcfCAkJwcDAgEWLFmFgYKC2QIN4ezNmzCA4OJj79+8DkJCQQGZmJgDBwcFkZGQwYcIEHjx4wNdff828efPo2bMnDg4OhRm2+BekR6mHnt8Xg/+7Pzls2DAqVarE2LFjcXBwoH379pw7d44aNWpgaWmJra2t6v0yFejf6dixI4GBgcyaNYuxY8fSsWNHatSowZEjR7h8+TIbN27ExMSEmjVrUrlyZfz8/Ao7ZPEvSaLUM7m5uRgaGnLlyhVWr15Namoq9evXp23btpQpUwaABg0aALBq1Sq8vLwYNWqUqicpSfLfyczMpHLlymzfvp3GjRtjYmLC8OHDqVatGjt27MDIyAgTExPWrFnDuXPnGD58uPQkiwF56q1Hnvck4+Pj6d69O61bt8ba2prNmzfz/vvvExAQwKZNmzh+/DgmJiakpaWxbds2jI2NJUnqwIs/wzVr1jB37lzMzMzw8vIiJCSE7Oxs2rdvj4ODA3fu3GHp0qXUrFmzkKMWuiCJUs88fvyYadOmYWNjw+jRowG4evUqw4YN48MPP6Rbt26cPHmSu3fv0r17d4yMjFTz/MQ/ExYWRqtWrXjvvfcAWLx4MREREfzyyy8YGhri5+dH3bp1GTduHNnZ2Vy9ehVnZ2fKly9fyJELXZFEWcRlZGTw22+/cf/+fapXr05MTAyrVq3C1dWVZcuWqdr9+uuvjBw5kh07dmBnZ6cqf/F+pvhn9u7dy9ixY9m4cSP79+9n5cqVrFmzRrWQRXx8PF988QWurq5MmTJF1vkshqSbUYSlpqbSo0cPnJycuHjxIj179qRBgwZkZWVx6NAhIiMjVZ/ZrlKlCtWrV88zkVmS5L/3wQcfYGJigq+vL6VKlWLbtm3Y29urVit3dHRk8eLFDB8+nIyMDEmUxZAkyiIqPT2dLl260LBhQ8aPH09SUhI2NjYoFArKlStHdnY2MTEx3L59m06dOjFv3jwyMjLkY3H5xNvbm5UrVzJgwACePHkCoLqdkZ2djbOzMxs3bpRbHMWUDL2LqMjISPbs2cOCBQtUZbt372bdunWYm5uTlJRE06ZN+emnn7CxscHd3Z3JkycDshpQfjp48CAjRoxg8eLF1K9fX61OPulUfMm/piLq0aNHqs2/jhw5wqRJk/jyyy9xcXHBw8ODUqVKkZycTNeuXbG3t6d69eqFHHHJ0Lx5c2bPns3gwYM5duyYWp0kyeJLepRF1J9//knXrl2pVq0aWVlZ2NjYMHjwYBo3bgxAaGgo1atXx9/fnx9//JEzZ87g5eXFF198UciRlwxRUVFMnjyZX375RbVivCi+5IZKEVW7dm02bNjAkSNH8PT0xMHBAUdHR1X933//TXZ2NlZWVnTq1ImMjAzOnj1LcnKyPEwoAK1ataJx48aSJEsI6VHqib///lv1yZtRo0Zx8eJFNm/erFrk4sGDBxgaGqraCCF0RxKlHoiJiaFv3764ublhbm5Oamoqa9euxdjYWCaTC1EAJFHqgSdPnhAdHc3169epUKECPj4+GBoaSpIUooBIotRT8okbIQqOJEohhNBA5lEKIYQGkiiFEEIDSZRCCKGBJEohhNBAEqUQQmggibKY8/Hxwc3NTfWqWbMmzZs3Z8aMGWRlZensPNHR0bi5uan2Dffx8WHdunUa35eZmcnatWv/8Xlv3bqFm5sbsbGxr6zXNo63bfu2cQj9JrOVS4CRI0eqFvjNycnh3LlzBAcHY2FhweDBg/PlnJs2bcLCwkJjux07djB//nx69OiRL3EIoQvSoywBrKysKFu2LGXLlqV8+fK0atWK9u3bs2fPnnw7Z5kyZbRaMEKm8Qp9IImyhDIyMlItqDFmzBhGjRrFJ598QsOGDTl16hRZWVlMnz6dJk2aUL9+fQYMGEB8fLzq/YmJiQwaNAhPT0/atm3LuXPn1I7/4jA2NzeX+fPn4+3tjaenJ3369OHGjRtER0czduxYHj58iJubG7du3UKpVLJkyRKaN2+Op6cnvXr14vz586rjPnnyhLFjx1KvXj2aN2/O4cOHtb7m1NRUQkNDadq0KTVr1sTHx4eIiAi1NnFxcXTu3JnatWvTrVs3Ll68qPb+kJAQGjRoQKNGjRgxYgSJiYna/9CF3pJEWcLk5ORw7Ngxtm3bRsuWLVXlP//8MwEBASxfvpzatWszZ84coqOjCQ8PZ8OGDZQtW5bevXuTkZEBwNChQ3n8+DHr1q1j7NixLF++/LXnnDdvHhEREYSGhrJ161asrKxUSTYkJIQyZcrw22+/YW9vz9q1a9mwYQOTJk1iy5YtNGjQgE8//ZQHDx4A8M033/Dnn3+yfPlyZs6cyY8//qj1tU+dOpXz58+zaNEifvnlFz766COmTJnCnTt3VG3WrVtH165diYyMpHz58gwaNEh1L3fcuHHcvXuXlStXsnLlStLS0ggMDJRecUmgFMVaixYtlLVq1VJ6eHgoPTw8lNWrV1fWqlVLOXbsWOXTp0+VSqVSOXr0aKWfn5/qPenp6cpatWopz5w5oyrLyclRNmvWTBkZGam8fPmysmrVqsq4uDhVfUREhLJq1arKjIwM1XnXrl2rzM3NVTZq1EgZERGhavvgwQPltGnTlCkpKcrNmzcrmzRpoqrz9vZW7ty5U+0aunbtqpw/f77y8ePHyurVqyt/++03Vd3BgweVVatWVXcuSuQAAAQ1SURBVF69evW117927VqlUqlUbt68WXnhwgVVXWZmprJq1aqq47Vo0UIZFhamqk9JSVF6eHgo9+7dq7xx44bSzc1N+fDhQ1V9amqqsmbNmsrff/9dGR8f/8Y4hH6ThzklQGBgIH5+fgCYmJhgZ2enGnY/V6FCBdXXN2/eJDMzk969e6ttb5CRkUFcXBwmJiZYWFjg7Oysqqtdu/Yrz/3o0SP+/vtvtXo7OzvVnuQvSktL486dO4wZM4aQkBBVeWZmJo6OjsTFxZGTk6O27cXrzvsqH3/8Mfv372fLli3ExcXx119/Ac962c/VqVNH9bWVlRXOzs7ExsZiaGiIUqmkVatWasfMzs4mLi5O9vAu5iRRlgBlypTBycnpjW1efPDyPHGsWrUqz2rp1tbWnDhxgtzcXLXylxOvpvJXeX7eGTNm4ObmplZnYWHB3bt3AfUHQG+zzNyYMWOIjo6mQ4cOdOzYEQ8PD1q0aKHW5uUVmXJzczExMSEnJwdTU1MiIyPzHLdMmTIkJydrHYfQP3KPUuRRsWJFjIyMSExMxMnJCScnJxwcHJg1axaXLl2iatWqZGRkcOnSJdV7Xnzg8iJra2tsbW1VvTeA5ORkGjVqRGxsrFqPtVSpUpQtW5b79++rzuvk5MSSJUs4ceIELi4uGBsbc+bMGY3nfdmjR4+IjIxk5syZDB8+nHbt2qm2nX0x8b748CY5OZkbN25QqVIlKlWqxNOnT3n69KkqLhsbG6ZOnUpCQoJWMQj9JT1KkYelpSXdu3dn4sSJGBkZ4ejoyMKFCzl+/Dhff/01//nPf2jWrBkhISFMmDCBx48fM2/evNce7/PPP2fevHnY29tToUIFvv/+e8qVK0elSpW4evUqqampxMbG4uTkxBdffEF4eDi2trbUrFmTn376iZ9//plPP/0UKysrPvnkE6ZOnUqpUqUwNjZm6tSpWl2TlZUVlpaW7Nmzh/Lly5OQkMCUKVMA1Cber1mzhmrVqlGtWjVmzZpFxYoVadasGQYGBvj4+DBq1ChCQ0MpVaoU06ZN4/r16zg7O6seNoniSRKleKVRo0ZhYGDw/9q7Q50FoTAO44/BQbW42QlsRJ3zMghGxhgUnBfgbDQhQFOL0Ni4IDKXQHJYDIZvs3075Qtf+P/u4JzwhPds5+V0OjFNE57nUdc1y+USgKqqyLKMIAhYLBZEUfRrtOI45vl8cj6fmaaJ7XbL7XZjNpux2+1wHAff92nbljAMeb1eFEXBOI44jsP9fsd1XeDn5TnPc9I0xbZtjscjWZYZzzOfzynLkjzP6bqO1WrFfr/Hsiz6vv/OHpMk4fF4MAwDm82G6/X63ZFeFAWXy4XD4cD7/Wa9XtM0DZZl/cGNy3+mj3tFRAw0oxQRMVAoRUQMFEoREQOFUkTEQKEUETFQKEVEDBRKEREDhVJExOADIpFi8oXz4moAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "spam_only_cnf_matrix = confusion_matrix(y_train, spam_only_y_pred, labels=[1, 0])\n", "\n", "plot_confusion_matrix(spam_only_cnf_matrix, classes=class_names,\n", " title='spam_only Confusion Matrix')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At the other extreme, `spam_only` predicts the training dataset has no ham emails, which the confusion matrix indicates is far from the truth with 758 false positives.\n", "\n", "Our main interest is the confusion matrix for `words_list_model`:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAEeCAYAAADsNlzkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xlczdn/wPFX+6LshsykULKViBBGdplkjH1fI8Iga9axG4yl7OuQbbJkxi7GzAiZxnz5WrIkhskaRelqu78//LpfTbjXuKp7vZ8e9/HonnPu5/O+1+3dOZ/lHAOlUqlECCHEGxnmdQBCCJHfSaIUQgg1JFEKIYQakiiFEEINSZRCCKGGJEohhFBDEuU7GDduHB07dtTa9oKCgqhXr57quZOTE1u3btXotXfv3qV37968ePFCa/Fo07t+Vrt27cLJyUnt+4mLi2P69Ok0adIEFxcXmjdvzty5c3ny5Mn7hpyDQqFgyJAhuLi4UL9+fdLT0997mz169GDEiBFaiO7NIiMjcXJywsnJiTt37ry2Tb9+/d7p+waaf+fedbu6wDivAxD/s337dmxtbTVqe+rUKU6dOvWBI8pfzp07x4ABAyhTpgxDhw6lVKlSxMTEsGLFCn777TdCQkIoXLiw1vZ37Ngxjhw5wjfffIOTkxPGxu//6zJlyhRMTEy0EJ16BgYGhIeH07t372zliYmJREZGvvP2NP3Ovcv3WFdIosxHXF1d8zqEfEuhUBAQEEDVqlVZuXKlKmnVqVMHDw8P2rRpw4oVKxg3bpzW9vn06VMAOnXqhIGBgVa26eDgoJXtaMLV1ZWjR4/mSJQ///wzZcuW5erVqx9sv/pG54bePj4+TJkyRfU8OjoaJycnVq5cqSo7evQozs7OPH/+HID9+/fTtm1bqlWrRpMmTVi9ejWv3pDk5OTEqlWraNGiBdWrVycyMhKlUsnKlStp0KAB1atXZ+bMmWRkZGSL5eeff6Zt27aqodmMGTNQKBT/+r29OmRJT09nzpw5fP755zg7O+Pj48P+/fuBl8PU8ePHA+Di4sKuXbs02n5QUBBdunThhx9+oHHjxri6uuLv78/Tp09ZsmQJderUoW7duixZsiTb6y5evEifPn2oWbMmderUYdKkSTx79kxVr8lnlZqayrfffkv9+vVxcXGhR48eREdHa/zZHD16lNu3bzNmzJgcPbuyZcsSEBCQLQk9evSIcePGUb9+fVxdXfH19SUmJibHZ7Fr1y6aNm2Ks7Mz3bp14/r168DLQwdZ37OKFSsSFBT02sMDMTExODk5qXpoycnJjB8/nnr16uHi4kLnzp2z9cL+OfR+3zjfplmzZvzxxx85DkscOnSIFi1a5Gh/9uxZ+vTpQ40aNVTfuaNHjwKv/85lDfG3bdtG3bp1qV+/PklJSarvcVpaGq1bt6Z169aqwxaJiYnUq1ePkSNHqo0/P9G5RFm/fn1+//131fMzZ84AL/+Ts5w8eRI3NzcsLS0JCQlh5MiRuLu7s3TpUtq2bcvixYuZN29etu0uX74cPz8/vvnmG1xcXFi9ejVBQUH06NGDRYsWERMTo0pUALdu3WLYsGG4u7uzevVqvv76a3bu3ElQUJBW3ufKlSvZvXs3AQEBrFmzBhcXF0aOHMm1a9fw9PRk0KBBAISEhODp6anxdqOjowkJCWHixIkEBgZy/Phx2rdvz6VLl5g/fz4tW7Zk6dKlREVFAXDhwgU6d+6MiYkJ8+fPJyAggKNHj+Lr66tKhuo+K4Dx48fzww8/MGjQIJYsWYKpqSk9e/bk3r17GsV96tQpPvnkE5ycnF5b36tXL9q3bw9AUlISnTt35s8//2T8+PHMnz+fJ0+e0LVrV+7evat6zdWrV1m7di0BAQEsXLiQu3fvqpLB4MGD6d+/P/ByKNmhQweN4pw5cya///47kydPZuXKlRQpUoRBgwbx+PHjHG21Eefb1KtXDzMzM37++WdVWXJyMhERETRv3jxb2zt37tC7d2+KFy/O0qVLWbRoEQUKFGDUqFE8ffr0rd+5DRs2MGfOHCZMmICVlZWq3MTEhOnTp3P9+nU2b94MwJw5c1AqlUycOFGjzzO/0Lmhd4MGDVi7di3x8fEUK1aMqKgoKlasyJ9//olSqcTAwICIiAg6dOhARkYGQUFBtG/fXvXFql+/PgYGBixfvpz+/ftTtGhRABo3bkzbtm0ByMzMZN26dfTo0YMBAwYAULt2bRo1aqSK48KFC6SmptKvXz8++eQTateujampqVYO+MPLxF+1alXatGkDQK1atbC2tiYjI4OiRYtSpkwZ4OVfdzMzM423+/z5c+bPn0+FChUA+Omnn7h8+TJhYWFYWlpSt25ddu7cyYULF6hZsybLly/n008/Zfny5RgZGQEve3DdunXj559/pnHjxmo/q2vXrrF3716+++47vvjiC+Dl/0OLFi1Yu3YtEyZMUBv3gwcPKF26tEbvcefOndy7d4+DBw/y2WefqWJq0qQJ69atU+0vKSmJJUuWUL58eQDi4+OZPHkyT548oUyZMqrXvstQ8uzZs3h4eKh6bJUrV2bFihWkpKR8kDiLFCnyxlhMTExo2LAh4eHhfPXVVwD88ssv2NjYqP7/s8TExFCrVi3mzp2LoeHL/pONjQ1t27bl0qVL1KlT543fub59+9KwYcPXxuDq6kqXLl0IDg6mZMmS7Nq1i8WLF6t+73SFzvUo3dzcsLCwUPUqo6Ki6NOnD4mJicTExHDv3j1iY2Np0KABN27cICEhgZYtW2bbRqtWrUhLS+PcuXOqsrJly6p+jo2N5cmTJ3z++eeqMnNzcxo0aKB67uLigqmpKR07dmT+/PmcPXuW1q1b065dO628z5o1a3LixAl69epFSEgI9+/fZ+zYsVSsWPG9tmthYZHtl6R48eLY29tjaWkJgJGREdbW1qqh9R9//EHz5s1VSTIrthIlSvDHH39o9Fll/V9lnTnO+mPi4eGh8UkFQ0PDHMP5N/njjz+oWrWqKvkAWFtb06BBA/744w9VmZWVlSr5AJQqVQrgtUlNUzVr1uSHH35g8ODB7Nixg4yMDMaOHcunn36aJ3E2a9aMiIgIVdsjR47k6E0CNGzYkLVr15KamsqlS5fYt28fW7ZsASAtLe2t+3j1d+d1Ro4ciYWFBSNGjKBFixY5fh91gc4lSlNTU2rVqsWZM2eIiYnh6dOntGjRgtKlS3P27FkiIiIoVaoUFSpUIDExEXiZDF5VrFgx4OVf6n+WAarX/fOv9attbG1tWb9+PQ4ODmzYsIEuXbrQtGlTfvnlF628zwEDBjB27FgePHjA9OnTadSoEb6+vsTHx7/XdgsUKJCjzMLC4o3tnz59mu19ZylWrBhJSUkafVYJCQkAuLu7U6VKFdXjhx9+4OHDhxrFXbp06bcO0588eUJqaupbYy5atCjJycmq5+bm5tnqs07YZGZmahTT60ycOBE/Pz8uX77MhAkTaNCgAaNGjXptUsuNOBs2bEhmZiYRERGkpqZy/Pjx1ybK9PR0pk2bRs2aNWnfvj3Lli1TxaxugjF1vUMrKys8PT3JzMzM9gdUl+jc0Bte9kxCQ0NxcnKicuXKWFhY4ObmxtmzZ0lLS6N+/foAFCpUCHh5wPxVWc+z6v8pq/yfx5WykkKWmjVrsmbNGpKTkzlx4gQrVqxgxIgRnD59GlNT0/d6j0ZGRvTt25e+ffty+/ZtDh48SHBwMIsXL2batGnvte13UbBgwdcm5/j4eAoVKqTRZ1WwYEFMTU1VPZRXaXrJjYeHB5s3b+bq1as5ho0A3377Lb/99hvHjx+nYMGCPHjw4I0x/1tZCerVnm3WCcMs5ubmDB8+nOHDh3P16lX27dvHqlWrcHR0ZODAgdnafqg4X2VlZUXdunU5evQoRkZGFClSBGdn5xztVqxYwZ49e1iyZAkeHh6Ym5sTExPD3r173zuG6Ohodu7cSYUKFfjuu+9o1qyZVi/jyg0616OEl4ny+vXrHD16FDc3N+DlMbzff/+dU6dOqf5qlStXjsKFC3Pw4MFsrz9w4ABGRka4uLi8dvvlypWjRIkSHD58WFWWnp7O6dOnVc/DwsJo0qQJaWlpFChQgBYtWuDr60tycnK2nuq/5efnx+zZs4GXvVdfX19q1aql6lVlHUf60GrUqMHhw4ezJYeoqCgePnyIq6urRp9V9erVSU1NJSMjA2dnZ9Vjz549Of5v3uTzzz/H1taWb7/9Nsdx4GvXrnHgwAGaNWuGsbExNWrU4MKFC9kutk5KSuLEiRPvdelK1uGJV3u2rw6RlUolX375JRs2bACgQoUKjBgxAnt7+9f2hj9UnP/UrFkzjh8/zuHDh2nWrNlr2/znP/+hRo0aNG7cWNWDjYiIUL0v+HffuczMTCZOnIizszMhISEAzJ0799+8jTylk4myfPnylCpVil9++YWaNWsCL3t3d+7cISEhAQ8PD+BlryzrWNHs2bOJiIhg6dKlLF++nB49erzxr5qBgQH+/v5s376dxYsX8+uvvzJs2LBsPVM3NzcePXrEyJEjiYiIIDw8nOXLl1OjRg2tHKiuUaMGmzdvZv369URGRrJhwwZOnz6t+qIXLFgQeJn0X9cr0RY/Pz/u3LnDoEGD+OWXX9ixYwfDhw/H2dkZT09PjT6rKlWq0KhRI4YNG0ZoaCinT59m2rRpbNq0SePrCk1NTZk5cyZRUVF0796dH3/8kVOnTrF27Vp69OjBZ599RkBAAADt2rXjk08+oV+/fuzfv5/w8HD69u1LZmYmvXr1+tefhbu7OyYmJsyYMYNTp06xefNmNm7cqKo3MDDAxcWFZcuW8cMPPxAZGcmiRYuIjY19bYL6UHH+U5MmTUhISOCnn3567bAboGrVqkRGRrJ9+3YiIyMJDg5m/vz5wP96zf/mO7dx40YuXbrElClTKFSoEAEBAezatSvbH1JdoJNDb3jZq9yxYwc1atQAXibPokWLYm9vr/oPhZeXjZiZmbF+/Xo2b96MjY0NI0aMoG/fvm/dfpcuXcjIyGDdunWsW7eOZs2a0bFjR86fPw+87OUtX76cRYsWMWTIEIyNjWnQoIFGl21oon///igUCjZt2sSDBw8oVaoUI0eOVF2mUrduXerUqcPEiRP5+uuv8fX11cp+/8nFxYX169ezYMEChgwZgrW1Nc2aNWPUqFGqYbO6zwpg4cKFLFy4kMWLF/P06VPs7e2ZN28erVu31jiW2rVrs23bNlatWsWCBQt48uQJNjY2fPXVV/j5+akuTbG2tmbTpk3MnTuXiRMnYmBgQM2aNdm+fftrT6poqkiRIixatIgFCxbg6+tLlSpVWLx4seqyJHh5GZSJiQnBwcE8fvwYOzs75syZo/rj/aoPFec/FS1aFDc3N27evEn16tVf22bAgAHcv3+f7777joyMDMqVK8fixYuZNWsW586dw8vLK8d37k0jsixxcXEsXryYzp07U6lSJeDlH4fQ0FCmTJnCjz/++E5XbOQlA1kKQggh3k5ne5T5WWZmptozkoaGhlo7zpjb+xPiYyOJ8gNYunQpwcHBb23Ttm1b5syZo5P7E+JjI0PvD+D+/ftqD3YXKVIk24XGurQ/IT42kiiFEEINOWglhBBqyDHKf0h68e9vXxPaIWOc/MHaXLv9KIvqQzRql/Ln24+35wVJlEKI3GGguwNYSZRCiNxhaKS+TT4liVIIkTu0tJxGXpBEKYTIHTL0FkIINWToLYQQasjQWwgh1JChtxBCqKHDQ2/dTfFCCN1iYKDZQwM//vgj1atXz/aoWLEikyZNIjExEX9/f9zc3PD09CQ0NFT1utTUVAIDA3F3d8fDw4Ply5drtD/pUQohcocWh94+Pj74+Pionp88eZIxY8bg7+/PpEmTsLS05OTJk1y5cgVfX18cHR1xdXVl4cKFxMXFcfToUeLj4+nbty92dna0atXqrfuTHqUQIncYGWn2eEfJycmMGzeOqVOnYm1tTXh4OMOGDcPMzAwXFxe8vb0JCwsDYM+ePQwcOBBra2vs7e3p3r07u3fvVrsPSZRCiNxhYKjZ4x2tWbOGChUq0LRpU27duoWxsTG2traq+rJly3Ljxg0SExOJj4/Ptk5TVp06MvQWQuSOD3B5UHJyMiEhIaxevRp4uRDaP9dBNzc3R6FQqNYpf3Ud+6w6daRHKYTIHR+gRxkeHk7p0qVVy/taWFjw4sWLbG0UCgWWlpaqBPpqYsyqU0cSpRAidxgaafZ4Bz///DNeXl6q53Z2dqSlpREXF6cqi42NxcHBgcKFC1OsWDFiY2Oz1ZUvX1596O8UlRBC/FtavDwoy7lz51S9SQArKyuaNGnCggULSElJ4fz58+zdu1e1LLKPjw9BQUEkJCRw8+ZNQkJCaNOmjdr9SKIUQuQOLQ+9MzIyuHv3LiVKlMhWPn36dNLT02nYsCHDhg1j9OjRVKtWDYDhw4djb2+Pl5cXXbt2pWPHjtl6pG8MXdbMyU5mOM978o3MH7Q+w3mrxRq1S9n/tVb3qw1y1lsIkTtkUgwhhFBDJsUQQgg1dHhSDEmUQojcIT1KIYRQQ45RCiGEGjL0FkKItzOQHqUQQrydJEohhFDDwFASpRBCvJX0KIUQQg1JlEIIoYYMvYUQQg3pUQohhBqSKIUQQg1JlEIIoYYuH6PU3bvUhRA6xcDAQKOHpu7du8fAgQOpUaMGn3/+ORs3bgQgMTERf39/3Nzc8PT0JDQ0VPWa1NRUAgMDcXd3x8PDg+XLl2u0L+lRCiFyhTaH3kqlksGDB1O7dm2Cg4O5efMm3bp1o2rVqmzYsAFLS0tOnjzJlStX8PX1xdHREVdXVxYuXEhcXBxHjx4lPj6evn37YmdnR6tWrd66P+lR6oGffgyjvF3pbGUJCQn4+/lS3q409p+VpH/vHjx8+DCPItRPqampzJ09A9eqTpQqZk292jXYteOH17Y9fTKCQpbG/Pbr8dwNMh8xMDTQ6KGJc+fO8eDBA0aNGoWJiQmOjo5s27aNkiVLEh4ezrBhwzAzM8PFxQVvb2/CwsIA2LNnDwMHDsTa2hp7e3u6d+/O7t271e5PEqWO+/1MJH79++QoH9CvFwcP7GPWnHksWbqC389E0q1Te2SJJO2ZOimQxd/No/+AQWwN3U29+p/Tu0cX9v20J1u71NRUhvoP/Og/e20OvS9evIijoyPz5s2jXr16tGjRgnPnzpGYmIixsTG2traqtmXLluXGjRskJiYSHx+Pg4NDjjp1ZOito9LT01m5fClTJwViYWGRrS4hIYED+/aybOUaOnXpBoBVASvaeLfkypVoKlaslBch65XU1FRWr1zGlGmzGDJsOACNGjflxo3rLAtewhet/7cE6vxvZ/Ps6dO8CjXf0ObQOzExkcjISOrUqcPPP//MhQsX6N+/P6tWrcLc3DxbW3NzcxQKBSkpKQDZfl+y6tSRHqWOOhVxglnTpzJ1+iwGDhqSrS71xQsAChYsqCorUrQoAE8eP869IPXY08REevTuSwuv7Me2HB2d+OvWTdXz6MuXCFq0gFlz5+dyhPmPoaGhRg9NmJqaUqhQIQYOHIipqSk1atSgRYsWLFmyhBf///3PolAosLS0VCXQVxNjVp3a2N/hfYp8pGKlypy/fB3/oV/n+Ev9ScmSePu0Yf63c7h+/Rpxf//NN5MnYG9fFreatfIoYv1SvEQJvlsUjKNjBVVZZmYm4UcO4ejkBLw84TB08EAG+PlT1dklr0LNPww0fGigbNmyZGRkkJGRoSrLyMigcuXKpKWlERcXpyqPjY3FwcGBwoULU6xYMWJjY7PVlS9fXu3+JFHqqBKffEKxYsXeWD9rznwSnjyhetWKOJUvw9k/oti+MwxTU9NcjPLj8u2cmVyJvsyQYSMAWLNqBffv32PchMl5HFn+oM1jlPXq1cPc3Jzg4GDS09M5e/YsR44coWXLljRp0oQFCxaQkpLC+fPn2bt3L61btwbAx8eHoKAgEhISuHnzJiEhIbRp00bN3iRR6qV7d+/SrHEDihQtyrYdu9kR9hNVnavRro03cX//ndfh6aUVy4KZNX0qw0eOpnGTZv/fiw/ku8XBOY4hf6y0OfQ2Nzdn06ZNnD9/Hg8PD0aNGsXEiRNxdXVl+vTppKen07BhQ4YNG8bo0aOpVq0aAMOHD8fe3h4vLy+6du1Kx44d8fLyUru/XD2ZExsby7fffktUVBTp6enY2trSo0cPOnTokJth6L2NG9aR9OwZu37cT/HixQFo8Lkn1Z0rsmTRAubM+y6PI9Qvs2Z8w5yZ0+g/cBDfzJgNwMjh/jRt3hLPRk1IT09XDRGzhotGRrq7fsy/puUbc+zs7Fi7dm2O8sKFC7N48eLXvsbc3Jxp06Yxbdq0d9pXriXKzMxM+vfvz1dffcXChQsxNTUlKiqKIUOGULBgQVq0aJFboei9O3duU7ZceVWSBLC0tMTZpRpXoi/nYWT6J2D4UFavXMbwkaOZNnOOqnz/3p8AclxX6dOqOfUbNGT/4WO5Gmd+IPd6a+DJkyfcuXMHHx8f1dknd3d3Ro8eTVpaGkFBQdy4cYP4+HjOnz+Po6Mj06ZNo1Kll5eybNy4kdDQUOLi4jAzM6NLly4MHToUACcnJ6ZNm8ayZct49uwZ/fr1w8bGhkWLFqFQKBg4cCD9+vXLrbea58o7OLJtSwgPHz6kRIkSwMuzexf/+1+at1Q/zBCamf/tbFavXMbEKdMYM25CtrrjJyKzPb/91y16dO3IoqBlNPjcMxejzD80HVbnR7mWKIsVK4a7uzt9+vTBx8cHd3d3XFxcVMPuoKAgDhw4wHfffcfatWtZuXIlgwcP5tChQ5w/f54VK1awZcsW7O3tiYqKonv37vj4+GBnZwdAREQEBw8e5OzZs/Tr1w8vLy8OHz5MZGQkgwYNomPHjlhbW+fW281TPXr1YVnwYtq1+YJRY8djamrK8uAlPH4cz+ChX+d1eHrh9l9/MXvGN9Su44FnoyaciTytqjM1NaWGW81s7a2srABwrOCEYwWnXI01v5AepYbWrFnD1q1bOXLkCKtWrQKgefPmTJo0CYC6deuq7rkcNGgQmzZt4uzZs1SrVo1du3ZRqlQpHj16RFpaGubm5jx48ECVKLt3746FhQV16tRBqVTSrVs3zM3NadCgARkZGdy/f/+jSZRFixbl4JHjjB8bgF//PhibmFCrljs//3YKJ6eKeR2eXjh8aD9paWlEnj5JU8962eo+KVmS6zfj3vDKj5ju5sncTZRmZmb07t2b3r178+LFC/744w/mz59PYGAglStXpkyZMqq2RkZGlChRgkePHmFoaMiyZcs4dOgQxYoVo2rVqsDL455ZChUqpHod/O9i66zu/qtt9U3gpCkETpqSraxsuXJsC1V/D6v4d/r5+tHP10/j9hWcKvI0JUN9Qz2my0PvXIt8//79NGnSRHW/q5mZGR4eHgwdOpTo6GgAHjx4oGqfnp7OgwcPKFWqFOvXr+fq1auEh4ezf/9+Zs6cSXp6erbt63K3XoiPgYGBZo/8KNcSZd26dUlOTmbmzJnEx8ejVCq5desWmzZtolGjRgD89ttvnDx5krS0NJYuXUqRIkWoXr06SUlJmJiYYGJiQnJyMnPnziUtLS1HshRC5F/ano8yN+VaoixSpAhbtmzhwYMHeHt74+rqSp8+fXB2dmbcuHEAuLi4sHr1amrXrk1UVBQrV67EyMiIPn36YGxsTN26dWnRogWpqanUqFGDmJiY3ApfCPGedLlHaaDMJ3M/BQUFce3aNZYsWZKncSS90N9jmboif3wjhbW5dvtRFccd0qhd9Jz8d021TLMmhMgVhjq8Zo4kSiFErsivw2pN5JtEmXWXjRBCP0mPUggh1MivZ7Q1IYlSCJErJFEKIYQaMvQWQgg1dLhDKYlSCJE7dHnorbt3qQshdIqhoYFGD02tXbuWqlWrUr16ddUjKiqKxMRE/P39cXNzw9PTk9DQUNVrUlNTCQwMxN3dHQ8PD5YvX67RvqRHKYTIFdruUF66dIkRI0bkmJR72LBhWFpacvLkSa5cuYKvry+Ojo64urqycOFC4uLiOHr0KPHx8fTt2xc7OzvV9I5vIj1KIUSu0PakGJcvX1atgJAlOTmZ8PBwhg0bhpmZGS4uLnh7exMWFgbAnj17GDhwINbW1tjb29O9e3d271Y/HaEkSiFErtDm0DslJYXY2Fg2btxIvXr18PLyYseOHdy6dQtjY2NsbW1VbcuWLcuNGzdITEwkPj4eBweHHHXqyNBbCJErtDn0fvToEW5ubnTp0oUlS5Zw/vx5/Pz86NOnj2pNrizm5uYoFApSUlIAsi0fnFWnzhsTZWxsrMZBly1bVuO2QoiPkzbPetva2hISEqJ6XrNmTdq0aUNUVBQvXrzI1lahUGBpaalKoAqFQrWGUVadOm9MlF5eXhgYGPCmWdiy6gwMDLh8WZZAFUK8nTYvOL948SIREREMGDBAVfbixQtsbGxIS0sjLi6O0qVLAy87fQ4ODhQuXJhixYoRGxurWso5NjaW8uXLq93fGxPl0aNH3/e9CCGEijZ7lJaWlgQHB1OmTBmaN29OZGQk+/btIyQkhGfPnrFgwQJmzJjBtWvX2Lt3r2oxQx8fH4KCgliyZAkJCQmEhIQwevRo9bG/y8S9Z8+eJTY2lhYtWnD37l3s7OwwNTX99+82H5KJe/OeTNybP2h74t6GCyM0avfLiHrqGwHHjh1j4cKF3L59m5IlSzJixAhatmxJQkICU6ZM4dSpU1haWjJkyBDat28PvBxqz5o1iyNHjmBgYEDPnj3x81O/SJxGifLx48f4+flx6dIlMjMzOXToEDNnziQmJoZ169ZlO8Ok6yRR5j1JlPmDthOl56KTGrU7PtxDq/vVBo0+iZkzZ1K8eHEiIyMxMzMDYO7cuZQpU4aZM2d+0ACFEPpB23fm5CaNEuXJkycZPnw4BQoUUJUVKlSIcePGERUV9cGCE0LoD11eXEyj6ygzMjLIzMw5JH327BlGRkZaD0oIoX8M82sW1IBGPcqmTZsyb948Hj9+rDpzdf36daZPn06TJk0+aIBCCP2g90PvwMBArKzd8IBkAAAgAElEQVSsqFevHs+fP6d169a0bt0aGxsbAgMDP3SMQgg9YGig2SM/0mjobWVlxeLFi7l9+zYxMTGkp6dTvnx5uSNHCKExXZ6PUuN7vTMyMrh69SoxMTGYmppibm4uiVIIoTFdPkapUaK8cuUKgwYNIiEhAXt7ezIzM7l16xb29vYEBwfz6aeffug4hRA6Lr8OqzWh0THKKVOmUK1aNX799Vd27dpFWFgYx48fp2TJkkyePPlDxyiE0APano8yN2mUKC9dusTQoUNVM27Ay+soAwIC5DpKIYRGjAwNNHrkRxolyooVK3Lu3Lkc5VeuXJHjlEIIjejlBefbt29X/Vy9enWmTp3KxYsXcXFxwcjIiOjoaEJCQnKsVyGEEK+TX4fVmnjjpBiNGzfWbAMGBno1JZtMipH3ZFKM/EHbk2J0+v5Pjdpt71Vdq/vVhjf2KI8dO5abcQgh9Jzu9iff4TrK+/fvc+PGDTIyMgBQKpWkpqZy8eJFhg0b9sECFELoB10eemuUKDdv3sysWbPIyMjItjyEgYEB1apVk0QphFArv57R1oRGByHWrl3LoEGD+O9//0uxYsU4fvw4e/fupWLFijRr1uxDxyiE0AMf6qz3o0ePqFu3Lj///DMAd+7coVevXlSvXp0WLVqoygESExPx9/fHzc0NT09PQkNDNdqHRonywYMHtGnTBhMTEypVqsR//vMfHBwcGD9+vMY7EkJ83D7UBecTJkwgISFB9fzrr7/GxcWFM2fOEBgYSEBAAHFxcQBMmjQJS0tLTp48yZIlS5g/fz7/+c9/1O5Do0RZuHBhnj17BrxcmvbKlSsAfPrpp9y7d++d35gQ4uPzIWYP2rp1KxYWFtjY2AAQExPD1atX8ff3x8TEhIYNG+Lu7s6+fftITk4mPDycYcOGYWZmhouLC97e3oSFhamPXZNgGjVqxOTJk4mOjqZOnTrs2bOHs2fPsmnTJlWAQgjxNoYGBho9NBUbG8v69euZOnWqquzGjRt8+umnqjW84WXn7saNG9y6dQtjY+Nsa3xl1amNXZOAxo0bR8WKFYmOjqZx48bUqlWLrl27Ehoayrhx4zR+Y0KIj5c2E2V6ejpjxoxhwoQJFC5cWFX+/PlzLCwssrU1NzdHoVDw/PnzbAn01Tp1NDrrXaBAAWbMmKF6PnfuXMaPH4+VlRXGxhpfYSSE+Ihp8+qgZcuWUalSJRo2bJit3MLCIkfiUygUWFpaYmFhwYsXL15bp45GtzCq06lTJ43bCiE+Ttpc5mH//v08fPiQ/fv3A5CUlMTIkSPx8/Pj77//JjU1FVNTU+DlEL127drY2dmRlpZGXFwcpUuXVtU5ODio3d8bE+XKlSs1CtjAwEASpRBCLW1O3Hvw4MFszxs3bsykSZNo1KgRhw4dYtGiRQwfPpxTp04RGRnJlClTsLKyokmTJixYsIAZM2Zw7do19u7dy6pVq9TuT25h/AdjI+3e3yreXZFaQ/I6BAGk/Bms1e3l1o05QUFBTJ48mbp161K8eHG+++471Unn6dOnM2XKFBo2bIilpSWjR4+mWrVqarf5xkkxPlaK9LyOQEiizB+0nSiH7r6sUbugtpW0ul9tkDMxQohcoff3egshxPvS4Vu9JVEKIXKH3k+KARAfH8+KFSsYO3Ys8fHx7N+/n6tXr37I2IQQeuRD3MKYWzReXKxFixYcP36cffv28fz5c06ePEmHDh04derUh45RCKEHdHnNHI0S5ezZs+nVqxfbtm3DxMQEgBkzZtCjRw/mz5//QQMUQugHYwMDjR75kUaJ8uLFi/j4+OQo79SpEzExMVoPSgihf/S+R1moUCHVfG6vunjxIkWLFtV6UEII/aPt2YNyk0Znvbt06cLkyZMZNWoU8HI9719//ZWgoCD69OnzQQMUQugHXb7pTaNEOWDAAAoUKMCcOXNISUlhyJAhFC9eHD8/P3r16vWhYxRC6IH82lvUhMbXUXbr1o1u3brx/PlzMjIysLa2/pBxCSH0jA7nSc0Spbqp0r/88kutBCOE0F/59RpJTWiUKP95CVB6ejpPnz7F1NSUihUrSqIUQqhlpMNdSo0S5YkTJ3KUJSYmMmnSJGrUqKH1oIQQ+keXe5T/+jxUoUKFGD58OGvWrNFmPEIIPfWhlqvNDe81KcadO3dISUnRVixCCD2m95cHBQQE5ChLSkrizJkzeHt7az0oIYT+0fblQfv37ycoKIh79+5RunRpRowYQdOmTUlMTCQwMJDTp09jbW2Nv78/HTp0ACA1NZWpU6cSHh6OsbExPXr0YNCgQWr3pVGizFqk51UlS5YkMDCQNm3avOPbE0J8jLR5jDI2NpbAwEDWrVtHjRo1OHnyJAMGDODXX39l6tSpWFpacvLkSa5cuYKvry+Ojo64urqycOFC4uLiOHr0KPHx8fTt2xc7OztatWr11v1plCidnJzw8vKiZMmSWnmTQoiPjzbPepctW5aIiAgKFChAeno6jx49okCBApiamhIeHs6hQ4cwMzPDxcUFb29vwsLCcHV1Zc+ePSxYsABra2usra3p3r07u3fvVpsoNTpqsHTpUlJTU7XyBoUQHydtT4pRoEABbt++jYuLC2PGjGHEiBH89ddfGBsbY2trq2pXtmxZbty4QWJiIvHx8dmWp82qU0ejRPn5558TEhLC06dPNX8XQgjxig8xca+NjQ3nzp1j/fr1zJ07l2PHjmFubp6tjbm5OQqFQnXi2cLCIkedOhoNvW/dusW+ffvYuHEjVlZWmJmZZat/3XWWQgjxqg+xFISx8csUVrduXZo3b86FCxd48eJFtjYKhQJLS0tVAlUoFFhZWWWrU7sfTYLp1q3bOwUvhBD/pM2z3r/88gvr169nw4YNqrK0tDTKlCnDr7/+SlxcHKVLlwZenvhxcHCgcOHCFCtWjNjYWIoXL66qK1++vNr9vTFRBgcH069fPywsLGjbtu17vi0hxMdOm1cHVa5cmQsXLhAWFoaPjw+//fYbv/zyCz/88AN3795lwYIFzJgxg2vXrrF3715WrVoFgI+PD0FBQSxZsoSEhARCQkIYPXq0+tiVSqXydRWVKlXixIkTFCtWTHvvTgco0vM6AlGk1pC8DkEAKX8Ga3V730fd1qhdr5q26hsBUVFRzJo1i5s3b2Jvb8+YMWOoU6cOCQkJTJkyhVOnTmFpacmQIUNo37498HKoPWvWLI4cOYKBgQE9e/bEz89P7b7emCgrVqxIRESEJEqR6yRR5g/aTpQbNUyUPTVMlLnprcco09LSNLos6HUXpAshxKv0duLeRo0aabSRy5cvayUYIYT+0uXZg96aKJcsWUKhQoVyKxYhhB7LrzMDaeKNidLAwIAaNWp8dMcohRAfhg5PHvTmRPmGczxCCPGv6GWPsm3btjnuwBFCiH9LL0/mzJ49OzfjEELoOb0cegshhDbp5dBbCCG0SW8vDxJCCG0xRHczpSRKPbJv709MmzKJq1ev8JmtLSNGjqZvf9+8DksvNHBz5PCar99Yb1F9CO2b12DT3L456nz8l3Lk5MubMjzdKzB50BdUcShNwrMU9h4/z+SgH0lO0f+JsXV45C2JUl8cOxpOx3Zf4jvAjznzFnDsaDj+gwZQrHhx2nwpsz+9r/9E36Zhz/nZyooUsmTrvP5sPxgFQBWH0vz36t8MmbE1W7vo2HsAVHP6jB+D/dl55CyzVh3k05KFmTbUh89KFqFTwOrceSN5SC/PegvdMmXSBL5q34FFQUsBaNS4CTEx1zkWfkQSpRY8S1Zw5r83s5V9P7s3fz9IYOTcUACqVviUPy7dytEuy+Aunly6cZc+E77/33aTFGye148yNkX46+6TDxV+viBDb5Gn7t+/T9TvZ5g+M/slXVu2heZRRPqvbrVydGxZk3ZfryBFkQZAVQcblm658sbXnL9yhxNnr2cru3rrPgBlShfT+0Spwx1Knb60Sfy/y5cuAmBsYsIXLZtRqIAZDmVtWb92TR5Hpr+mDmnNb39cY/+vFwCwsjSjjE1RaruU5eKPU3h6ZjG/bhxFrap2qtcs3XqcTT+ezradlg2qkpmZyfVbD3I1/rxgaGCg0SM/kkSpBx4+fAhA7x5dqetRjz17D9Da50sG+/ly6OCBPI5O/1R1LM3nNR1Z+P3RbGWGhobY2hRlxNxQOgWs5kVaOj8tG4JNiddPLFOpXClG92nG9gNR3Huk/wv3fYjFxXKLJEo9kJ72cuj3Zdt2TJw8Fc9GjVm4OIgmTZsxd/bMPI5O//T9qh43bj/kwG8XVGWXY+7RduhyvvAL4nDEJQ78doEvhyxDkZrGsO6Nc2zD0e4T9i4fQtyDRNUxTn1noOE/TUVFRdGhQwfc3Nxo2rQp27ZtAyAxMRF/f3/c3Nzw9PQkNPR/n29qaiqBgYG4u7vj4eHB8uXLNdqXHKPUAwX+f0W5Zs1bZCv3bNyE+XPlVlRt827orDrTnSUxKYWDJy5mK0tOSSXyXCxVHEpnK3erXIZdQYNIfJaC96BgEp6lfPCY8wNtDqsTExMZPHgwkyZN4osvvuDy5cv06dOHMmXKsG3bNiwtLTl58iRXrlzB19cXR0dHXF1dWbhwIXFxcRw9epT4+Hj69u2LnZ0drVq1envsWov8PTg5OXH16tUc5bVr1yYyMjIPItIt5cq9XEXun8t0pqel6fRtY/lRxXKlsLUpyp6j57KVu1T4lJ5t6uRob2ZmTHLK//5fPq/pyIFVw7j/6CnN+i3k7wcJHzzm/EKbQ++4uDgaNmxI69atMTQ0pEqVKtSuXZuzZ88SHh7OsGHDMDMzw8XFBW9vb8LCwgDYs2cPAwcOxNraGnt7e7p3787u3bvVx/4+b1zkD5WrVKGUjQ27dmQfwh0+dJDadermUVT6qUblMrxITeM/V7Kv/+Li9Bkrp3bHqWxJVVmJIlbUrVaOiP8/0+1o9wmhCwdwJfYezfsv4n78s1yNPa9pc+hdqVIl5s2bp3qemJhIVNTLXr6xsTG2tv9bd6ds2bLcuHGDxMRE4uPjcXBwyFGnjs4MvS9dusS3337L9evXefbsGW5ubnz77bcUL16ccePGUbRoUc6ePUt0dDRVq1ZlzJgxzJo1iytXruDq6kpQUJBq0XN9Y2hoyKTJ3+A/aAClbGxo3qIlodu3EXn6FIeP/pLX4emVyuVsiL0TT3p6Zrby3eF/Mq5/S7bN9+WbZXtRKpUEDvQiPiGZtbsiAFgwpgOmJsbMXXuICvYls70+OvYeT5MUufY+8sKHGtw8e/YMPz8/Va9y48aN2erNzc1RKBSkpLw8xGFhYZGjTp18kyg7d+6MoWH2Dm5SUpLq56+//pqePXuyfv16EhISGDBgACEhIQwfPhyAXbt2ERISgo2NDe3atWPw4MF8//33FC9enI4dO7Jnzx66deuWq+8pN/Xt74uxsTEL5s9lxbJgKlRwYsfuH6lXv35eh6ZXihexIjEp5zHF5JRUWvktYdbwtiyZ0AkzE2OORV5hzPydpCjSsDA3oUkdJwwNDQldODDH61+9zVFfGX2ATHn79m38/PywtbVl0aJFxMTE5DgEpVAosLS0xNzcXPU8q9OUVadOvkmU27Zto0KFCtnKateurfp57dq1fPbZZ6SkpHD//n2KFCnC/fv3VfWNGjVSdamdnZ0xNTWlfPmXx+6qVavG33//nQvvIm/17N2Hnr375HUYes3vm81vrPvr7hO6j1332roURRoF3IZ9qLB0grbT5MWLF+nfvz8+Pj6MHTsWQ0ND7OzsSEtLIy4ujtKlX55Ei42NxcHBgcKFC1OsWDFiY2MpXry4qi4rT7yNzhyjPH/+PF5eXjRv3px58+bx+PHjbMtVvLoImpGREQULFlQ9NzQ0lKUthMhrBho+NPDo0SP69+9Pnz59GD9+vGo0amVlRZMmTViwYAEpKSmcP3+evXv30rp1awB8fHwICgoiISGBmzdvEhISQps2bdTuL9/0KN/m3r17jB07li1btlCtWjUAxo8fny35ydldIfI3bV4etGPHDh4/fszy5cuzXQvZs2dPpk+fzpQpU2jYsCGWlpaMHj1alTeGDx/OrFmz8PLywsDAgJ49e+Ll5aV2fzqRKJOTk1EqlZibm6NUKvn11185ePAgjRvnvJBXCJE/abMr4+fnh5+f3xvrFy9e/Npyc3Nzpk2bxrRp095pfzqRKMuXL8/gwYPp1asXmZmZlCtXjs6dO3P69Gn1LxZC5A86POgzUMrBu2wU6XkdgShSa0hehyCAlD+Dtbq9P25qdj+7m31B9Y1ymU70KIUQuk+HO5SSKIUQuUSHM6UkSiFErsivc01qQhKlECJX6G6alEQphMgtOpwpJVEKIXKFDL2FEEIN3U2TkiiFELlFhzOlJEohRK6QobcQQqihu2lSEqUQIrfocKaURCmEyBXvshRtfiOJUgiRKzRdYTE/kkQphMgdOpwodWYpCCGEbtPmcrWvOn/+PPVfWUQvMTERf39/3Nzc8PT0JDT0f8s4p6amEhgYiLu7Ox4eHtlmR38b6VEKIXKFtofeSqWSnTt3MmfOHIyMjFTlkyZNwtLSkpMnT3LlyhV8fX1xdHTE1dWVhQsXEhcXx9GjR4mPj6dv377Y2dnRqlWrt8eu3dCFEOINtLi4GMCKFSvYuHFjtiUhkpOTCQ8PZ9iwYZiZmeHi4oK3tzdhYWEA7Nmzh4EDB2JtbY29vT3du3dn9+7davcliVIIkSu0PfRu164de/bswdnZWVV269YtjI2NsbW1VZWVLVuWGzdukJiYSHx8vGpZ61fr1JGhtxAiV2h76P3JJ5/kKHv+/Dnm5ubZyszNzVEoFKSkpABgYWGRo04d6VEKIXKHlofer2NhYcGLFy+ylSkUCiwtLVUJ9NXEmFWnjiRKIUSu+FBnvV9lZ2dHWloacXFxqrLY2FgcHBwoXLgwxYoVIzY2Nltd+fLl1W5XEqUQIlcYGmj2eB9WVlY0adKEBQsWkJKSwvnz59m7dy+tW7cGwMfHh6CgIBISErh58yYhISG0adNGfezvF5YQQmjGwECzx/uaPn066enpNGzYkGHDhjF69GiqVasGwPDhw7G3t8fLy4uuXbvSsWNHvLy81Mcu63pnJ+t65z1Z1zt/0Pa63neepGrU7rMiplrdrzbIWW8hRK6Qe72FEEINHZ63VxKlECJ3yDRrQgihhvQohRBCDUmUQgihhgy9hRBCHd3Nk5IohRC5Qy4PEkIINWToLYQQasjJHCGEUEMSpRBCqCFDbyGEUEN6lEIIoYYkSiGEUEOG3kIIoYb0KIUQQg1JlEIIoYYuD71lKQghhFBDFhcTQgg1JFEKIYQakiiFEEINSZRCCKGGJEohhFBDEqUQQqghiVIIIdSQRCmEEGpIohRCCDUkUQohhBqSKIUQQg1JlB8pucU/d2VmZqp+ls9e90ii/EhkZGQAkJ6eDoCBgYH8wuaS9PR0DA0NVT8L3SOzB30ElEolBgYGXL16lU2bNmFgYEBAQACFChXK69A+GpmZmQwcOBCFQkG9evXw8vLCzs4ur8MSGpIepZ7LSpL379+nW7duGBgY8Oeff9K+fXtiY2PzOjy9ltWLBxg0aBAFCxbEw8OD33//nc2bN3Pz5s28C068E+lR6rGMjAyMjIx4/vw5f/75J+fOnWPw4MEADBgwgNjYWNasWSM9mw8g67NXKpXcuXOH0NBQRo4cCcD+/fv58ccfKVOmDF27dsXe3j5vgxVqSaLUc5cvX2b27Nk8evSIEiVKsHLlSszNzQHw9fXlr7/+YtmyZZQvXz6PI9UfWb34zMxMOnbsyKNHj7h37x7Tp0+nQ4cOAOzbt4+DBw9SsGBBBg4cSJkyZfI4avE2MvTWQ1lDvqSkJAICAqhWrRqtWrUiMjKSw4cPq9qtXr0aa2trlixZkleh6p2sJAkwdepUKleuzIoVK2jZsiVnzpwhPDwcgC+++IJGjRqhUCiwtLTMy5CFBqRHqadu3brF0qVLsbGxYcSIEQBs3bqVadOmMXv2bL788ktV28zMTNVZWaEd69ev59ixY0yYMIGKFSsSHR3Nhg0bSE1Nxdvbm8aNGwMv/5hZWVnlcbRCHaOpU6dOzesghHa82ptJSUlhypQpXL9+HW9vbwoUKICzszNFixblm2++oXDhwjg7OwOohokGurxMXh7LyMhQ/bFJS0vj+PHjnD59mtTUVGrVqoWNjQ1lypTh4sWLnDlzhoIFC2Jvb4+pqWkeRy40IYlST2T9oiYlJfH06VNKlixJu3bt2LZtG7dv36ZBgwaYmJjg7OyMqakphw4d4quvvlIlR0mS/156ejrGxsYolUquXLmChYUFjRo1wsDAgAsXLvDkyRMqVqxIqVKlKF26NLdv36Zly5bSk9QhMvTWA1lD5+joaKZOnUpCQgJt2rRh0KBB3L9/ny+//BIPDw+mT5+uOh6W1ft8tRcq3l3W2e2MjAw6d+7Ms2fPKFGiBF26dKFVq1asXr2aM2fOUK9ePTp06ECBAgVITU2VnqSOkQNTesDQ0JCYmBj69u1L48aN6dy5M6tWrWLp0qWULFmSsLAwIiMjGTp0KC9evFC9TpLk+8u6BKh37944OjqydOlSihQpwubNm/npp5/w9fWlTp06HDx4kN27d6NUKjExMcnrsMU7Ms7rAMT7S09PZ9OmTfj6+tKnTx/+/vtvjh8/zpo1a1AoFAQEBLBz506mTJmi+iWVBPl+nj17hrW1NQCRkZGYmpoya9YsAAoUKEBycjI7duzAwsKCfv36YW5urhqOC90jPUo9YGxszNWrV7G0tCQjI4OAgADatWvHzJkzWb16NYGBgRgaGrJixQoMDQ2zTdAg3t28efMYPXo0Dx48ACAuLo7U1FQARo8ejUKhYNq0aTx8+JCJEycSHBxMt27dKF26dF6GLd6D9Ch1UNZxMfjf8cnhw4dTrlw5xo8fT+nSpWndujUXLlygcuXKFChQgGLFiqleL5cCvZ+vvvoKPz8/FixYwPjx4/nqq6+oXLkyERERXL16ldDQUExNTalSpQrly5fH29s7r0MW70kSpY7JzMzEyMiIa9eusWnTJpKSkqhZsyYtW7akaNGiANSqVQuAjRs34u7uzpgxY1Q9SUmS7yc1NZXy5cuzd+9e6tati6mpKSNGjKBixYrs27cPY2NjTE1N2bx5MxcuXGDEiBHSk9QDctZbh2T1JG/fvk2XLl1o3rw51tbW7Ny5k88//5w+ffqwY8cOTp8+jampKcnJyezZswcTExNJklrw6me4efNmlixZgrm5Oe7u7gQGBpKenk7r1q0pXbo0d+/eZc2aNVSpUiWPoxbaIIlSxzx9+pQ5c+ZQqFAhxo4dC8D169cZPnw4X3zxBZ07dyYqKop79+7RpUsXjI2NVdf5iX9n6tSpNG3alPr16wOwcuVKQkJCOHDgAEZGRnh7e1OjRg0mTJhAeno6169fx97enlKlSuVx5EJbJFHmcwqFghMnTvDgwQMqVarE2bNn2bhxIw4ODqxdu1bV7rfffmPUqFHs27eP4sWLq8pfPZ4p/p0jR44wfvx4QkNDOXbsGBs2bGDz5s2qiSxu375N//79cXBwYNasWTLPpx6SbkY+lpSURNeuXbGzsyM6Oppu3bpRq1Yt0tLS+OWXXwgLC1Pds+3o6EilSpVyXMgsSfL9NWvWDFNTU7y8vChYsCB79uzBxsZGNVu5ra0tK1euZMSIESgUCkmUekgSZT6VkpJCx44dqV27NlOmTCEhIYFChQphYGBAyZIlSU9P5+zZs/z999+0b9+e4OBgFAqF3Bb3gTRs2JANGzYwcOBAnj9/DqA6nJGeno69vT2hoaFyiENPydA7nwoLC+Pw4cMsW7ZMVXbo0CG2bt2KhYUFCQkJ1KtXjx9++IFChQrh4uLCzJkzAZkN6EM6fvw4AQEBrFy5kpo1a2arkzud9Jf8NuVTT548US3+FRERwYwZM/j6668pW7Ysrq6uFCxYkMTERDp16oSNjQ2VKlXK44g/Dp6ennz33Xf4+/tz6tSpbHWSJPWX9Cjzqf/+97906tSJihUrkpaWRqFChfD396du3boATJo0iUqVKuHj48P333/PuXPncHd3p3///nkc+cchPDycmTNncuDAAdWM8UJ/yQGVfMrZ2Znt27cTERFB9erVKV26NLa2tqr6x48fk56ejpWVFe3bt0ehUHD+/HkSExPlZEIuaNq0KXXr1pUk+ZGQHqWOePz4serOmzFjxhAdHc3OnTtVk1w8fPgQIyMjVRshhPZIotQBZ8+epV+/fjg5OWFhYUFSUhJbtmzBxMRELiYXIhdIotQBz58/JzIykps3b/LZZ5/RuHFjjIyMJEkKkUskUeooueNGiNwjiVIIIdSQ6yiFEEINSZRCCKGGJEohhFBDEqUQQqghiVIIIdSQRKnnGjdujJOTk+pRpUoVPD09mTdvHmlpaVrbT2RkJE5OTqp1wxs3bszWrVvVvi41NZUtW7b86/3euXMHJycnYmJiXluvaRzv2vZd4xC6Ta5W/giMGjVKNcFvRkYGFy5cYPTo0VhaWuLv7/9B9rljxw4sLS3Vttu3bx9Lly6la9euHyQOIbRBepQfASsrK0qUKEGJEiUoVaoUTZs2pXXr1hw+fPiD7bNo0aIaTRghl/EKXSCJ8iNlbGysmlBj3LhxjBkzhnbt2lG7dm3++OMP0tLSmDt3Lh4eHtSsWZOBAwdy+/Zt1evj4+MZPHgw1atXp2XLlly4cCHb9l8dxmZmZrJ06VIaNmxI9erV6du3L7du3SIyMpLx48fz6NEjnJycuHPnDkqlklWrVuHp6Un16tXp3r07Fy9eVG33+fPnjB8/Hjc3Nzw9Pfn11181fs9JSUlMmjSJevXqUaVKFRo3bkxISEi2NrGxsXTo0AFnZ2c6d+5MdHR0ttcHBgZSq1Yt6tSpQ0BAAPHx8Zp/6IsXnXcAAAXXSURBVEJnSaL8yGRkZHDq1Cn27NlDkyZNVOU//vgjffr0Yd26dTg7O7Nw4UIiIyMJCgpi+/btlChRgp49e6JQKAAYOnQoT58+ZevWrYwfP55169a9cZ/BwcGEhIQwadIkdu/ejZWVlSrJBgYGUrRoUU6cOIGNjQ1btmxh+/btzJgxg127dlGrVi169OjBw4cPAZg8eTL//e9/WbduHfPnz+f777/X+L3Pnj2bixcvsmLFCg4cOMCXX37JrFmzuHv3rqrN1q1b6dSpE2FhYZQqVYrBgwerjuVOmDCBe/fusWHDBjZs2EBycjJ+fn7SK/4YKIVea9SokbJq1apKV1dXpaurq7JSpUrKqlWrKsePH6988eKFUqlUKseOHav09vZWvSYlJUVZtWpV5blz51RlGRkZygYNGijDwsKUV69eVVaoUEEZGxurqg8JCVFWqFBBqVAoVPvdsmWLMjMzU1mnTh1lSEiIqu3Dhw+Vc+bMUT579ky5c+dOpYeHh6quYcOGyv3792d7D506dVIuXbpU+fTpU2WlSpWUJ06cUNUdP35cWaFCBeX169ff+P63bNmiVCqVyp07dyovXbqkqktNTVVWqFBBtb1GjRopp06dqqp/9uyZ0tXVVXnkyBHlrVu3lE5OTspHjx6p6pOSkpRVqlRR/v7778rbt2+/NQ6h2+RkzkfAz88Pb29vAExNTSlevLhq2J3ls88+U/38119/kZqaSs+ePbMtb6BQKIiNjcXU1BRLS0vs7e1Vdc7Ozq/d95MnT3j8+HG2+uLFi6vWJH9VcnIyd+/eZdy4cQQGBqrKU1NTsbW1JTY2loyMjGzLXrxpv6/Ttm1bjh07xq5du4iNjeXy5cvAy152lmrVqql+trKywt7enpiYGIyMjFAqlTRt2jTbNtPT04mNjZU1vPWcJMqPQNGiRbGzs3trm1dPvGQljo0bN+aYLd3a2pozZ86QmZmZrfyfiVdd+etk7XfevHk4OTllq7O0tOTevXtA9hNA7zLN3Lhx44iMjKRNmzZ89dVXuLq60qhRo2xt/jkjU2ZmJqampmRkZGBmZkZYWFiO7RYtWpTExP9r745dGgfDOI5/LQ0VWiNSrFa00VAk0FGRLl0cnQRdBJWiDoqTS0VRKAomBSOC1YKoW0GcxNG/QNwEl4IVXToIKpVSLTjccBDqeRKHG+6O57O/ed83w4+H9wl5y99eh/j3yBml+CQSieD1enl8fETTNDRNo6OjA9u2KRQK9Pb28vb2RqFQcMbUN1zqNTU1EQwGneoNoFwuE4/HKRaLHypWVVVpbW3l4eHBmVfTNPb397m8vKSnpwdFUbi6unKd91fPz8+cnp6yubnJwsICQ0NDzrWz9cFb37wpl8vc39+j6zq6rlOr1ajVas66mpubMU2TUqn0rTWIf5dUlOITv9/P2NgY6+vreL1eurq6yOVyXFxcsLKyQigUIpFIsLy8zNraGi8vL2Sz2S+fl0wmyWazhMNhOjs72d7epq2tDV3Xubm5oVKpUCwW0TSNmZkZdnZ2CAaDxGIxTk5OODs7Y2JigkAgwMjICKZpoqoqiqJgmua39hQIBPD7/Zyfn9Pe3k6pVGJjYwPgw4f3+XwewzAwDAPbtolEIiQSCTweD4ODg6RSKVZXV1FVFcuyuLu7o7u722k2if+TBKX4rVQqhcfjYXFxkWq1SiwW4/DwkFAoBMDW1hbpdJrx8XFaWlpIJpNfhtbU1BSVSoWlpSWq1SoDAwPs7e3R0NBAPB4nGo0yPDxMPp9ncnKS19dXMpkMT09PRKNRcrkchmEAPzvPlmUxOztLY2Mj8/PzpNNp1/0oioJt21iWxfHxMeFwmNHRUXw+H9fX187Z4/T0NAcHB9ze3tLf38/u7q5zR3omk8E0Tebm5nh/f6evr4+joyN8Pt8feOPibyY/7hVCCBdyRimEEC4kKIUQwoUEpRBCuJCgFEIIFxKUQgjhQoJSCCFcSFAKIYQLCUohhHDxA0DtpuIzoVihAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "words_list_model_cnf_matrix = confusion_matrix(y_train, words_list_model_y_pred, labels=[1, 0])\n", "\n", "plot_confusion_matrix(words_list_model_cnf_matrix, classes=class_names,\n", " title='words_list_model Confusion Matrix')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The row totals match those of the `ham_only` and `spam_only` confusion matrices as expected since the true labels in the training dataset are unaltered for all models.\n", "\n", "Of the 42 spam emails, `words_list_model` correctly classifies 18 of them, which is a poor performance. Its high accuracy is buoyed by the large number of true negatives, but this is insufficient because it does not serve its purpose of reliably filtering out spam emails.\n", "\n", "This emails dataset is an example of a **class-imbalanced dataset**, in which a vast majority of labels belong to one class over the other. In this case, most of our emails are ham. Another common example of class imbalance is disease detection when the frequency of the disease in a population is low. A medical test that always concludes a patient doesn't have the disease will have a high accuracy because most patients truly won't have the disease, but its inability to identify individuals with the disease renders it useless. \n", "\n", "We now turn to sensitivity and specificity, two metrics that are better suited for evaluating class-imbalanced datasets." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sensitivity\n", "\n", "**Sensitivity** (also called **true positive rate**) measures the proportion of data belonging to the positive class that the classifier correctly labels.\n", "\n", "$$ \\text{Sensitivity} = \\frac{TP}{TP + FN} $$\n", "\n", "From our discussion of confusion matrices, you should recognize the expression $TP + FN$ as the sum of the entries in the first row, which is equal to the actual number of data points belonging to the positive class in the dataset. Using confusion matrices allows us to easily compare the sensitivities of our models:\n", "\n", "* `ham_only`: $\\frac{0}{0 + 42} = 0$\n", "* `spam_only`: $\\frac{42}{42 + 0} = 1$\n", "* `words_list_model`: $\\frac{18}{18 + 24} \\approx .429$\n", "\n", "Since `ham_only` has no true positives, it has the worst possible sensitivity value of 0. On the other hand, `spam_only` has an abysmally low accuracy but it has the best possible sensitivity value of 1 because it labels all spam emails correctly. The low sensitivity of `words_list_model` indicates that it frequently fails to mark spam emails as such; nevertheless, it significantly outperforms `ham_only`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specificity\n", "\n", "**Specificity** (also called **true negative rate**) measures the proportion of data belonging to the negative class that the classifier correctly labels.\n", "\n", "$$ \\text{Specificity} = \\frac{TN}{TN + FP} $$ \n", "\n", "The expression $TN + FP$ is equal to the actual number of data points belonging to the negative class in the dataset. Again the confusion matrices help to compare the specificities of our models:\n", "\n", "* `ham_only`: $\\frac{758}{758 + 0} = 1$\n", "* `spam_only`: $\\frac{0}{0 + 758} = 0$\n", "* `words_list_model`: $\\frac{752}{752 + 6} \\approx .992$\n", "\n", "As with sensitivity, the worst and best specificities are 0 and 1 respectively. Notice that `ham_only` has the best specificity and worst sensitivity, while `spam_only` has the worst specificity and best sensitivity. Since these models only predict one label, they will misclassify all instances of the other label, which is reflected in the extreme sensitivity and specificity values. The disparity is much smaller for `words_list_model`. \n", "\n", "Although sensitivity and specificity seem to describe different characteristics of a classifier, we draw an important connection between these two metrics using the classification threshold." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification Threshold\n", "\n", "The **classification threshold** is a value that determines what class a data point is assigned to; points that fall on opposite sides of the threshold are labeled with different classes. Recall that logistic regression outputs a probability that the data point belongs to the positive class. If this probability is greater than the threshold, the data point is labeled with the positive class, and if it is below the threshold, the data point is labeled with the negative class. For our case, let $f_{\\hat{\\theta}}$ be the logistic model and $C$ the threshold. If $f_{\\hat{\\theta}}(x) > C$, $x$ is labeled spam; if $f_{\\hat{\\theta}}(x) < C$, $x$ is labeled ham. Scikit-learn breaks ties by defaulting to the negative class, so if $f_{\\hat{\\theta}}(x) = C$, $x$ is labeled ham.\n", "\n", "We can assess our model's performance with classification threshold $C$ by creating a confusion matrix. The `words_list_model` confusion matrix displayed earlier in the section uses scikit learn's default threshold $C = .50$. \n", "\n", "Raising the threshold to $C = .70$, meaning we label an email $x$ as spam if the probability $f_{\\hat{\\theta}}(x)$ is greater than .70, results in the following confusion matrix:" ] }, { "cell_type": "code", "execution_count": 88, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEeCAYAAADow4zZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8TNf/+PHXZF8lhBJKEkISayIEsYfaGqGW2lJFLbGWqi2K2Cmqldi3IqjaEkUtsbS2T1AtpRUVoYilCQnZt/n94Zf5msqYSZrI4v30uI+HnHvmnve9mbznnHvv3KNQKpVKhBBCvEKvsAMQQoiiShKkEEJoIAlSCCE0kAQphBAaSIIUQggNJEEKIYQGkiCFEEIDSZBCCKGBJMj/SO6zF2+SvN/erGKXIL28vFi8eHG+bc/JyYnt27e/8n9dHD16lHnz5uVbLPkpL8dJ1/3/4Ycf8PX1pWHDhnh4ePDRRx/x008/5TVUjW7fvk23bt2oU6cOo0aNypdt5vZ3nBdeXl44OTkRGBiY4/r9+/fj5OTEhx9+mKvt6vp+y699/OGHHxg0aBBNmjShdu3atG7dmi+++IKoqKj/vO3c+uijj3BycspxmT59ulrdu3fv4ufnh5ubGx4eHkyePJm4uLg8tWuQH8GXFDt27KBy5co619+8eTNly5YtwIiKFqVSyaRJkzhy5Aj9+vVj6NChZGZmsn//foYOHcrMmTPp3bt3vrW3ceNG7t+/z/Lly6lYsWK+bDO3v+O8UigUhIWFMXr06FfWHTlyJE/b1PX99l/3MTU1lU8//ZTTp0/TrVs3+vbti4WFBREREWzZsoWbN2/y3Xff5Xn7eTFjxgwSEhLUyk6ePMnKlStp3bq1qiwhIYGPP/4Ya2trli5dSlJSEosXL2b48OFs27YNhUKRq3YlQb7E1dW1sEMo0nbs2MEPP/zAxo0bady4saq8devWGBkZMW/ePNq1a0eZMmXypb3nz59TvXp1WrRokS/bgzf3O3Z1deXXX3/l3r17vPvuu6rylJQUTp06RY0aNQq07f9i4sSJnDt3jnXr1qn9nhs3bkzv3r05derUfw0x1xwdHV8pW7VqFWXKlKF58+aqsu+//57Hjx+zfft2ypcvD0CFChXo06cPJ0+eVEumusjTENvHx4cZM2aofr5+/TpOTk6sXr1aVXbs2DHq1KlDUlISaWlpBAUF0a5dO+rWrcsHH3zAiRMnVHWdnJxYs2YN7du3x83NjfDwcODFp8HkyZNp0KABzZo1Y8+ePWpx/Prrr/Tu3Rs3NzcaN27MpEmTePr0aV52SRXHy0OT123/o48+4vz58xw8eBAnJ6dctRESEoKfnx/16tXDy8uLgwcPcuPGDXr16kW9evXo1asXkZGRqtdoO37ajhO86P2tWbMGLy8v6tSpQ7du3VTHWVebNm2ibdu2an802UaNGkXv3r1JTEzUKebsYxEaGsqoUaNwdXWladOmBAUFAS+GqQcOHODChQs4OTmpYs1p+Pjhhx8yefJkQPt74t+v1+W9qSnG16lfvz42NjaEhYWplZ86dYrSpUtTq1YttfJnz54xc+ZMWrRoQe3atWnWrBnz5s0jPT0dyPn9punvJnsfs+sePHhQ1c6qVauoXbs2EREROcYdFhbGoUOH+Oyzz3L8PRsbG9O2bVut+1/Qnjx5wqlTp3j//fcxMPi/ft6JEydo2LChKjnCi99FpUqVXnn/6SJPCbJZs2ZcuHBB9fP58+cBuHTpkqrs7NmzuLu7Y2Zmxueff86GDRvw9fUlKCgIR0dHhg8frhbwypUr8fPzY+bMmdStWxeAsWPHcvr0aaZNm8bUqVNZvnw5jx49Al70Lvz8/KhUqRIrVqxg2rRpnDlzhpkzZ+Zll16hbfszZsygZs2aeHp6smPHjlxte/bs2dSsWZNVq1ZRpUoVpk6dypgxY+jatStff/019+/fZ/bs2ar62o7f645TtqVLlxIYGEivXr1Yvnw5Dg4ODB48mGvXrukU86NHj7h165bap/XLKlWqhL+/v2pop8vvHGDOnDlUqVKFlStX0qlTJwIDA/npp58ICgrC09OTmjVrsmPHjlcSSk7y8p7QJU5NMb6Onp4eXl5eHDt2TK38yJEjtG/f/pX6n332GWfPnmXKlCmsXbuWbt26sWnTJvbu3Qtofr/l9HeTrVOnTrRo0YKFCxeSlJREVFQUK1asYPjw4Ro/1NetW4e1tTV9+vR57f79m1KpJCMjQ+uSXxeZ9u/fT0ZGBl26dFErj4yMzLG36ejoyM2bN3PdTp6G2M2bN2f9+vXExsZiY2PDxYsXcXZ25tdff0WpVKJQKDhz5gw9e/bk+vXrHD58mEWLFuHj4wNAixYtePz4MV9//bWqy+vl5cUHH3ygauP69eucOnWK1atX06pVKwAqV65M9+7dgRcHIi4uDl9fX9zc3ACwsLDg1q1bedmlV2jbvqOjIxYWFlhbW+d6SNO0aVPGjBkDgL6+Ph999BE+Pj6qN2W/fv1Yt24dgNbjZ2tr+9rjBBAXF8fGjRsZPXo0Q4cOVW2jX79+LF++nBUrVmiNOTvh2traaq2r6+8cXnzYTpw4EXgxhDt06BA///wz06ZNw9ramoyMDJ2Pb27fE7rGqSnGli1bvjae9957j+HDh/P06VNKly5Neno6J06cYO3atWpJLiUlhczMTGbOnKnqtTVp0oSTJ0/yyy+/8OGHH2p8v/377+bfAgIC8Pb2Zs2aNfzyyy9UrVpV9R74tydPnvDbb7/Rt29fjIyMXrtv/7Z3716mTJmitd78+fPp1q1brradk9DQUKpVq0adOnXUyp89e0apUqVeqV+qVCnu37+f63bylCDd3d0xNTXlwoULdOjQgYsXLzJx4kQmTZpEZGQkFhYWREVF0bx5cy5cuIBCoaBDhw5q2+jUqRPTp09XnXh1cHBQW3/p0iX09PRo1qyZqqx27dqqP1BHR0esra0ZPnw43t7eeHl50bRpU61vWl0V5PZf/qS3sbEBUOshWVtbq47LL7/88trjd+HChdceJ4DLly+TlpZG8+bNycjIUJU3a9ZMlYi10dfXByArK0trXW0xJyQkYGFhAUC9evVU6xUKBeXLlycpKUmnmP4tt78zXeL8LzE2adIEU1NTTpw4Qbdu3Th37hympqa4urqqJUgTExM2btyIUqnk77//JioqioiICGJjY1VDbE3+/Xfzb5UqVWLMmDF8+eWX6OnpsWvXLgwNDXOsGxERgVKpzNP50datW7Nr1y6t9V4+H5tXkZGRXL16lc8+++w/b0ubPCVIIyMjGjZsyPnz56levTrPnj2jffv2fPPNN1y6dAl9fX0qVKhAjRo1CAsLw9LS8pVPpOzEkH3OKvvnbM+ePcPS0lLt/MLL9SwsLNiyZQtBQUHs2rWLLVu2YGNjw/jx49V6T3lVkNs3Nzd/pczExCTHuvHx8VqP3+uOE6C6xaFr1645tpGcnIypqelrY85OuA8ePNBY5+HDh1SoUEGnmLMTpLGxsVodPT29PA/Dcvs70yXO/xKjkZERLVu2JCwsjG7dunHkyBHee++9HK+khoWFMXfuXKKjoylbtiyurq4YGxtrbefffzc56dixI19++SW2trY5Dj+zPXv2DCBPd2ZYW1tjaWmptV72B+1/sW/fPhQKBZ07d35lXalSpVT78bJnz55hZWWV67byfBW7WbNm7Ny5EycnJ2rWrImpqSnu7u5cunSJ9PR0VY/GysqK58+fk5aWpvZGjImJUa3PiZWVFc+ePXvldfHx8ar/16hRg2XLlpGWlsa5c+dYv349X3zxBU2bNqVChQp53bU3tn1daDt+2W+I1x2n7CHH+vXrczzeugynypQpg7OzM2fPns3xVp779+/j5eXF9OnT8/w719W/e7Ev9+Zy8zsr6DjhxTB78uTJJCYmcvz4cb766qtX6ty+fZuxY8fSt29fhgwZQrly5QDo2bPnf24fYN68eVSqVImYmBjWrFnDyJEjc6yXnWz/ff5aF29qiK1UKvnhhx/w8PDI8davatWqqV3gzHbz5k2aNm2a6/byfKN4s2bNuHnzJseOHcPd3R2Ahg0bcuHCBc6dO6c6mV+/fn2USiWHDh1Se/2PP/6Ii4uLxp6Th4cHSqVS7SpgZGSk6jzCuXPnaNKkCU+ePFF9Uo8fP56srCweP36c191S0WX7enoFf5+9tuPXqFGj1x4neDGkNzAwID4+njp16qiWM2fOsH37dp0/1X19fTl69KjaBbpsgYGBGBoa0rZt2zz/znVhZmbGw4cPVT8/efKE27dvA7l/TxRknNlatGiBUqlkxYoVZGVl0bBhw1fq/PHHH6SnpzNs2DBVcoyJieHGjRtqHwZ5eb8dP36cw4cPM2PGDPz8/Fi1apXGG73r1q1L6dKl2b17d46nUh49esTff/+d42uzh9jaltzeZvNvFy5c4P79+69cnMnWqlUrzp8/r5bkf/vtN+7fv5+ntvPcg6xWrRoVKlTgp59+olevXgA0aNCA6dOno6+vj6enJwAuLi60bduWgIAA4uLicHBwYP/+/YSHh7/2dolq1arRqVMnAgICSEpKwsrKiq+//lp1/qR27dooFArGjBnDkCFDAFizZg3vvvsuLi4ued0tFV22X6pUKSIiIggPD8fDwyPXN6HqQtvx03ac4EXPoHfv3gQEBBATE4OTkxPnz59n5cqVfPrppzrH0r17d44dO8aQIUPo378/jRo1IjExkb1793L8+HHmzJlD+fLlKV++fJ5+57po3rw533//PU5OTpibm7NixQrVKYvcvify+t7MDXNzczw9Pdm0aRNdu3bN8cPI2dkZfX19FixYQPfu3Xn8+DGrVq0iNTWV5ORkVb1/v9+0SUhIYObMmbRr147mzZvTqFEj9uzZw/Tp09myZcsr9Y2MjPjiiy+YMGECffv2pU+fPlSsWJGYmBjOnTvHvn37CA0NzbGt0qVLU7p06VwcGXUhISH4+/vz7bffvnbfQkNDMTExyfFOAIBevXoRHBzMiBEjGD16NMnJySxevBg3N7c3myDhRS9y165d1K9fH3iR1MqUKYO9vb3alaQlS5bw9ddfs2bNGp49e0aNGjVeuQM+JwsWLGDBggUsWrQIgEGDBqlum7C0tGTt2rUsWrSI8ePHk5mZScOGDVm/fr3Gk9C5ocv2P/74Y8aOHcvQoUM5fPhwgQ27tR2/1x2nbP7+/pQuXZpNmzbxzz//ULFiRSZNmsSAAQN0jkNPT4+goCCCg4MJCQlh69atGBgY4OTkxIYNG9SGMHn9nWszdepUZsyYwdSpU7GysuKTTz7h999/B/L2niioOF/23nvvceLECd57770c11etWpV58+axfPlyDh8+TPny5enQoQPt2rVjx44dZGZmoq+v/8r7TZulS5cSHx+vGvoaGRkxdepUhg4dys6dO3Mcwnt7e1OuXDnWrFnDnDlzSElJoVy5cjg7OzN9+nSqVKny3w6GBllZWWRmZr72nGtqaiqHDx+mbdu2qnPY/2ZhYcGmTZuYO3cu48aNw8DAgNatWzNlypQ8dWAUMquhEELkrER+1VCpVJKZmfnaOgqFIl+uqBVGe0KIN6NEJsjz58/Tv3//19apVKkSx48fL5btCSHejBI5xE5ISND6SCYjI6NcfYe6KLUnhHgzSmSCFEKI/FAih9j/RXK6fF4UpuS015/LFQWvjHn+pgVTN+0PO07+NX9uq8pvkiCFEAVLUewmLlCRBCmEKFh6xffuDUmQQoiCVQDfMHtTJEEKIQqWDLGFEEIDGWILIYQGMsQWQggNZIgthBAayBBbCCE0KMZD7OLb9xVCFA8KPe2Ljvbt24ebm5va4uzszLRp04iPj2fkyJG4u7vTqlUrdu7cqXpdWloa/v7+eHh44OnpycqVK3VqT3qQQoiClY+P+fPx8VFN0Qtw9uxZJk6cyMiRI5k2bRpmZmacPXuWiIgIhgwZQvXq1XF1dWXp0qVER0dz7NgxYmNjGTRoEHZ2dnTq1Om17UkPUghRsPKxB/myxMREJk+eTEBAAJaWloSFhTFmzBiMjY2pW7cu3t7ehISEAC+mahg2bBiWlpbY29vj6+vL3r17tbYhCVIIUbAUCu1LHqxbt44aNWrQtm1b7ty5g4GBAZUrV1atd3Bw4NatW8THxxMbG6s25W32Om1kiC2EKFgFcBU7MTGR4OBg1q5dC7yY+vffs1CamJiQkpKimvjs5bnfs9dpIz1IIUTBKoAhdlhYGBUrVsTV1RV4kfxSU1PV6qSkpGBmZqZKnC8nxOx12kiCFEIUrAIYYp84cYKOHTuqfrazsyM9PZ3o6GhVWVRUFI6OjlhbW2NjY6P21P+oqCiqVaumtR1JkEKIgqWnr33JpcuXL6t6j/Biutc2bdqwZMkSkpOTuXLlCvv376dz587Ai6vfgYGBxMXFcfv2bYKDg+nSpYv20HMdmRBC5EY+D7EzMzN58OAB5cqVUyufPXs2GRkZtGzZkjFjxjBhwgTq1asHwNixY7G3t6djx4707duXDz/8UK0HqjF0mZNGnUy5ULhkyoXCl+9TLry/TGud5ANj8rXN/CJXsYUQBUuv+KaZ4hu5EKJ4KMbfxZYEKYQoWPK4MyGE0EB6kEIIoYE8D1IIIXKmkB6kEELkTBKkEEJooNCTBCmEEDmSHqQQQmggCVIIITSQIbYQQmggPUghhNBAEqQQQmigpydfNRRCiJwV3w6kJEghRMEqzkPs4tv3FUIUC3p6elqX3Hj48CHDhg2jfv36tGjRgs2bNwMQHx/PyJEjcXd3p1WrVuzcuVP1mrS0NPz9/fHw8MDT05OVK1fqFnuuIhNF1qNHjzAz0ntlmeY/ubBDK5FSU1MJmDaF2jXsqVKhNH17duV21It5lkcOG4SNhWGOyyi/Two58kKg0GHRkVKpZMSIEVStWpXw8HDWr19PUFAQly5dYtq0aZiZmXH27FmWLVvG4sWL+e233wBYunQp0dHRHDt2jG3btrFz504OHjyotT0ZYpcQ167+jkKh4HDYCYyMjFTlFStWKsSoSi7/iZ8RuncXM+csoIJtRRYvnEvX99tx9uIVPp80lQGfDFWrv2/vblYtX0bvvv0LKeLCk59D7MuXL/P48WM+//xz9PX1qV69Ot999x3GxsaEhYVx+PBhjI2NqVu3Lt7e3oSEhODq6kpoaChLlizB0tISS0tLfH192bt3L506dXpte5IgS4irv1/B3sGBZs1bFHYoJV58XBxbN29k0dJA+vUfCICTswv1XKpxPOwI3j5dcaj6f1OKPnz4gG3Bm/j0s4k0a9GysMIuNPl5FfvatWtUr16dRYsW8cMPP2BhYYGfnx9OTk4YGBhQuXJlVV0HBweOHDlCfHw8sbGxODo6qq3bunWr1vYkQZYQV69epVatOoUdxlvB3MKCwydO41D1//7gDP9/rz3tX5PXAyyYE4CZmTmfTZzypkIsUvKzBxkfH094eDiNGzfmxIkTXL16lcGDB7NmzRpMTEzU6pqYmJCSkkJycjIApqamr6zTRs5BlhDXrv5OfHwczT0bYW1hQk2nagRv3lTYYZVIBgYG1HOtT6lSpcjMzORGxHXGDB9CBduKtG3XQa1u1K1Itm3ZxOQvZmBmZlZIEReyfDwHaWRkhJWVFcOGDcPIyIj69evTvn17li1bRuq/PpxSUlIwMzNTJc6XE2L2Om0kQZYAmZmZXP/zD27ciGDU6E/Zu+8Arb3aMHTwQA4e2F/Y4ZVo0/0n0sS9DsfDjjBj9jxKWVmprV+/ZiVly71Dz159CynCwpefV7EdHBzIzMwkM/P/pgfOzMykZs2apKenEx0drSqPiorC0dERa2trbGxsiIqKUltXrVo1tJEEWULsDvmB4ydP06tPX1p7tWH5yjW079iJubNnFnZoJVqvvr6EHDhK/4GDGTFkIKF7dqnWpaens33rZgYMGqJ24exto1AotC66atq0KSYmJgQFBZGRkcGlS5c4evQoHTp0oE2bNixZsoTk5GSuXLnC/v376dy5MwA+Pj4EBgYSFxfH7du3CQ4OpkuXLtpjVyqVyjzveS5FRUXx5ZdfcvHiRTIyMqhcuTIfffQRPXv2fFMhaJWc/sYOR4FbHvgNX/hP5smzpGJzs25yWqb2SkVUt84dSExM4PDx0wD8dOI43Tq359wvv1PDybmQo9NdGfP8vTRReVSo1jp3g7Qnq2x37txh1qxZ/P7771hYWDBy5Ei6d+9OXFwcM2bM4Ny5c5iZmTFq1Ch69OgBvBhSz5s3j6NHj6JQKOjfvz9+fn5a23pjF2mysrIYPHgw3bp1Y+nSpRgZGXHx4kVGjRpFqVKlaN++/ZsKpcR58OABBw/8QNcPumNjY6MqT0lJwdzcvNgkx+Li0aOHhB05RLcevdRO/NeuU5d9IXtUP584doQaTi7FKjkWhPz+LradnR3r169/pdza2ppvvvkmx9eYmJgwa9YsZs2alau23tgQ++nTp9y7dw8fHx9MTEzQ09PDw8ODCRMmkJ6eTmBgIOPGjaN///64urrSs2dP/vzzT9XrN2/eTOfOnXF3d8fT05PAwEDVOicnJ3bs2EHLli2pX78+y5cvZ8+ePbRo0QIPD48cD2ZJkpaayugRfuze9b1a+Q/7QvFs2ryQoiq54p4+ZczwIfy4f5+qLCsri9OnfsLZpaaq7NdLv9DAw6MwQixS8nOI/aa9sR6kjY0NHh4eDBw4EB8fHzw8PKhbt65qeB0YGMiPP/7IV199xfr161m9ejUjRozg8OHDXLlyhVWrVrFt2zbs7e25ePEivr6++Pj4YGdnB8CZM2c4dOgQly5d4pNPPqFjx44cOXKE8PBwhg8fzocffoilpeWb2t03ys7enh49ezF96hSUSiUODlXZtHEDl3/7lZOnzhV2eCWOk7MLnby7MGXCOJJTkqlQwZZNG9bx57WrLFr6fx/cEX/+gVfbdoUYadFQlBOgNm/0Is26devw9fUlPDycwYMH4+HhwWeffcbTp08BaNKkCZ06dcLQ0JDhw4eTlJTEpUuXqFWrFnv27MHe3p6YmBjS09MxMTHh8ePHqm37+vpiampK48aNUSqV9OvXDxMTE5o3b05mZiaPHj16k7v6xq1au55PBg/lq8Vf0qvHB9y/f48fDh6mnqtrYYdWIq1av4mevfuyYM5M+vfpQWzMP4QcOIJ7gxc9RqVSyZMnsVj966r220ihp9C6FFVv9EZxY2NjBgwYwIABA0hNTeWXX35h8eLF+Pv7U7NmTapUqaKqq6+vT7ly5YiJiUFPT48VK1Zw+PBhbGxsqF27NvBiWJMt+42or/9ikvJSpUoB/3f+4+W6JZGZmRlzF3zJ3AVfFnYobwVzc3PmLFjMnAWLc1yvUCh4HK/9RuS3gfQgdXDw4EHatGlD9kVzY2NjPD09GT16NNevXwdQ6xFmZGTw+PFjKlSowMaNG7lx4wZhYWEcPHiQuXPnkpGRobb94vxLEKIkUyi0L0XVG0uQTZo0ITExkblz5xIbG4tSqeTOnTts2bKF1q1bA3Dq1CnOnj1Leno6y5cvp3Tp0ri5uZGQkIChoSGGhoYkJiaycOFC0tPTX0mSQoiiR09PoXUpqt5YgixdujTbtm3j8ePHeHt74+rqysCBA6lTpw6TJ794JFfdunVZu3YtjRo14uLFi6xevRp9fX0GDhyIgYEBTZo0oX379qSlpVG/fn0iIyPfVPhCiDwqzlex3+iN4q8TGBjIX3/9xbJlywo1jpJ0o3hxVJxvFC8p8vtGcefJh7XWub6gaN4HLU/zEUIUKH39ottD1EYSpBCiQBXlIbQ2RSZBjh49urBDEEIUgGKcH4tOghRClEwyL7YQQmggPUghhNBAzkEKIYQGRflGcG2K78kBIUSxkN9fNVy/fj21a9fGzc1NtVy8eJH4+HhGjhyJu7s7rVq1YufOnarXpKWl4e/vj4eHB56enqxcuVKntqQHKYQoUPk9xP7jjz8YN24cn3zyiVr5mDFjMDMz4+zZs0RERDBkyBCqV6+Oq6srS5cuJTo6mmPHjhEbG8ugQYOws7PTOi+29CCFEAUqv7+L/eeff+Li4qJWlpiYSFhYGGPGjMHY2Ji6devi7e1NSEgIAKGhoQwbNgxLS0vs7e3x9fVl79692mPPVWRCCJFL+TnETk5OJioqis2bN9O0aVM6duzIrl27uHPnDgYGBlSuXFlV18HBgVu3bhEfH09sbCyOjo6vrNNGhthCiAKVn0PsmJgY3N3d6dOnD8uWLePKlSv4+fkxcOBA1fzX2UxMTEhJSSE5ORlAbf6g7HXaaEyQL88hq42Dg4POdYUQb5f8vIpduXJlgoODVT83aNCALl26cPHiRVJTU9XqpqSkYGZmpkqcKSkpWFhYqK3TRmOC7NixIwqFAk0P+8lep1Ao1CbXEkKIl+XnNZpr165x5swZhg4dqipLTU3F1taW9PR0oqOjqVixIvCik+fo6Ii1tTU2NjZERUVRtmxZ1bpq1appbU9jgjx27Nh/3RchhMjXIbaZmRlBQUFUqVKFdu3aER4ezoEDBwgODub58+csWbKEOXPm8Ndff7F//37WrFkDgI+PD4GBgSxbtoy4uDiCg4OZMGGC9thz8zzIS5cuERUVRfv27Xnw4AF2dnYYGRnlfW+LIHkeZOGS50EWvvx+HmTLpWe01vlpXFOdt3f8+HGWLl3K3bt3KV++POPGjaNDhw7ExcUxY8YMzp07h5mZGaNGjaJHjx7AiyH1vHnzOHr0KAqFgv79++Pn56e1LZ0S5JMnT/Dz8+OPP/4gKyuLw4cPM3fuXCIjI9mwYYPalaPiThJk4ZIEWfjyO0G2+vqs1jonx3rma5v5RafbfObOnUvZsmUJDw/H2NgYgIULF1KlShXmzp1boAEKIYq3Ej9p19mzZxk7dizm5uaqMisrKyZPnszFixcLLDghRPFXnCft0qkvnZmZmeO80s+fP1fNQy2EEDnRK8pdRC106kG2bduWRYsW8eTJE9UVqZs3bzJ79mzatGlToAEKIYq3Ej/E9vf3x8LCgqZNm5KUlETnzp3p3Lkztra2+Pv7F3SMQohiTF9PoXUpqnQaYls3AoKUAAAgAElEQVRYWPDNN99w9+5dIiMjycjIoFq1avINGiGEVm/FA3MzMzO5ceMGkZGRGBkZYWJiIglSCKFVMc6PuiXIiIgIhg8fTlxcHPb29mRlZXHnzh3s7e0JCgqiUqVKBR2nEKKY0i/GGVKnc5AzZsygXr16/Pzzz+zZs4eQkBBOnjxJ+fLlmT59ekHHKIQoxhQKhdalqNIpQf7xxx+MHj1a9SQMeHEf5Pjx4+U+SCHEa5X4q9jOzs5cvnz5lfKIiAg5DymEeK0SeRV7x44dqv+7ubkREBDAtWvXqFu3Lvr6+ly/fp3g4OBX5oUQQoiXFeUhtDYaH1bh5eWl2wYUihL1aDR5WEXhkodVFL78flhFz28vaa2zc0D9fG0zv2g8EsePH3+TcQghSqjifBVb54+KR48ecevWLTIzX3zCK5VK0tLSuHbtGmPGjCmwAIUQxVtxHmLrlCC3bt3KvHnzyMzMVJuGQaFQUK9ePUmQQgiNivA1GK10uoq9fv16hg8fzu+//46NjQ0nT55k//79ODs789577xV0jEKIYqygHncWExNDkyZNOHHiBAD37t3j448/xs3Njfbt26vKAeLj4xk5ciTu7u60atWKnTt36ha7LpUeP35Mly5dMDQ0xMXFhd9++w1HR0emTJmic0NCiLdTQd0oPnXqVOLi4lQ/f/rpp9StW5fz58/j7+/P+PHjiY6OBmDatGmYmZlx9uxZli1bxuLFi/ntt9+0tqFTgrS2tub58+fAiyleIyIiAKhUqRIPHz7M9Y4JId4eegrtS25t374dU1NTbG1tAYiMjOTGjRuMHDkSQ0NDWrZsiYeHBwcOHCAxMZGwsDDGjBmDsbExdevWxdvbm5CQEO2x6xJM69atmT59OtevX6dx48aEhoZy6dIltmzZogpQCCFyoqdQaF1yIyoqio0bNxIQEKAqu3XrFpUqVVLNgQ0vOnO3bt3izp07GBgYqM2dlb1Oa+y6BDR58mScnZ25fv06Xl5eNGzYkL59+7Jz504mT56ci10TQrxt8jNBZmRkMHHiRKZOnYq1tbWqPCkpCVNTU7W6JiYmpKSkkJSUpJY4X16njU5Xsc3NzZkzZ47q54ULFzJlyhQsLCwwMMjfm0qFECVLft7ls2LFClxcXGjZsqVauamp6SsJLyUlBTMzM0xNTUlNTc1xnTY6fdVQm169eulcVwjxdsnPSbkOHjzIP//8w8GDBwFISEjgs88+w8/Pj/v375OWloaRkRHwYijeqFEj7OzsSE9PJzo6mooVK6rWOTo6am1PY4JcvXq1TgErFApJkEIIjfJz0q5Dhw6p/ezl5cW0adNo3bo1hw8f5uuvv2bs2LGcO3eO8PBwZsyYgYWFBW3atGHJkiXMmTOHv/76i/3797NmzRqt7clXDf+lON/1XxJUaja2sEN46yX/GpSv23tTf1KBgYFMnz6dJk2aULZsWb766ivVReTZs2czY8YMWrZsiZmZGRMmTKBevXpat6nxYRVvq5SMwo7g7Va64ajCDuGtl98JcvTeP7XWCfzAJV/bzC9yhUUIUaCK86hMEqQQokAV5+9iS4IUQhSoovzEcG10ulEcIDY2llWrVjFp0iRiY2M5ePAgN27cKMjYhBAlQEF81fBN0XnSrvbt23Py5EkOHDhAUlISZ8+epWfPnpw7d66gYxRCFGMlftKu+fPn8/HHH/Pdd99haGgIwJw5c/joo49YvHhxgQYohCjeDBQKrUtRpVOCvHbtGj4+Pq+U9+rVi8jIyHwPSghRcpT4HqSVlZXquWovu3btGmXKlMn3oIQQJUd+P83nTdLpKnafPn2YPn06n3/+OfBiPuyff/6ZwMBABg4cWKABCiGKN32dLwUXPTolyKFDh2Jubs6CBQtITk5m1KhRlC1bFj8/Pz7++OOCjlEIUYwV5R6iNjrfB9mvXz/69etHUlISmZmZWFpaFmRcQogSohjnR90SpLZHk3ft2jVfghFClDwlfl7sf9/Kk5GRwbNnzzAyMsLZ2VkSpBBCo6J8I7g2OiXI06dPv1IWHx/PtGnTqF+/fr4HJYQoOYpzgszz9SUrKyvGjh3LunXr8jMeIUQJo6+n0LoUVf/pAvy9e/dITk7Or1iEECVQft8ofvDgQTp27Iibmxvvv/8+YWFhwItR7ciRI3F3d6dVq1bs3LlT9Zq0tDT8/f3x8PDA09OTlStX6tSWTkPs8ePHv1KWkJDA+fPn8fb21qkhIcTbKT9v84mKisLf358NGzZQv359zp49y9ChQ/n5558JCAjAzMyMs2fPEhERwZAhQ6hevTqurq4sXbqU6Ohojh07RmxsLIMGDcLOzo5OnTq9tj2dEmT2JDgvK1++PP7+/nTp0iVveyqEeCvk543iDg4OnDlzBnNzczIyMoiJicHc3BwjIyPCwsI4fPgwxsbG1K1bF29vb0JCQnB1dSU0NJQlS5ZgaWmJpaUlvr6+7N27N38SpJOTEx07dqR8+fL5spNCiLeHHvl7jtHc3Jy7d+/Svn17srKyCAgI4O+//8bAwIDKlSur6jk4OHDkyBHi4+OJjY1Vm8XQwcGBrVu36hC7DpYvX05aWloedkUI8bYriIdV2NracvnyZTZu3MjChQs5fvw4JiYmanVMTExISUlRXScxNTV9ZZ02OiXIFi1aEBwczLNnz3KzD0IIgYGeQuuS620aGGBoaEiTJk1o164dV69eJTU1Va1OSkoKZmZmqsT5ckLMXqe1HV2CuXPnDgcOHGDz5s1YWFhgbGystj6n+ySFEALy96uGP/30Exs3buTbb79VlaWnp1OlShV+/vlnoqOjqVixIvDigo6joyPW1tbY2NgQFRVF2bJlVeuqVaumtT2dEmS/fv3ysCtCCJG/V7Fr1qzJ1atXCQkJwcfHh1OnTvHTTz/x/fff8+DBA5YsWcKcOXP466+/2L9/P2vWrAHAx8eHwMBAli1bRlxcHMHBwUyYMEFrexrnxQ4KCuKTTz5RG7e/DWRe7MIl82IXvvyeF/vbC39rrTOgYRWdt3fx4kXmzZvH7du3sbe3Z+LEiTRu3Ji4uDhmzJjBuXPnMDMzY9SoUfTo0QN4MaSeN28eR48eRaFQ0L9/f/z8/LS2pTFBuri4cPr0aWxsbHQOvCSQBFm4JEEWvvxOkJsu3tVa5+MGlbXWKQwah9ga8qYQQuRK0f0ioXavPQeZnp6u0+09Od1ILoQQUIIfmNu6dWudNvLnn3/mSzBCiJKnCD+LQqvXJshly5ZhZWX1pmIRQpRAipLYg1QoFNSvX/+tu0gjhMhfxXjOLrlII4QoWCXyHOQHH3zwyjdmhBAit0rkEHv+/PlvMg4hRAlVIofYQgiRH0rkEFsIIfJDMc6PkiCFEAUrvx+Y+yZJgiyB9oWGMGakH7fvPSzsUIq95u7VObLuU43rTd1G0aNdfbYsHPTKOp+Ryzl69k+uH5iJXcWcb5cbPH0LW38Iz7d4iyIZYosi43x4OEMGffzWPYWpoPx2/S4t+y9WKyttZcb2RYPZcegiALUcK/L7jfuMmrNdrd71qBcfUL0+W4uxkfqf2qzRPtSwL8+R09cKMPqioRjnR0mQJUVGRgYrlwcx/Yspkhzz0fPEFM7/flutbNP8Adx/HMdnC19MK1q7RiV++ePOK/WyXY64p/bze54utGxYg/f9gvjnaUJBhF2kFOchdnG+Ai9ecvbMaebMmsGsOfMZPnJ0YYdTYjWpV5UPOzRg0pI9JKekA1Db0ZZrf0Xr9Ho9PQVfju9OyLHfOB5+vSBDLTL09LQvRVURDk3khrNLTf64cYvRn44t1jfmFnUBozpz6pe/OPjzVQAszIypYluGRnUduLZvBs/Of8PPmz+nYW27HF/f930Pqtu9w9RvQt5k2IVKocO/okoSZAnxzjvvyPfmC1jt6hVp0aA6SzcdUyvT09Ojsm0Zxi3cSa/xa0lNz+CHFaOwLffqg15G9GnFvhOXuXU35k2GXqj0FNqX3Lh48SI9e/bE3d2dtm3b8t133wEQHx/PyJEjcXd3p1WrVuzcuVP1mrS0NPz9/fHw8MDT05OVK1fq1JacgxRCR4O6NeXW3X/48dRVVdmfkQ/5YPRKTv3yF4nJL56d+vPFG1z7IYAxvl5MWbpXVdfNpTJuLpWZuGT3G4+9MOXnVez4+HhGjBjBtGnTeP/99/nzzz8ZOHAgVapU4bvvvsPMzIyzZ88SERHBkCFDqF69Oq6urixdupTo6GiOHTtGbGwsgwYNws7Ojk6dOr0+9nyL/D9wcnLixo0br5Q3atSI8PCSfQuEKD68W9ZhT9ivamXxCckcOn1NlRwBEpPTCL8cRS3Himp1O7euy4N/4jn9y803Em9RkZ9D7OjoaFq2bEnnzp3R09OjVq1aNGrUiEuXLhEWFsaYMWMwNjambt26eHt7ExLy4lRGaGgow4YNw9LSEnt7e3x9fdm7d6+W1opIghSiqHOuWoHKtmUIPXZZrbxujUr079L4lfrGxgYkJqvP0/xeExf2nbj8St2SLj+H2C4uLixatEj1c3x8PBcvvrjdysDAgMqV/29uGwcHB27dukV8fDyxsbE4Ojq+sk5r7LqHVrj++OMPBgwYQLNmzahXrx6DBg0iJubFeZzJkyfz5Zdf0rt3b1xdXfH19eXKlSv07t0bNzc3Bg4cSEJCyb+dQhSc+jWrkJqWzm8R6hNQ1XV6l9UBvjg5lFeVlSttQZN6VTlz6f96igYGetSpUUnjrUAlmZ5CoXXJi+fPn+Pn56fqRZqYmKitNzExISUlheTkZAC129+y12mNPU+RFYDevXvToEEDtSU+Pl61/tNPP6VNmzacOnWKkydP8vz5c4KDg1Xr9+zZw5w5czhz5gwxMTGMGDGCuXPncvLkSaKjowkNDS2M3RIlRM2qtkTdiyUjI0utfG/Yr0T+/Q/fLR5C1zaudPGqx/5Vo4iNS2T9njOqetUql8PYyJC/bj9606EXOoUOS27dvXuX3r17Y2VlRVBQEGZmZqSmqvfYU1JSMDMzUyXOlxNi9jptisxFmu+++44aNWqolTVq1Ej1//Xr1/Puu++SnJzMo0ePKF26NI8e/d+brXXr1qoudJ06dTAyMqJatWoA1KtXj/v377+BvRAlVdnSFsQnJL9SnpicRie/Zcwb+wHLpvbC2NCA4+ERTFy8W3WfZPbrAeKev7qNki6/v2p47do1Bg8ejI+PD5MmTUJPTw87OzvS09OJjo6mYsUX536joqJwdHTE2toaGxsboqKiKFu2rGpddn54nSKTILW5cuUKQ4YMITExEScnJ+Lj4ylTpoxq/ctz5+jr61OqVCnVz3p6em/VE9K/mB7AF9MDCjuMEsVv5laN6/5+8BTfSRte+/ozlyIxdXs75/zOz/wYExPD4MGDGThwIEOHDlWVW1hY0KZNG5YsWcKcOXP466+/2L9/P2vWrAHAx8eHwMBAli1bRlxcHMHBwUyYMEFre8UiQT58+JBJkyaxbds26tWrB8CUKVPUkp7cHC1E0ZSfN4Lv2rWLJ0+esHLlSrV7Gfv378/s2bOZMWMGLVu2xMzMjAkTJqjyxdixY5k3bx4dO3ZEoVDQv39/OnbsqLW9YpEgExMTUSqVmJiYoFQq+fnnnzl06BBeXl6FHZoQQov87Lv4+fnh5+encf0333yTY7mJiQmzZs1i1qxZuWqvWCTIatWqMWLECD7++GOysrKoWrUqvXv35n//+19hhyaE0KI4D+4Uyrfp5JwOUjIKO4K3W+mGb+d5uqIk+degfN3exahnWus0cCiltU5hKBY9SCFE8VWce5CSIIUQBUoSpBBCaFCUH2emjSRIIUSBkh6kEEJoIAlSCCE0kCG2EEJoID1IIYTQQBKkEEJoIENsIYTQQHqQQgihgSRIIYTQQIbYQgihQXHuQRaZOWmEECWTQqF9yYsrV67QrFkz1c/x8fGMHDkSd3d3WrVqxc6dO1Xr0tLS8Pf3x8PDA09PT7WH7b6O9CCFEAUqv4fYSqWS3bt3s2DBAvT19VXl06ZNw8zMjLNnzxIREcGQIUOoXr06rq6uLF26lOjoaI4dO0ZsbCyDBg3Czs6OTp06vbYt6UEKIQpUfvcgV61axebNm9WeLJ6YmEhYWBhjxozB2NiYunXr4u3tTUhICAChoaEMGzYMS0tL7O3t8fX1Ze/evVrbkgQphChQ+Z0gu3fvTmhoKHXq1FGV3blzBwMDAypXrqwqc3Bw4NatW8THxxMbG6ua9fTlddrIEFsIUaDye4j9zjvvvFKWlJSkmv86m4mJCSkpKSQnv5hq19TU9JV12kgPUghRoArqIs3LTE1NSU1NVStLSUnBzMxMlThfTojZ67SRBCmEKFBvIkHa2dmRnp5OdHS0qiwqKgpHR0esra2xsbEhKipKbV21atW0blcSpBCiQCl0+PdfWVhY0KZNG5YsWUJycjJXrlxh//79dO7cGQAfHx8CAwOJi4vj9u3bBAcH06VLF63blQQphChQb6IHCTB79mwyMjJo2bIlY8aMYcKECdSrVw+AsWPHYm9vT8eOHenbty8ffvghHTt21B67TPuqTqZ9LVwy7Wvhy+9pX+89TdVa593SxvnaZn6Rq9hCiAJWfL9rKAlSCFGgivN3sSVBCiEKlJ4kSCGEyJk87kwIITQpvvlREqQQomDJEFsIITSQIbYQQmhSfPOjJEghRMGSIbYQQmggQ2whhNBAbhQXQggNJEEKIYQGMsQWQggNpAcphBAaSIIUQggNZIgthBAaSA9SCCE0kAQphBAaFOchtsxJI4QQGsishkIIoYEkSCGE0EASpBBCaCAJUgghNJAEKYQQGkiCFEIIDSRBCiGEBpIghRBCA0mQQgihgSRIIYTQQBKkEEJoIAnyLSRfv39zsrKyVP+X4178SIJ8C2RmZgKQkZEBgEKhkD/WNyAjIwM9PT3V/0XxI0/zKeGUSiUKhYIbN26wZcsWFAoF48ePx8rKqrBDeytkZWUxbNgwUlJSaNq0KR07dsTOzq6wwxI6kh5kCZadHB89ekS/fv1QKBT8+uuv9OjRg6ioqMIOr8TK7rEDDB8+nFKlSuHp6cmFCxfYunUrt2/fLrzgRK5ID7KEyszMRF9fn6SkJH799VcuX77MiBEjABg6dChRUVGsW7dOejP5LPu4K5VK7t27x86dO/nss88AOHjwIPv27aNKlSr07dsXe3v7wg1WaCUJsgT7888/mT9/PjExMZQrV47Vq1djYmICwJAhQ/j7779ZsWIF1apVK+RIS4bsHntWVhYffvghMTExPHz4kNmzZ9OzZ08ADhw4wKFDhyhVqhTDhg2jSpUqhRy1eB0ZYpcw2cO7hIQExo8fT7169ejUqRPh4eEcOXJEVW/t2rVYWlqybNmywgq1RMlOjgABAQHUrFmTVatW0aFDB86fP09YWBgA77//Pq1btyYlJQUzM7PCDFnoQHqQJdCdO3dYvnw5tra2jBs3DoDt27cza9Ys5s+fT9euXVV1s7KyVFdaxX+3ceNGjh8/ztSpU3F2dub69et8++23pKWl4e3tjZeXF/DiA8zCwqKQoxXa6AcEBAQUdhDiv3u5B5OcnMyMGTO4efMm3t7emJubU6dOHcqUKcPMmTOxtramTp06AKohoaI4Tz1XiDIzM1UfMOnp6Zw8eZL//e9/pKWl0bBhQ2xtbalSpQrXrl3j/PnzlCpVCnt7e4yMjAo5cqELSZAlQPYfaUJCAs+ePaN8+fJ0796d7777jrt379K8eXMMDQ2pU6cORkZGHD58mG7duqmSoiTHvMnIyMDAwAClUklERASmpqa0bt0ahULB1atXefr0Kc7OzlSoUIGKFSty9+5dOnToID3HYkSG2MVc9hD5+vXrBAQEEBcXR5cuXRg+fDiPHj2ia9eueHp6Mnv2bNU5r+ze5su9TpE72VerMzMz6d27N8+fP6dcuXL06dOHTp06sXbtWs6fP0/Tpk3p2bMn5ubmpKWlSc+xmJGTT8Wcnp4ekZGRDBo0CC8vL3r37s2aNWtYvnw55cuXJyQkhPDwcEaPHk1qaqrqdZIc/5vsW3kGDBhA9erVWb58OaVLl2br1q388MMPDBkyhMaNG3Po0CH27t2LUqnE0NCwsMMWuWRQ2AGI/yYjI4MtW7YwZMgQBg4cyP379zl58iTr1q0jJSWF8ePHs3v3bmbMmKH6A5XEmHfPnz/H0tISgPDwcIyMjJg3bx4A5ubmJCYmsmvXLkxNTfnkk08wMTFRDbtF8SM9yGLOwMCAGzduYGZmRmZmJuPHj6d79+7MnTuXtWvX4u/vj56eHqtWrUJPT0/t4QkidxYtWsSECRN4/PgxANHR0aSlpQEwYcIEUlJSmDVrFv/88w9ffPEFQUFB9OvXj4oVKxZm2OI/kB5kMZN97gv+7/zj2LFjqVq1KlOmTKFixYp07tyZq1evUrNmTczNzbGxsVG9Xm7pybtu3brh5+fHkiVLmDJlCt26daNmzZqcOXOGGzdusHPnToyMjKhVqxbVqlXD29u7sEMW/5EkyGIkKysLfX19/vrrL7Zs2UJCQgINGjSgQ4cOlClTBoCGDRsCsHnzZjw8PJg4caKq5yjJMe/S0tKoVq0a+/fvp0mTJhgZGTFu3DicnZ05cOAABgYGGBkZsXXrVq5evcq4ceOk51gCyFXsYiK753j37l369OlDu3btsLS0ZPfu3bRo0YKBAweya9cu/ve//2FkZERiYiKhoaEYGhpKcvyPXj5+W7duZdmyZZiYmODh4YG/vz8ZGRl07tyZihUr8uDBA9atW0etWrUKOWqRHyRBFiPPnj1jwYIFWFlZMWnSJABu3rzJ2LFjef/99+nduzcXL17k4cOH9OnTBwMDA9W9eiL3AgICaNu2Lc2aNQNg9erVBAcH8+OPP6Kvr4+3tzf169dn6tSpZGRkcPPmTezt7alQoUIhRy7yiyTIIiwlJYXTp0/z+PFjXFxcuHTpEps3b8bR0ZH169er6p06dYrPP/+cAwcOULZsWVX5y+crRe4dPXqUKVOmsHPnTo4fP863337L1q1bVQ+YuHv3LoMHD8bR0ZF58+bJMzZLIOlaFFEJCQn07dsXOzs7rl+/Tr9+/WjYsCHp6en89NNPhISEqL5TXb16dVxcXF65CVmS43/z3nvvYWRkRMeOHSlVqhShoaHY2tqqng5euXJlVq9ezbhx40hJSZEEWQJJgiyCkpOT+fDDD2nUqBEzZswgLi4OKysrFAoF5cuXJyMjg0uXLnH//n169OhBUFAQKSkp8hW2AtCyZUu+/fZbhg0bRlJSEoDqlEVGRgb29vbs3LlTTmOUUDLELoJCQkI4cuQIK1asUJUdPnyY7du3Y2pqSlxcHE2bNuX777/HysqKunXrMnfuXECezlNQTp48yfjx41m9ejUNGjRQWyffSiq55C+pCHr69KlqUq0zZ84wZ84cPv30UxwcHHB1daVUqVLEx8fTq1cvbG1tcXFxKeSIS75WrVrx1VdfMXLkSM6dO6e2TpJjySU9yCLo999/p1evXjg7O5Oeno6VlRUjR46kSZMmAEybNg0XFxd8fHzYtGkTly9fxsPDg8GDBxdy5CVfWFgYc+fO5ccff1Q9nV2UXHLipAiqU6cOO3bs4MyZM7i5uVGxYkUqV66sWv/kyRMyMjKwsLCgR48epKSkcOXKFeLj4+VCQQFr27YtTZo0keT4lpAeZDHw5MkT1TdlJk6cyPXr19m9e7fq4RP//PMP+vr6qjpCiPwhCbKIu3TpEp988glOTk6YmpqSkJDAtm3bMDQ0lJvAhShgkiCLuKSkJMLDw7l9+zbvvvsuXl5e6OvrS3IU4g2QBFkMyTdkhHgzJEEKIYQGch+kEEJoIAlSCCE0kAQphBAaSIIUQggNJEEKIYQGkiBLKC8vL5ycnFRLrVq1aNWqFYsWLSI9PT3f2gkPD8fJyUk157aXlxfbt2/X+rq0tDS2bduW53bv3buHk5MTkZGROa7XNY7c1s1tHKJ4kzuNS7DPP/9c9VDdzMxMrl69yoQJEzAzM2PkyJEF0uauXbswMzPTWu/AgQMsX76cvn37FkgcQuQH6UGWYBYWFpQrV45y5cpRoUIF2rZtS+fOnTly5EiBtVmmTBmdHuQgt9+K4kAS5FvGwMBA9ZCLyZMnM3HiRLp3706jRo345ZdfSE9PZ+HChXh6etKgQQOGDRvG3bt3Va+PjY1lxIgRuLm50aFDB65evaq2/ZeHq1lZWSxfvpyWLVvi5ubGoEGDuHPnDuHh4UyZMoWYmBicnJy4d+8eSqWSNWvW0KpVK9zc3PD19eXatWuq7SYlJTFlyhTc3d1p1aoVP//8s877nJCQwLRp02jatCm1atXCy8uL4OBgtTpRUVH07NmTOnXq0Lt3b65fv672en9/fxo2bEjjxo0ZP348sbGxuh90UWxJgnxLZGZmcu7cOUJDQ2nTpo2qfN++fQwcOJANGzZQp04dli5dSnh4OIGBgezYsYNy5crRv39/UlJSABg9ejTPnj1j+/btTJkyhQ0bNmhsMygoiODgYKZNm8bevXuxsLBQJVd/f3/KlCnD6dOnsbW1Zdu2bezYsYM5c+awZ88eGjZsyEcffcQ///wDwPTp0/n999/ZsGEDixcvZtOmTTrv+/z587l27RqrVq3ixx9/pGvXrsybN48HDx6o6mzfvp1evXoREhJChQoVGDFihOpc7dSpU3n48CHffvst3377LYmJifj5+Ukv+G2gFCVS69atlbVr11a6uroqXV1dlS4uLsratWsrp0yZokxNTVUqlUrlpEmTlN7e3qrXJCcnK2vXrq28fPmyqiwzM1PZvHlzZUhIiPLGjRvKGjVqKKOiolTrg4ODlTVq1FCmpKSo2t22bZsyKytL2bhxY2VwcLCq7j///KNcsGCB8vnz58rdu3crPT09VetatmypPHjwoNo+9OrVS7l8+c7/Hf8AAAQ8SURBVHLls2fPlC4uLsrTp0+r1p08eVJZo0YN5c2bNzXu/7Zt25RKpVK5e/du5R9//KFal5aWpqxRo4Zqe61bt1YGBASo1j9//lzp6uqqPHr0qPLOnTtKJycnZUxMjGp9QkKCslatWsoLFy4o7969+9o4RPEmF2lKMD8/P7y9vQEwMjKibNmyquF1tnfffVf1/7///pu0tDT69++vNo1ASkoKUVFRGBkZYWZmhr29vWpdnTp1cmz76dOnPHnyRG192bJlVfN5vywxMZEHDx4wefJk/P39VeVpaWlUrlyZqKgoMjMz1aaW0NRuTj744AOOHz/Onj17iIqK4s8//wRe9Kqz1atXT/V/CwsL7O3tiYyMRF9fH6VSSdu2bdW2mZGRQVRUlMyBXcJJgizBypQpg52d3WvrvHxBJTthbN68+ZUnk1taWnL+/HmysrLUyv+dcLWV5yS73UWLFuHk5KS2zszMjIcPHwLqF3Zy86i3yZMnEx4eTpcuXejWrRuurq60bt1arc6/n46UlZWFkZERmZmZGBsbExIS8sp2y5QpQ3x8vM5xiOJHzkEKlSpVqmBgYEBsbCx2dnbY2dlRsWJFlixZQkREBDVq1CAlJYWIiAjVa16+kPIyS0tLbGxsVL01gPj4eBo3bkxkZKRaD7VUqVKUK1eOx48fq9q1s7NjzZo1nD9/HgcHBwwNDbl8+bLWdv/t6dOnhISEsHjxYsaNG0enTp1U07e+nHBfvigTHx/PnTt3qFq1KlWrViU1NZXU1FRVXFZWVsyfP5/o6GidYhDFl/QghYq5uTl9+vRh9uzZGBgYULlyZVauXMn//vc/vvjiC9555x2aN2+Ov78/s2bN4tmzZwQFBWnc3oABAwgKCsLW1pZ3332Xr7/+mvLly1O1alVu3rxJQkICkZGR2NnZMXjwYAIDA7GxsaFWrVp8//337Nu3j48++ggLCwu6d+/O/PnzKVWqFIaGhsyfP1+nfbKwsMDc3JwjR45QoUIFoqOjmTdvHoDaDfNbt27F2dkZZ2dnlixZQpUqVWjevDl6enp4eXkxceJEpk2bRqlSpViwYAG3b9/G3t5edRFJlEySIIWaiRMnoqenx6RJk0hKSqJWrVqsX7+ed955B4CvvvqKgIAAfH19KV26NAMGDNCYrAYNGkRCQgJTpkwhKSkJDw8PVqxYgUKhoHHjxjg6OtK1a1e2bt1K//79SU5OZuHChTx58gRHR0dWrlyJs7Mz8OJK8oIFC/Dz88PExISRI0cSEBCgdX8MDQ1ZsmQJCxYs4LvvvsPW1pYePXpgbGzM1atXVecWP/nkE9atW8etW7do0KABy5cvV80vvnDhQubPn8/w4cPJyMjA3d2dDRs2YGxsnA9HXBRl8sBc8f/asWMBAAAAAGH+1hFksEF0BAwPEmAIJMAQSIAhkABDIAGGQAIMgQQYAgkwAjoP9SpCpdTIAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "words_list_prediction_probabilities = words_list_model.predict_proba(X_train)[:, 1]\n", "\n", "words_list_predictions = [1 if pred >= .70 else 0 for pred in words_list_prediction_probabilities]\n", "\n", "high_classification_threshold = confusion_matrix(y_train, words_list_predictions, labels=[1, 0])\n", "\n", "plot_confusion_matrix(high_classification_threshold, classes=class_names,\n", " title='words_list_model Confusion Matrix $C = .70$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By raising the bar for classifying an email as spam, 13 spam emails that were correctly classified with $C = .50$ are now mislabeled. \n", "\n", "$$ \\text{Sensitivity } (C = .70) = \\frac{5}{42} \\approx .119 \\\\\n", "\\text{Specificity } (C = .70) = \\frac{757}{758} \\approx .999\n", "$$\n", "\n", "Compared to the default, a higher threshold of $C = .70$ increases specificity but decreases sensitivity.\n", "\n", "Lowering the threshold to $C = .30$, meaning we label an email $x$ as spam if the probability $f_{\\hat{\\theta}}(x)$ is greater than .30, results in the following confusion matrix:" ] }, { "cell_type": "code", "execution_count": 89, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEeCAYAAADow4zZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8TNf/+PHXJBFJJBLbh1BCJRJbFksQa0JtjVBqX1oUIShVW5TEFhSlkth3sdUWLWqJfWtUtZQWRayxJZqQZWSb3x/5Zb5Sxkxisno/Pe7jYe45c+/73pm855y7nKtQqVQqhBBCvMYgrwMQQoj8ShKkEEJoIAlSCCE0kAQphBAaSIIUQggNJEEKIYQGkiCFEEIDSZBCCKGBJMh3JNfZi9wk37fcVeASpIeHB/PmzdPb8uzt7dm8efNr/9fFoUOHCAgI0Fss+pSd/aTr9v/000/06dOH+vXr4+rqSt++fTl+/Hh2Q9Xo9u3bdO7cmdq1azN8+HC9LDOrn3F2eHh4YG9vT2Bg4BvL9+zZg729Pd26dcvScnX9vulrG3/66ScGDBhAo0aNqFWrFu7u7nzzzTdERES887Kz6uTJk/Tt2zdTLJMmTeLRo0ev1b137x7e3t64uLjg6urKhAkTiImJydZ6jd418MJk69atVKxYUef669evp3Tp0jkYUf6iUqkYP348Bw8epHfv3gwePJjU1FT27NnD4MGDmTp1Kj169NDb+tasWcODBw8IDg6mfPnyellmVj/j7FIoFISFhTFixIjXyg4ePJitZer6fXvXbXz58iVffvklp06donPnzvTq1Qtzc3OuXbvGhg0buHHjBlu2bMn28rMjNjaW2rVr07dvX6ysrLhz5w7BwcGcPXuWvXv3YmpqCkBcXByfffYZVlZWLFiwgISEBObNm8fQoUPZtGkTCoUiS+uVBPkKZ2fnvA4hX9u6dSs//fQTa9asoWHDhur57u7uGBsbExAQQOvWrSlZsqRe1vfixQvs7Oxo1qyZXpYHufcZOzs78/vvv3P//n0++OAD9XylUsnJkyepVq1ajq77XYwbN46zZ8+ycuXKTJ9zw4YN6dGjBydPnnzXELPM09MTT09P9WtXV1esra0ZOHAg4eHhtGjRAoAffviBJ0+esHnzZsqWLQtAuXLl6NmzJ8eOHcPd3T1L681WF9vLyws/Pz/166tXr2Jvb8+yZcvU8w4fPkzt2rVJSEggKSmJoKAgWrdujaOjI5988glHjx5V17W3t2f58uW0adMGFxcXwsPDgfRfgwkTJlCvXj2aNGnCzp07M8Xx+++/06NHD1xcXGjYsCHjx4/n33//zc4mqeN4tWvytuX37duXc+fOsW/fPuzt7bO0jtDQULy9vXFycsLDw4N9+/Zx/fp1unfvjpOTE927d+fmzZvq92jbf9r2E6S3/pYvX46Hhwe1a9emc+fO6v2sq3Xr1tGqVatMfzQZhg8fTo8ePYiPj9cp5ox9sXv3boYPH46zszONGzcmKCgISO+m7t27l19//RV7e3t1rG/qPnbr1o0JEyYA2r8T/32/Lt9NTTG+TZ06dShVqhRhYWGZ5p88eZISJUpQs2bNTPOfP3/O1KlTadasGbVq1aJJkyYEBASQnJwMvPn7punvJmMbM+ru27dPvZ6lS5dSq1Ytrl279sa4w8LC2L9/P1999dUbP+eiRYvSqlUrrdufG6ysrAAwNDRUzzt69Cj169dXJ0dI/ywqVKjw2vdPF9lKkE2aNOHXX39Vvz537hwAFy5cUM87c+YMdevWxczMjK+//prVq1fTp08fgoKCsLW1ZejQoZkCXrJkCd7e3kydOhVHR0cARo0axalTp5g8eTKTJk0iODiYx48fA+mtC29vbypUqMDixYuZPHkyp0+fZurUqdnZpNdoW76fnx81atTAzc2NrVu3ZmnZ06dPp0aNGixdupRKlSoxadIkRo4cSadOnVi4cCEPHjxg+vTp6vra9t/b9lOGBQsWEBgYSPfu3QkODqZKlSp88cUXXLlyRaeYHz9+zK1bt2jatOkbyytUqICvr6+6a6fLZw4wY8YMKlWqxJIlS2jfvj2BgYEcP36coKAg3NzcqFGjBlu3bn0tobxJdr4TusSpKca3MTAwwMPDg8OHD2eaf/DgQdq0afNa/a+++oozZ84wceJEVqxYQefOnVm3bh27du0CNH/f3vR3k6F9+/Y0a9aMOXPmkJCQQEREBIsXL2bo0KEaf9RXrlyJlZUVPXv2fOv2/ZdKpSIlJUXr9K4nmVJTU0lKSuL69evMmTOHatWq0aBBA3X5zZs3sbW1fe19tra23LhxI8vry1YXu2nTpqxatYro6GhKlSrF+fPncXBw4Pfff0elUqFQKDh9+jRdu3bl6tWrHDhwgLlz5+Ll5QVAs2bNePLkCQsXLlQ3eT08PPjkk0/U67h69SonT55k2bJl6uZzxYoV6dKlC5C+I2JiYujTpw8uLi4AmJubc+vWrexs0mu0Ld/W1hZzc3OsrKyy3KVp3LgxI0eOBNJ//fr27YuXl5f6S9m7d29WrlwJoHX/WVtbv3U/AcTExLBmzRpGjBjB4MGD1cvo3bs3wcHBLF68WGvMGQnX2tpaa11dP3NI/7EdN24ckN6F279/PydOnGDy5MlYWVmRkpKi8/7N6ndC1zg1xdi8efO3xvPRRx8xdOhQ/v33X0qUKEFycjJHjx5lxYoVmZKcUqkkNTWVqVOnqlttjRo14tixY/z2229069ZN4/ftv383/+Xv74+npyfLly/nt99+48MPP1R/B/7r2bNn/PHHH/Tq1QtjY+O3btt/7dq1i4kTJ2qtN2vWLDp37pylZb/q448/Vp8kql27NqtXr84U6/PnzylevPhr7ytevDgPHjzI8vqylSDr1q2Lqakpv/76K23btuX8+fOMGzeO8ePHc/PmTczNzYmIiKBp06b8+uuvKBQK2rZtm2kZ7du3Z8qUKcTFxQFQpUqVTOUXLlzAwMCAJk2aqOfVqlVL/Qdqa2uLlZUVQ4cOxdPTEw8PDxo3bqz1S6urnFz+q7/0pUqVAsjUQrKyslLvl99+++2t++/XX399634CuHjxIklJSTRt2pSUlBT1/CZNmqgTsTYZ3Zi0tDStdbXFHBcXh7m5OQBOTk7qcoVCQdmyZUlISNAppv/K6memS5zvEmOjRo0wNTXl6NGjdO7cmbNnz2Jqaoqzs3OmBGliYsKaNWtQqVTcvXuXiIgIrl27RnR0tLqLrcl//27+q0KFCowcOZJvv/0WAwMDtm/fTpEiRd5Y99q1a6hUqmwdH3V3d2f79u1a6716PDY7AgMDiY+PJyIigmXLljFgwAA2b96s/j7pW7YSpLGxMfXr1+fcuXPY2dnx/Plz2rRpw/fff8+FCxcwNDSkXLlyVKtWjbCwMCwsLF77RcpIDBnHrDJeZ3j+/DkWFhYYGRm98X3m5uZs2LCBoKAgtm/fzoYNGyhVqhRjxozJ1HrKrpxcfrFixV6bZ2Ji8sa6sbGxWvff2/YToL7EoVOnTm9cR2JiovosoCYZCffhw4ca6zx69Ihy5crpFHPGF7po0aKZ6hgYGGS7G5bVz0yXON8lRmNjY5o3b05YWBidO3fm4MGDfPTRR288kxoWFsbMmTOJjIykdOnSODs7U7RoUa3r+e/fzZu0a9eOb7/9Fmtr6zd2PzM8f/4cIFtXZlhZWWFhYaG13qvHC7PDzs4OSD8R1bBhQzw8PNi6dSsDBw4E0luKGdvxqufPn2NpaZnl9WX7LHaTJk3Ytm0b9vb21KhRA1NTU+rWrcuFCxdITk5Wt2gsLS158eIFSUlJmb6IUVFR6vI3sbS05Pnz56+9LzY2Vv3/atWqsWjRIpKSkjh79iyrVq3im2++oXHjxpQrVy67m5Zry9eFtv2X8YV4237K6HKsWrXqjftbl+5UyZIlcXBw4MyZM2+8lOfBgwd4eHgwZcqUbH/muvpvK/bV1lxWPrOcjhPSu9kTJkwgPj6eI0eO8N13371W5/bt24waNYpevXoxaNAgypQpA0DXrl3fef0AAQEBVKhQgaioKJYvX46Pj88b62Uk2/8ev9ZFbnWxX2VtbU3JkiW5c+eOel7VqlUzneDMcOPGDRo3bpzldWT7QvEmTZpw48YNDh8+TN26dQGoX78+v/76K2fPnlUfzK9Tpw4qlYr9+/dnev/PP/9M9erVNbacXF1dUalUmc4C3rx5U30c4ezZszRq1Ihnz56pf6nHjBlDWloaT548ye5mqemyfAODnL/OXtv+a9CgwVv3E6R36Y2MjNTXkmVMp0+fZvPmzTr/qvfp04dDhw5lOkGXITAwkCJFitCqVatsf+a6MDMzy3Rx8LNnz7h9+zaQ9e9ETsaZoVmzZqhUKhYvXkxaWhr169d/rc5ff/1FcnIyQ4YMUSfHqKgorl+/nunHIDvftyNHjnDgwAH8/Pzw9vZm6dKlGi/0dnR0pESJEuzYseONh1IeP37M3bt33/jejC62timrl9m8TUREBFFRUVSqVEk9r0WLFpw7dy5Tkv/jjz948OBBttad7RZk1apVKVeuHMePH6d79+4A1KtXjylTpmBoaIibmxsA1atXp1WrVvj7+xMTE0OVKlXYs2cP4eHhb71comrVqrRv3x5/f38SEhKwtLRk4cKF6uMntWrVQqFQMHLkSAYNGgTA8uXL+eCDD6hevXp2N0tNl+UXL16ca9euER4ejqura5YvQtWFtv2nbT9BesugR48e+Pv7ExUVhb29PefOnWPJkiV8+eWXOsfSpUsXDh8+zKBBg+jXrx8NGjQgPj6eXbt2ceTIEWbMmEHZsmUpW7Zstj5zXTRt2pQffvgBe3t7ihUrxuLFi9WHLLL6ncjudzMrihUrhpubG+vWraNTp05v/DFycHDA0NCQ2bNn06VLF548ecLSpUt5+fIliYmJ6nr//b5pExcXx9SpU2ndujVNmzalQYMG7Ny5kylTprBhw4bX6hsbG/PNN98wduxYevXqRc+ePSlfvjxRUVGcPXuWH3/8kd27d79xXSVKlKBEiRJZ2DOZhYaG4uvry9q1a9+4bT4+PtSsWRN7e3vMzMy4ceMGq1evxtramk8//VRdr3v37oSEhDBs2DBGjBhBYmIi8+bNw8XFJXcTJKS3Irdv306dOnWA9KRWsmRJKleunOlM0vz581m4cCHLly/n+fPnVKtWjSVLlmgNePbs2cyePZu5c+cCMGDAAPVlExYWFqxYsYK5c+cyZswYUlNTqV+/PqtWrdJ4EDordFn+Z599xqhRoxg8eDAHDhzIsW63tv33tv2UwdfXlxIlSrBu3TqePn1K+fLlGT9+PJ9//rnOcRgYGBAUFERISAihoaFs3LgRIyMj7O3tWb16daYuTHY/c20mTZqEn58fkyZNwtLSkoEDB/Lnn38C2ftO5FScr/roo484evQoH3300RvLP/zwQwICAggODubAgQOULVuWtm3b0rp1a7Zu3UpqaiqGhoavfd+0WbBgAbGxsequr7GxMZMmTWLw4MFs27btjV14T09PypQpw/Lly5kxYwZKpZIyZcrg4ODAlClTMrXW9CktLY3U1FSNx1ydnJz4+eefWbVqFampqVhbW9OmTRsGDRqkvh4S0o9Dr1u3jpkzZzJ69GiMjIxwd3dn4sSJ2WrAKOSphkII8WaF8lZDlUpFamrqW+soFIp3PqOWV+sTQuSOQpkgz507R79+/d5ap0KFChw5cqRArk8IkTsKZRc7Li5O65BMxsbGWbqHOj+tTwiROwplghRCCH0olF3sd5GQJL8XecnAQP+XSomsMdFzVjB10T7YceLv+rmsSt8kQQohcpaiwD24QE0SpBAiZxkU3Ks3JEEKIXJWDtxhllskQQohcpZ0sYUQQoMC3MUuuKldCFEwKBTaJx39+OOPuLi4ZJocHByYPHkysbGx+Pj4ULduXVq0aMG2bdvU70tKSsLX1xdXV1fc3NxYsmSJTuuTFqQQImfpsYvt5eWlfjwGpD/7aty4cfj4+DB58mTMzMw4c+YM165dY9CgQdjZ2eHs7MyCBQuIjIzk8OHDREdHM2DAAGxsbGjfvv1b1yctSCFEzjIw1D5lQ3x8PBMmTMDf3x8LCwvCwsIYOXIkRYsWxdHREU9PT0JDQwHYvXs3Q4YMwcLCgsqVK9OnTx/1A9HeGnq2IhNCCF3psYv9qpUrV1KtWjVatWrFnTt3MDIyUj9VE9Kf13Pr1i1iY2OJjo7O9LiJjDJtpIsthMhZOXAWOz4+npCQEFasWAGkP3bjvyPAm5iYoFQq1YMOv/rcpYwybSRBCiFyVg4M8xcWFkb58uXVj8A1NTXl5cuXmeoolUrMzMzUiVOpVKofFpdRpo10sYUQOUthoH3KoqNHj9KuXTv1axsbG5KTk4mMjFTPi4iIUD8KuFSpUplG3IqIiKBq1apa1yMJUgiRs3LgGOTFixfVrUdIf9RCy5YtmT9/PomJiVy6dIk9e/bQoUMHIP3sd2BgIDExMdy+fZuQkBA6duyodT2SIIUQOUvPZ7FTU1N5+PCh+gmQGaZPn05KSgrNmzdn5MiRjB07FicnJwBGjRpF5cqVadeuHb169aJbt26ZWqCayHiQ/yHDneUtGe4s7+l9uLPWc7XWSTw4Vr8r1RM5SSOEyFkyWIUQQmhQgO/FlgQphMhZMpqPEEJoIF1sIYTQwKDgppmCG7kQomCQFqQQQmggxyCFEEIDaUEKIYQGcpmPEEK8mUJakEII8WaSIIUQQgNFAb6/XhKkECJHSQtSCCE0kAQphBAaSBdbCCE0kBakEEJoUJATZMG9B0gIUSAYGBhonbLi0aNHDBkyhDp16tCsWTPWr18PQGxsLD4+PtStW5cWLVqwbds29XuSkpLw9fXF1dUVNzc3lixZotO6pAUphMhZemxAqlQqhg0bRoMGDQgKCuL27dv07t2bWrVqsXbtWszMzDhz5gzXrl1j0KBB2NnZ4ezszIIFC4iMjOTw4cNER0czYMAAbGxsaN++/VvXJy1IIUSOUigUWiddXbx4kSdPnvD1119TpEgR7Ozs2LJlC2XLliUsLIyRI0dStGhRHB0d8fT0JDQ0FIDdu3czZMgQLCwsqFy5Mn369GHXrl1a1ycJUgiRo/TZxb5y5Qp2dnbMnTuXxo0b06ZNGy5evEhsbCxGRkZUrFhRXbdKlSrcunWL2NhYoqOjsbW1fa1Ma+xZ21SRHyQlJTE7YDqONapRpoQ5Deu7sGP7D2+se/bMacxNDDlx/FjuBvke+XF3KJU/KJdpXkxMDEMHf0HlD8pR0boM/fv14enTp3kUYR5T6DDpKDY2lvDwcEqUKMHRo0eZNWsW06dPJyEhARMTk0x1TUxMUCqVJCYmAmBqavpamTaSIAugKd9MZMH8uQwaMpSt20Np0rQZ/Xr3YM+PuzPVS0pKYvjQwciTfXPOufBwBg347LX5X/Tvx8/79jDr2/kELVnOuXO/0KNr5/fys9BnF9vY2BhLS0uGDBmCsbExderUoU2bNixatIiXL19mqqtUKjEzM1MnzlcTYkaZNnKSpoBJSkpi+dLFTJ0ewIgvRwPg0bIVt27eJDjoezy9Oqrrzp0TwPMXz/Mq1EItJSWFJcFBTPlmYqaWCaS3Hvfu+YllK1bTs1dvAMzNzfFs15prV6/iUL16XoScZ7J6lvptqlSpQmpqKqmpqRgapg+jlpqaSo0aNTh//jyRkZGUL18egIiICGxtbbGysqJUqVJERERQunRpdVnVqlW1x663yEWuiI2N5bPPB9C23ceZ5ttVq8ad27fVr//++y++XzCf2d/Oz+UI3w9nTp9ixjQ/ps2YxVCfEZnKMloyFsWLq+eVKFESgGfPnuVekPmEPluQjRs3xsTEhKCgIFJSUrhw4QKHDh2ibdu2tGzZkvnz55OYmMilS5fYs2cPHTp0AMDLy4vAwEBiYmK4ffs2ISEhdOzYUcvaQKF6H9v8b5GQVPB2R1paGvVdalOxkg2hP+1DpVLRyr0pbo2b0Kfv59RxqsHPB4/QrHmLvA5VK4MCclvakydPMDQ0pFSpUsyY5s/K5Uu5ff+Rurzbp59w/+5d1m/cgqmZGd6DBnDjxj9cvHwVY2PjPIxcOxM99yvLe+/UWidyaWedl3fnzh2mTZvGn3/+ibm5OT4+PnTp0oWYmBj8/Pw4e/YsZmZmDB8+nE8//RRI71IHBARw6NAhFAoF/fr1w9vbW+u6pItdCMyZNYOrV//m2/kLAVixbAmPHz3C9xs/7t65k8fRFU7/+9//3lo+59v5tG/bito1qgFQokQJDh05ke+TY07QZxcbwMbGhlWrVr0238rKiu+///6N7zExMWHatGlMmzYtS+uSLnYBtyQ4kBnT/Bk9ZiwtW31E5IMH+E32ZcGi4NeOjYnc8fDhQzyaN6ZkiZJs27mbXT/upbajE5282vPgwYO8Di/X6bOLndtyNUFGREQwdOhQ6tevj4uLC15eXpluBxJZM3O6P19/9SWDhwxl+szZAIwa6UPrNu1w92hJSkoKqWmpAOoD2yLnrVuzihcvXrB77348O3jRtl17dv24l7S0NBZ+Ny+vw8t9erzMJ7flWhc7LS2NL774gs6dO7NgwQKMjY05f/48w4cPp3jx4rRp0ya3QikUvvpyOMuWLmb0mLHMCJijnr93z48AbN+2NVN9z3Yf0bRZc/YfOpqrcb6P7t+7x4cfVlWfMQUwMzPDycmZa1f/zsPI8oa+u9i5KdcS5L///sv9+/fx8vJSX5fk6urK2LFjSU5OJjAwkFu3bhEdHc2lS5ews7Nj2rRpVP//l0SsX7+ebdu2ERkZSdGiRenZsycjRqSfPbS3t2fatGksXryYFy9eMHDgQKytrVm4cCFKpZIhQ4YwcODA3NrUHDd3TgDLli5miv80xk/8JlPZyTPnMr2+e/cOvXt0ZVHQEpo2a5GLUb6/qtrZsWnjBp4+fUqZMmWA9JMEf/55ibZt337vb2GUn7vQ2uRagixVqhSurq70798fLy8vXF1dcXR0pGvXrgAEBgby888/891337Fq1SqWLVvGsGHDOHDgAJcuXWLp0qVs2rSJypUrc/78efr06YOXlxc2NjYAnD59mv3793PhwgUGDhxIu3btOHjwIOHh4QwdOpRu3bphYWGRW5ubY+7dvcvM6VNp2MgNd49WnAv/RV1WxNiYOnXrZapfrJg5AHbV7Klmb5+rsb6vPvt8AEGLFtLJsx3jJk7C2NiY4MDveRYdzfCRo/I6vFwnCVJHK1euZPPmzRw6dIjly5cD0Lp1ayZPngxAo0aN1KNrDB06lA0bNnDhwgWcnJzYuXMn5cqVIyoqiuTkZExMTHjy5Ik6Qfbp0wdTU1MaNmyISqWid+/emJiY0LRpU1JTU3n8+HGhSJAH9u8jOTmZX86ewb2ZW6ay/5UtS8Tdh3kUmchQsmRJwo6cYNzYrxg04DOKFClCfdcGnDgdjr2DQ16Hl+tkRHEdFS1alM8//5zPP/+cly9f8ttvvzFv3jx8fX2pUaMGlSpVUtc1NDSkTJkyREVFYWBgwOLFizlw4AClSpWiVq1aQPpxzQyWlpbq9wEU//8X6WYc/3i1bkH2xWBvvhis/fqtDPYODsS/LBzbnl99M8Wfb6b4Z5pX5cMP2bYjNG8CymcKcgsy146e7tu3j5YtW6rvRS1atChubm6MGDGCq1evAukX32ZISUnhyZMnlCtXjjVr1nD9+nXCwsLYt28fM2fOJCUlJdPyC/KHIERhplBon/KrXEuQjRo1Ij4+npkzZxIdHY1KpeLOnTts2LABd3d3AE6ePMmZM2dITk4mODiYEiVK4OLiQlxcHEWKFKFIkSLEx8czZ84ckpOTX0uSQoj8x8BAoXXKr3ItQZYoUYJNmzbx5MkTPD09cXZ2pn///tSuXZsJEyYA4OjoyIoVK2jQoAHnz59n2bJlGBoa0r9/f4yMjGjUqBFt2rQhKSmJOnXqcPPmzdwKXwiRTQX5QvF8cy92YGAg//zzD4sWLcrTOArivdiFSX5uTbwv9H0vtsOEA1rrXJ2dP6+DlnuxhRA5ytCw4P7oSYIUQuSo/NyF1ibfJMiMu2KEEIVLAc6P+SdBCiEKJ7kXWwghNJAWpBBCaFCQj0EW3LavEKJA0PeF4qtWraJWrVq4uLiop/PnzxMbG4uPjw9169alRYsWmcaaTUpKwtfXF1dXV9zc3FiyZIlO65IWpBAiR+m7AfnXX38xevTo14YwHDlyJGZmZpw5c4Zr164xaNAg7OzscHZ2ZsGCBURGRnL48GGio6MZMGAANjY26sFxNJEWpBAiR+n7Tpq///5bPU5shvj4eMLCwhg5ciRFixbF0dERT09PQkPTBwzZvXs3Q4YMwcLCgsqVK9OnTx927dqldV2SIIUQOUqfXezExEQiIiJYv349jRs3pl27dmzfvp07d+5gZGRExYoV1XWrVKnCrVu3iI2NJTo6Gltb29fKtJEuthAiR+mzix0VFUXdunXp2bMnixYt4tKlS3h7e9O/f3/1kwoymJiYoFQqSUxMBMj0ELuMMm0kQQohcpQ+z2JXrFiRkJAQ9et69erRsWNHzp8/z8uXLzPVVSqVmJmZqROnUqnE3Nw8U5k2GhNkRESEzkFXqVJF57pCiPeLPgcguXLlCqdPn2bw4MHqeS9fvsTa2prk5GQiIyMpX748kJ7DbG1tsbKyolSpUkRERKgfpBYREUHVqlW1rk9jgmzXrh0KhQJNg/1klCkUCv7++/17UpsQQjf67GKbmZkRFBREpUqVaN26NeHh4ezdu5eQkBBevHjB/PnzmTFjBv/88w979uxRP9rFy8uLwMBAFi1aRExMDCEhIYwdO1Z77JqGO8vKA84rVKigc938ToY7y1sy3Fne0/dwZ03nn9Ja5+SYJjov78iRIyxYsIB79+5RtmxZRo8eTdu2bYmJicHPz4+zZ89iZmbG8OHD+fTTT4H0LnVAQACHDh1CoVDQr18/vL21P7okS+NBXrhwgYiICNq0acPDhw+xsbHB2NhY5w0rCCRB5i1JkHlP3wmy+YLTWuscH91YvyvVE512xbNnz/D29uavv/4iLS0NV1dX5s9OvoiiAAAgAElEQVSfz82bN1m9enWmU+tCCPGqQn+r4cyZMyldujTh4eEULVoUgDlz5lCpUiVmzpyZowEKIQq2Qv/QrjNnzjBq1CiKFSumnmdpacmECRM4f/58jgUnhCj4CvJDu3TqYqempr7xudIvXrxQP4daCCHexCA/NxG10KkF2apVK+bOncuzZ8/UxxNu3LjB9OnTadmyZY4GKIQo2Ap9F9vX1xdzc3MaN25MQkICHTp0oEOHDlhbW+Pr65vTMQohCjBDA4XWKb/SqYttbm7O999/z71797h58yYpKSlUrVpV7qARQmhVkM9i63zFU2pqKtevX+fmzZsYGxtjYmIiCVIIoVUBzo+6Jchr164xdOhQYmJiqFy5Mmlpady5c4fKlSsTFBRUqO6kEULol2EBzpA6HYP08/PDycmJEydOsHPnTkJDQzl27Bhly5ZlypQpOR2jEKIA0/eAublJpwT5119/MWLECPVQQZB+HeSYMWPkOkghxFsV+rPYDg4OXLx48bX5165dk+OQQoi3KpRnsbdu3ar+v4uLC/7+/ly5cgVHR0cMDQ25evUqISEhrz04RwghXpWfu9DaaBzNx8PDQ7cFKBQcPnxYr0HlJRnNJ2/l59vO3hf6Hs2n69oLWuts+7yOfleqJxp3xZEjR3IzDiFEIVWQz2Lr/Fvx+PFjbt26RWpqKgAqlYqkpCSuXLnCyJEjcyxAIUTBVpC72DolyI0bNxIQEEBqamqmxzAoFAqcnJwkQQohNMqpoyZRUVF06NCBgIAA3N3duX//PpMmTeLSpUv873//Y8KECbi7uwMQGxuLr68vv/zyCxYWFvj4+NC1a1ftsesSyKpVqxg6dCh//vknpUqV4tixY+zZswcHBwc++uijd9tKIUShllPDnU2aNImYmBj16y+//BJHR0fOnTuHr68vY8aMITIyEoDJkydjZmbGmTNnWLRoEfPmzeOPP/7QHrsugTx58oSOHTtSpEgRqlevzh9//IGtrS0TJ05k27Zt2do4IcT7IScuFN+8eTOmpqZYW1sDcPPmTa5fv46Pjw9FihShefPmuLq6snfvXuLj4wkLC2PkyJEULVoUR0dHPD09CQ0N1boenRKklZUVL168ANIf8Xrt2jUg/WFdjx49yvLGCSHeHwYK7VNWREREsGbNGvz9/dXzbt26RYUKFdTPwIb0XHXr1i3u3LmDkZFRpkfDZJRpjV2XgNzd3ZkyZQpXr16lYcOG7N69mwsXLrBhwwZ1BhdCiDcxUCi0TrpKSUlh3LhxTJo0CSsrK/X8hIQETE1NM9U1MTFBqVSSkJCQKXG+WqY1dl2CmjBhAg4ODly9ehUPDw/q169Pr1692LZtGxMmTNBlEUKI95Q+E+TixYupXr06zZs3zzTf1NT0tYSnVCoxMzPD1NSUly9fvrFMG53OYhcrVowZM2aoX8+ZM4eJEydibm6OkZGeryoVQhQq+rzKZ9++fTx9+pR9+/YBEBcXx1dffYW3tzcPHjwgKSlJ/SjqiIgIGjRogI2NDcnJyURGRlK+fHl1ma2trdb16XSroTbdu3fXua4Q4v2iz7uj9u/fn+m1h4cHkydPxt3dnQMHDrBw4UJGjRrF2bNnCQ8Px8/PD3Nzc1q2bMn8+fOZMWMG//zzD3v27GH58uVa16cxQS5btkyngBUKhSRIIYRGufXQrsDAQKZMmUKjRo0oXbo03333nfocyfTp0/Hz86N58+aYmZkxduxYnJyctC5T473Y7ytlSl5H8H4rUX94Xofw3kv8PUivyxu28y+tdRZ3rqHXdeqLHEAUQuSo9+JebCGEyI5Cfy+2EEJkV0EewU4SpBAiR+XnEcO10elCcYDo6GiWLl3K+PHjiY6OZt++fVy/fj0nYxNCFAL6vtUwN+n80K42bdpw7Ngx9u7dS0JCAmfOnKFr166cPXs2p2MUQhRghf6hXbNmzeKzzz5jy5YtFClSBIAZM2bQt29f5s2bl6MBCiEKNiOFQuuUX+mUIK9cuYKXl9dr87t3787Nmzf1HpQQovAo9C1IS0tL9cCTr7py5QolS5bUe1BCiMJDn4NV5DadzmL37NmTKVOm8PXXXwPpz8M+ceIEgYGB9O/fP0cDFEIUbIY6nwrOf3RKkIMHD6ZYsWLMnj2bxMREhg8fTunSpfH29uazzz7L6RiFEAVYfm4haqPzdZC9e/emd+/eJCQkkJqaioWFRU7GJYQoJApwftQtQWp7dkOnTp30EowQovAp9Pdi//dSnpSUFJ4/f46xsTEODg6SIIUQGuXnC8G10SlBnjp16rV5sbGxTJ48mTp16ug9KCFE4VGQE2S2zy9ZWloyatQoVq5cqc94hBCFjKGBQuuUX73TYBX3798nMTFRX7EIIQqhAnwIUrcEOWbMmNfmxcXFce7cOTw9PfUelBCi8ND3ZT779u0jMDCQR48eUb58eUaPHk2rVq2IjY3F19eXX375BQsLC3x8fOjatSsASUlJ+Pv7ExYWhpGREX379mXo0KFa16VTgsx4StirypYti6+vLx07dszi5gkh3if6vFA8IiICX19fVq9eTZ06dThz5gyDBw/mxIkT+Pv7Y2ZmxpkzZ7h27RqDBg3Czs4OZ2dnFixYQGRkJIcPHyY6OpoBAwZgY2ND+/bt37o+nRKkvb097dq1o2zZsnrZSCHE+8MA/bUgq1SpwunTpylWrBgpKSlERUVRrFgxjI2NCQsL48CBAxQtWhRHR0c8PT0JDQ3F2dmZ3bt3M3/+fCwsLLCwsKBPnz7s2rVLa4LUKbcHBweTlJSklw0UQrxf9D1YRbFixbh37x6Ojo6MGzeO0aNHc/fuXYyMjKhYsaK6XpUqVbh16xaxsbFER0dneg52Rpk2OiXIZs2aERISwvPnz7O2JUKI956RgULrlFXW1tZcvHiRNWvWMGfOHI4cOYKJiUmmOiYmJiiVSvWJZFNT09fKtMauSzB37txh7969rF+/HnNzc4oWLZqp/E3XSQohBOTMWWwjo/TU1ahRI1q3bs3ly5d5+fJlpjpKpRIzMzN14lQqlZibm2cq07oeXYLp3bt3loIXQogM+jyLffz4cdasWcPatWvV85KTk6lUqRInTpwgMjKS8uXLA+kndGxtbbGysqJUqVJERERQunRpdVnVqlW1rk9jggwKCmLgwIGYmpryySefvONmCSHeV4Z6bEHWqFGDy5cvExoaipeXFydPnuT48eP88MMPPHz4kPnz5zNjxgz++ecf9uzZw/LlywHw8vIiMDCQRYsWERMTQ0hICGPHjtW6PoVKpVK9qaB69eqcOnWKUqVK6W/rCgBlSl5H8H4rUX94Xofw3kv8PUivy1t3/p7WOp/Vq6i1Tobz588TEBDA7du3qVy5MuPGjaNhw4bExMTg5+fH2bNnMTMzY/jw4Xz66adAepc6ICCAQ4cOoVAo6NevH97e3lrXpTFBOjg4cPr0aUmQIldJgsx7+k6Q63VIkP2ykCBz01uPQSYnJ+t0ec+bLiQXQggoxAPmuru767SQv//+Wy/BCCEKn3w8FoVWb02QixYtwtLSMrdiEUIUQorC2IJUKBTUqVPnvTsGKYTQrwL8zC7NCVLDuRshhMiSQnkM8pNPPnntjhkhhMiqQtnFnjVrVm7GIYQopAplF1sIIfShUHaxhRBCHwpwfpQEKYTIWfocMDe3FeTDA++tpKQkZs2cTq3qdpSyLEaDus5s3/aDujwhIYGRPkOpaF2GcqWtGPLFAGJjY/Mw4oKraV07En8P0jgBlC9jycZvB/Dg2BzuHpnFd+O7YmaS+e6yBo5VOLJmNE9OzeNS6BS8uzfLi83JEwYKhdYpv5IWZAE0edJEVq9czmS/adSqXZu9e36ib6/uFC1alA5eHfHxHszRI2HM/e57VCoVE8eN4dm/z9i2IzSvQy9w/rh6j+b95mWaV8LSjM1zv2Dr/vMUMTJkZ2D6w58G+4VQzNSYWaM/4X8lLegzfjUAlaxL8NNiH05duEHPr1fiUqMS88Z+ysukFNbsOpPr25Tb8nH+00oSZAGTlJTEsiXBTJsxi5GjRgPg0bIVt27eIGjRQmrUqMnWLZvYEfoT7dp/DECFChVo08qdv65coUbNmnkZfoHzIl7JuT9vZ5q3btbnPHgSw1dztuHR0B4n+w+o6eXPrXtRABgZGbJiah9KFDfj3+cJdGjhhKGBAb3HrSJRmczhX65Ss6o1/T9xey8SpHSxRa6JjY3l8/4D1ckvg101e+7cuc3xY0cxNjam1Uet1WVNmzXH0tKSsEMHczvcQqeR04d0a1uP8fN3kqhM5tRvN2jeb546OQIkJ6diYGCAcRFDAEyKGpGckory5f8NFfXv8wRKWGof0bowMDDQPuVX+Tg08SZlypRhYWAwdtWqqeelpaVx6OB+7O0d+Oef61SysaFIkSLqcoVCQaVKNty6eSMvQi5U/Id34ORv/7DvxGUA4hOT1C3MosZGNHapit8wT/aduMzj6BcAbDtwgTSVCr9hnlhZmNK0rh09P67P9gMX8mozcpVCh3/5lXSxC4HZATO4+vffzPvue3bt2I6FucVrdcwtLHjx4kUeRFd41LIrT7N6dnQeufSN5cfWjcHZoSJR/8YxJfBH9fy7D58xccEugr/pyfgv2gCw/9QVpi/dmytx57WCPJqPtCALuMVBgUyf6sdXX4+jZauPSEtL03hUvCDf8pUfDOjcmFv3nvLzyctvLB83bydePsH8dfMhh1aNosoH6c8/6evVkKV+vVmy5ThtBn3P8BmbqVfThsWTe+Vm+HmmIJ/FzhcJ0t7enuvXr782v0GDBoSHh+dBRAXDjGn+jBk9kiHew5gRMBuA4paWxMfHvVY37sULisvQde/Es3ltdob9rrH85G//cOjM33T5cikqlYr+n7gBMH5gG7Yf+I2x83Zw4vw/rNpxGu+pG/msUyNqV6uQW+HnGX13sc+fP0/Xrl2pW7curVq1YsuWLUD68XkfHx/q1q1LixYt2LZtm/o9SUlJ+Pr64urqipubG0uWLNFpXdLFLqBGjRzOsiXBfPX1OGbOmqOeb2trx727d0lNTcXQMP0kgUql4u7dO9jZVdO0OKGFw4flqGhdkt2HL2aaX9O2PA5VyrLj0P8lzriEl0Q8iKZc6eIAfFDOimU/nMj0vl8upj+03qFKOf68/iCHo89b+uxix8bGMmzYMCZPnszHH3/M33//Tf/+/alUqRJbtmzBzMyMM2fOcO3aNQYNGoSdnR3Ozs4sWLCAyMhIDh8+THR0NAMGDMDGxob27du/PXb9hZ6z/vrrLz7//HOaNGmCk5MTAwYMICoq/czhhAkT+Pbbb+nRowfOzs706dOHS5cu0aNHD1xcXOjfvz9xca+3qgqqb2cHsGxJMH5Tp2dKjgAt3D1ISEjgyOEw9byTJ44TGxtLs+YtcjnSwqNOjUq8TErmj2uZn6/i5vIhawM+p8L/rNTzypUujkOVsvx98yEAN+4+paFTlUzvq1fLBoA7kdE5HHne02cXOzIykubNm9OhQwcMDAyoWbMmDRo04MKFC4SFhTFy5EiKFi2Ko6Mjnp6ehIamX/u7e/duhgwZgoWFBZUrV6ZPnz7s2rVLe+zZ3mo969GjB/Xq1cs0vXr3x5dffknLli05efIkx44d48WLF4SEhKjLd+7cyYwZMzh9+jRRUVEMGzaMmTNncuzYMSIjI9m9e3debJbe3b17lxnT/GnYyA13j1aE//KLevr9wgVs7ezw6vQJAz7rw8YN69myeROf9emJp1dHataqldfhF1g1PrQm4n40KSlpmeZv/fk89x//y9YFg2nfrBaftHLmp8XDefrsBat2ngZg9or9dP6oDvPHfUqzenb0/8SNZf59OBp+7bVrLAsjhQ6TrqpXr87cuXPVr2NjYzl//jwARkZGVKz4fw//qlKlCrdu3SI2Npbo6GhsbW1fK9Mm33Sxt2zZQrVqmbuADRo0UP9/1apVfPDBByQmJvL48WNKlCjB48eP1eXu7u7qHVC7dm2MjY3VDwZ3cnLiwYPC0Y058PM+kpOT+eXsGVo0bZSprGzZsty+/4iVq9cxZtRIvho1AiMjIzw7dGTudwvzKOLCoXQJc2LjEl+b/zxOSZtBi5gzpjPLp/bFyNCAsLN/M37+Tp7HKQHYfjD9cp5xA9swsEtjHjyOYeNP4cxYti9XtyGv5NRJmBcvXuDt7a1uRa5fvz5TuYmJCUqlksTE9M/N1NT0tTJt8k2C1ObSpUsMGjSI+Ph47O3tiY2NpWTJkuryV5+dY2hoSPHixdWvDQwMCs0I6YOGeDNoyNuf52thYcHyVWtYvmpNLkVV+HlP3aix7O7DZ/T8euVb37/94AV1onzf5ER+vHfvHt7e3lSsWJGFCxdy8+ZNXr58mamOUqnEzMwMExMT9Wtzc/NMZdrkmy722zx69Ijx48fz7bffcurUKVatWoWdnV2mOnIJixD5k77PYl+5coVu3brRpEkTFi9ejImJCTY2NiQnJxMZGamuFxERga2tLVZWVpQqVYqIiIhMZRk9zLcpEAkyPj4elUqFiYkJKpWK48ePs3//fpKTk/M6NCGEFgqF9klXUVFRfPHFF/Tv35+JEydi8P/vUzQ3N6dly5bMnz+fxMRELl26xJ49e+jQoQMAXl5eBAYGEhMTw+3btwkJCaFjx45a11cguthVq1Zl2LBhfPbZZ6SlpfHhhx/So0cPfvnll7wOTQihhT47d9u3b+fZs2csWbIk07WM/fr1Y/r06fj5+dG8eXPMzMwYO3YsTk5OAIwaNYqAgADatWuHQqGgX79+tGvXTnvsqsJycE5PlCna64icU6L+8LwO4b2XMc6lvpyPeK61Tr0qxbXWyQsFogUphCi4CvLpAUmQQogcJQlSCCE0yM/DmWkjCVIIkaOkBSmEEBpIghRCCA2kiy2EEBpIC1IIITSQBCmEEBpIF1sIITSQFqQQQmggCVIIITSQLrYQQmggLUghhNBAEqQQQmggXWwhhNCgILcgC8QjF4QQBZc+H7nwqkuXLtGkSRP169jYWHx8fKhbty4tWrRg27Zt6rKkpCR8fX1xdXXFzc0t02jkbyMtSCFEjtJ3F1ulUrFjxw5mz56NoaGhev7kyZMxMzPjzJkzXLt2jUGDBmFnZ4ezszMLFiwgMjKSw4cPEx0dzYABA7CxsaF9+/ZvXZe0IIUQOUrfLcilS5eyfv16vL3/7/HH8fHxhIWFMXLkSIoWLYqjoyOenp6EhoYCsHv3boYMGYKFhQWVK1emT58+7Nq1S+u6JEEKIXKUvhNkly5d2L17N7Vr11bPu3PnDkZGRlSsWFE9r0qVKty6dYvY2Fiio6OxtbV9rUwb6WILIXKUvrvY//vf/16bl5CQgImJSaZ5JiYmKJVKEhMTATA1NX2tTBtpQQohclROnaR5lampKS9fvsw0T6lUYmZmpk6crybEjDJtJEEKIXKUgUL79K5sbGxITk4mMjJSPS8iIgJbW1usrKwoVaoUERERmcqqVq2qPfZ3D00IId5GocP0bszNzWnZsiXz588nMTGRS5cusWfPHjp06ACAl5cXgYGBxMTEcPv2bUJCQujYsaPW5UqCFELkqNzoYgNMnz6dlJQUmjdvzsiRIxk7dixOTk4AjBo1isqVK9OuXTt69epFt27daNeunfbYVSqVSj/hFQ7KlLyO4P1Wov7wvA7hvZf4e5Bel/cwNklrHWtLY72uU1/kLLYQIkfJvdhCCKFJwc2PkiCFEDlLH2ep84okSCFEjpIuthBCaFJw86MkSCFEzpIuthBCaCBdbCGE0KAgjyguCVIIkaMkQQohhAbSxRZCCA2kBSmEEBpIghRCCA2kiy2EEBpIC1IIITSQBCmEEBoU5C62DJgrhBAayCMXhBBCA0mQQgihgSRIIYTQQBKkEEJoIAlSCCE0kAQphBAaSIIUQggNJEEKIYQGkiCFEEIDSZBCCKGBJEghhNBAEuR7SG6/zz1paWnq/8t+L3gkQb4HUlNTAUhJSQFAoVDIH2suSElJwcDAQP1/UfDIaD6FnEqlQqFQcP36dTZs2IBCoWDMmDFYWlrmdWjvhbS0NIYMGYJSqaRx48a0a9cOGxubvA5L6EhakIVYRnJ8/PgxvXv3RqFQ8Pvvv/Ppp58SERGR1+EVWhktdoChQ4dSvHhx3Nzc+PXXX9m4cSO3b9/Ou+BElkgLspBKTU3F0NCQhIQEfv/9dy5evMiwYcMAGDx4MBEREaxcuVJaM3qWsd9VKhX3799n27ZtfPXVVwDs27ePH3/8kUqVKtGrVy8qV66ct8EKrSRBFmJ///03s2bNIioqijJlyrBs2TJMTEwAGDRoEHfv3mXx4sVUrVo1jyMtHDJa7GlpaXTr1o2oqCgePXrE9OnT6dq1KwB79+5l//79FC9enCFDhlCpUqU8jlq8jXSxC5mM7l1cXBxjxozBycmJ9u3bEx4ezsGDB9X1VqxYgYWFBYsWLcqrUAuVjOQI4O/vT40aNVi6dClt27bl3LlzhIWFAfDxxx/j7u6OUqnEzMwsL0MWOpAWZCF0584dgoODsba2ZvTo0QBs3ryZadOmMWvWLDp16qSum5aWpj7TKt7dmjVrOHLkCJMmTcLBwYGrV6+ydu1akpKS8PT0xMPDA0j/ATM3N8/jaIU2hv7+/v55HYR4d6+2YBITE/Hz8+PGjRt4enpSrFgxateuTcmSJZk6dSpWVlbUrl0bQN0lVBTkR8/lodTUVPUPTHJyMseOHeOXX34hKSmJ+vXrY21tTaVKlbhy5Qrnzp2jePHiVK5cGWNj4zyOXOhCEmQhkPFHGhcXx/PnzylbtixdunRhy5Yt3Lt3j6ZNm1KkSBFq166NsbExBw4coHPnzuqkKMkxe1JSUjAyMkKlUnHt2jVMTU1xd3dHoVBw+fJl/v33XxwcHChXrhzly5fn3r17tG3bVlqOBYh0sQu4jC7y1atX8ff3JyYmho4dOzJ06FAeP35Mp06dcHNzY/r06epjXhmtzVdbnSJrMs5Wp6am0qNHD168eEGZMmXo2bMn7du3Z8WKFZw7d47GjRvTtWtXihUrRlJSkrQcCxg5+FTAGRgYcPPmTQYMGICHhwc9evRg+fLlBAcHU7ZsWUJDQwkPD2fEiBG8fPlS/T5Jju8m41Kezz//HDs7O4KDgylRogQbN27kp59+YtCgQTRs2JD9+/eza9cuVCoVRYoUyeuwRRYZ5XUA4t2kpKSwYcMGBg0aRP/+/Xnw4AHHjh1j5cqVKJVKxowZw44dO/Dz81P/gUpizL4XL15gYWEBQHh4OMbGxgQEBABQrFgx4uPj2b59O6ampgwcOBATExN1t1sUPNKCLOCMjIy4fv06ZmZmpKamMmbMGLp06cLMmTNZsWIFvr6+GBgYsHTpUgwMDDINniCyZu7cuYwdO5YnT54AEBkZSVJSEgBjx45FqVQybdo0nj59yjfffENQUBC9e/emfPnyeRm2eAfSgixgMo59wf8dfxw1ahQffvghEydOpHz58nTo0IHLly9To0YNihUrRqlSpdTvl0t6sq9z5854e3szf/58Jk6cSOfOnalRowanT5/m+vXrbNu2DWNjY2rWrEnVqlXx9PTM65DFO5IEWYCkpaVhaGjIP//8w4YNG4iLi6NevXq0bduWkiVLAlC/fn0A1q9fj6urK+PGjVO3HCU5Zl9SUhJVq1Zlz549NGrUCGNjY0aPHo2DgwN79+7FyMgIY2NjNm7cyOXLlxk9erS0HAsBOYtdQGS0HO/du0fPnj1p3bo1FhYW7Nixg2bNmtG/f3+2b9/OL7/8grGxMfHx8ezevZsiRYpIcnxHr+6/jRs3smjRIkxMTHB1dcXX15eUlBQ6dOhA+fLlefjwIStXrqRmzZp5HLXQB0mQBcjz58+ZPXs2lpaWjB8/HoAbN24watQoPv74Y3r06MH58+d59OgRPXv2xMjISH2tnsg6f39/WrVqRZMmTQBYtmwZISEh/PzzzxgaGuLp6UmdOnWYNGkSKSkp3Lhxg8qVK1OuXLk8jlzoiyTIfEypVHLq1CmePHlC9erVuXDhAuvXr8fW1pZVq1ap6508eZKvv/6avXv3Urp0afX8V49Xiqw7dOgQEydOZNu2bRw5coS1a9eyceNG9QAT9+7d44svvsDW1paAgAAZY7MQkqZFPhUXF0evXr2wsbHh6tWr9O7dm/r165OcnMzx48cJDQ1V31NtZ2dH9erVX7sIWZLju/noo48wNjamXbt2FC9enN27d2Ntba0eHbxixYosW7aM0aNHo1QqJUEWQpIg86HExES6detGgwYN8PPzIyYmBktLSxQKBWXLliUlJYULFy7w4MEDPv30U4KCglAqlXILWw5o3rw5a9euZciQISQkJACoD1mkpKRQuXJltm3bJocxCinpYudDoaGhHDx4kMWLF6vnHThwgM2bN2NqakpMTAyNGzfmhx9+wNLSEkdHR2bOnAnI6Dw55dixY4wZM4Zly5ZRr169TGVyV1LhJX9J+dC///6rfqjW6dOnmTFjBl9++SVVqlTB2dmZ4sWLExsbS/fu3bG2tqZ69ep5HHHh16JFC7777jt8fHw4e/ZspjJJjoWXtCDzoT///JPu3bvj4OBAcnIylpaW+Pj40KhRIwAmT55M9erV8fLyYt26dVy8eBFXV1e++OKLPI688AsLC2PmzJn8/PPP6tHZReElB07yodq1a7N161ZOnz6Ni4sL5cuXp2LFiuryZ8+ekZKSgrm5OZ9++ilKpZJLly4RGxsrJwpyWKtWrWjUqJEkx/eEtCALgGfPnqnvlBk3bhxXr15lx44d6sEnnj59iqGhobqOEEI/JEHmcxcuXGDgwIHY29tjampKXFwcmzZtokiRInIRuBA5TBJkPpeQkEB4eDi3b9/mgw8+wMPDA0NDQ0mOQuQCSZAFkNwhI0TukAQphBAayOMkCKQAAAeSSURBVHWQQgihgSRIIYTQQBKkEEJoIAlSCCE0kAQphBAaSIIspDw8PLC3t1dPNWvWpEWLFsydO5fk5GS9rSc8PBx7e3v1M7c9PDzYvHmz1vclJSWxadOmbK/3/v372Nvbc/PmzTeW6xpHVutmNQ5RsMmVxoXY119/rR5UNzU1lcuXLzN27FjMzMzw8fHJkXVu374dMzMzrfX27t1LcHAwvXr1ypE4hNAHaUEWYubm5pQpU4YyZcpQrlw5WrVqRYcOHTh48GCOrbNkyZI6DeQgl9+KgkAS5HvGyMhIPcjFhAkTGDduHF26dKFBgwb89ttvJCcnM2fOHNzc3KhXrx5Dhgzh3r176vdHR0czbNgwXFxcaNu2LZcvX860/Fe7q2lpaQQHB9O8eXNcXFwYMGAAd+7cITw8nIkTJxIVFYW9vT33799HpVKxfPlyWrRogYuLC3369OHKlSvq5SYkJDBx4kTq1q1LixYtOHHihM7bHBcXx+TJk2ncuDE1a9bEw8ODkJCQTHUiIiLo2rUrtWvXpkePHly9ejXT+319falfvz4NGzZkzJgxREdH677TRYElCfI9kZqaytmzZ9m9ezctW7ZUz//xxx/p378/q1evpnbt2ixYsIDw8HACAwPZunUrZcqUoV+/fiiVSgBGjBjB8+fP2bx5MxMnTmT16tUa1xkUFERISAiTJ09m165dmJubq5Orr68vJUuW5NSpU1hbW7Np0ya2bt3KjBkz2LlzJ/Xr16dv3748ffoUgClTpvDnn3+yevVq5s2bx7p163Te9lmzZnHlyhWWLl3Kzz//TKdOnQgICODhw4fqOps3b6Z79+6EhoZSrlw5hg0bpj5WO2nSJB49esTatWtZu3Yt8fHxeHt7Syv4faAShZK7u7uqVq1aKmdnZ5Wzs7OqevXqqlq1aqkmTpyoevnypUqlUqnGjx+v8vT0VL8nMTFRVatWLdXFixfV81JTU1VNmzZVhYaGqq5fv66qVq2aKiIiQl0eEhKiqlatmkqpVKrXu2nTJlVaWpqqYcOGqpCQEHXdp0+fqmbPnq168eKFaseOHSo3Nzd1WfPmzVX79u3LtA3du3dXBQcHq54/f66qXr266tSpU+qyY8eOqapVq6a6ceOGxu3ftGmTSqVSqXbs2KH666+/1GVJSUmqatWqqZfn7u6u8vf3V5e/ePFC5ezsrDp06JDqzp07/6+9ewltYovjOP61zUOah5g+NKXNxBBsIEgLlZJNFoorEfpyI2go6iLiyk2blhaKgklpIwWjBWldCAFxIcVlF1236UoQSsA4PiCK2JRIrGlom7uQDok61gt3cVv+n9UwZzLnTBY/TuYfzim3tbWVv3z5orUXCoWy3+8vr6yslD98+PDHcYj9TYo0B1g4HObChQsAmEwmGhoatJ/Xu1paWrTj9+/fUyqVCIVCVdsIFItFVFXFZDJRV1eH2+3W2k6dOvXbvtfX18nlclXtDQ0N2n7elb59+8bHjx+JRCKMjIxo50ulEq2traiqyvb2dtXWEnr9/k5vby+Li4s8f/4cVVVZXV0Ffsyqd7W3t2vHVqsVt9tNJpOhtraWcrnMuXPnqu65tbWFqqqyB/YBJwF5gDkcDhRF+eM1lQWV3cB48uTJLyuT22w2UqkUOzs7Ved/Dty9zv/Obr+Tk5O0tbVVtdXV1fHp0yegurDzb5Z6i0QiLC8v093dTV9fHx0dHZw5c6bqmp9XR9rZ2cFkMrG9vY3ZbGZ+fv6X+zocDvL5/F+PQ+w/8g5SaFwuFwaDgbW1NRRFQVEUmpubicfjpNNpTp48SbFYJJ1Oa5+pLKRUstls1NfXa7M1gHw+TyAQIJPJVM1Q7XY7jY2NfP78WetXURQePXpEKpXixIkTGI1GXr58uWe/P1tfX2d+fp6pqSlu3brF+fPnte1bKwO3siiTz+d59+4dHo8Hj8fD5uYmm5ub2riOHDlCNBolm83+1RjE/iUzSKGxWCxcunSJO3fuYDAYaG1tZWZmhqWlJUZHR2lqaiIYDDIyMsLt27f5+vUriURC934DAwMkEgmcTictLS1MT09z7NgxPB4Pr1+/plAokMlkUBSF69evc//+ferr6/H7/Tx79owXL15w5coVrFYr/f39RKNR7HY7RqORaDT6V89ktVqxWCwsLCxw/Phxstksd+/eBaj6w3wymcTn8+Hz+YjH47hcLoLBIDU1NZw9e5bBwUHGxsaw2+3EYjHevn2L2+3WikjiYJKAFFUGBwepqalhaGiIjY0N/H4/c3NzNDU1AXDv3j3Gx8e5fPkyR48eZWBgQDesrl69SqFQYHh4mI2NDbq6unj48CGHDh0iEAjg9Xrp6ekhmUwSCoX4/v07ExMT5HI5vF4vMzMz+Hw+4EclORaLEQ6HOXz4MDdv3mR8fHzP5zEajcTjcWKxGE+fPsXpdHLx4kXMZjOvXr3S3i1eu3aN2dlZ3rx5w+nTp3nw4IG2v/jExATRaJQbN26wtbVFZ2cnjx8/xmw2/wffuPg/kwVzhRBCh7yDFEIIHRKQQgihQwJSCCF0SEAKIYQOCUghhNAhASmEEDokIIUQQocEpBBC6PgHBINE2R2iZeUAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "words_list_predictions = [1 if pred >= .30 else 0 for pred in words_list_prediction_probabilities]\n", "\n", "low_classification_threshold = confusion_matrix(y_train, words_list_predictions, labels=[1, 0])\n", "\n", "plot_confusion_matrix(low_classification_threshold, classes=class_names,\n", " title='words_list_model Confusion Matrix $C = .30$')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By lowering the bar for classifying an email as spam, 6 spam emails that were mislabeled with $C = .50$ are now correct. However, there are more false positives.\n", "\n", "$$ \\text{Sensitivity } (C = .30) = \\frac{24}{42} \\approx .571 \\\\\n", "\\text{Specificity } (C = .30) = \\frac{738}{758} \\approx .974\n", "$$\n", "\n", "Compared to the default, a lower threshold of $C = .30$ increases sensitivity but decreases specificity.\n", "\n", "We adjust a model's sensitivity and specificity by changing the classification threshold. Although we strive to maximize both sensitivity and specificity, we can see from the confusion matrices created with varying classification thresholds that there is a tradeoff. Increasing sensitivity leads to a decrease in specificity and vice versa." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ROC Curves\n", "\n", "We can calculate sensitivity and specificity values for all classification thresholds between 0 and 1 and plot them. Each threshold value $C$ is associated with a (sensitivity, specificity) pair. A **ROC (Receiver Operating Characteristic) curve** is a slight modification of this idea; instead of plotting (sensitivity, specificity) it plots (sensitivity, 1 - specificity) pairs, where 1 - specificity is defined as the false positive rate.\n", "\n", "$$ \\text{False Positive Rate } = 1 - \\frac{TN}{TN + FP} = \\frac{TN + FP - TN}{TN + FP} = \\frac{FP}{TN + FP} $$\n", "\n", "A point on the ROC curve represents the sensitivity and false positive rate associated with a specific threshold value. \n", "\n", "The ROC curve for `words_list_model` is calculated below using scikit-learn's [ROC curve function](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html):" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import roc_curve\n", "\n", "words_list_model_probabilities = words_list_model.predict_proba(X_train)[:, 1]\n", "false_positive_rate_values, sensitivity_values, thresholds = roc_curve(y_train, words_list_model_probabilities, pos_label=1)" ] }, { "cell_type": "code", "execution_count": 86, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/plain": [ "Text(0.5,1,'words_list_model ROC Curve')" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEhCAYAAABhpec9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XlcTfn/B/DXvbeuVpItxKiQNaWkUrQopPhGSL4xFJUxjGUsQ2MZhrETSpaRMmWdsg3GMo0ty5AtWaohzNBmabvVvef3R7/O11Udt+7ttr2fj8d96J57lvf7lPO+n88553N4DMMwIIQQQirAr+kACCGE1G5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoURGHmz5+P0aNHK2x9wcHB6NevH/ve2NgYUVFRMi37zz//4Msvv4RIJFJYPIpU2X115MgRGBsbV5jPtWvXYGxsXObVu3dveHp64uzZs2WWEYlECAsLg7u7O0xMTGBjY4OAgABcv3693G2IxWLs27cPnp6e6N27N2xsbDBlyhQkJCTIlMOrV6/www8/wMnJCSYmJnBxccFPP/2E7OxsmfcDqRkqNR0AIbLav38/2rVrJ9O8V69exdWrV6s5otpnw4YNaNOmDQCAYRikp6cjIiIC06dPx8GDB9G9e3cAQE5ODiZNmoRnz57hyy+/hJmZGT58+IBjx45h/PjxmD9/Pr788kt2vYWFhZgyZQoePnyICRMmYObMmcjLy0N0dDTGjRuHrVu3wt7evsK47ty5gylTpqB9+/b4+uuvoaenh+TkZISGhuLixYuIjIyEjo5Ode4aIgcqFKTOMDU1rekQaj1jY2MYGRlJTevXrx+srKxw/PhxtlCsXLkSz58/x6FDh6SKr7OzM7Zs2YKffvoJvXv3homJCYCS1t2dO3dw4MABdOrUiZ3fyckJvr6+WLRoEc6fPw+hUFgmpoKCAsyePRs9evTA9u3boaJSctixsrKCjY0Nhg8fjtDQUMyfP1/h+4MoBnU91TPDhg3D4sWL2fdJSUkwNjbG9u3b2Wnnzp1Dz549kZeXBwA4efIkPDw80KtXLzg5OWHHjh34+IZ9Y2NjhIWFYdCgQTAzM8O1a9fAMAy2b98OOzs7mJmZYcWKFRCLxVKxXLhwAR4eHjAxMYGtrS2WL1+OgoKCKuf2cddTcXExVq1ahf79+6Nnz54YNmwYTp48CaCkm2bBggUAABMTExw5ckSm9QcHB2Ps2LE4cOAAHB0dYWpqiq+++grv37/H5s2bYWVlBWtra2zevFlquQcPHmDixImwsLCAlZUVgoKC8OHDB/ZzWfZVYWEhVq9eDVtbW5iYmMDHxwdJSUlV3lcfEwqFUgfwjIwMxMTEICAgoNwWWmBgIPT19bFz5042tqioKIwZM0aqSAAAn8/HzJkz4eHhgZycnHK3f+7cOaSlpWHu3LlskShlYGCA2bNno2PHjgD+14WWnJzMziMSiWBsbMz+Hkt/T4sXL4aZmRkmT54MR0dHLF++XGrdDx48gLGxMe7duwcAeP36Nb755huYm5vD3Nwc3377LXV7yYhaFPWMra0t/vjjD/Z9aX/zrVu32GlXrlyBubk5NDQ0EBkZieXLl2PChAmYPXs2EhISsGnTJmRnZ2Pu3LnsMiEhIfj+++8hEAhgYmKCHTt2IDg4GNOnT4exsTHCw8Nx7do19hvrs2fPMH36dHh7e2P+/Pl4/vw5fvzxRzRq1Ajffvut3Hlu374dv/76K7777jvo6enh2LFjmDVrFjp16gR7e3sEBgYiJCQEkZGRZb5hc0lKSkJkZCQWLVqEjIwMLF26FJ6enjA0NMTatWtx7tw5bN26FTY2NrCwsMD9+/cxduxYWFtbY+3atUhPT8eGDRvw5MkT7Nu3DwKB4LP7CgAWLFiAuLg4zJw5E23btkVERATGjx+Po0ePQk9PT+b4JRIJiouL2Z9fv36N0NBQ5OXlYfDgwQBK/iaKi4thZ2dX7joEAgGcnJywf/9+AMC9e/fw4cOHCuc3MTFhWx7luXr1Klq2bAljY+NyP58wYYLM+ZW6e/cutLS0sG3bNgDApUuXcOLECSxcuBA8Hg8AcPr0abRv3579UjR+/HgIBAKsWLECEokE69evh7+/P6KioiAQCCodQ0NChaKesbOzw65du5CZmYlmzZrh5s2b6NKlC27fvg2GYcDj8XD58mWMGjUKYrEYwcHB8PT0ZL+B29ragsfjISQkBH5+ftDV1QUAODo6wsPDA0DJAWj37t3w8fHBlClTAAB9+/aFg4MDG8f9+/dRWFgIX19ftGzZEn379oVQKGQPYvK6desWevTogeHDhwMA+vTpA21tbYjFYujq6qJ9+/YASg5ijRo1knm9eXl5WLt2LTp37gwAOHbsGB4+fIiYmBhoaGjA2toahw8fxv3792FhYYGQkBC0bdsWISEh7MHGwMAA48aNw4ULF+Do6PjZffXkyRMcP34c69evx9ChQwGU/B4GDRqEXbt2YeHChTLH7+bmVmZa586dsWXLFvTq1QtAyUllAGjbtm2F69HX10deXh7evn2LN2/eAABat24tcxwfe/PmDXveRFGKi4uxcOFCdOjQAQDQpEkT7Ny5E/fu3WOL1unTp+Hq6goA+PXXX/Hy5UucPn2azbtbt24YMmQILly4gIEDByo0vvqGup7qGXNzc6irq+PGjRsAgJs3b2LixIl49+4dkpOT8e+//yI1NRV2dnZISUnB27dv2W+apVxdXVFUVIQ7d+6w0wwMDNifU1NTkZ2djf79+7PT1NTUpL5xmpiYQCgUYvTo0Vi7di1u3boFd3d3jBw5UiF5WlhY4NKlS5gwYQIiIyPx+vVrzJs3D126dJFrverq6myRAIDmzZujQ4cO0NDQAFDybVtbW5vtWvrrr7/g4uIi9Y3UwsICLVq0wF9//SXTvir9Xdna2qK4uJgtpjY2Nrh27Vql4t+8eTMOHTqEvXv3wtraGnp6elizZg2cnJzYeUq7Fbm+RZd+xjAM+7NEIqlULKX4fH6ZrjZ5qaioSHWbdevWDR06dMCZM2cAlLQM//77bwwZMgRAyT7u1KkTWrVqxe5jfX19tG/fHvHx8QqNrT6iQlHPCIVC9OnTB9evX0dycjLev3+PQYMGoU2bNrh16xYuX74MPT09dO7cGe/evQNQcjD8WLNmzQBAqs+5dBoAdrmmTZuWuxwAtGvXDj///DM6duyIPXv2YOzYsRg4cCDi4uIUkueUKVMwb948vHnzBj/88AMcHBwwefJkZGZmyrVeTU3NMtPU1dUrnP/9+/dSeZdq1qwZcnJyZNpXb9++BQBYWlqie/fu7OvAgQNIT0+vVPwdO3ZEz5490bdvX2zfvh3a2tqYPHkysrKy2HlKv1H/888/Fa7n5cuXUFdXR9OmTdmWREXzSyQSttVRnjZt2uDff/+t8PPs7GwUFhZy5vUpHR2dMoXO1dWVLRSnT5+GoaEh+8Xh7du3SExMlNq/3bt3x99//13pfdwQUaGoh2xtbXH9+nXcvHkT3bp1g7q6OszNzXHr1i1cuXIFtra2AEqa60DJyc2Plb4v/fxTpdM/PvgA/ysgpSwsLLBz505cu3YNmzdvRpMmTTBz5sxKHxTKIxAIMGnSJPz22284e/YsZs+ejevXr2PTpk1yr7syGjduXG5xyszMRJMmTWTaV40bN4ZQKMShQ4fKvHbv3l3l2Bo1aoQlS5bgzZs3WLt2LTvdxsYGqqqqOHfuXLnLMQyDCxcusK2gbt26QUdHB1euXCl3/r/++gt2dna4cOFCuZ/b2NggPT0djx8/Lvfz1atXw9HREcXFxez5hY9bL7m5uZ9PFiWF4tmzZ3j8+DHOnDnDdjsBJfvYzMys3H08e/ZsmdbfkFGhqIdsbW3x9OlTnDt3Dubm5gBK+vBv3LiBq1evst0ehoaG0NHRwalTp6SW/+2339iT1uUxNDREixYt2G9vQEmf8cdN+JiYGDg5OaGoqAiampoYNGgQJk+ejNzc3AqvjqmMgIAArFy5EkBJ62Xy5Mno06cP+82Vz1fOn3bv3r1x5swZqa6VmzdvIj09HaampjLtKzMzMxQWFkIsFqNnz57sKzY2tszvprIsLCwwZMgQ/Prrr+xVVDo6OhgzZgxCQkKQmppaZpmdO3ciJSWFvY9CIBDAy8sL0dHRSElJkZpXIpFgy5Yt0NHRgY2NTbkx9O/fH+3atcPq1avLnKN68uQJfvvtNzg7O0NFRYXt4vu4BfLxhRhcOnXqhM6dOyMyMhJPnz6VKhRmZmZ49uwZDAwM2P3bqVMnBAcHs1dFkYrRyex6yMjICHp6eoiLi8OYMWMAlBwwSq9aKv0PLRAIMHXqVKxcuRKampro378/EhISEBISAh8fnwpvgOLxePjqq6+wbNkyNG3aFGZmZoiOjkZGRgbbjWVubo6MjAzMmjULXl5eyM/PR0hICHr37s2eIJdH7969sXnzZujp6aFbt254+PAh4uPj2UuDGzduDKCk6NnY2KBly5Zyb7M8AQEB8Pb2RmBgIMaNG4f09HRs3LgRPXv2hL29vUz7qnv37nBwcMD06dPx9ddfo127djhz5gz27dsn1RKoqlmzZuHs2bNYs2YNdu3axU57/PgxxowZA19fX5iamiI3NxcnTpzAyZMnMWfOHPTu3Ztdh7+/P65cuYKxY8di0qRJMDExQVZWFvbt24c7d+4gNDS0wosGhEIhVqxYAX9/f/z3v/+Ft7c3WrRogcTEROzYsQP6+vrst/ouXbqgRYsW2LBhAxiGQWZmJoKDg6GmpiZTrkOGDEFwcDA6d+4sdbXbyJEjER4eDj8/P/j6+kIoFGL37t24e/cu3b8hAyoU9ZStrS0OHTrE/mc3MjKCrq4uOnTowB5EgZJLExs1aoSff/4Z+/btQ+vWrTFz5kxMmjSJc/1jx46FWCzG7t27sXv3bjg7O2P06NG4e/cugJJv+SEhIdi4cSOmTZsGFRUV2NnZsVdXycvPzw8FBQWIiIjAmzdvoKenh1mzZmHUqFEAAGtra1hZWWHRokWYMWMGJk+erJDtfsrExAQ///wz1q1bh2nTpkFbWxvOzs6YM2cOe8/A5/YVUHJH9YYNG7Bp0ya8f/8eHTp0wJo1a+Du7i53jO3bt4eXlxciIiJw+fJl9OvXD5qamti1axeioqLw66+/IjQ0FGpqaujVqxfCw8NhaWkptQ4NDQ3s2bMHP//8M44ePYrQ0FBoaGigZ8+eiI6ORs+ePTlj6Nu3L6KjoxEWFoZ169YhOzsbrVu3xogRIxAQEAAtLS0AJSepN27ciB9//BFTp06FgYEBVqxYgXnz5smU69ChQ7Fp0yap1gRQ8sUhIiICq1evZgtDjx49EB4eDkNDQ1l3ZYPFo0ehEkII4UItCqJ0Eonks5da8vl8hZ1nUPb2CKlvqFAQpdu6dSu2bNnCOY+HhwdWrVpVJ7dHSH1DXU9E6V6/fs153T1Qct+Bvr5+ndweIfUNFQpCCCGcqFOWEEIIp3p3jiI9/cPnZyoHj8dDs2aayMzMRUNpZFHOlHN9RTlXPucWLbQr/IxaFP+Pzy/Z0Q3pwhfKuWGgnBuG6sy5Ae1GQgghVUGFghBCCCcqFIQQQjhRoSCEEMKpRgrF3bt32WcilOf48eNwcnKCqakp/P39yzwvgRBCiPIotVAwDINDhw5h0qRJKCoqKneepKQkLF68GOvXr0d8fDyaN2+usBFHCSGEVJ5SC0VoaCj27t2LgICACuc5duwYnJyc0KtXL6ipqWHOnDm4ePEitSoIIYQDw5S8qoNSb7gbOXIkAgICcP369QrnSUlJgZmZGfu+adOmaNKkCVJTU8s827k8Vb2OmM/nSf3bEFDODQPlXP8xDJCUxEfjxoC+Pg+fPE5cbkotFLI8ZSw/P7/M06zU1dWRn58v0zaaNdNkn7tbFTo6mlVetq6inBsGyrn+EokANTWgsBDQ0NBEBQ8brLJaN4SHmpoaCgoKpKbl5+ezz9L9nMzM3Cq3KHR0NPH2bS4kkoZxyz/lTDnXVw0tZ5EI+PBBAG1tdbx9mwtV1crnrKurVeFnta5QGBkZST3wPSsrC+/evZN6/i0XhmHw0XPuK00iYSAW1/8/rI9Rzg0D5Vx/icVgC2J15Fzr7qNwc3PDmTNncPPmTYhEIqxfvx79+/dH06ZNazo0QgipEQxT0mqo6FXBRaQKUytaFN9//z0AYNmyZejatSt++OEHLFy4EOnp6bCwsMDKlStrOEJCCKkZDAM8fMj/bDGozgEQ692Di6o6zLhAwIOurhaysnIaRFMVoJwp5/qrPuUsEpVc0fQ5fD4PLVpooE2bnCqdl+EaZrxWtCgIIYR8npGRBKqq5X8mEPCgpwdkZyt+u1QoCCGkjlBVRYWXvgoEgBx3BnCqdSezCSGE1C5UKAghhHCiQkEIIYQTFQpCCCGc6GQ2UTiGKRlzpjYSCP53k5I8d/DXJZRzTUcjn+q+mU4WVCiIQsl6c1BN4fN5aNIEePeO3yDGAAIo54aSc3WirieiUIWFteMbECH1jaoqIBTWzLapRUGqDdfNQTWl5I5dICtLUufv2JUV5Vw/chYKq+8+ic+hQkGqDdfNQTVFICiJqVGjut93LSvKuaajqfuo64kQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnujO7jlHkyKzVMcImjfNESP1DhaIOUfTIrDTCJiFEFtT1VIfUpZFZa3KkS0KIYlGLoo4yNJTIfSCuzhE2a3KkS0KIYlGhqKOEQvlHZqURNgkhsqCuJ0IIIZyoUBBCCOFEhYIQQggnKhSEEEI4KbVQJCYmwtPTE6amphg+fDgSEhLKnW/btm2ws7ODhYUFfH19kZaWpswwCSGEfERphUIkEiEgIAAjRozAjRs34OPjg8DAQOTm5krNd/78ecTExODw4cO4evUq2rdvj4ULFyorzBrBMP+7Q5rrVVfuoSCE1C9Kuzw2Pj4efD4f3t7eAABPT0+Eh4cjLi4Orq6u7Hx///03JBIJJBIJGIaBQCCAmpqassJUOkXfbU0IIYqmtEKRmpoKIyMjqWkGBgZISUmRmjZ06FDs378fAwYMgEAgQMuWLREVFaWsMJWuKndb013PhBBlUlqhyMvLg7q6utQ0NTU1FBQUSE0rLCxE7969sX37drRo0QIrV67EzJkzERUVBZ4Mt/ryeDzwq9ChxufzpP5VFoHgf9s0MpLtbuuSu57lj7Omcq5JlHPDQDkrltIKhbq6epmiUFBQAA0NDalpy5cvh7OzMzp06AAAWLRoEXr37o3Hjx/D2Nj4s9tp1kxTroOojo5mlZetCpEIaNKk5Gc9Pfnvtq4KZedcG1DODQPlrBhKKxSGhoaIjIyUmpaamgo3Nzepaa9evULhR+No8/l88Pl8qKjIFmpmZm6VWxQ6Opp4+zZXqSOpFhSUjN4KlIy5pMxCUVM51yTKmXKur+TNWVdXq8LPlFYorK2tUVhYiIiICHh5eSE2NhYZGRmwtbWVms/e3h67du2CnZ0dWrVqhXXr1qFTp04wMDCQaTsMw8g1bpFEwih8gLyKMAyQmMgHULI9sVi+2KtKmTnXFpRzw0A5K4bSLo8VCoXYsWMHTpw4AUtLS0RGRiIkJAQaGhrw8/NDaGgoAODrr7+Gi4sLvL29YWdnh+fPn2Pr1q3gV6WZUMt9+gAiOkFNCKmNeAzD1Ktym57+oUrLlQy5rYWsrBylfQMRiYCkpJICaGwsgbKvAq6JnGsa5Uw511fy5tyihXaFn9W/r+l1BMNIXxZLz24ghNRW9DyKGkA32RFC6hJqUdSAT2+yoxvoCCG1GbUoapiRkQSamtT1RAipvahFUcNUValIEEJqNyoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCONEQHgrAMGWfLcGFBgMkhNQlVCjkRCPBEkLqO+p6ktOnI8FWBo0aSwipC6hFoUCGhpJKHfiFQhoQkBBS+1GhUCChEGjUqKajIIQQxaKuJ0IIIZyoUBBCCOFEhYIQQggnKhSEEEI4yVwoxGJxdcZBCCGklpK5UPTr1w9Lly7FrVu3qjOeOoNhAJGocndkE0JIXSRzoVi5ciU+fPgAPz8/ODo6Yv369Xjy5El1xlZrld6NnZTER0oK9d4RQuo3me+jcHBwgIODAwoKCnD+/HmcOnUKXl5eaNu2LYYNGwY3Nzfo6elVZ6y1Rnl3Y9Nd1oSQ+qrSN9ypqanB1dUVenp6aNOmDaKiohASEoJNmzahX79+WLhwIdq1a1cdsdZKRkYStkjQXdaEkPqoUv0m9+7dw08//QQHBwdMnDgRr169wpo1a3D16lX88ccfEAgEmDp1anXFWiupqpbcjU1FghBSX8ncohg4cCBevXoFc3NzTJ06FYMHD4a2tjb7ebNmzeDu7o7FixdXS6CEEEJqhswtCi8vL5w/fx4REREYNWqUVJEoZW9vj4sXL1a4jsTERHh6esLU1BTDhw9HQkJCufP9/vvvGDx4MMzMzDB69GgkJSXJGiYhhBAFk7lQREVFQU1Nrcz0169fw8rKCkDJ+QthBWd0RSIRAgICMGLECNy4cQM+Pj4IDAxEbm6u1HyJiYn47rvvsHz5cvz1118YOHAgZsyYUZmcCCGEKBBn19PJkydx7tw5AMCrV6+waNEiNPpkeNSXL19CVVX1sxuKj48Hn8+Ht7c3AMDT0xPh4eGIi4uDq6srO190dDRGjRoFCwsLAMDEiRNhY2MDiUQCPp8uRSWEEGXjLBRWVlZsVxLDMFBVVS3TYujRowcWLlz42Q2lpqbCyMhIapqBgQFSUlKkpiUmJsLe3h7jx4/Ho0eP0K1bN3z//fcyFwkej4eq1BM+nyf1LxeB4H/zCQQ8CASV315tUJmc6wvKuWGgnBWLs1Do6upi5cqVAIC2bdvC19cX6urqVdpQXl5emWXV1NRQUFAgNe3du3eIjo5GSEgIjI2NsXnzZgQGBuL48eNQUfn8ufdmzTTBk+MSJB0dzc/OIxIBTZqU/KyrW/efQSFLzvUN5dwwUM6KwXnkvXTpEqysrKCiogJTU1P89ddfFc5ra2vLuSF1dfUyRaGgoAAaGhpS04RCIZydndGzZ08AwIwZM7Bnzx6kpKSgc+fOnNsAgMzM3Cq3KHR0NPH2bS4kEoZzXpEIePeuZCNZWZI6Wygqk3N9QTlTzvWVvDnr6mpV+BlnofDz88Ply5fRrFkz+Pn5VTgfj8fDw4cPOYMwNDREZGSk1LTU1FS4ublJTTMwMEDhRwMoMQzDvmTBMAzkGb9QImEgFnNvSywG+4sQi+XbXm0gS871DeXcMFDOisFZKD6+LFXeS1Stra1RWFiIiIgIeHl5ITY2FhkZGWVaIh4eHpg3bx6GDRuGrl27YuPGjejQoYNMrQlCCCGKJ3MnzezZsxEXF4fi4uIqbUgoFGLHjh04ceIELC0tERkZiZCQEGhoaMDPzw+hoaEAACcnJwQFBWHevHmwtLTE3bt3sXXrVrnOOyhC6WixIlHZcZ4IIaQ+4zEy9uksWLAA586dA5/Ph4uLC9zc3GBpaVnd8VVaevqHKi0nEPCgq6uFrKycMs220tFiyysQXbrU3XMUXDnXV5Qz5VxfyZtzixZlb6IuJfMQHitXrkRxcTEuXbqEU6dO4auvvoKGhgZcXV3h7u6Obt26VTqwuqK80WIBGjGWENIwVGr0WBUVFdjb28Pe3h5FRUWIiIjAli1bsGfPns+ezK4vSkeLBWjEWEJIw1DpYcYfPHiAU6dO4dSpU8jIyICjo2OZK5fqs9LRYgkhpKGQuVCsXbsWp0+fxj///IN+/fph+vTpGDhwYJVvwCOEEFI3yFwobt++jUmTJmHw4MFo2rRpdcZECCGkFpG5UOzbt6864yCEEFJLcRYKW1tbHDt2DE2bNv3sEB2XLl1SaGCEEEJqB85CMXv2bGhqarI/E0IIaXg4C4WHhwf7M4/Hg6ura5lhxvPy8nDgwIHqiY4QQkiN4ywUb968YZ9At2DBAnzxxRfQ0dGRmicpKQnr16/Hl19+WW1BEkIIqTmchSIhIQHTp09nx1kaO3ZsufN93PIghBBSv3AWChcXF5w/fx4SiQQDBw7EwYMHoaury37O4/GgoaFRppVBCCGk/vjs5bFt2rQBIP8w44QQQuomzkLh5eWFsLAwNG7cGF5eXpwrio6OVmhghBBCaofP3keh+v8j4H3uPgpCCCH1E2ehmDZtWrk/E0IIaThkfsJdcXExIiIi8OLFCwDA+vXrMXDgQHzzzTd4+/ZttQVICCGkZslcKFatWoXt27fjw4cPOHfuHHbv3o3Ro0cjIyMDP/zwQ3XGWGNKH39Kjz4lhDRkMg8K+NtvvyE4OBhdu3bFrl27YGNjgylTpsDe3h7e3t7VGWON4Hr8KSGENCQytyjy8vLQqlUrSCQSXLx4EQMGDAAACAQC8Pkyr6bOKO/xp/ToU0JIQyRzi6J79+4ICQmBjo4OPnz4ACcnJ7x8+RJr1qyBmZlZdcZY4wwNJRAK6dGnhJCGSeamwOLFi3H//n1ER0djzpw50NPTQ3h4ONLT0xEUFFSdMdY4obDk8adUJAghDZHMLYpOnTohJiZGatqcOXPKjCZLCCGkfpG5UADAkydPcO/ePRQXF4NhGKnPxowZo9DACCGE1A4yF4qwsDCsX78eTZo0YR9mVIrH41GhIISQekrmQrFnzx7Mnj0bkydPrs54CCGE1DIyn8zOz8/H4MGDqzOWWoNh6CY7QggpJXOhcHFxwdGjR6szllqh9Ea75OT6d28IIYRUhcxdT9ra2ggJCcGpU6dgYGDAjipbat26dQoPriZ8eqMd3WRHCGnoZP7anJubC3d3d/To0QOampoQCoVSL1kkJibC09MTpqamGD58OBISEjjnP3ToEPr27StriApnZCRB164Sun+CENKgydyiWLlypVwbEolECAgIQEBAAEaNGoXY2FgEBgbi7NmzZa6iAoC0tDSsWrUKAoFAru3KQ1WVbrIjhJBKdcRwjcJgAAAgAElEQVQ/efIEQUFB8PHxwevXrxEZGYnLly/LtGx8fDz4fD68vb2hqqoKT09PNG/eHHFxcWXmFYvFmDt3Ll1ySwghtYDMheLq1avw9PREXl4eEhISUFhYiDdv3mDKlCk4efLkZ5dPTU2FkZGR1DQDAwOkpKSUmTcsLAydOnVC//79ZQ2PEEJINZG562n9+vWYO3cuxo0bxw4COGvWLOjq6mLr1q1wdXXlXD4vLw/q6upS09TU1FBQUCA17f79+zh69CgOHTqE+/fvyxoei8fjoSqD2fL5PPbf0p8FAh5qsOer2n2cc0NBOTcMlLNiyVwonjx5Uu43fCcnJ6xfv/6zy6urq5cpCgUFBdDQ0JB6P3/+fCxfvrzc8xayaNZMEzw5Tizo6GiiSZOSn3V1SwYDrO90dKq2r+syyrlhoJwVQ+ZC0apVKzx69Ajt2rWTmh4fH4/WrVt/dnlDQ0NERkZKTUtNTYWbmxv7/v79+0hLS4O/vz+AknMV+fn5sLCwwNGjR9GmTZvPbiczM7fKLQodHU28fZuLd+9KCk1WlqReF4qPc5ZImM8vUA9QzpRzfSVvzrq6WhV+JnOhmDJlCoKCgvD8+XNIJBL8+eefePnyJfbt24dFixZ9dnlra2sUFhYiIiICXl5eiI2NRUZGBmxtbdl5LCwscOfOHfb9tWvXMH36dFy7dk3WMMEwDMRimWcvQyJhIJGU/CwWy7euukIiYSAWN4z/TKUo54aBclYMmb97jxw5EqtWrcL58+ehrq6OTZs24datW1i3bh1GjRr12eWFQiF27NiBEydOwNLSEpGRkQgJCYGGhgb8/PwQGhoqVyKEEEKqB4/5dLzwT4hEIsTFxcHW1pY9nxAREYHLly9DV1cX48ePR5cuXZQSrCzS0z9UaTmBgAddXS38808OHjwo6Xrq0qV+dz2V5pyVldNgvnVRzpRzfSVvzi1aaFf4GWeL4tWrVxgyZAhmz56N9PR0AMDq1avx448/QiAQQCwWY9y4cbh3716lgyKEEFI3cJ6j2LRpEwwMDHD06FFoaWkhKysLe/fuhbOzMzZv3gyg5J6HTZs2YefOnUoJuLpxt68IIaTh4WxRXLp0CTNmzICWVsnZ8IsXL0IsFuM///kPO4+dnR1u375dvVEqCcMASUk0aiwhhHyM86j4/v17NG/enH1/7do1CAQCWFlZsdO0tLQgKb1MqI4rLJR+T6PGEkLIZwpFmzZtkJqaCqDknoY///wTFhYWUjfJXbt2Dfr6+tUbZQ0wNqZRYwkhBPhMofDw8MDy5ctx8uRJLFmyBBkZGRg7diz7+c2bN7Fx40YMGTKk2gNVNioShBBSgvNk9uTJk/H+/XssXboUfD4fs2bNwqBBgwAAy5cvR2RkJFxcXOg52oQQUo999j6Kijx69AgSiQRdu3ZVdExykec+Ck1NLVy+nAeJhKn391AAdK055Vx/Uc6KvY9C5iE8PmVsbFzVRQkhhNQhdC0oIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOGk1EKRmJgIT09PmJqaYvjw4UhISCh3vm3btsHe3h4WFhbw8fHB48ePlRkmIYSQjyitUIhEIgQEBGDEiBG4ceMGfHx8EBgYiNzcXKn5jhw5gtjYWERERCA+Ph7W1tbw9/eHRCJRVqiEEEI+orRCER8fDz6fD29vb6iqqsLT0xPNmzdHXFyc1HzZ2dkICAhAu3btoKKigvHjx+PVq1f4999/lRUqIYSQj6goa0OpqakwMjKSmmZgYICUlBSpab6+vlLvz58/Dx0dHejp6cm0HR6PB34Vyh+fz5P6VyDgQSCo/Hrqkk9zbggo54aBclYspRWKvLw8qKurS01TU1NDQUFBhctcv34dixcvxrJly8CX8ejfrJkmeLyq7SiRCNDWLolRVxdo1KhKq6lzdHQ0azoEpaOcGwbKWTGUVijU1dXLFIWCggJoaGiUO39MTAyWLl2KoKAguLu7y7ydzMzcKrco1NU18eFDPiQSBllZknpfKPh8HnR0NPH2bS4kEqamw1EKyplyrq/kzVlXV6vCz5RWKAwNDREZGSk1LTU1FW5ubmXm3bp1K/bu3Ytt27bB2tq6UtthGAZicdXjlEgYSCQMxGL51lOXlObbkFDODQPlrBhKO5ltbW2NwsJCREREoKioCIcOHUJGRgZsbW2l5jt8+DDCw8Pxyy+/VLpIEEIIUTylFQqhUIgdO3bgxIkTsLS0RGRkJEJCQqChoQE/Pz+EhoYCAMLCwpCbmwtPT0+YmZmxr+TkZGWFSggh5CNK63oCgC5duiA6OrrM9J07d7I/nz59WpkhEUII+QwawoMQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEheIjTMMaEoYQQmRCheL/MQzw4EFNR0EIIbUPFYr/V1go/V4orJk4CCGktqFCUQ5jYwmq+OwjQgipd6hQlIOKBCGE/A8VCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhJNSC0ViYiI8PT1hamqK4cOHIyEhodz59uzZAzs7O/Tu3Rtz5sxBXl6eMsMkhBDyEaUVCpFIhICAAIwYMQI3btyAj48PAgMDkZubKzXfhQsXsGvXLuzduxdxcXF49+4dVq9erawwCSGEfEJphSI+Ph58Ph/e3t5QVVWFp6cnmjdvjri4OKn5YmNj4enpCQMDA2hra2PGjBmIjY2FWCxWVqiEEEI+orRCkZqaCiMjI6lpBgYGSElJkZqWkpKCjh07Ss2Tl5eH169fV2t8DFOtqyeEkDpLRVkbysvLg7q6utQ0NTU1FBQUSE3Lz8+Hmpoa+750mfz8fJm2w+PxwK9k+WMY4PFjAbS1AT6fB4GAB4Ggcuuoi/h8ntS/DQHl3DBQzoqltEKhrq5epigUFBRAQ0NDapqamhpEIhH7vrRAaGpqyrSdZs00weNVbkcxDNCsGVBYCDRrpg49PaCSq6jTdHRk27f1CeXcMFDOiqG0QmFoaIjIyEipaampqXBzc5OaZmRkJNUdlZqaisaNG6Nly5YybSczM7fSLQoA0NfnQUNDE3l5ucjObhj9UHw+Dzo6mnj7NhcSCeVcX1HOlLMsdHW1KvxMaYXC2toahYWFiIiIgJeXF2JjY5GRkQFbW1up+YYNG4bFixdj0KBBaN26NTZv3gw3NzfwZTz6MwyDqpz3FgiARo2A3FwGYnHD+MMqJZFQzg0B5dwwVEfOSjuZLRQKsWPHDpw4cQKWlpaIjIxESEgINDQ04Ofnh9DQUACAo6MjJk+eDH9/f9jb20NbWxtz585VVpiEEEI+wWOY+nW9T3r6hyotJxDwoKurhaysnAbzDYRyppzrK8q58jm3aKFd4Wc0hAchhBBOVCgIIYRwokJBCCGEExUKQgghnKhQEEII4VTvrnoihBCiWNSiIIQQwokKBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCODW4QpGYmAhPT0+Ymppi+PDhSEhIKHe+PXv2wM7ODr1798acOXOQl5en5EgVR9act23bBnt7e1hYWMDHxwePHz9WcqSKI2vOpQ4dOoS+ffsqKbrqIWvOv//+OwYPHgwzMzOMHj0aSUlJSo5UMSrzd21nZwcLCwv4+voiLS1NyZEq3t27d8s89O1jx48fh5OTE0xNTeHv74+MjAz5Nsg0IAUFBYydnR2zb98+prCwkDl48CBjZWXF5OTkSM13/vx5xtbWlklJSWHev3/P+Pn5MYsXL66ZoOUka86HDx9mXFxcmOfPnzNFRUXM1q1bGXt7e0YsFtdQ5FUna86lnj9/zpibmzOWlpZKjlRxZM35wYMHjIWFBXPjxg1GLBYz27dvZ1xcXGoo6qqTNd9z584xzs7OzOvXr5nCwkJmyZIljI+PTw1FLT+JRMIcPHiQ8+/14cOHTO/evZmEhAQmPz+f+e677xg/Pz+5ttugWhTx8fHg8/nw9vaGqqoqPD090bx5c8TFxUnNFxsbC09PTxgYGEBbWxszZsxAbGwsxFV5xmoNkzXn7OxsBAQEoF27dlBRUcH48ePx6tUr/PvvvzUUedXJmjMAiMVizJ07F2PGjKmBSBVH1pyjo6MxatQoWFhYgM/nY+LEiVi3bh0kEkkNRV41sub7999/QyKRQCKRgGEYCAQCqKmp1VDU8gsNDcXevXsREBBQ4TzHjh2Dk5MTevXqBTU1NcyZMwcXL16Uq1XRoApFamoqjIyMpKYZGBggJSVFalpKSgo6duwoNU9eXh5ev36tlDgVSdacfX194eHhwb4/f/48dHR0oKenp5Q4FUnWnAEgLCwMnTp1Qv/+/ZUVXrWQNefExERoaGhg/Pjx6Nu3L6ZMmQJNTU2Zn0lfW8ia79ChQyEQCDBgwACYmpri7NmzWLp0qTJDVaiRI0ciNjYWPXv2rHCeT49fTZs2RZMmTZCamlrl7datvw455eXlQV1dXWqampoaCgoKpKbl5+dLfesoXSY/P7/6g1QwWXP+2PXr17F48WIsWrSozh1AANlzvn//Po4ePYp58+YpM7xqIWvO7969Q3R0NL799ltcvHgR3bt3R2BgIIqLi5UZrtxkzbewsBC9e/fG6dOncePGDdja2mLmzJlg6uhYqC1btgSPx+Oc59PjF1ByDJPn+FX3jgJyUFdXL/OHVFBQAA0NDalpampqEIlE7PvSHaypqVn9QSqYrDmXiomJgb+/P4KCguDu7q6MEBVOlpwLCgowf/58LF++vE7+Xj8l6+9ZKBRi2LBh6NmzJ4RCIWbMmIEXL16U29qqzWTNd/ny5TA3N0eHDh2gqamJRYsW4e7du3X6Qo3PqejLb0X/52XRoAqFoaFhmeZXamqqVDMNAIyMjKT+46SmpqJx48Zo2bKlUuJUJFlzBoCtW7di5cqV2LZtG0aMGKGsEBVOlpzv37+PtLQ0+Pv7w8LCAgEBAXj37h0sLCzw6tUrZYcsN1l/zwYGBigsLGTfMwzDvuoSWfN99eqVVL58Ph98Ph8qKipKibMmGBkZSe2brKwsvHv3rkxXXWU0qEJhbW2NwsJCREREoKioCIcOHUJGRkaZy8yGDRuG/fv348mTJ8jJycHmzZvh5uZWJ7thZM358OHDCA8Pxy+//AJra+sailYxZMnZwsICd+7cwc2bN3Hz5k2EhoaiSZMmuHnzJtq0aVOD0VeNrL9nDw8PxMTE4O7duygqKsLGjRvRoUMHdO7cuYYirxpZ87W3t8euXbuQlpaGwsJCrFu3Dp06dYKBgUENRV793NzccObMGdy8eRMikQjr169H//790bRp06qvVK5rpuqghw8fMmPGjGFMTU2Z4cOHM7dv32YYhmF8fX2ZkJAQdr7w8HDGwcGBMTc3Z2bNmsXk5eXVVMhykyVnFxcXplu3boypqanU6+nTpzUZepXJ+nsuFR8fX6cvj2UY2XOOiYlhBg8ezJiamjLjxo1jUlNTayhi+ciSr0gkYlatWsXY2toylpaWTEBAAPPy5cuaDFshPv17DQoKYoKCgtj3J06cYFxcXBgzMzNm8uTJTEZGhlzboyfcEUII4VT3+lIIIYQoFRUKQgghnKhQEEII4USFghBCCCcqFIQQQjhRoSCEEMKJCkUd5ejoCGNj4zIvCwsLmZafP38+Zs6cqfC4rl27ViamXr16YcSIEeWO3loVPj4+WLt2LYCSsXx++eUX9jNl5tW9e3f0798fK1asQFFRkczrOn36tNwDTIaFhSE0NLTM9O+++47dN/IoLi5GaGgoBg0ahB49esDGxgZz587Fy5cv5V63LIKDgzF69Gj2fXh4OPr06QNzc3P8/vvvMDY2lhpmpyKOjo6IiooCAOTm5uLQoUMybT88PBxbtmypWvD1kVx3YZAa4+DgwISFhTFv3ryResl6Y828efOYb775RuFxxcfHM507d2ZevHjBxvTs2TNm2bJlTPfu3Zlnz57JvY3s7Gz2uQNHjhxhbGxs2M/ev3/PvH//Xu5tfKq8vNLS0pjDhw8zPXr0YIKDg2Vaz4sXL5jOnTvLdSNjWloa4+joWOYm0C1btjCdO3dm1qxZU+V1l/rpp58YFxcXJi4ujklLS2Nu377NTJo0iXFwcFDKzac5OTlMdnY2wzAlz2Do2bMnExwczKSlpTEikYh58+aNTOvJzMxk8vPzGYZhmODgYGbUqFEyLScSiRgXF5c6ezOiolGLog7T0tJCixYtpF7NmjWr6bAAAM2bN2djat++PebPnw+hUIjz58/LvW4dHR12ID/mk/tFtbW1oa2tLfc2KvJxXvr6+hgxYgSGDx+Os2fPyrT8p/FWxc6dO+Hm5saOnpqVlQV/f3/s3bsXrVu3lnv9QMmQLtOnT0f//v2hr68PU1NTbNq0CW/evFFYy5CLpqYmdHR0AABFRUUQiUSwsLCAvr4+hEIhWrRoIdN6dHV12ZFUK7PvhUIhPDw8EBYWVvng6yEqFPXYzp074ezsjB49eqBv3774/vvvy+0iycnJwcyZM2FpaQkzMzMEBgZKPbDo9u3bGDNmDExMTDBo0CCEh4dX+oBXOgibqqoqu81ly5bB1taW3ebHg/GdOXMGrq6u6NmzJ5ydnREdHc1+Vtr1dO3aNSxYsAAZGRkwNjbGixcv2K6n3Nxc9OrVq8xBzcnJiV2XIvIqzUkgELDv4+LiMHLkSJiYmMDMzAy+vr7s/nRycgIAuLq64siRIwCACxcuwN3dHSYmJnB3d8fx48cr3FZOTg5iY2MxcOBAdlpycjJUVVURExODdu3aVTr+8vB4PFy5ckVq+HEtLS0cO3aMHU8pODgY06ZNw9KlS2FmZgYHBwfs379faj2HDx+Gi4sLevXqhZEjR+LKlSvsZxKJBFu3bsWAAQNgZmaGSZMm4dmzZ+y6R48ejRcvXrDPXpgwYQLmz5/PdgOWdj29evUKU6dOhZmZGfr164c1a9awD2Iq7Xo6cuQItmzZgjt37sDY2BgnT56EmZmZ1NDbjx8/Rrdu3ZCZmQmg5Hd1/PhxvH37ViH7tC6jQlFPxcbGIiwsDEFBQTh9+jSWLFmCmJgYnDp1qsy8mzZtwvPnz7F3714cOnQIHz58wA8//AAAyMjIgJ+fH5ydnXHs2DHMnTsXO3bskDov8Dk5OTlYu3YtioqKYGdnBwD4+uuvcevWLWzevBlRUVEQiUTw9/eHWCxGZmYmZs2ahf/+9784deoUvvrqKyxZsqTMs53NzMzw3XffQVdXF5cuXZL6Nq2pqQkHBwepfO/evYvXr19j8ODBCsmLYRhcvXoVR48exaBBgwAAaWlp+OqrrzB8+HCcPHkSO3bswIsXL7B161YAwMGDBwEA+/btg6urKx49eoSZM2diwoQJOH78OHx9ffH9999X+K39xo0bEAqF6NGjBzutT58+2LJli8JaEwAwceJEHDp0CPb29liwYAFiYmKQlZUFAwMDaGlpsfP98ccfyMzMxMGDBxEYGIhly5axrca4uDisXr0as2fPxtGjR/Gf//wH/v7+ePToEQBgy5YtiIyMRFBQEH799VdoaWlh6tSpUnG0bt2aXV9wcDAWLlwo9XlhYSEmTpyIoqIiREVFYcOGDYiNjcXu3bul5nN1dcWkSZPQvXt3XLp0CY6OjuDxeLhw4QI7z4kTJ2BjY8O2yjt16gQdHR1cu3ZNQXu17qq/Y+02AD/++CNWr14tNW3fvn3o1q0bWrVqhZUrV7JPbmvbti327t2Lp0+fllnPixcvoKGhAX19fWhpaWHVqlXIzs5m12dubg4/Pz8AwBdffIGMjAzs3LkT48aNqzA2KysrACUH0/z8fOjp6WHlypVo3749Hj16hCtXriA2NhZdunQBAKxbtw4ODg74448/0Lp1axQVFaFVq1Zo27Yt2rZtCz09PbRq1UpqG0KhENra2uDz+eV2Rbi5ueG7775DUVERVFVV8dtvv6Ffv37Q0dHBpk2b5MoLKDlINW/eHBMnToSvry+AkkerLliwgF2Hvr4+Bg8ezB5sdHV1AZQ8dUxNTQ27du2Ch4cHPD09AQDt27dHSkoKfv75ZwwYMKDM9u/fvw8jI6PPPrxGXv7+/mjfvj2ioqJw9OhRHDlyhH1E7ty5c9nta2pqYtWqVdDQ0EDHjh1x8+ZN7N+/H46OjggLC8PkyZPZIurj44OEhATs3bsXy5cvR1RUFKZNm8a2jr7//nvs2rULOTk5bBwCgQDNmzcHADRp0qRMt+KVK1fwzz//IDo6mh0ddcmSJfjw4YPUfGpqatDQ0ICKigr7t+Li4oKTJ0/C1dUVQEmhmDZtmtRyHTt2xL1799gcGioqFHVYQEAA3NzcpKaVfqu0srLCvXv3sGHDBqSkpODx48d49uwZzM3Ny6znyy+/RGBgIKytrWFpaYmBAweyj0VNTk7G5cuXYWZmxs4vFotRVFSEwsJCCIXCcmM7ePAgVFVVwePxoKmpKXXuJDk5GWpqamyRAEoOnAYGBkhOToajoyMcHR0xdepU6Ovrw8HBAR4eHpUeJrm0SF65cgUDBgzA6dOn2Sui5M0rLS0NS5YsgampKQICAtiupw4dOkBdXR1hYWF4/PgxUlJS8OjRI3Tv3r3c9T19+hSPHz9GTEwMO624uJgtKJ/KzMyUa7joo0ePYvHixex7f3//Cp+/PGTIEAwZMgQ5OTmIj49HTEwMdu/ejdatW2P8+PEAgK5du0o9EKdnz54IDw9nc7t79y7bmgJKzjeYmJggOzsbWVlZUo/0bN68eaWfNvj06VO0a9dOap983C3Hxd3dHYGBgcjJyUFKSgoyMjLg7OwsNY+Ojg7bFdWQUaGow3R1dfHFF1+U+9nhw4exbNkyeHp6wt7eHl9//TWWLFlS7rx9+/bFn3/+iQsXLiAuLg5r165FbGwsfvnlFxQXF2PIkCH4+uuvyyzH9fCXdu3aoVGjRuV+VtF0sVgMsVgMHo+HkJAQPHjwABcuXMD58+cRFRWFLVu2wMHBocJtfkooFMLFxQWnTp1C06ZNkZWVxZ4jkDevL774AqGhoRgxYgSaNm3Kdok8evQIXl5esLOzQ58+fTB27Fj88ccfFXZfiMViTJgwQepSUAAVPvuEz+ez/e9V4ejoiF69erHvmzRpUmaepKQkHDx4EEFBQQBKzk0MHDgQAwcOxNSpU3H58mW2UHx8bqY0n9JpYrEYc+fOLfM8cqFQyJ6rkpc867G2tkbjxo1x/vx5JCYmwsHBoczTDsVicZ18Do2i0R6op/bs2YMpU6YgKCgII0eOhKGhIZ4/f17uydo9e/bg1q1bcHd3x9q1a7Fr1y7cvn0b//zzD/u0vy+++IJ93b9/Hzt27KjyfyBDQ0MUFBRInXPIysrCs2fP2FbFihUr0L17d0ybNg1HjhyBlZUVzpw5U2Zdn+uCcXNzQ1xcHE6dOgVHR0f2268i8urYsSOmTZuGiIgIJCQkAAAOHDgAExMTbN68GT4+PjA3N5fa75/Ga2RkhLS0NKk4/vjjjwqv92/evDnbLVgVWlpaUtsqvbLoYxKJBJGRkbh+/Xq5y3/c2nn8+LHUCe979+7B2NiYze3Vq1dS2zt8+DB+//13aGtro1mzZnj48CG77Lt372BlZYXk5GSZ8+nQoQNevHiB9+/fs9P27duHSZMmlZn3033P5/MxdOhQXLhwgb2g4FPZ2dls11dDRoWinmrZsiXi4+ORnJyMpKQkfPvtt0hPT5d6LGSp169fY/ny5bh58ybS0tJw9OhRtGjRAi1btsS4ceOQkpKCH3/8ESkpKYiLi8OyZcvk6v4wMDCAk5MT5s+fj1u3biEpKQlz5sxBixYtMGDAADRp0gQHDx7Exo0bkZaWhuvXryMpKanc7hsNDQ3k5OQgOTlZ6oBVytLSEioqKoiKisLQoUPZ6YrKa+LEiTAwMMCyZcsgkUjQqlUrJCcn49atW3j+/DlCQkJw+vRpdr+XFqpHjx4hNzcXkyZNwtmzZxEWFoZnz57h6NGjWLt2bYUnprt3744nT57I1ar4nG7dusHZ2RnffPMNDh48iOfPnyMxMRFhYWH4/fff2dYEALx58wbLly9HSkoKfvnlF5w+fRo+Pj4AAD8/P+zbtw8HDhzA8+fPsWfPHuzYsQMdOnQAUNLluWXLFvz5559ISUlBUFAQWrVqBUNDQ5ljtbW1hb6+PhYuXIgnT57g6tWrCA0NZS+a+JiGhgYyMjKQlpbGTnN3d8e5c+fw9u3bcpd59OiRVPdYQ0WFop5auHAhiouLMWLECEyePBlaWloYN24cHjx4UGbeGTNmwMrKCtOnT4erqysePnyI7du3Q1VVFXp6eti5cyfu3LmD4cOHIygoCF5eXvjmm2/kim/lypUwNjbGlClT4O3tDU1NTUREREBdXR3NmzfH1q1b8eeff8LNzQ2zZs3CyJEj4e3tXWY9VlZW6NixI/7zn/8gMTGxzOd8Ph+urq5QVVWVOhAoKi9VVVUEBQXhwYMHOHDgAHx8fGBpaYnJkydj1KhR+Ouvv7Bw4UL8/fffyM3NRdOmTTFixAjMnTsXBw4cQI8ePbBp0yYcO3YMQ4cOxcaNGzF79mx4eXmVuz0rKyuIxWKpb+LVYf369fD29saePXvg7u6OcePG4dq1a9izZw+6du3KztetWzdIJBJ4eHggPDwca9euRZ8+fQAAzs7OWLhwIXbu3AlXV1fs33/hdJkAAAEvSURBVL8fq1evZruiJk2aBA8PDyxYsAAjR46ESCTCtm3bKnWiXiAQICQkBPn5+fD09MTcuXMxatQofPnll2XmdXFxgYqKCtzc3JCeng4A6NGjB1q3bo1BgwaV6cZ6+vQpcnNz6/yjgRWBnnBHSB2zZMkSqKmpYf78+TUaR3BwMC5evIgDBw7UaBzyKCoqgq2tLbZu3Vpm+JsNGzYgPT0dP/74Yw1FV3tQi4KQOsbX1xe//fYbcnNzazqUOu306dNYunQpWrZsWaZIiEQiHDt2jL18uqGjQkFIHdOuXTt4eXmxl6GSqtm0aRMuXbqEFStWlPnsl19+wfDhwyt1vqQ+o64nQgghnKhFQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDC6f8AlRjZvx7FW34AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.step(false_positive_rate_values, sensitivity_values, color='b', alpha=0.2,\n", " where='post')\n", "plt.xlabel('False Positive Rate (1 - Specificity)')\n", "plt.ylabel('Sensitivity')\n", "plt.title('words_list_model ROC Curve')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that as we move from left to right across the curve, sensitivity increases and the specificity decreases. Generally, the best classification threshold corresponds to high sensitivity and specificity (low false positive rate), so points in or around the northwest corner of the plot are preferable. \n", "\n", "Let's examine the four corners of the plot:\n", "* (0, 0): Specificity $=1$, which means all data points in the negative class are correctly labeled, but sensitivity $= 0$, so the model has no true positives. (0,0) maps to the classification threshold $C = 1.0$, which has the same effect as `ham_only` since no email can have a probability greater than $1.0$.\n", "* (1, 1): Specificity $=0$, which means the model has no true negatives, but sensitivity $= 1$, so all data points in the positive class are correctly labeled. (1,1) maps to the classification threshold $C = 0.0$, which has the same effect as `spam_only` since no email can have a probability lower than $0.0$.\n", "* (0, 1): Both specificity $=1$ and sensitivity $= 1$, which means there are no false positives or false negatives. A model with an ROC curve containing (0, 1) has a $C$ value at which it is a perfect classifier!\n", "* (1, 0): Both specificity $=0$ and sensitivity $= 0$, which means there are no true positives or true negatives. A model with an ROC curve containing (1, 0) has a $C$ value at which it predicts the wrong class for every data point!\n", "\n", "A classifier that randomly predicts classes has a diagonal ROC curve containing all points where sensitivity and the false positive rate are equal:" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/plain": [ "Text(0.5,1,'Random Classifier ROC Curve')" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEgCAYAAACq+TSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XlcFPX/B/DXLrJyyiF4IKKAgAen8vXEC5WUMFMxTdNSUcDMO4+UNLMsr1JQDLRUNDWPxDK1TCMr8UjxQkABEcSDQ5D72JnfH/tjcthdWGDZhd338/HgoczO7rzfu8u+9/2Zmc8IWJZlQQghhMghVHcAhBBCmjYqFIQQQmpEhYIQQkiNqFAQQgipERUKQgghNaJCQQghpEZUKDTE1KlT4eTkxPtxdnbGoEGDsHLlSuTn5zd6DBkZGXBycsKff/7Z6NuSJTMzE59++imGDRsGV1dX+Pj44Msvv8SLFy+4dS5fvgwnJyckJyerJKbq2ystLcXcuXPh6uoKLy8vHDlyBE5OTigrK2uU7Vb/6dmzJ/z9/XHu3Dmp+5SVlSEiIgKjR4+Gq6sr+vfvj6CgIFy5ckXmNsRiMQ4cOAB/f3/07NkT/fv3x+zZsxEXF6dQjIq8XqRpaKHuAIjy9O/fH/Pnz+d+Ly0txa1bt7B9+3bk5+cjLCxMjdE1rps3b2L27NmwsbHBBx98gHbt2iE5ORk7d+7ExYsXsX//fpiamqo8rh49euDw4cOwtrYGAJw/fx6//fYbPvnkEzg5OaFTp05wcHCASCRqlO1/9dVXsLKyAgCwLIusrCxERUVh3rx5OHLkCHr06AEAKCwsxIwZM5CWlob33nsPHh4eKCgowE8//YRp06Zh+fLleO+997jHLS8vx+zZs3Hv3j28++67WLhwIYqLi3Ho0CFMmTIF27dvx5AhQ+TG1VRfLyIHSzTCO++8wy5YsEDmbVu2bGGdnJzYwsLCRo0hPT2ddXR0ZGNiYhp1O9WVlJSww4YNY2fMmMFWVFTwbktJSWFdXFzY9evXsyzLsrGxsayjoyP74MEDlcZY5eDBg6yjoyPLMEyjbqemPAsLC1lnZ2f2iy++4JZ99NFHbJ8+fdhHjx5JrR8aGsp27dqVvXnzJrds06ZNrLu7O5uUlMRbVywWs++99x47YMAAtqysTGZsdXm9SNNAQ09awMjICIDkG2XVv7t27YKvry+cnZ3h6emJuXPn4tmzZ9x9nJycEB0djblz58Ld3R0DBgyQ6kgePHiAd999F+7u7nj99ddx584dqW1funQJkyZNgoeHBwYOHIiNGzeivLycu93b2xvfffcdli5dCg8PDwwYMABRUVF4+vQpZs6cCTc3N4wePRrXr1+Xm9/vv/+O9PR0LF26FC1a8JtkW1tbLF68GF26dJF5X0Weixs3bnA59O3bF8uWLeMNj1y4cAFjx47lhpPWrVuH0tJSAPyhp+XLl2P16tUAgK5duyI0NBTHjx+XGno6duwYRo4cCWdnZ/j6+uL06dPcbVXDe/v27cOgQYPQu3dvPHz4UO5zI4tIJOJ1MNnZ2Thx4gSCgoLQsWNHqfWDg4NhbW2NXbt2AZB0EwcPHsTEiRPh4ODAW1coFGLhwoUYO3YsCgsLZW6/Lq+XrKHCsrIyODk54fjx4wCA0NBQvP3221i9ejU8PDwwa9YseHt7Y926dbzHvnv3LpycnHD79m0AwLNnz7BgwQL06tULvXr1wocffkjDXnJQodAgLMuisrKS+ykoKEBMTAy+++47DBo0iCsYkZGRCAsLw5QpU/Dtt99i0aJFiI2NxYYNG3iPt27dOtjY2CA8PBy+vr4IDQ1FTEwMAKCgoADvvvsuysrK8NVXX2HixIlYuXIl7/7nz5/H9OnT0alTJ2zbtg0BAQH4/vvv8eGHH/LWCw0NhaGhIXbs2IE+ffrgs88+w/Tp09GnTx9s374dYrEYy5Ytk5v3pUuX0KZNGzg5Ocm8/d1334W/v7/M22p7LgoKChAUFIQOHTpgx44dCAkJwd9//41PPvkEAJCWloZ58+ahd+/eiIyMxPz583Hs2DGEhoZKbWvOnDkICAgAABw+fBgTJkyQWufw4cNYtWoVvL29ER4ejgEDBmDhwoU4f/48b72IiAisWrUKH330ETp37iz3uWEYhns/lJeXIz09HWvWrEFxcTFGjhwJALhy5QoqKysxcOBAmY+ho6ODYcOG4eLFiwCA27dvo6CgQO76rq6uWLx4MczNzWXe3pDXS55bt24hMzMTO3bswIwZMzBq1CicO3eO+3IEAGfPnoWNjQ1cXFxQXFyMadOmISkpCZ999hk+/fRT3LhxA4GBgRCLxXXatjagfRQa5PTp07xvnwBgaGiI1157DcuXL+eWPX/+HPPmzcOUKVMAAL1790ZKSgp+//133n29vLywdOlSAEDfvn1x5swZ/Pnnnxg8eDB+/PFHFBYWYseOHdwHQmVlJb788kvu/tu2bUO/fv24ZQMHDoSJiQmWLVuGe/fuoVu3bgCAzp07c9+0bWxscOrUKfTs2ROzZ88GAAQFBeHDDz9Efn4+TExMpPJ+/vw5Nw5fV7U9F8nJycjLy8M777wDDw8PAJIOLSUlBQBw584dlJeXY+bMmWjTpg369OkDkUiEyspKqW3Z2Nhw+yrc3d2lbmcYBtu2bcOECRO4533gwIHIy8vD1q1b4e3tza07fvx4+Pj41Jqfn5+f1DJHR0eEhYXBzc0NgGSnMgB06NBB7uNYW1ujuLgYeXl5eP78OQCgffv2tW5floa8XvJUVlZi5cqVXNE0MTHBrl27cPv2bbi6ugKQFApfX18AwI8//ojHjx/j7NmzXN7du3fHqFGjcOHCBQwfPlyp8TV3VCg0iJeXFxYsWACWZXH79m1s2LAB48aNw8qVKyEQCLj1Vq1aBUAy5JCcnIwHDx7g+vXrqKio4D1e1QcJAAgEArRt2xbFxcUAgOvXr8PZ2Zn3rXHEiBFcUSgqKkJCQgL3zbuKr68vli9fjn///ZcrFK9ux8LCAoDkj7ZK1U7NgoICmYVCKBTW+1tgbc9Fly5dYGpqiuDgYPj5+cHb2xsDBgzA4MGDAUi+PYtEIrz11lvc7aNHj4ZQWPdmPTU1FdnZ2Rg0aBCv0AwcOBAnT55EXl4et8zW1lahx9y2bRusrKxQXFyM8PBwpKamYuPGjejatSu3TtW3bh0dHbmPU3Uby7Lc/xmGUTy5VzTk9ZKnRYsWvGGz7t27o3Pnzvj111/h6uqKhIQEPHz4EKNGjQIAXL16FQ4ODmjbti33XFtbW8PGxgaxsbFUKKqhQqFBWrVqBRcXFwCSDzBDQ0MsW7YMlpaWCAwM5NZ78OABVq5cibi4OBgaGqJHjx5o2bIlr00HgJYtW/J+FwqF3DovX76EmZkZ7/bWrVtz/y8oKADLsrxlgGR83MjIiDd+bWhoKJWLvr6+wnlbWVnJ3D9S5cWLFzA0NJR5ZFFtz4WRkRGioqIQFhaGo0ePIioqCq1bt8bixYsxfvx4dOzYEd999x127tyJPXv2IDIyEh06dMDq1au5YqKoqvHx999/X+bt2dnZ0NPTAwC5wzrVdenSBfb29gAkXcz48eMxa9YsREdHc49R9Y36yZMnsLGxkfk4jx8/hr6+PszMzLhO4smTJzL3/TAMg+zsbLRp00bmYzXk9ZLH1NRUqtD5+vri1KlTWLJkCc6ePQs7OzuuQObl5SE+Pp476utVrxZRIkH7KDTYm2++if79+yMsLAxpaWkAJH/EwcHBEIlEOH36NP79919ERUWhZ8+edXpsExMT5OTk8Ja9eq6GkZERBAKB1DplZWVyO4P66t+/P7KyspCUlCTz9g0bNsDb21tqOEjR58LR0RHbtm3DlStXEBERgS5dumDVqlV4+vQpAMDT0xO7du3C5cuXsW3bNpiYmGDhwoW8nfaKaNWqFQBg/fr1OHr0qNRP1bBVfbVs2RJr1qzB8+fPsWnTJm55//79oaurKzX0WIVlWVy4cAGDBg0CIPm2bmpqin/++Ufm+v/++y8GDhyICxcuyLy9Lq9XVSf8avdSVFRUe7KQFIq0tDQkJSXh119/5YadAMlz7eHhIfN5Xrx4sUKPr02oUGi4FStWoLKyEps3bwYA5Obm4tGjR3j77bdhZ2cHgUAAhmFw6dKlOg0l/O9//8Pt27e5D0sA3M5OQFIonJyccObMGd79qvahVI33K8OgQYPQsWNHbNiwQaoY3L9/H6dPn8aIESOkjrBR5Lm4dOkS+vXrh9zcXIhEIgwePBiLFy8GwzB4/vw5Tpw4gWHDhqGiooLbHzRr1iwUFRXJPepHHjs7O5iamiI7OxsuLi7cT2JiIiIiIuo1nFWdp6cnRo0ahR9//BEJCQkAJN/GJ06cyA1NVbdr1y6kpKRw51Ho6Ohg0qRJOHToELevpgrDMAgLC4OpqSn69+8vM4a6vF4GBgYAwHuf1XQE3KscHBzg6OiI/fv348GDB7xC4eHhgbS0NNja2nLPs4ODA0JDQ7mjosh/aOhJwzk6OuLNN9/E8ePHcePGDbi7u6N9+/bYvXs3DA0NwTAMDh48iPj4eG5YQxFjx47F7t27MXv2bMybNw/Z2dlSR/p88MEHeP/997Fs2TL4+fkhNTUVX3/9NUaMGKHU9l4kEuGzzz5DYGAg3nnnHUyePBmWlpaIj49HZGQkrK2tZX5LbN26da3PhbOzMwQCAebNm4dZs2YBkBxxZG1tjW7dusHMzAzZ2dlYtGgRJk2ahJKSEoSHh6Nnz54KDw9VadGiBQIDA/H111+joqICvXr1QkJCAr766iu88cYbSjspb9GiRTh37hw2btyI3bt3c8uSkpIwceJEzJw5E+7u7igqKsKpU6fwyy+/YMmSJbxOKzAwEP/88w/efvttzJgxA66ursjNzcWBAwdw8+ZN7Ny5U2roskpdXq+uXbvC0tISX331FViWRU5ODkJDQxV+r44aNQqhoaFwdHTkhuAAycEAe/fuRUBAAGbOnAmRSIRvv/0Wt27d4h34QSSoo9ACCxYsgJ6eHjZs2ACBQIBt27ZBKBRi3rx5+Pjjj2FkZIQtW7agpKQEiYmJCj2mvr4+9u7di3bt2mHJkiXYvXs31qxZw1tn+PDhCAsLQ0JCAoKDg7F7926888472LJli9Jz7NOnDw4dOgQrKyts3rwZgYGB+OGHHzBu3Dh8//333KHBr1LkuTA2NkZkZCRatGiBxYsXY8GCBTA0NMTu3buhq6uLjh07Ijw8HM+ePcPcuXOxYsUKODg4YNu2bfXKY8aMGVi+fDlOnjyJgIAA7Nu3D9OnT8fHH3/c0KeIY2Njg0mTJuGvv/7C33//DQBcTu+//z5Onz6NoKAgrFy5EkVFRdi7dy9mzpzJewwDAwPs2bMHU6dOxcmTJzFnzhx8/vnnaNWqFQ4dOiT30Nkqir5eLVq0wNdffw1Acnjxt99+i88++0zhocvXX38dDMPwuglAMvQUFRUFS0tLLF++HIsWLQIA7N27F3Z2dgo9tjYRsNX3YBJCCCGvoI6CEEJIjahQEEIIqREVCkIIITWiQkEIIaRGGnd4bFZWQb3uJxAI0Lq1IXJyiqTOUNZUlDPlrKko57rnbGlpLPc26ij+n1AoeaKVcE5Ts0E5awfKWTs0Zs5a9DQSQgipDyoUhBBCakSFghBCSI2oUBBCCKmRWgrFrVu34OXlJff2n3/+GcOGDYO7uzsCAwORnZ2twugIIYS8SqWFgmVZHD16FDNmzJC6mlqVhIQErF69Glu2bEFsbCwsLCywYsUKVYZJCCHkFSotFDt37sS+ffsQFBQkd52ffvoJw4YNg5ubG/T09LBkyRJcvHiRugpCCFETlZ5wN378eAQFBeHKlSty10lJSeFd1MbMzAwmJiZITU3lrqdck/oeRywUCnj/agPKWTtQzprv9m0hWFYABwfA0FD5Oau0UMi7hu6rSkpKpC5Koq+vj5KSEoW20bq1IXf5xPowNZW+frOmo5y1A+WseQoLgcREoOpyK0+fAr16KT/nJjeFh56eHkpLS3nLSkpKuEsi1iYnp6jeHYWpqSHy8orAMNpxyj/lTDlrKm3IOS6O/0EnFArg7q5f75zNzaUv7lWlyRUKe3t73nV7c3NzkZ+fz7uMYU1YloVYXP/tMwwLsVgz31jyUM7agXLWDPn5wMOHQgD8vNzcGJiaArm5ys+5yZ1H4efnh19//RXXrl1DWVkZtmzZgkGDBsHMzEzdoRFCiNqwLHDzpvD/i8R/bGwYuLkxaMCIe62aREdRdT3gtWvXolu3bvj000+xcuVKZGVlwdPTE+vXr1dzhIQQoj5ZWQJkZkpXAjc3RiXb17hrZtd3mnEdHQHMzY2Qm1uoca2qPJQz5aypNCVnhpEc0VSdvT3D7cCu0tCca5pmvEl0FIQQQvgyMwXIylJfF/EqKhSEENKEiMXAnTvSXYSjIwN9fTUEBCoUhBDSZKSmCvDyJb+LaNkS6NpV9V3Eq6hQEEKImpWXA/fuSXcR3boxEInUEFA1VCgIIUSNEhKEKCvjL2vVioWtbdPZCU+FghBC1KCkBEhKku4inJ0Z6OioIaAaUKEghBAVu3lTukBYWrKwsmo6XcSrqFAQQoiKFBYCycnSRcLFhanXHHWqQoWCEEJUQFYXYWXFwtKyaXYRr6JCQQghjejFC+DRI+ki4erauPMzKRMVCkIIaQQsC9y6JV0gOnWSzPLanFChIIQQJXv+XIAnT5rG9BvKQIWCEEKUpC6T+DUnVCgIIUQJMjIEyMnhdxFCoeSIpuaOCgUhhDRAZSVw9650F+HkxEBPTw0BNQIqFIQQUk8pKQIUFPC7CH19yUyvmoQKBSGE1JG8Sfy6d2egq6uGgBoZFQpCCKmDe/eEKC/nLzMxYdG5c9M/ca6+qFAQQogCiouB+/ebxyR+ykaFghBCaiFr+o02bVi0b6+5XcSrqFAQQogcBQVASkrzm8RP2ahQEEKIDLK6iA4dWFhYaEcX8SoqFIQQ8orcXCA9vXlP4qdsVCgIIQTyJ/GztWXQqpUaAmpCqFAQQrTes2cCPH2qOZP4KRsVCkKI1pI3iV+XLgwMDdUQUBNFhYIQopXS0wXIzeV3ETo6kvMiCB8VCkKIVpE3iV/XrgxatlRDQM0AFQpCiNZIThagsJDfRRgYAA4O1EXUhAoFIUTjlZUBCQnaM4mfsqn03ML4+Hj4+/vD3d0dY8aMQVxcnMz1duzYgYEDB8LT0xMzZ85Eenq6KsMkhGiQ+HihVJEwNWXh5kZFQlEqKxRlZWUICgrCuHHjcPXqVUydOhXBwcEoKirirXf+/HmcOHECx44dw6VLl2BjY4OVK1eqKkxCiIYoLgbi4oSoqOAvd3Zm0KmT9p1d3RAqKxSxsbEQCoWYPHkydHV14e/vDwsLC8TExPDWe/jwIRiGAcMwYFkWOjo60NOUy0QRQlQiLk6Ie/f4y9q2lXQRmj7Ta2NQ2T6K1NRU2Nvb85bZ2toiJSWFt+z111/H4cOHMXjwYOjo6KBNmzY4ePCgwtsRCAT1mqxLKBTw/tUGlLN20KacX76UTOJXPWc3t6rpNzT3OWjM11llhaK4uBj6+vq8ZXp6eigtLeUtKy8vR8+ePfHNN9/A0tIS69evx8KFC3Hw4EEIFJhopXVrQ4XWk8fUVPvOsqGctYMm58yywPXrkv+bmPy33NlZHxYW6olJXRrjdVZZodDX15cqCqWlpTAwMOAtW7duHUaMGIHOnTsDAFatWoWePXsiKSkJTk5OtW4nJ6eo3h2Fqakh8vKKwDDaMX5JOVPOmiAnR3oSP6FQgMGD9ZGfX4TcXM3LWZaGvs7m5kZyb1NZobCzs8P+/ft5y1JTU+Hn58dblpmZifJXrjMoFAohFArRooViobIsC7G4/nEyDAuxWDveWFUoZ+2gaTnzJ/H7Ly9bWwZmZgIIBJqXsyIaI2eV7czu168fysvLERUVhYqKChw9ehTZ2dnw8vLirTdkyBDs3r0b6enpKC8vx+bNm+Hg4ABbW1tVhUoIaeKePhXInOnVzY1mem0MKusoRCIRIiMjsWbNGmzZsgWdOnVCeHg4DAwMEBAQAE9PTwQFBeGDDz5AZWUlJk+ezO2v2L59O4TadDkpQohMYjFw5470Z4GDA4Nqo9hEiQQsy2pUX5aVVVCv++noCGBuboTc3EKtaVUpZ8q5OXn0SIAXL/gHqrRoAfToIT39hqbkXBcNzdnS0ljubTSFByGkSauokJxdXR1N4qc6VCgIIU3W/ftCFBfzlxkZsbC3144uoamgQkEIaXLkTeLXowcDBQ+AJEpETzkhpEm5fVsIptpuB3NzFh07UhehLlQoCCFNQlER8OCBdBfh4sLU6yRaojxUKAghanfzpnQlaNeORdu21EU0BVQoCCFqk58PPHwoXSRcXasm8SNNARUKQojK8aff+E/HjgzMzdUQEKkRFQpCiEplZwvw+LF0u+DmRtetbqqoUBBCVIJhJEc0VWdnx8BY/knBpAmgQkEIaXRPngjw/Dl1Ec0VFQpCSKOhSfw0AxUKQkijePhQgPx8fhchEgHdulEX0dxQoSCEKJW8Sfy6dWMgEqkhINJgVCgIIUqTlCRESQl/mbExCzs7OnGuOaNCQQhpsNJSIDGRJvHTVPQSEkIaRNYkfhYWLDp0oC5CU1ChIITUS2EhkJxMk/hpAyoUhJA6kzWJX/v2LNq0oS5CE1GhIIQoLC8PSEujSfy0DRUKQkit5E3i16kTA1NTNQREVIoKBSGkRllZAmRm0vQb2owKBSFEJnmT+NnbMzAyUkNARG0ULhRisRg6OjqNGQshpInIzBQgK4vfRQgEkn0RRPsoXCgGDBiAUaNGYfTo0ejZs2djxkQIURN5k/g5OjLQ11dDQKRJUPho5/Xr16OgoAABAQHw9vbGli1bcP/+/caMjRCiQqmpAqkioacn2RdBRUK7KdxRDB06FEOHDkVpaSnOnz+PM2fOYNKkSejQoQPeeOMN+Pn5oV27do0ZKyGkEZSXA/fuSX9n7N6dga6uGgIiTU6dz5/U09ODr68v3nvvPUyYMAFpaWkIDw/HiBEjEBQUhPT09MaIkxDSCBIThVJFolUrFm5uVCTIf+pUKG7fvo0vv/wSQ4cOxfTp05GZmYmNGzfi0qVL+OOPP6Cjo4M5c+Y0VqyEECUpKZGcXV1ayl/u7MzA1pbOriZ8Cg89DR8+HJmZmejVqxfmzJmDkSNHwviVC922bt0ao0ePxurVqxslUEKIcsiafsPSkoWVFRUIIpvCHcWkSZNw/vx5REVFYcKECbwiUWXIkCG4ePGi3MeIj4+Hv78/3N3dMWbMGMTFxclc77fffsPIkSPh4eGBt956CwkJCYqGSQiRo6BAdpFwcWGoSJAaKVwoDh48CD09Panlz549Q9++fQFI9l+I5FzCqqysDEFBQRg3bhyuXr2KqVOnIjg4GEVFRbz14uPj8dFHH2HdunX4999/MXz4cMyfP78uORFCqvn3X+mZXq2sJPsiaKZXUpsah55++eUX/P777wCAzMxMrFq1Ci1btuSt8/jxY+gqsNcrNjYWQqEQkydPBgD4+/tj7969iImJga+vL7feoUOHMGHCBHh6egIApk+fjv79+4NhGAgVeEcLBIJ6vfGFQgHvX21AOWu+Fy+A9HQdGBvzc3Zzq5rETzOfB217nYHGzbnGQtG3b19uKIllWejq6kp1DM7Ozli5cmWtG0pNTYW9vT1vma2tLVJSUnjL4uPjMWTIEEybNg2JiYno3r07Pv74Y4WKBAC0bm0IQQOmsTQ1Naz3fZsrylnzsCxw/brk/1WjxMbG+rC3h1ZN4qfpr7MsjZFzjYXC3Nwc69evBwB06NABM2fOhH49z7wpLi6Wuq+enh5Kqx12kZ+fj0OHDiE8PBxOTk7Ytm0bgoOD8fPPP6OFAtdUzMkpqndHYWpqiLy8IjCMdozXUs6amfPz5/xJ/IRCwf8XCUnOublqDE5FtOF1rq6hOZuby5/Aq8ZP3r/++gt9+/ZFixYt4O7ujn///Vfuul5eXjUGoa+vL1UUSktLYWBgwFsmEokwYsQIuLi4AADmz5+PPXv2ICUlBY6OjjVuA5B0PmJxravJxTAsxGLteGNVoZw1A38Sv/9y69KFgY0NkJureTnXRhNf59o0Rs41FoqAgAD8/fffaN26NQICAuSuJxAIcO/evRo3ZGdnh/379/OWpaamws/Pj7fM1tYW5eXl3O8sy3I/hBDZMjIEyMnhD7kKhZIjmnR0tGecnjSOGgvFq4elNvQQ1X79+qG8vBxRUVGYNGkSoqOjkZ2dLdWJjB07FsuWLcMbb7yBbt264euvv0bnzp0V6iYI0TaVlcDdu9JjrU5ODGQcpEhIvSg8mr948WLExMSgsrKyXhsSiUSIjIzEqVOn0Lt3b+zfvx/h4eEwMDBAQEAAdu7cCQAYNmwYQkJCsGzZMvTu3Ru3bt3C9u3bG7SDmhBNlJIikCoS+vqSI5qoSBBlErAKjumsWLECv//+O4RCIXx8fODn54fevXs3dnx1lpVVUK/76egIYG5uhNzcQq0Z06Scm2fOdZ3ETxNyrivKue45W1pKn0RdReEpPNavX4/Kykr89ddfOHPmDN5//30YGBjA19cXo0ePRvfu3escGCGkbuLjhaio4C8zMWHRubN2fBgS9ajTpVBbtGiBIUOGYMiQIaioqEBUVBTCwsKwZ8+eWndmE0Lqr7gYuH9fuotwdmZAF54kja3O18y+e/cuzpw5gzNnziA7Oxve3t5SRy4RQpRH1vxMbdqwaN+eugiiGgoXik2bNuHs2bN48uQJBgwYgHnz5mH48OH1PgGPEFKzggIgJUW6SLi6Vk2/QYhqKFwobty4gRkzZmDkyJEwMzNrzJgI0WosC9y6JV0gOnRgYWFBXQRRPYULxYEDBxozDkIIgNxcID1duki4uTFqiIYQiRoLhZeXF3766SeYmZnVOkXHX3/9pdTACNEm8roIW1sGrVqpISBCXlFjoVi8eDEMDQ25/xNClO/ZMwGePpXe6UBdBGkqaiwUY8eO5f4vEAjg6+srNc14cXExfvjhh8aJjhANxp/E7z9dujAw1L7ZsUmQ0NRwAAAgAElEQVQTVmOheP78OXcFuhUrVqBTp04wrTaZfUJCArZs2YL33nuv0YIkRNOkpwuQm8vvInR0JOdFENLU1Fgo4uLiMG/ePG6epbffflvmeq92HoQQ+eRN4te1K4NqF48kpMmosVD4+Pjg/PnzYBgGw4cPx5EjR2Bubs7dLhAIYGBgINVlEEKkPXggQFERv4swNGTRpQsd8kqatloPj7WysgLQ8GnGCdFWZWVAQoJ0F9GjBwMFLtpIiNrV+DadNGkSIiIi0KpVK0yaNKnGBzp06JBSAyNEE9y9K0T1mfnNzFjY2FAXQZqPWs+j0P3/eYtrO4+CEPKfoiLgwQPpLsLFhanXNd0JUacaC8XcuXNl/p8QIp+sSfzatWPRti11EaR5Uvi7TWVlJaKiopCRkQEA2LJlC4YPH44FCxYgLy+v0QIkpLl4+VJ2kXB1ZahIkGZN4ULxxRdf4JtvvkFBQQF+//13fPvtt3jrrbeQnZ2NTz/9tDFjJKRJY1lJgUhN5f85dezIwM2NZnolzZ/Cx1ycPn0aoaGh6NatG3bv3o3+/ftj9uzZGDJkCCZPntyYMRLSZOXkCJCRIV0JaCpwokkU7iiKi4vRtm1bMAyDixcvYvDgwQAAHR0dCGnvHNEyVV1E9SJhZ0ddBNE8CncUPXr0QHh4OExNTVFQUIBhw4bh8ePH2LhxIzw8PBozRkKalKdPBXj2jCbxI9pD4UKxevVqfPjhh8jIyMCSJUvQrl07fP7558jKysLWrVsbM0ZCmgSxGLhzR7p7dnBgYGCghoAIURGFC4WDgwNOnDjBW7ZkyRKp2WQJ0USPHgnw4gW/i9DVBbp3py6CaL46TSBw//593L59G5WVlWBZ/uF+EydOVGpghDQFFRVAfDxN4ke0m8KFIiIiAlu2bIGJiQl3MaMqAoGACgXROLIm8TMyYmFvT+dEEO2icKHYs2cPFi9ejFmzZjVmPISoHU3iRwifwm/7kpISjBw5sjFjIUTtbt8Wgqm228HcnEXHjtRFEO2l8AkQPj4+OHnyZGPGQojaFBVJzouoXiRcXBgqEkTrKdxRGBsbIzw8HGfOnIGtrS03q2yVzZs3Kz04QlSBJvEjpGYKdxRFRUUYPXo0nJ2dYWhoCJFIxPtRRHx8PPz9/eHu7o4xY8YgLi6uxvWPHj2KPn36KBoiIXWSn0+T+BGiCIU7ivXr1zdoQ2VlZQgKCkJQUBAmTJiA6OhoBAcH49y5c1JHUQFAeno6vvjiC+jo6DRou4RUx7JAXJwQDMMvBjY2DMzM1BQUIU1YnSZpun//PkJCQjB16lQ8e/YM+/fvx99//63QfWNjYyEUCjF58mTo6urC398fFhYWiImJkVpXLBZj6dKldMgtUbrsbOD6denlbm5UJAiRR+GO4tKlSwgKCsLw4cMRFxeH8vJyPH/+HOvXr8fGjRvh6+tb4/1TU1Nhb2/PW2Zra4uUlBSpdSMiIuDg4IBBgwbh6NGjioYIQHJOR33mKBQKBbx/tYE25cwwwK1bQgiFAhgb/5ezvT0DY2MA0NznQJte5yqUs3IpXCi2bNmCpUuXYsqUKdwkgIsWLYK5uTm2b99ea6EoLi6Gvr4+b5menh5KS0t5y+7cuYOTJ0/i6NGjuHPnjqLhcVq3NoSgAVN3mppKD4NpOk3P+fFj4OlTwMTkv2XGxvro1Ut9MamDpr/OslDOyqFwobh//z4GDRoktXzYsGHYsmVLrffX19eXKgqlpaUweGU2tdLSUixfvhzr1q2Tud9CETk5RfXuKExNDZGXVyQ1dq2pND1nsVhyXsSrhEIB+vTRR1lZEXJzNS9nWTT9dZaFcq57zubmRnJvU7hQtG3bFomJiejYsSNveWxsLNq3b1/r/e3s7LB//37estTUVPj5+XG/37lzB+np6QgMDAQg2VdRUlICT09PnDx5ElZWVrVuh2VZiMWKZCQbw7AQi7XjjVVFE3N++FCA/HwBgP/yEokAZ2cG+vpASYnm5VwbTXyda0M5K4fChWL27NkICQnBo0ePwDAM/vzzTzx+/BgHDhzAqlWrar1/v379UF5ejqioKEyaNAnR0dHIzs6Gl5cXt46npydu3rzJ/X758mXMmzcPly9frmNaRFvJm8SvWzcGkqO4tWfMmhBlUbhQjB8/HhYWFoiMjIS+vj62bt0KOzs7bN68GcOHD6/1/iKRCJGRkVizZg22bNmCTp06ITw8HAYGBggICICnpyeCgoIalAzRbomJQlQb3YSxMQs7O+36RkmIsgnY6vOFV1NWVoaYmBh4eXlx+xOioqLw999/w9zcHNOmTUPXrl1VEqwisrIK6nU/HR0BzM2NkJtbqDWtqqbkXFoqKRLVyZrET1NyrgvKmXJWhKWlsdzbatztm5mZiVGjRmHx4sXIysoCAGzYsAGff/45dHR0IBaLMWXKFNy+fbvOQRGiDLduCaWKhIUFCzc3mumVEGWp8U9p69atsLW1xcmTJ2FkZITc3Fzs27cPI0aMwLZt2wBIznnYunUrdu3apZKACQGAwkIgOVn6e46LC1Ovo94IIfLV+Cf1119/Yf78+TAykhw2dfHiRYjFYrz55pvcOgMHDsSNGzcaN0pCXnHzplCqSLRvL+kiqEgQonw1dhQvX76EhYUF9/vly5eho6ODvn37csuMjIzAVJ+bmZBGkJcHpKXJnsSvAedYEkJqUWOhsLKyQmpqKqysrCAWi/Hnn3/C09OTd5Lc5cuXYW1t3eiBEu3FspJ9EdV16sTA1FQNARGiZWosFGPHjsW6devwwQcf4NKlS8jOzkZISAh3+7Vr1/D1119j8uTJjR4o0U5ZWQJkZkq3C25u1MUSoio1FopZs2bh5cuX+OSTTyAUCrFo0SK89tprAIB169Zh//798PHxoetoE6VjGOnpNwDJJH5G8mcaIIQ0glrPo5AnMTERDMOgW7duyo6pQeg8CsU11ZwfPxYgO5vfRQgEkn0RDdVUc25MlDPlrIiazqOo95HmTk5O9b0rITKJxcCdO9JdhKOjZH4mQoh60ClJpElISRGgoIDfRejpAU5OtC+CEHWjQkHUqrwcuHdPuovo3p2Brq4aAiKESKFCQdQmIUGIsjL+slatWNjaaseYMiHNBRUKonIlJUBSknQX4ezMQEdHDQERQmpEhYKo1M2b0gXC0pKFlRV1EYQ0VVQoiErQJH6ENF9UKEijk9VFWFmxsLSkLoKQ5oAKBWk0L14Ajx7RJH6ENHdUKIjSyZvEr3NnBiYmagiIENIgVCiIUj17JsDTpzSJHyGahAoFUQqaxI8QzUWFgjRYRoYAOTn8LkIolBzRRAhp/qhQkHqrrATu3pXuIpycGOjpqSEgQkijoEJB6kXWJH76+pKZXgkhmoUKBakTmsSPEO1DhYIo7N49IcrL+ctMTVl06kQnzhGiyahQkFoVFwP379MkfoRoKyoUpEaypt9o25ZFu3bURRCiLahQEJkKCoCUFJp+gxBChYLIIKuLsLZm0bo1dRGEaCMqFISTmwukp1MXQQjhU+mVAOLj4+Hv7w93d3eMGTMGcXFxMtfbsWMHhgwZAk9PT0ydOhVJSUmqDFPrsKyki6heJGxtGbi5UZEgRNuprFCUlZUhKCgI48aNw9WrVzF16lQEBwejqKiIt97x48cRHR2NqKgoxMbGol+/fggMDATD0IlcjeHJE9lDTW5uDFq1UkNAhJAmR2VDT7GxsRAKhZg8eTIAwN/fH3v37kVMTAx8fX259V68eIGgoCB07NgRADBt2jRs3boVT58+hZWVVa3bEQgE9bpimlAo4P2r6cRi4NYtHRgb83N2dGRgYAAAmvk8aNvrDFDO2qIxc1ZZoUhNTYW9vT1vma2tLVJSUnjLZs6cyfv9/PnzMDU1Rbt27RTaTuvWhhA0YKzE1NSw3vdtLh4+BHJyAGNjye/Gxvpo0QJwc1NrWCqlDa9zdZSzdmiMnFVWKIqLi6Gvr89bpqenh9LSUrn3uXLlClavXo21a9dCqGCbkJNTVO+OwtTUEHl5RWAYzTy6p7ISuHPnvydHKBTA2Fgf1tZF0NVlkZurxuBURBte5+ooZ8pZEebm8q8HoLJCoa+vL1UUSktLYSAZ55By4sQJfPLJJwgJCcHo0aMV3g7LshCL6x8nw7AQizXvjfXggQBFRQIA/+VmaMiiVy8gN1czc66Jpr7ONaGctUNj5Kyyndl2dnZITU3lLUtNTUWXLl2k1t2+fTvWr1+PHTt2YNy4caoKUSOVlUl2VkuKxH+6d2fg4KBdf0CEkPpRWaHo168fysvLERUVhYqKChw9ehTZ2dnw8vLirXfs2DHs3bsX33//Pfr166eq8DTSnTtCJCTwX2IzMxZubjTTKyFEcSorFCKRCJGRkTh16hR69+6N/fv3Izw8HAYGBggICMDOnTsBABERESgqKoK/vz88PDy4n+TkZFWF2uwVF0u6iOpDcM7ODGxsqIsghNSNgGVZjfrkyMoqqNf9dHQEMDc3Qm5uYbMe05R1TkS7dizatpXOSVNyrgvKmXLWVA3N2dLSWO5tNIWHhnj5EkhNpek3CCHKR4WimWNZ4NYt6QLRsSMDc3M1BEQI0ThUKJqxnBwBMjKk2wU3N5ruhBCiPFQomiF5XYSdHcOdbU0IIcpChaKZefpUgGfPqIsghKgOFYpmQizmT79RxcGhahI/QghpHFQomoG0NAHy8vhdhK6u5OxqQghpbFQomrCKCiA+XrqL6NqVQcuWagiIEKKVqFA0UffvC1FczF9mbMzCzk47Th4ihDQdVCiamNJSIDFRuovo0YNBC3q1CCFqQB89Tcjt20JUv+Jr69YsrK2piyCEqA8ViiagsBBITpbuIlxcmHpdhIkQQpSJCoWa1WUSP0IIUQcqFGqSnw88fEiT+BFCmj4qFComb/oNGxsGZmZqCIgQQmpBhUKFsrIEyMyk6TcIIc0LFQoVYBjJEU3V2dszMDJSQ0CEEFIHVCgaWWamAFlZ1EUQQpovKhSNRN4kfo6ODPT11RAQIYTUExWKRvDwoQD5+fwuQiQCunWjLoIQ0vxQoVCi8nLg3j3pLqJbNwYikRoCIoQQJaBCoSSJiUKUlvKX0SR+hBBNQIWigeRN4ufszEBHRw0BEUKIklGhaABZk/hZWLDo0IG6CEKI5qBCUQ80iR8hRJtQoagjWZP4tW/Pok0b6iIIIZqJCoWC8vKAtDSaxI8Qon2oUNRC3iR+nToxMDVVQ0CEEKJiVChq8Py5AE+e0PQbhBDtptJdr/Hx8fD394e7uzvGjBmDuLg4mevt2bMHAwcORM+ePbFkyRIUFxerMkwwjGRfRPUiYW/PUJEghGgdlRWKsrIyBAUFYdy4cbh69SqmTp2K4OBgFBUV8da7cOECdu/ejX379iEmJgb5+fnYsGGDqsLE48cCqZlehUJJF0EzvRJCtJHKCkVsbCyEQiEmT54MXV1d+Pv7w8LCAjExMbz1oqOj4e/vD1tbWxgbG2P+/PmIjo6GWCxu1PjEYuDffyE106uTEwMXF+oiCCHaS2X7KFJTU2Fvb89bZmtri5SUFN6ylJQUjBgxgrdOcXExnj17Bisrq1q3IxAI6nUuQ1KSELq6gFAoKRR6eiy6dmUBaO4hTVW5Vv2rDShn7UA5K5fKCkVxcTH0q82vraenh9JqEySVlJRAT0+P+73qPiUlJQptp3VrQwjqcbyqgwPw6BFgbKwPFxdo1SR+pqaG6g5B5Shn7UA5K4fKCoW+vr5UUSgtLYWBgQFvmZ6eHsrKyrjfqwqEoaFiyefkFNWro9DREcDDwxB5eUUoLNSOk+eEQgFMTSU5MwzlrKkoZ8pZEebm8nfCqqxQ2NnZYf/+/bxlqamp8PPz4y2zt7fnDUelpqaiVatWaNOmjULbYVkWDdmdwTAsxGLteGNVoZy1A+WsHRojZ5XtzO7Xrx/Ky8sRFRWFiooKHD16FNnZ2fDy8uKt98Ybb+Dw4cO4f/8+CgsLsW3bNvj5+UFIkygRQohaqOzTVyQSITIyEqdOnULv3r2xf/9+hIeHw8DAAAEBAdi5cycAwNvbG7NmzUJgYCCGDBkCY2NjLF26VFVhEkIIqUbAsqxG9WVZWQX1up+OjgDm5kbIzS3UmlaVcqacNRXlXPecLS2N5d5G4zmEEEJqRIWCEEJIjahQEEIIqREVCkIIITXSuJ3ZhBBClIs6CkIIITWiQkEIIaRGVCgIIYTUiAoFIYSQGlGhIIQQUiMqFIQQQmpEhYIQQkiNqFAQQgipERUKQgghNdK6QhEfHw9/f3+4u7tjzJgxiIuLk7nenj17MHDgQPTs2RNLlixBcXGxiiNVHkVz3rFjB4YMGQJPT09MnToVSUlJKo5UeRTNucrRo0fRp08fFUXXOBTN+bfffsPIkSPh4eGBt956CwkJCSqOVHnq8t4eOHAgPD09MXPmTKSnp6s4UuW7deuW1IXfXvXzzz9j2LBhcHd3R2BgILKzs+u/MVaLlJaWsgMHDmQPHDjAlpeXs0eOHGH79u3LFhYW8tY7f/486+XlxaakpLAvX75kAwIC2NWrV6sn6AZSNOdjx46xPj4+7KNHj9iKigp2+/bt7JAhQ1ixWKymyOtP0ZyrPHr0iO3Vqxfbu3dvFUeqPIrmfPfuXdbT05O9evUqKxaL2W+++Yb18fFRU9QNo2jOv//+OztixAj22bNnbHl5ObtmzRp26tSpaoq64RiGYY8cOVLje/bevXtsz5492bi4OLakpIT96KOP2ICAgHpvU6s6itjYWAiFQkyePBm6urrw9/eHhYUFYmJieOtFR0fD398ftra2MDY2xvz58xEdHQ1xQy7GrSaK5vzixQsEBQWhY8eOaNGiBaZNm4bMzEw8ffpUTZHXn6I5A4BYLMbSpUsxceJENUSqPIrmfOjQIUyYMAGenp4QCoWYPn06Nm/eDIZh1BR5/Sma88OHD8EwDBiGAcuy0NHRgZ6enpqibridO3di3759CAoKkrvOTz/9hGHDhsHNzQ16enpYsmQJLl68WO+uQqsKRWpqKuzt7XnLbG1tkZKSwluWkpKCLl268NYpLi7Gs2fPVBKnMima88yZMzF27Fju9/Pnz8PU1BTt2rVTSZzKpGjOABAREQEHBwcMGjRIVeE1CkVzjo+Ph4GBAaZNm4Y+ffpg9uzZMDQ0bJbXpFc059dffx06OjoYPHgw3N3dce7cOXzyySeqDFWpxo8fj+joaLi4uMhdp/pnmJmZGUxMTJCamlqvbTa/d0cDFBcXQ19fn7dMT08PpaWlvGUlJSW8bxxV9ykpKWn8IJVM0ZxfdeXKFaxevRqrVq1qlh8giuZ8584dnDx5EsuWLVNleI1C0Zzz8/Nx6NAhfPjhh7h48SJ69OiB4OBgVFZWqjJcpVA05/LycvTs2RNnz57F1atX4eXlhYULF4JtphNnt2nTBgKBoMZ1qn+GAZLPsfp+hjW/T4EG0NfXl3oTlZaWwsDAgLdMT08PZWVl3O9VT66hoWHjB6lkiuZc5cSJEwgMDERISAhGjx6tihCVTpGcS0tLsXz5cqxbt65Zvq7VKfo6i0QivPHGG3BxcYFIJML8+fORkZEhs9tq6hTNed26dejVqxc6d+4MQ0NDrFq1Crdu3WrWB2vURt4XYHl/97XRqkJhZ2cn1XqlpqbyWjQAsLe35/3hpKamolWrVmjTpo1K4lQmRXMGgO3bt2P9+vXYsWMHxo0bp6oQlU6RnO/cuYP09HQEBgbC09MTQUFByM/Ph6enJzIzM1UdcoMp+jrb2tqivLyc+51lWe6nuVE058zMTF7OQqEQQqEQLVq0UEmc6mBvb897bnJzc5Gfny81VKcorSoU/fr1Q3l5OaKiolBRUYGjR48iOztb6hCzN954A4cPH8b9+/dRWFiIbdu2wc/Pr1kOwyia87Fjx7B37158//336Nevn5qiVQ5Fcvb09MTNmzdx7do1XLt2DTt37oSJiQmuXbsGKysrNUZfP4q+zmPHjsWJEydw69YtVFRU4Ouvv0bnzp3h6OiopsjrT9GchwwZgt27dyM9PR3l5eXYvHkzHBwcYGtrq6bIG5+fnx9+/fVXXLt2DWVlZdiyZQsGDRoEMzOz+j1gvY+Xaqbu3bvHTpw4kXV3d2fHjBnD3rhxg2VZlp05cyYbHh7Orbd371526NChbK9evdhFixaxxcXF6gq5wRTJ2cfHh+3evTvr7u7O+3nw4IE6Q683RV/nKrGxsc368FiWVTznEydOsCNHjmTd3d3ZKVOmsKmpqWqKuOEUybmsrIz94osvWC8vL7Z3795sUFAQ+/jxY3WGrRTV37MhISFsSEgI9/upU6dYHx8f1sPDg501axabnZ1d723RpVAJIYTUqPmNpRBCCFEpKhSEEEJqRIWCEEJIjahQEEIIqREVCkIIITWiQkEIIaRGVCiaKW9vbzg5OUn9eHp6KnT/5cuXY+HChUqP6/Lly1Ixubm5Ydy4cTJnb62PqVOnYtOmTQAk8/h8//333G2qzKtHjx4YNGgQPvvsM1RUVCj8WGfPnm3wBJMRERHYuXOn1PKPPvqIe24aorKyEjt37sRrr70GZ2dn9O/fH0uXLsXjx48b/NiKCA0NxVtvvcX9vnfvXvzvf/9Dr1698Ntvv8HJyYk3zY483t7eOHjwIACgqKgIR48eVWj7e/fuRVhYWP2C10QNON+DqNHQoUPZiIgI9vnz57wfRU+qWbZsGbtgwQKlxxUbG8s6OjqyGRkZXExpaWns2rVr2R49erBpaWkN3saLFy+4aw4cP36c7d+/P3fby5cv2ZcvXzZ4G9XJyis9PZ09duwY6+zszIaGhir0OBkZGayjo2ODTmRMT09nvb29pU4CDQsLYx0dHdmNGzfW+7GrfPnll6yPjw8bExPDpqenszdu3GBnzJjBDh06VCUnnxYWFrIvXrxgWVZy/QUXFxc2NDSUTU9PZ8vKytjnz58r9Dg5OTlsSUkJy7IsGxoayk6YMEGh+5WVlbE+Pj7N+mREZaKOohkzMjKCpaUl76d169bqDgsAYGFhwcVkY2OD5cuXQyQS4fz58w1+bFNTU24iP7ba+aLGxsYwNjZu8DbkeTUva2trjBs3DmPGjMG5c+cUun/1eOtj165d8PPz42ZOzc3NRWBgIPbt24f27ds3+PEByZQu8+bNw6BBg2BtbQ13d3ds3boVz58/V1pnWBNDQ0OYmpoCACoqKlBWVgZPT09YW1tDJBLB0tJSoccxNzfnZlGty3MvEokwduxYRERE1D14DUSFQoPt2rULI0aMgLOzM/r06YOPP/5Y5hBJYWEhFi5ciN69e8PDwwPBwcG8CxbduHEDEydOhKurK1577TXs3bu3zh94VROw6erqcttcu3YtvLy8uG2+Ohnfr7/+Cl9fX7i4uGDEiBE4dOgQd1vV0NPly5exYsUKZGdnw8nJCRkZGdzQU1FREdzc3KQ+1IYNG8Y9ljLyqspJR0eH+z0mJgbjx4+Hq6srPDw8MHPmTO75HDZsGADA19cXx48fBwBcuHABo0ePhqurK0aPHo2ff/5Z7rYKCwsRHR2N4cOHc8uSk5Ohq6uLEydOoGPHjnWOXxaBQIB//vmHN/24kZERfvrpJ24updDQUMydOxeffPIJPDw8MHToUBw+fJj3OMeOHYOPjw/c3Nwwfvx4/PPPP9xtDMNg+/btGDx4MDw8PDBjxgykpaVxj/3WW28hIyODu+7Cu+++i+XLl3PDgFVDT5mZmZgzZw48PDwwYMAAbNy4kbsQU9XQ0/HjxxEWFoabN2/CyckJv/zyCzw8PHjTbiclJaF79+7IyckBIHmtfv75Z+Tl5SnlOW3OqFBoqOjoaERERCAkJARnz57FmjVrcOLECZw5c0Zq3a1bt+LRo0fYt28fjh49ioKCAnz66acAgOzsbAQEBGDEiBH46aefsHTpUkRGRvL2C9SmsLAQmzZtQkVFBQYOHAgA+OCDD3D9+nVs27YNBw8eRFlZGQIDAyEWi5GTk4NFixbhnXfewZkzZ/D+++9jzZo1Utd29vDwwEcffQRzc3P89ddfvG/ThoaGGDp0KC/fW7du4dmzZxg5cqRS8mJZFpcuXcLJkyfx2muvAQDS09Px/vvvY8yYMfjll18QGRmJjIwMbN++HQBw5MgRAMCBAwfg6+uLxMRELFy4EO+++y5+/vlnzJw5Ex9//LHcb+1Xr16FSCSCs7Mzt+x///sfwsLClNZNAMD06dNx9OhRDBkyBCtWrMCJEyeQm5sLW1tbGBkZcev98ccfyMnJwZEjRxAcHIy1a9dyXWNMTAw2bNiAxYsX4+TJk3jzzTcRGBiIxMREAEBYWBj279+PkJAQ/PjjjzAyMsKcOXN4cbRv3557vNDQUKxcuZJ3e3l5OaZPn46KigocPHgQX331FaKjo/Htt9/y1vP19cWMGTPQo0cP/PXXX/D29oZAIMCFCxe4dU6dOoX+/ftzXbmDgwNMTU1x+fJlJT2rzZfmzrOrBT7//HNs2LCBt+zAgQPo3r072rZti/Xr13NXbuvQoQP27duHBw8eSD1ORkYGDAwMYG1tDSMjI3zxxRd48eIF93i9evVCQEAAAKBTp07Izs7Grl27MGXKFLmx9e3bF4Dkw7SkpATt2rXD+vXrYWNjg8TERPzzzz+Ijo5G165dAQCbN2/G0KFD8ccff6B9+/aoqKhA27Zt0aFDB3To0AHt2rVD27ZtedsQiUQwNjaGUCiUORTh5+eHjz76CBUVFdDV1cXp06cxYMAAmJqaYuvWrQ3KC5B8SFlYWGD69OmYOXMmAMmlVVesWME9hrW1NUaOHMl92JibmwOQXHFMT08Pu3fvxtixY+Hv7w8AsLGxQUpKCr777jsMHjxYavt37tyBvb19rReuaajAwEDY2Njg4MGDOHnyJI4fP85dInfp0qXc9g0NDfHFF1/AwMAAXbp0wbVr13D48GF4e35pNEcAAAiiSURBVHsjIiICs2bN4oro1KlTERcXh3379mHdunU4ePAg5s6dy3VHH3/8MXbv3o3CwkIuDh0dHVhYWAAATExMpIYV//nnHzx58gSHDh3iZkZds2YNCgoKeOvp6enBwMAALVq04N4rPj4++OWXX+Dr6wtAUijmzp3Lu1+XLl1w+/ZtLgdtRYWiGQsKCoKfnx9vWdW3yr59++L27dv46quvkJKSgqSkJKSlpaFXr15Sj/Pee+8hODgY/fr1Q+/evTF8+HDusqjJycn4+++/4eHhwa0vFotRUVGB8vJyiEQimbEdOXIEurq6EAgEMDQ05O07SU5Ohp6eHlckAMkHp62tLZKTk+Ht7Q1vb2/MmTMH1tbWGDp0KMaOHVvnKZKriuQ///yDwYMH4+zZs9wRUQ3NKz09HWvWrIG7uzuCgoK4oafOnTtDX18fERERSEpKQkpKChITE9GjRw+Zj/fgwQMkJSXhxIkT3LLKykquoFSXk5NT/6miAZw8eRKrV6/mfg8MDJR77eVRo0Zh1KhRKCwsRGxsLE6cOIFvv/0W7du3x7Rp0wAA3bp1410Mx8XFBXv37uVyu3XrFtdNAZL9Da6urnjx4gVyc3N5l/O0sLCo89UGHzx4gI4dO/Kek1eH5WoyevRoBAcHo7CwECkpKcjOzsaIESN465iamnJDUdqMCkUzZm5ujk6dOsm87dixY1i7di38/f0xZMgQfPDBB1izZo3Mdfv06YM///wTFy5cQExMDDZt2oTo6Gh8//33qKysxKhRo/DBBx9I3a+mC7907NgRLVu2lHmbvOVisRhisRgCgQDh4eG4e/cuLly4gPPnz+PgwYMICwvD0KFD5W6zOpFIBB8fH5w5cwZmZmbIzc3l9hE0NK9OnTph586dGDduHMzMzLghkcTEREyaNAkDBw7E//73P7z99tv4448/5A5fiMVivPvuu7xDQQHIvfaJUCjkxt/rw9vbG25ubtzvJiYmUuskJCTgyJEjCAkJASDZNzF8+HAMHz4cc+bMwd9//80Vilf3zVTlU7VMLBZj6dKlUtcjF4lE3L6qhmrI4/Tr1w+tWrXC+fPnER8fj6FDh0pd7VAsFjfL69AoGz0DGmrPnj2YPXs2QkJCMH78eNjZ2eHRo0cyd9bu2bMH169fx+jRo7Fp0ybs3r0bN27cwJMnT7ir/XXq1In7uXPnDiIjI+v9B2RnZ4fS0lLePofc3FykpaVxXcVnn32GHj16YO7cuTh+/Dj69u2LX3/9VeqxahuC8fPzQ0xMDM6cOQNvb2/u268y8urSpQvmzp2LqKgoxMXFAQB++OEHuLq6Ytu2bZg6dSp69erFe96rx2tvb4/09HReHH/88Yfc4/0tLCy4YcH6MDIy4m2r6siiVzEMg/379+PKlSsy7/9qt5OUlMTb4X379m04OTlxuWVmZvK2d+zYMfz2228wNjZG69atce/ePe6++fn56Nu3L5KTkxXOp3PnzsjIyMDLly+5ZQcOHMCMGTOk1q3+3AuFQrz++uu4cOECd0BBdS9evOCGvrQZFQoN1aZNG8TGxiI5ORkJCQn48MMPkZWVxbskZJVnz55h3bp1uHbtGtLT03Hy5ElYWlqiTZs2mDJlClJSUvD5558jJSUFMTExWLt2bYOGP2xtbTFs2DAsX74c169fR0JCApYsWQJLS0sMHjwYJiYmOHLkCL7++mukp6fjypUrSEhIkDl8Y2BggMLCQiQnJ/M+sKr07t0bLVq0wMGDB/H6669zy5WV1/Tp02Fra4u1a9eCYRi0bdsWycnJuH79Oh49eoTw8HCcPXuWe96rClViYiKKioowY8YMnDt3DhEREUhLS8PJkyexadMmuTume/Togfv37zeoq6hN9+7dMWLECCxYsABHjhzBo0ePEB8fj4iICPz2229cNwEAz58/x7p165CSkoLvv/8eZ8+exdSpUwEAAQEBOHDgAH744Qc8evQIe/bsQWRkJDp37gxAMuQZFhaGP//8EykpKQgJCUHbtm1hZ2encKxeXl6wtrbGypUrcf/+fVy6dAk7d+7kDpp4lYGBAbKzs5Gens4tGz16NH7//Xfk5eXJvE9iYiJveExbUaHQUCtXrkRlZSXGjRuHWbNmwcjICFOmTMHdu3el1p0/fz769u2LefPmwdfXF/fu3cM333wDXV1dtGvXDrt27cLNmzcxZswYhISEYNKkSViwYEGD4lu/fj2cnJwwe/ZsTJ48GYaGhoiKioK+vj4sLCywfft2/Pnnn/Dz88OiRYswfvx4TJ48Wepx+vbtiy5duuDNN99EfHy81O1CofD/2rtjFcWhKAzAP2jA1kaw8AGUtIqFrQpRkaiBoAii+ARCUIKghRYiahC1sYiNYNoUYq2NhaWdhX3qIEGU7YTBJTC7Mzjunq/NhZw0+XPPveSC4zgwDPPhRfBVz8UwDJrNJo7HIzRNQ7FYRCgUQrVahSAIOBwOkGUZ5/MZpmnC7XYjk8lAkiRomgaWZaEoCnRdRyKRwGg0Qq1WgyiKv71fOBzG7Xb78CX+HQaDAfL5PFRVRSqVQqFQwH6/h6qq8Pv9j3GBQAD3+x08z2OxWKDf7yMYDAIAotEoZFnGfD4Hx3FYrVbo9XqPVlS5XAbP82g0Gshms7AsC9Pp9FML9Q6HA7PZDJfLBblcDpIkQRAElEqlp7GxWAxOpxPJZBKGYQAAWJaF1+tFPB5/amOdTieYpvn2RwN/BTrhjpA302q14HK5UK/XX1rHeDzGdruFpmkvreNvXK9XRCIRTCaTp9/fDIdDGIaBbrf7oup+DppREPJmKpUK1us1TNN8dSlvbbPZoN1uw+PxPIWEZVnQdf2xffp/R0FByJvx+XwQRfGxDZX8GUVRsNvt0Ol0nq4tl0uk0+lPrZf8y6j1RAghxBbNKAghhNiioCCEEGKLgoIQQogtCgpCCCG2KCgIIYTY+gXYMMkHI6pc9AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.step(np.arange(0, 1, 0.001), np.arange(0, 1, 0.001), color='b', alpha=0.2,\n", " where='post')\n", "plt.xlabel('False Positive Rate (1 - Specificity)')\n", "plt.ylabel('Sensitivity')\n", "plt.title('Random Classifier ROC Curve')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Intuitively, a random classifier that predicts probability $p$ for input $x$ will result in either a true positive or false positive with chance $p$, so sensitivity and the false positive rate are equal. \n", "\n", "We want our classifier's ROC curve to be high above the random model diagnoal line, which brings us to the AUC metric." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AUC\n", "\n", "The **Area Under Curve (AUC)** is the area under the ROC curve and serves as a single number performance summary of the classifier. The AUC for `words_list_model` is shaded below and calculating using scikit-learn's [AUC function](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score):" ] }, { "cell_type": "code", "execution_count": 91, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/plain": [ "Text(0.5,1,'words_list_model ROC Curve')" ] }, "execution_count": 91, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEhCAYAAABhpec9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xlcjfn7P/DXOdVRp1ISso4KzVhSSiplWhRSTITkk62ojGEsYxkayzCMnVCyjJQpxJRtMJZpbFmGbNlrCDO0kPZT59y/P/p2/xzV3alOJ3Wu5+NxHpz7vO/7vq67uq/zvpf3zWMYhgEhhBBSCX59B0AIIeTTRoWCEEIIJyoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkHkZt68eRg5cqTclhcSEoK+ffuy701MTBAdHS3TvP/++y/Gjx+PoqIiucUjT9XdVocOHYKJiUml+Vy5cgUmJiblXr169YKXlxdOnz5dbp6ioiKEh4fDw8MDpqamsLW1RWBgIK5evVrhOsRiMfbu3QsvLy/06tULtra2mDx5MpKSkmTK4dWrV/jxxx/h7OwMU1NTuLq64ueff8bbt29l3g6kfqjWdwCEyGrfvn1o3769TG0vX76My5cv13FEn57169ejTZs2AACGYZCeno7IyEhMmzYNBw4cQLdu3QAAubm5mDhxIp49e4bx48fD3NwcOTk5OHLkCMaOHYt58+Zh/Pjx7HJFIhEmT56M+/fvY9y4cZgxYwby8/MRExODMWPGYMuWLXBwcKg0rlu3bmHy5Mno0KEDvvnmGxgYGODp06cICwvD+fPnERUVBV1d3brcNKQWqFCQBsPMzKy+Q/jkmZiYwNjYWGpa3759YW1tjaNHj7KFYsWKFXj+/DliY2Oliq+Liws2b96Mn3/+Gb169YKpqSmA0t7drVu3sH//fnTu3Jlt7+zsDD8/PyxcuBBnz56FQCAoF1NhYSFmzZqF7t27Y9u2bVBVLd3tWFtbw9bWFkOHDkVYWBjmzZsn9+1B5IMOPTUyQ4YMwaJFi9j3Dx48gImJCbZt28ZOO3PmDHr06IH8/HwAwPHjx+Hp6YmePXvC2dkZ27dvx4c37JuYmCA8PBwDBgyAubk5rly5AoZhsG3bNtjb28Pc3BzLly+HWCyWiuXcuXPw9PSEqakp7OzssGzZMhQWFtY4tw8PPZWUlGDlypXo168fevTogSFDhuD48eMASg/TzJ8/HwBgamqKQ4cOybT8kJAQjB49Gvv374eTkxPMzMzw9ddf4/3799i0aROsra1hY2ODTZs2Sc137949TJgwAZaWlrC2tkZwcDBycnLYz2XZViKRCKtWrYKdnR1MTU3h6+uLBw8e1HhbfUggEEjtwDMyMhAXF4fAwMAKe2hBQUFo164dduzYwcYWHR2NUaNGSRUJAODz+ZgxYwY8PT2Rm5tb4frPnDmDtLQ0zJkzhy0SZQwNDTFr1ix06tQJwP8/hPb06VO2TVFREUxMTNifY9nPadGiRTA3N8ekSZPg5OSEZcuWSS373r17MDExwZ07dwAAr1+/xrfffgsLCwtYWFjgu+++o8NeMqIeRSNjZ2eHP//8k31fdrz5xo0b7LRLly7BwsICQqEQUVFRWLZsGcaNG4dZs2YhKSkJGzduxNu3bzFnzhx2ntDQUPzwww9QUVGBqakptm/fjpCQEEybNg0mJiaIiIjAlStX2G+sz549w7Rp0+Dj44N58+bh+fPn+Omnn9CkSRN89913tc5z27Zt+O233/D999/DwMAAR44cwcyZM9G5c2c4ODggKCgIoaGhiIqKKvcNm8uDBw8QFRWFhQsXIiMjA0uWLIGXlxeMjIywZs0anDlzBlu2bIGtrS0sLS1x9+5djB49GjY2NlizZg3S09Oxfv16PH78GHv37oWKikqV2woA5s+fj4SEBMyYMQNt27ZFZGQkxo4di8OHD8PAwEDm+CUSCUpKStj/v379GmFhYcjPz8fAgQMBlP5OlJSUwN7evsJlqKiowNnZGfv27QMA3LlzBzk5OZW2NzU1ZXseFbl8+TJatmwJExOTCj8fN26czPmVuX37NrS0tLB161YAwIULF3Ds2DEsWLAAPB4PAHDy5El06NCB/VI0duxYqKioYPny5ZBIJFi3bh0CAgIQHR0NFRWVasegTKhQNDL29vbYuXMnMjMz0bx5c1y/fh2ff/45bt68CYZhwOPxcPHiRYwYMQJisRghISHw8vJiv4Hb2dmBx+MhNDQU/v7+0NPTAwA4OTnB09MTQOkOaNeuXfD19cXkyZMBAH369IGjoyMbx927dyESieDn54eWLVuiT58+EAgE7E6stm7cuIHu3btj6NChAIDevXtDW1sbYrEYenp66NChA4DSnViTJk1kXm5+fj7WrFmDLl26AACOHDmC+/fvIy4uDkKhEDY2Njh48CDu3r0LS0tLhIaGom3btggNDWV3NoaGhhgzZgzOnTsHJyenKrfV48ePcfToUaxbtw6DBw8GUPpzGDBgAHbu3IkFCxbIHL+7u3u5aV26dMHmzZvRs2dPAKUnlQGgbdu2lS6nXbt2yM/Px7t37/DmzRsAQOvWrWWO40Nv3rxhz5vIS0lJCRYsWICOHTsCAHR0dLBjxw7cuXOHLVonT56Em5sbAOC3337Dy5cvcfLkSTbvrl27YtCgQTh37hz69+8v1/gaGzr01MhYWFhAQ0MD165dAwBcv34dEyZMQHZ2Np4+fYr//vsPqampsLe3R0pKCt69e8d+0yzj5uaG4uJi3Lp1i51maGjI/j81NRVv375Fv3792Gnq6upS3zhNTU0hEAgwcuRIrFmzBjdu3ICHhweGDx8ulzwtLS1x4cIFjBs3DlFRUXj9+jXmzp2Lzz//vFbL1dDQYIsEAOjr66Njx44QCoUASr9ta2trs4eW/v77b7i6ukp9I7W0tESLFi3w999/y7Styn5WdnZ2KCkpYYupra0trly5Uq34N23ahNjYWOzZswc2NjYwMDDA6tWr4ezszLYpO6zI9S267DOGYdj/SySSasVShs/nlzvUVluqqqpSh826du2Kjh074tSpUwBKe4b//PMPBg0aBKB0G3fu3BmtWrVit3G7du3QoUMHJCYmyjW2xogKRSMjEAjQu3dvXL16FU+fPsX79+8xYMAAtGnTBjdu3MDFixdhYGCALl26IDs7G0DpzvBDzZs3BwCpY85l0wCw8zVr1qzC+QCgffv2+OWXX9CpUyfs3r0bo0ePRv/+/ZGQkCCXPCdPnoy5c+fizZs3+PHHH+Ho6IhJkyYhMzOzVsvV1NQsN01DQ6PS9u/fv5fKu0zz5s2Rm5sr07Z69+4dAMDKygrdunVjX/v370d6enq14u/UqRN69OiBPn36YNu2bdDW1sakSZOQlZXFtin7Rv3vv/9WupyXL19CQ0MDzZo1Y3sSlbWXSCRsr6Mibdq0wX///Vfp52/fvoVIJOLM62O6urrlCp2bmxtbKE6ePAkjIyP2i8O7d++QnJwstX27deuGf/75p9rbWBlRoWiE7OzscPXqVVy/fh1du3aFhoYGLCwscOPGDVy6dAl2dnYASrvrQOnJzQ+VvS/7/GNl0z/c+QD/v4CUsbS0xI4dO3DlyhVs2rQJOjo6mDFjRrV3ChVRUVHBxIkT8fvvv+P06dOYNWsWrl69io0bN9Z62dXRtGnTCotTZmYmdHR0ZNpWTZs2hUAgQGxsbLnXrl27ahxbkyZNsHjxYrx58wZr1qxhp9va2kJNTQ1nzpypcD6GYXDu3Dm2F9S1a1fo6uri0qVLFbb/+++/YW9vj3PnzlX4ua2tLdLT0/Ho0aMKP1+1ahWcnJxQUlLCnl/4sPeSl5dXdbIoLRTPnj3Do0ePcOrUKfawE1C6jc3NzSvcxrNmzZJp+cqMCkUjZGdnhydPnuDMmTOwsLAAUHoM/9q1a7h8+TJ72MPIyAi6uro4ceKE1Py///47e9K6IkZGRmjRogX77Q0oPWb8YRc+Li4Ozs7OKC4uhqamJgYMGIBJkyYhLy+v0qtjqiMwMBArVqwAUNp7mTRpEnr37s1+c+XzFfOr3atXL5w6dUrq0Mr169eRnp4OMzMzmbaVubk5RCIRxGIxevTowb7i4+PL/Wyqy9LSEoMGDcJvv/3GXkWlq6uLUaNGITQ0FKmpqeXm2bFjB1JSUtj7KFRUVODt7Y2YmBikpKRItZVIJNi8eTN0dXVha2tbYQz9+vVD+/btsWrVqnLnqB4/fozff/8dLi4uUFVVZQ/xfdgD+fBCDC6dO3dGly5dEBUVhSdPnkgVCnNzczx79gyGhobs9u3cuTNCQkLYq6JI5ehkdiNkbGwMAwMDJCQkYNSoUQBKdxhlVy2V/UGrqKhgypQpWLFiBTQ1NdGvXz8kJSUhNDQUvr6+ld4AxePx8PXXX2Pp0qVo1qwZzM3NERMTg4yMDPYwloWFBTIyMjBz5kx4e3ujoKAAoaGh6NWrF3uCvDZ69eqFTZs2wcDAAF27dsX9+/eRmJjIXhrctGlTAKVFz9bWFi1btqz1OisSGBgIHx8fBAUFYcyYMUhPT8eGDRvQo0cPODg4yLStunXrBkdHR0ybNg3ffPMN2rdvj1OnTmHv3r1SPYGamjlzJk6fPo3Vq1dj586d7LRHjx5h1KhR8PPzg5mZGfLy8nDs2DEcP34cs2fPRq9evdhlBAQE4NKlSxg9ejQmTpwIU1NTZGVlYe/evbh16xbCwsIqvWhAIBBg+fLlCAgIwP/+9z/4+PigRYsWSE5Oxvbt29GuXTv2W/3nn3+OFi1aYP369WAYBpmZmQgJCYG6urpMuQ4aNAghISHo0qWL1NVuw4cPR0REBPz9/eHn5weBQIBdu3bh9u3bdP+GDKhQNFJ2dnaIjY1l/9iNjY2hp6eHjh07sjtRoPTSxCZNmuCXX37B3r170bp1a8yYMQMTJ07kXP7o0aMhFouxa9cu7Nq1Cy4uLhg5ciRu374NoPRbfmhoKDZs2ICpU6dCVVUV9vb27NVVteXv74/CwkJERkbizZs3MDAwwMyZMzFixAgAgI2NDaytrbFw4UJMnz4dkyZNkst6P2ZqaopffvkFa9euxdSpU6GtrQ0XFxfMnj2bvWegqm0FlN5RvX79emzcuBHv379Hx44dsXr1anh4eNQ6xg4dOsDb2xuRkZG4ePEi+vbtC01NTezcuRPR0dH47bffEBYWBnV1dfTs2RMRERGwsrKSWoZQKMTu3bvxyy+/4PDhwwgLC4NQKESPHj0QExODHj16cMbQp08fxMTEIDw8HGvXrsXbt2/RunVrDBs2DIGBgdDS0gJQepJ6w4YN+OmnnzBlyhQYGhpi+fLlmDt3rky5Dh48GBs3bpTqTQClXxwiIyOxatUqtjB0794dERERMDIyknVTKi0ePQqVEEIIF+pREIWTSCRVXmrJ5/Pldp5B0esjpLGhQkEUbsuWLdi8eTNnG09PT6xcubJBro+QxoYOPRGFe/36Ned190DpfQft2rVrkOsjpLGhQkEIIYQTHZQlhBDCqdGdo0hPz6m6UQV4PB6aN9dEZmYelKWTRTlTzo0V5Vz9nFu00K70M+pR/B8+v3RDK9OFL5SzcqCclUNd5qxEm5EQQkhNUKEghBDCiQoFIYQQTlQoCCGEcKqXQnH79m32mQgVOXr0KJydnWFmZoaAgIByz0sghBCiOAotFAzDIDY2FhMnTkRxcXGFbR48eIBFixZh3bp1SExMhL6+vtxGHCWEEFJ9Ci0UYWFh2LNnDwIDAyttc+TIETg7O6Nnz55QV1fH7Nmzcf78eepVEEJIPVHoDXfDhw9HYGAgrl69WmmblJQUmJubs++bNWsGHR0dpKamlnu2c0Vqeh0xn8+T+lcZUM7KgXJWFjxIJHWTs0ILhSxPGSsoKCj3NCsNDQ0UFBTItI7mzTXZ5+7WhK6uZo3nbagoZ+VAOTduGRlAVhagry//nD+5ITzU1dVRWFgoNa2goIB9lm5VMjPzatyj0NXVxLt3eZBIlOOWf8qZcm6slDHn7GwedHRqnrOenlaln31yhcLY2Fjqge9ZWVnIzs6Wev4tF4Zh8MFz7qtNImEgFivHL1YZylk5UM6NW9mzueoi50/uPgp3d3ecOnUK169fR1FREdatW4d+/fqhWbNm9R0aIYQopU+iR/HDDz8AAJYuXYovvvgCP/74IxYsWID09HRYWlpixYoV9RwhIYTUH4YBqjpNW1hYdyfuG92Di2o6zLiKCg96elrIyspVmq4q5Uw5N1aNLef8fODvv/kQCCpvU1zMg6urEEVFNcuZa5jxT6JHQQghhJtAAHx0QaiU0mHG62bdn9w5CkIIIZ8WKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBNdHkvkTpabg+qLikrpJYb5+ajVUC8NCeVc39HUXlFR/a6fCgWRu4KCqm8Oqi98Pg86OkB2Nl9pBoujnBtHzk2a1N+6qVCQOlHVzUH1hc8vjauo6P8PotbYUc71HU3DR+coCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcKJCQQghhBMVCkIIIZxorKcGRp4js9bVCJv1PdIlIUS+qFA0MPIcmbUuR9isz5EuCSHyRYWiAZLXyKw0wiYhRBZ0joIQQggnKhSEEEI4UaEghBDCiQoFIYQQTlQoCCGEcFJooUhOToaXlxfMzMwwdOhQJCUlVdhu69atsLe3h6WlJfz8/JCWlqbIMAkhhHxAYYWiqKgIgYGBGDZsGK5duwZfX18EBQUhLy9Pqt3Zs2cRFxeHgwcP4vLly+jQoQMWLFigqDAJIYR8RGGFIjExEXw+Hz4+PlBTU4OXlxf09fWRkJAg1e6ff/6BRCKBRCIBwzBQUVGBujxuGviEMUzp3dGyvOiuZ0KIoinshrvU1FQYGxtLTTM0NERKSorUtMGDB2Pfvn348ssvoaKigpYtWyI6OlpRYdaL6t5tTXc9E0IUSWGFIj8/HxoaGlLT1NXVUVhYKDVNJBKhV69e2LZtG1q0aIEVK1ZgxowZiI6OBo/Hq3I9PB4P/Br0k/h8ntS/ilQ65hJPLndbV0d95lxfKGflQDnLl8IKhYaGRrmiUFhYCKFQKDVt2bJlcHFxQceOHQEACxcuRK9evfDo0SOYmJhUuZ7mzTVlKiiV0dXVrPG8NaWuDujoyGdYjprQ1taoulEjQzkrB2XKuWz3Whf7MIUVCiMjI0RFRUlNS01Nhbu7u9S0V69eQSQSse/5fD74fD5UVWULNTMzr8Y9Cl1dTbx7lyf3AfKqkp9fOjCfos8/8Pk8aGtrICenQOE51xfKmXJurEQiHgCNGu/D9PS0Kv1MYYXCxsYGIpEIkZGR8Pb2Rnx8PDIyMmBnZyfVzsHBATt37oS9vT1atWqFtWvXonPnzjA0NJRpPQzD1GrIbImEgVisuF+sshPZEglTbwPzla5bOf6YylDOykGZci7bf9TFPkxhVz0JBAJs374dx44dg5WVFaKiohAaGgqhUAh/f3+EhYUBAL755hu4urrCx8cH9vb2eP78ObZs2QJ+TboJDUBBAXD3Lp9OUBNCPlk8hmEaVblNT8+p0XwqKjzo6WkhKytXoT2K/Hzgzh1+vZyfKH0ehRDZ2flK862LcqacGyuRiAcHByEKC2u2D2vRQrvSzxrn13RCCCFyQw8uqidljzSlG+gIIZ86KhT15MOb7Oj8BCHkU0aFoh7J65GmhBBSl+gcBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkEIIYQTjfUkB2UjwVYHjRpLCGkoqFDIwYcjwVYHjRpLCGkIqFDICY0ESwhprOgcBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhJHOhEIvFdRkHIYSQT5TMhaJv375YsmQJbty4UZfxEEII+cTIXChWrFiBnJwc+Pv7w8nJCevWrcPjx4/rMrZPGsMA+fmlL7rLmhDSmMl8w52joyMcHR1RWFiIs2fP4sSJE/D29kbbtm0xZMgQuLu7w8DAoC5j/aR8fDc23WVNCGmsqn0yW11dHW5ubhg/fjxGjBiBZ8+eITQ0FC4uLggMDERaWlpdxPlJKrsbW10d4PHqOxpCCKkb1SoUd+7cwc8//wxHR0dMmDABr169wurVq3H58mX8+eefUFFRwZQpU+oqVkIIIfVA5kNP/fv3x6tXr2BhYYEpU6Zg4MCB0NbWZj9v3rw5PDw8sGjRojoJlBBCSP2QuUfh7e2Ns2fPIjIyEiNGjJAqEmUcHBxw/vz5SpeRnJwMLy8vmJmZYejQoUhKSqqw3R9//IGBAwfC3NwcI0eOxIMHD2QNkxBCiJzJXCiio6OhXsHwqK9fv4a1tTWA0vMXgkrG2i4qKkJgYCCGDRuGa9euwdfXF0FBQcjLy5Nql5ycjO+//x7Lli3D33//jf79+2P69OnVyYkQQogccR56On78OM6cOQMAePXqFRYuXIgmH13e8/LlS6ipqVW5osTERPD5fPj4+AAAvLy8EBERgYSEBLi5ubHtYmJiMGLECFhaWgIAJkyYAFtbW0gkEvD5dCM5IYQoGmehsLa2Zg8lMQwDNTW1cj2G7t27Y8GCBVWuKDU1FcbGxlLTDA0NkZKSIjUtOTkZDg4OGDt2LB4+fIiuXbvihx9+kLlI8Hg81KSe8Pk8qX+roqJS2rYh167q5twYUM7KgXKWL85CoaenhxUrVgAA2rZtCz8/P2hoaNRoRfn5+eXmVVdXR2FhodS07OxsxMTEIDQ0FCYmJti0aROCgoJw9OhRqKpWfe69eXNN8GpxraqurqZM7dTVAR2dxvGwIm3tmv1MGzLKWTkoU85lu1JZ92HVwbnnvXDhAqytraGqqgozMzP8/ffflba1s7PjXJGGhka5olBYWAihUCg1TSAQwMXFBT169AAATJ8+Hbt370ZKSgq6dOnCuQ4AyMzMq3GPQldXE+/e5UEiYapsn58PZGfzG/Rd2Xw+D9raGsjJKZAp58aAcqacGyuRiAdAQ+Z92Mf09LQq/YyzUPj7++PixYto3rw5/P39K23H4/Fw//59ziCMjIwQFRUlNS01NRXu7u5S0wwNDSESidj3DMOwL1kwDIPajF8okTAQi6tel1hc2lYiqfm6PhWleSjHH1MZylk5KFPOZfsiWfdh1cFZKD68LLW2l6ja2NhAJBIhMjIS3t7eiI+PR0ZGRrmeiKenJ+bOnYshQ4bgiy++wIYNG9CxY0eZehOEEELkT+aDNLNmzUJCQgJKSkpqtCKBQIDt27fj2LFjsLKyQlRUFEJDQyEUCuHv74+wsDAAgLOzM4KDgzF37lxYWVnh9u3b2LJlS63OOxBCCKk5HiPjMZ358+fjzJkz4PP5cHV1hbu7O6ysrOo6vmpLT8+p0XwqKjzo6WkhKyu3wm4bw5QOBFimqAh49IjfoE9m8/k86OgIkZ2drzTdc8qZcm6sRCIeHByEKCyseB9WlRYtyt9EXUbmITxWrFiBkpISXLhwASdOnMDXX38NoVAINzc3eHh4oGvXrtUOrCH5eLRYgEaMJYQoB5kLBQCoqqrCwcEBDg4OKC4uRmRkJDZv3ozdu3dXeTK7MSgbLZYQQpRJtQoFANy7dw8nTpzAiRMnkJGRAScnp3JXLhFCCGk8ZC4Ua9aswcmTJ/Hvv/+ib9++mDZtGvr371/jG/AIIYQ0DDIXips3b2LixIkYOHAgmjVrVpcxEUII+YTIXCj27t1bl3EQQgj5RHEWCjs7Oxw5cgTNmjWrcoiOCxcuyDUwQgghnwbOQjFr1ixoamqy/yeEEKJ8OAuFp6cn+38ejwc3N7dyw4zn5+dj//79dRMdIYSQesdZKN68ecM+gW7+/Pn47LPPoKurK9XmwYMHWLduHcaPH19nQRJCCKk/nIUiKSkJ06ZNY8dZGj16dIXtPux5EEIIaVw4C4WrqyvOnj0LiUSC/v3748CBA9DT02M/5/F4EAqF5XoZhBBCGo8qL49t06YNgNoPM04IIaRh4iwU3t7eCA8PR9OmTeHt7c25oJiYGLkGRggh5NNQ5X0Uampq7P8JIYQoH85CMXXq1Ar/TwghRHnI/IS7kpISREZG4sWLFwCAdevWoX///vj222/x7t27OguQEEJI/ZK5UKxcuRLbtm1DTk4Ozpw5g127dmHkyJHIyMjAjz/+WJcxEkIIqUcyDwr4+++/IyQkBF988QV27twJW1tbTJ48GQ4ODvDx8anLGOvNh48/LSqq31gIIaS+yFwo8vPz0apVK0gkEpw/fx7Tpk0DAKioqIDPl7lj0qB8/PhTevQpIUQZyVwounXrhtDQUOjq6iInJwfOzs54+fIlVq9eDXNz87qMsV7R408JIcpO5q7AokWLcPfuXcTExGD27NkwMDBAREQE0tPTERwcXJcxEkIIqUcy9yg6d+6MuLg4qWmzZ88uN5osIYSQxkXmQgEAjx8/xp07d1BSUgKGYaQ+GzVqlFwDI4QQ8mmQuVCEh4dj3bp10NHRYR9mVIbH41GhIISQRkrmQrF7927MmjULkyZNqst4CCGEfGJkPpldUFCAgQMH1mUshBBCPkEyFwpXV1ccPny4LmP5ZDAMkJ9PN9kRQghQjUNP2traCA0NxYkTJ2BoaMiOKltm7dq1cg+uvnx4ox3dZEcIUXYy9yjy8vLg4eGB7t27Q1NTEwKBQOoli+TkZHh5ecHMzAxDhw5FUlISZ/vY2Fj06dNH1hDlquxGu/97CiwhhCgtmXsUK1asqNWKioqKEBgYiMDAQIwYMQLx8fEICgrC6dOny11FBQBpaWlYuXIlVFRUarVeQgghtVOtQZoeP36M4OBg+Pr64vXr14iKisLFixdlmjcxMRF8Ph8+Pj5QU1ODl5cX9PX1kZCQUK6tWCzGnDlz6JJbQgj5BMhcKC5fvgwvLy/k5+cjKSkJIpEIb968weTJk3H8+PEq509NTYWxsbHUNENDQ6SkpJRrGx4ejs6dO6Nfv36yhkcIIaSOyHzoad26dZgzZw7GjBnDDgI4c+YOKETbAAAgAElEQVRM6OnpYcuWLXBzc+OcPz8/HxoaGlLT1NXVUVhYKDXt7t27OHz4MGJjY3H37l1Zw2PxeDzUZDBbPp/H/quiUvpvIx0Ul/VhzsqCclYOlLN8yVwoHj9+XOE3fGdnZ6xbt67K+TU0NMoVhcLCQgiFQqn38+bNw7Jlyyo8byGL5s01wavFGWhdXU0IBICOjvKMGqutrVF1o0aGclYOypRz2e5VV7dm+04uMheKVq1a4eHDh2jfvr3U9MTERLRu3brK+Y2MjBAVFSU1LTU1Fe7u7uz7u3fvIi0tDQEBAQBKz1UUFBTA0tIShw8fRps2bapcT2ZmXo17FLq6mnj3Lg+5uQyys/mN/j4KPp8HbW0N5OQUQCJhqp6hEaCcKefGSiTiAdDAu3d5NcpZT0+r0s9kLhSTJ09GcHAwnj9/DolEgr/++gsvX77E3r17sXDhwirnt7GxgUgkQmRkJLy9vREfH4+MjAzY2dmxbSwtLXHr1i32/ZUrVzBt2jRcuXJF1jDBMAzEYpmblyORMBCLGUgkDCSSmi+nISnNVTn+mMpQzspBmXIu21+V7cPkSebv3sOHD8fKlStx9uxZaGhoYOPGjbhx4wbWrl2LESNGVDm/QCDA9u3bcezYMVhZWSEqKgqhoaEQCoXw9/dHWFhYrRIhhBBSN3jMx+OFf6SoqAgJCQmws7NjzydERkbi4sWL0NPTw9ixY/H5558rJFhZpKfn1Gg+FRUe9PS0kJWVi5wcBnfu8Bv9OQo+nwcdHSGys/OV5lsX5Uw5N1YiEQ8ODkIUFubWqEfRooV2pZ9x9ihevXqFQYMGYdasWUhPTwcArFq1Cj/99BNUVFQgFosxZswY3Llzp9pBEUIIaRg4z1Fs3LgRhoaGOHz4MLS0tJCVlYU9e/bAxcUFmzZtAlB6z8PGjRuxY8cOhQRMCCFEsTh7FBcuXMD06dOhpVV6Nvz8+fMQi8X46quv2Db29va4efNm3UapQAxDo8YSQsiHOAvF+/fvoa+vz76/cuUKVFRUYG1tzU7T0tKCpBFdHlRQANy9y6dRYwkh5P9wFoo2bdogNTUVQOk9DX/99RcsLS2lbpK7cuUK2rVrV7dRKphAQKPGEkJIGc5C4enpiWXLluH48eNYvHgxMjIyMHr0aPbz69evY8OGDRg0aFCdB0oIIaR+cJ7MnjRpEt6/f48lS5aAz+dj5syZGDBgAABg2bJliIqKgqurKz1HmxBCGrEq76OozMOHDyGRSPDFF1/IO6Zaqe19FC9e5CIpidfo76EAlPNac8qZcm6s6vI+CpmH8PiYiYlJTWclhBDSgDTygbQJIYTUFhUKQgghnKhQEEII4USFghBCCCcqFIQQQjhRoSCEEMKJCgUhhBBOVCgIIYRwokJBCCGEExUKQgghnKhQEEII4USFghBCCCcqFIQQQjhRoSCEEMKJCgUhhBBOVCgIIYRwokJBCCGEExUKQgghnKhQEEII4USFghBCCCcqFIQQQjgptFAkJyfDy8sLZmZmGDp0KJKSkipst3XrVjg4OMDS0hK+vr549OiRIsMkhBDyAYUViqKiIgQGBmLYsGG4du0afH19ERQUhLy8PKl2hw4dQnx8PCIjI5GYmAgbGxsEBARAIpEoKlRCCCEfUFihSExMBJ/Ph4+PD9TU1ODl5QV9fX0kJCRItXv79i0CAwPRvn17qKqqYuzYsXj16hX+++8/RYVKCCHkA6qKWlFqaiqMjY2lphkaGiIlJUVqmp+fn9T7s2fPQldXFwYGBjKth8fjgV+D8sfn89h/S1/VX0ZD82HOyoJyVg6Us3wprFDk5+dDQ0NDapq6ujoKCwsrnefq1atYtGgRli5dCr6Me+7mzTXB49V8Q+nqakJHB1BXr/EiGhxtbY2qGzUylLNyUKacy3alurqacl+2wgqFhoZGuaJQWFgIoVBYYfu4uDgsWbIEwcHB8PDwkHk9mZl5Ne5R6Opq4t27PGRn81BUVP1lNDR8Pg/a2hrIySmARMLUdzgKQTlTzo2VSMQDoIF37/JqlLOenlalnymsUBgZGSEqKkpqWmpqKtzd3cu13bJlC/bs2YOtW7fCxsamWuthGAZicc3jlEgYSCSAMp07L81ZOf6YylDOykGZci7bZ0kkDMRi+eassCPxNjY2EIlEiIyMRHFxMWJjY5GRkQE7OzupdgcPHkRERAR+/fXXahcJQggh8qewQiEQCLB9+3YcO3YMVlZWiIqKQmhoKIRCIfz9/REWFgYACA8PR15eHry8vGBubs6+nj59qqhQCSGEfIDHMEyj6pelp+fUaD4VFR709LTw4kUukpJ4SnEym8/nQUdHiOzsfKXpnlPOlHNjJRLx4OAgRGFhbo0OPbVooV3pZ0pwESghhJDaoEJBCCGEExUKQgghnKhQEEII4USFghBCCCcqFIQQQjhRoSCEEMKJCgUhhBBOVCgIIYRwokLxAYaBUowaSwgh1UGF4gMFBcCdO3w0aVLfkRBCyKeDCsVHBAKgFs89IoSQRocKBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCOFGhIIQQwokKBSGEEE4KLRTJycnw8vKCmZkZhg4diqSkpArb7d69G/b29ujVqxdmz56N/Px8RYZJCCHkAworFEVFRQgMDMSwYcNw7do1+Pr6IigoCHl5eVLtzp07h507d2LPnj1ISEhAdnY2Vq1apagwCSGEfERhhSIxMRF8Ph8+Pj5QU1ODl5cX9PX1kZCQINUuPj4eXl5eMDQ0hLa2NqZPn474+HiIxWJFhUoIIeQDCisUqampMDY2lppmaGiIlJQUqWkpKSno1KmTVJv8/Hy8fv1aIXESQgiRpqqoFeXn50NDQ0Nqmrq6OgoLC6WmFRQUQF1dnX1fNk9BQYFM6+HxeODXqPzxkJUFFBfzajJzg8Tn81BYCIhEPEgk9R2NYlDO9R2NYihjziUlpfsuPl/++zCFFQoNDY1yRaGwsBBCoVBqmrq6OoqKitj3ZQVCU1NTpvU0b64JHq/6G0oiKX3176+BGszewGlU3aTRoZyVg3LlrKEBCIWy7SurQ2GFwsjICFFRUVLTUlNT4e7uLjXN2NhY6nBUamoqmjZtipYtW8q0nszMvBr1KPh8HvT1NfHuXR4kEqb6C2iA+HwedHUp58aOclaenIXCmuesp6dV6WcKKxQ2NjYQiUSIjIyEt7c34uPjkZGRATs7O6l2Q4YMwaJFizBgwAC0bt0amzZtgru7O/gy7v0ZhkFtzntLJAzEYuX4xSpDOSsHylk51EXOCjuZLRAIsH37dhw7dgxWVlaIiopCaGgohEIh/P39ERYWBgBwcnLCpEmTEBAQAAcHB2hra2POnDmKCpMQQshHeAzDNKpym56eU6P5VFR40NPTQlZWrtJ8A6GcKefGinKufs4tWmhX+hkN4UEIIYQTFQpCCCGcqFAQQgjhRIWCEEIIJyoUhBBCODW6q54IIYTIF/UoCCGEcKJCQQghhBMVCkIIIZyoUBBCCOFEhYIQQggnKhSEEEI4UaEghBDCiQoFIYQQTkpXKJKTk+Hl5QUzMzMMHToUSUlJFbbbvXs37O3t0atXL8yePRv5+fkKjlR+ZM1569atcHBwgKWlJXx9ffHo0SMFRyo/suZcJjY2Fn369FFQdHVD1pz/+OMPDBw4EObm5hg5ciQePHig4Ejlozq/1/b29rC0tISfnx/S0tIUHKn83b59u9xD3z509OhRODs7w8zMDAEBAcjIyKjdChklUlhYyNjb2zN79+5lRCIRc+DAAcba2prJzc2Vanf27FnGzs6OSUlJYd6/f8/4+/szixYtqp+ga0nWnA8ePMi4uroyz58/Z4qLi5ktW7YwDg4OjFgsrqfIa07WnMs8f/6csbCwYKysrBQcqfzImvO9e/cYS0tL5tq1a4xYLGa2bdvGuLq61lPUNSdrvmfOnGFcXFyY169fMyKRiFm8eDHj6+tbT1HXnkQiYQ4cOMD5+3r//n2mV69eTFJSElNQUMB8//33jL+/f63Wq1Q9isTERPD5fPj4+EBNTQ1eXl7Q19dHQkKCVLv4+Hh4eXnB0NAQ2tramD59OuLj4yGuzTNW64msOb99+xaBgYFo3749VFVVMXbsWLx69Qr//fdfPUVec7LmDABisRhz5szBqFGj6iFS+ZE155iYGIwYMQKWlpbg8/mYMGEC1q5dC4lEUk+R14ys+f7zzz+QSCSQSCRgGAYqKipQV1evp6hrLywsDHv27EFgYGClbY4cOQJnZ2f07NkT6urqmD17Ns6fP1+rXoVSFYrU1FQYGxtLTTM0NERKSorUtJSUFHTq1EmqTX5+Pl6/fq2QOOVJ1pz9/Pzg6enJvj979ix0dXVhYGCgkDjlSdacASA8PBydO3dGv379FBVenZA15+TkZAiFQowdOxZ9+vTB5MmToampKfMz6T8VsuY7ePBgqKio4Msvv4SZmRlOnz6NJUuWKDJUuRo+fDji4+PRo0ePStt8vP9q1qwZdHR0kJqaWuP1NqzfjlrKz8+HhoaG1DR1dXUUFhZKTSsoKJD61lE2T0FBQd0HKWey5vyhq1evYtGiRVi4cGGD24EAsud89+5dHD58GHPnzlVkeHVC1pyzs7MRExOD7777DufPn0e3bt0QFBSEkpISRYZba7LmKxKJ0KtXL5w8eRLXrl2DnZ0dZsyYAaaBjoXasmVL8Hg8zjYf77+A0n1YbfZfDW8vUAsaGhrlfpEKCwshFAqlpqmrq6OoqIh9X7aBNTU16z5IOZM15zJxcXEICAhAcHAwPDw8FBGi3MmSc2FhIebNm4dly5Y1yJ/rx2T9OQsEAgwZMgQ9evSAQCDA9OnT8eLFiwp7W58yWfNdtmwZLCws0LFjR2hqamLhwoW4fft2g75QoyqVffmt7G9eFkpVKIyMjMp1v1JTU6W6aQBgbGws9YeTmpqKpk2bomXLlgqJU55kzRkAtmzZghUrVmDr1q0YNmyYokKUO1lyvnv3LtLS0hAQEABLS0sEBgYiOzsblpaWePXqlaJDrjVZf86GhoYQiUTse4Zh2FdDImu+r169ksqXz+eDz+dDVVVVIXHWB2NjY6ltk5WVhezs7HKH6qpDqQqFjY0NRCIRIiMjUVxcjNjYWGRkZJS7zGzIkCHYt28fHj9+jNzcXGzatAnu7u4N8jCMrDkfPHgQERER+PXXX2FjY1NP0cqHLDlbWlri1q1buH79Oq5fv46wsDDo6Ojg+vXraNOmTT1GXzOy/pw9PT0RFxeH27dvo7i4GBs2bEDHjh3RpUuXeoq8ZmTN18HBATt37kRaWhpEIhHWrl2Lzp07w9DQsJ4ir3vu7u44deoUrl+/jqKiIqxbtw79+vVDs2bNar7QWl0z1QDdv3+fGTVqFGNmZsYMHTqUuXnzJsMwDOPn58eEhoay7SIiIhhHR0fGwsKCmTlzJpOfn19fIdeaLDm7uroyXbt2ZczMzKReT548qc/Qa0zWn3OZxMTEBn15LMPInnNcXBwzcOBAxszMjBkzZgyTmppaTxHXjiz5FhUVMStXrmTs7OwYKysrJjAwkHn58mV9hi0XH/++BgcHM8HBwez7Y8eOMa6uroy5uTkzadIkJiMjo1broyfcEUII4dTwjqUQQghRKCoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkEIIYQTFYoGysnJCSYmJuVelpaWMs0/b948zJgxQ+5xXblypVxMPXv2xLBhwyocvbUmfH19sWbNGgClY/n8+uuv7GeKzKtbt27o168fli9fjuLiYpmXdfLkyVoPMBkeHo6wsLBy07///nt229RGSUkJwsLCMGDAAHTv3h22traYM2cOXr58WetlyyIkJAQjR45k30dERKB3796wsLDAH3/8ARMTE6lhdirj5OSE6OhoAEBeXh5iY2NlWn9ERAQ2b95cs+Abo1rdhUHqjaOjIxMeHs68efNG6iXrjTVz585lvv32W7nHlZiYyHTp0oV58eIFG9OzZ8+YpUuXMt26dWOePXtW63W8ffuWfe7AoUOHGFtbW/az9+/fM+/fv6/1Oj5WUV5paWnMwYMHme7duzMhISEyLefFixdMly5danUjY1paGuPk5FTuJtDNmzczXbp0YVavXl3jZZf5+eefGVdXVyYhIYFJS0tjbt68yUycOJFxdHRUyM2nubm5zNu3bxmGKX0GQ48ePZiQkBAmLS2NKSoqYt68eSPTcjIzM5mCggKGYRgmJCSEGTFihEzzFRUVMa6urg32ZkR5ox5FA6alpYUWLVpIvZo3b17fYQEA9PX12Zg6dOiAefPmQSAQ4OzZs7Vetq6uLjuQH/PR/aLa2trQ1tau9Toq82Fe7dq1w7BhwzB06FCcPn1apvk/jrcmduzYAXd3d3b01KysLAQEBGDPnj1o3bp1rZcPlA7pMm3aNPTr1w/t2rWDmZkZNm7ciDdv3sitZ8hFU1MTurq6AIDi4mIUFRXB0tIS7dq1g0AgQIsWLWRajp6eHjuSanW2vUAggKenJ8LDw6sffCNEhaIR27FjB1xcXNC9e3f06dMHP/zwQ4WHSHJzczFjxgxYWVnB3NwcQUFBUg8sunnzJkaNGgVTU1MMGDAAERER1d7hlQ3Cpqamxq5z6dKlsLOzY9f54WB8p06dgpubG3r06AEXFxfExMSwn5Uderpy5Qrmz5+PjIwMmJiY4MWLF+yhp7y8PPTs2bPcTs3Z2ZldljzyKstJRUWFfZ+QkIDhw4fD1NQU5ubm8PPzY7ens7MzAMDNzQ2HDh0CAJw7dw4eHh4wNTWFh4cHjh49Wum6cnNzER8fj/79+7PTnj59CjU1NcTFxaF9+/bVjr8iPB4Ply5dkhp+XEtLC0eOHGHHUwoJCcHUqVOxZMkSmJubw9HREfv27ZNazsGDB+Hq6oqePXti+PDhuHTpEvuZRCLBli1b8OWXX8Lc3BwTJ07Es2fP2GWPHDkSL168YJ+9MG7cOMybN489DFh26OnVq1eYMmUKzM3N0bdvX6xevZp9EFPZoadDhw5h8+bNuHXrFkxMTHD8+HGYm5tLDb396NEjdO3aFZmZmQBKf1ZHjx7Fu3fv5LJNGzIqFI1UfHw8wsPDERwcjJMnT2Lx4sWIi4vDiRMnyrXduHEjnj9/jj179iA2NhY5OTn48ccfAQAZGRnw9/eHi4sLjhw5gjlz5mD79u1S5wWqkpubizVr1qC4uBj29vYAgG+++QY3btzApk2bEB0djaKiIgQEBEAsFiMzMxMzZ87E//73P5w4cQJff/01Fi9eXO7Zzubm5vj++++hp6eHCxcuSH2b1tTUhKOjo1S+t2/fxuvXrzFw4EC55MUwDC5fvozDhw9jwIABAIC0tDR8/fXXGDp0KI4fP47t27fjxYsX2LJlCwDgwIEDAIC9e/fCzc0NDx8+xIwZMzBu3DgcPXoUfn5++OGHHyr91n7t2jUIBAJ0796dnda7d29s3rxZbr0JAJgwYQJiY2Ph4OCA+fPnIy4uDllZWTA0NISWlhbb7s8//0RmZiYOHDiAoKAgLF26lO01JiQkYNWqVZg1axYOHz6Mr776CgEBAXj48CEAYPPmzYiKikJwcDB+++03aGlpYcqUKVJxtG7dml1eSEgIFixYIPW5SCTChAkTUFxcjOjoaKxfvx7x8fHYtWuXVDs3NzdMnDgR3bp1w4ULF+Dk5AQej4dz586xbY4dOwZbW1u2V965c2fo6uriypUrctqqDVfjHWtXCfz0009YtWqV1LS9e/eia9euaNWqFVasWME+ua1t27bYs2cPnjx5Um45L168gFAoRLt27aClpYWVK1fi7du37PIsLCzg7+8PAPjss8+QkZGBHTt2YMyYMZXGZm1tDaB0Z1pQUAADAwOsWLECHTp0wMOHD3Hp0iXEx8fj888/BwCsXbsWjo6O+PPPP9G6dWsUFxejVatWaNu2Ldq2bQsDAwO0atVKah0CgQDa2trg8/kVHopwd3fH999/j+LiYqipqeH3339H3759oauri40bN9YqL6B0J6Wvr48JEybAz88PQOmjVefPn88uo127dhg4cCC7s9HT0wNQ+tQxdXV17Ny5E56envDy8gIAdOjQASkpKfjll1/w5Zdfllv/3bt3YWxsXOXDa2orICAAHTp0QHR0NA4fPoxDhw6xj8idM2cOu35NTU2sXLkSQqEQnTp1wvXr17Fv3z44OTkhPDwckyZNYouor68vkpKSsGfPHixbtgzR0dGYOnUq2zv64YcfsHPnTuTm5rJxqKioQF9fHwCgo6NT7rDipUuX8O+//yImJoYdHXXx4sXIycmRaqeurg6hUAhVVVX2d8XV1RXHjx+Hm5sbgNJCMXXqVKn5OnXqhDt37rA5KCsqFA1YYGAg3N3dpaaVfau0trbGnTt3sH79eqSkpODRo0d49uwZLCwsyi1n/PjxCAoKgo2NDaysrNC/f3/2sahPnz7FxYsXYW5uzrYXi8UoLi6GSCSCQCCoMLYDBw5ATU0NPB4PmpqaUudOnj59CnV1dbZIAKU7TkNDQzx9+hROTk5wcnLClClT0K5dOzg6OsLT07PawySXFclLly7hyy+/xMmTJ9kromqbV1paGhYvXgwzMzMEBgayh546duwIDQ0NhIeH49GjR0hJScHDhw/RrVu3Cpf35MkTPHr0CHFxcey0kpIStqB8LDMzs1bDRR8+fBiLFi1i3wcEBFT6/OVBgwZh0KBByM3NRWJiIuLi4rBr1y60bt0aY8eOBQB88cUXUg/E6dGjByIiItjcbt++zfamgNLzDaampnj79i2ysrKkHumpr69f7acNPnnyBO3bt5faJh8eluPi4eGBoKAg5ObmIiUlBRkZGXBxcZFqo6uryx6KUmZUKBowPT09fPbZZxV+dvDgQSxduhReXl5wcHDAN998g8WLF1fYtk+fPvjrr79w7tw5JCQkYM2aNYiPj8evv/6KkpISDBo0CN988025+bge/tK+fXs0adKkws8qmy4WiyEWi8Hj8RAaGop79+7h3LlzOHv2LKKjo7F582Y4OjpWus6PCQQCuLq64sSJE2jWrBmysrLYcwS1zeuzzz5DWFgYhg0bhmbNmrGHRB4+fAhvb2/Y29ujd+/eGD16NP78889KD1+IxWKMGzdO6lJQAJU++4TP57PH32vCyckJPXv2ZN/r6OiUa/PgwQMcOHAAwcHBAErPTfTv3x/9+/fHlClTcPHiRbZQfHhupiyfsmlisRhz5swp9zxygUDAnquqrdosx8bGBk2bNsXZs2eRnJwMR0fHck87FIvFDfI5NPJGW6CR2r17NyZPnozg4GAMHz4cRkZGeP78eYUna3fv3o0bN27Aw8MDa9aswc6dO3Hz5k38+++/7NP+PvvsM/Z19+5dbN++vcZ/QEZGRigsLJQ655CVlYVnz56xvYrly5ejW7dumDp1Kg4dOgRra2ucOnWq3LKqOgTj7u6OhIQEnDhxAk5OTuy3X3nk1alTJ0ydOhWRkZFISkoCAOzfvx+mpqbYtGkTfH19YWFhIbXdP47X2NgYaWlpUnH8+eeflV7vr6+vzx4WrAktLS2pdZVdWfQhiUSCqKgoXL16tcL5P+ztPHr0SOqE9507d2BiYsLm9urVK6n1HTx4EH/88Qe0tbXRvHlz3L9/n503Ozsb1tbWePr0qcz5dOzYES9evMD79+/ZaXv37sXEiRPLtf142/P5fAwePBjnzp1jLyj42Nu3b9lDX8qMCkUj1bJlSyQmJuLp06d48OABvvvuO6Snp0s9FrLM69evsWzZMly/fh1paWk4fPgwWrRogZYtW2LMmDFISUnBTz/9hJSUFCQkJGDp0qW1OvxhaGgIZ2dnzJs3Dzdu3MCDBw8we/ZstGjRAl9++SV0dHRw4MABbNiwAWlpabh69SoePHhQ4eEboVCI3NxcPH36VGqHVcbKygqqqqqIjo7G4MGD2enyymvChAkwNDTE0qVLIZFI0KpVKzx9+hQ3btzA8+fPERoaipMnT7LbvaxQPXz4EHl5eZg4cSJOnz6N8PBwPHv2DIcPH8aaNWsqPTHdrVs3PH78uFa9iqp07doVLi4u+Pbbb3HgwAE8f/4cycnJCA8Pxx9//MH2JgDgzZs3WLZsGVJSUvDrr7/i5MmT8PX1BQD4+/tj79692L9/P54/f47du3dj+/bt6NixI4DSQ56bN2/GX3/9hZSUFAQHB6NVq1YwMjKSOVY7Ozu0a9cOCxYswOPHj3H58mWEhYWxF018SCgUIiMjA2lpaew0Dw8PnDlzBu/evatwnocPH0odHlNWVCgaqQULFqCkpATDhg3DpEmToKWlhTFjxuDevXvl2k6fPh3W1taYNm0a3NzccP/+fWzbtg1qamowMDDAjh07cOvWLQwdOhTBwcHw9vbGt99+W6v4VqxYARMTE0yePBk+Pj7Q1NREZGQkNDQ0oK+vjy1btuCvv/6Cu7s7Zs6cieHDh8PHx6fccqytrdGpUyd89dVXSE5OLvc5n8+Hm5sb1NTUpHYE8spLTU0NwcHBuHfvHvbv3w9fX19YWVlh0qRJGDFiBP7++28sWLAA//zzD/Ly8tCsWTMMGzYMc+bMwf79+9G9e3ds3LgRR44cweDBg7FhwwbMmjUL3t7eFa7P2toaYrFY6pt4XVi3bh18fHywe/dueHh4YMyYMbhy5Qp2796NL774gm3XtWtXSCQSeHp6IiIiAmvWrEHv3r0BAC4uLliwYAF27NgBNzc37Nu3D6tWrWIPRU2cOBGenp6YP38+hg8fjqKiImzdurVaJ+pVVFQQGhqKgoICeHl5Yc6cORgxYgTGjx9frq2rqytUVVXh7u6O9PR0AED37t3RunjTejsAAADgSURBVHVrDBgwoNxhrCdPniAvL6/BPxpYHugJd4Q0MIsXL4a6ujrmzZtXr3GEhITg/Pnz2L9/f73GURvFxcWws7PDli1byg1/s379eqSnp+Onn36qp+g+HdSjIKSB8fPzw++//468vLz6DqVBO3nyJJYsWYKWLVuWKxJFRUU4cuQIe/m0sqNCQUgD0759e3h7e7OXoZKa2bhxIy5cuIDly5eX++zXX3/F0KFDq3W+pDGjQ0+EEEI4UY+CEEIIJyoUhBBCOFGhIIQQwokKBSGEEE5UKAghhHCiQkEIIYTT/wNMObQ8Oq+POAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.fill_between(false_positive_rate_values, sensitivity_values, step='post', alpha=0.2,\n", " color='b')\n", "\n", "plt.xlabel('False Positive Rate (1 - Specificity)')\n", "plt.ylabel('Sensitivity')\n", "plt.title('words_list_model ROC Curve')" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9057984671441136" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.metrics import roc_auc_score\n", "\n", "roc_auc_score(y_train, words_list_model_probabilities)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "AUC is interpreted as the probability that the classifier will assign a higher probability to a randomly chosen data point truly belonging to the positive class than a randomly chosen data point truly belonging to the negative class. A perfect AUC value of 1 corresponds to a perfect classifier (the ROC curve would contain (0, 1). The fact that `words_list_model` has an AUC of .906 means that roughly 90.6% of the time it is more likely to classify a spam email as spam than a ham email as spam.\n", "\n", "By inspection, the AUC of the random classifier is 0.5, though this can vary slightly due to the randomness. An effective model will have an AUC much higher than 0.5, which `words_list_model` achieves. If a model's AUC is less than 0.5, it performs worse than random predictions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "AUC is an use metric for evaluating models on class-imbalanced datasets. After training a model, it is best practice to generate an ROC curve and calculate AUC to determine the next step. If the AUC is sufficiently high, use the ROC curve to identify the best classification threshold. However, if the AUC is not satisfactory, consider doing further EDA and feature selection to improve the model." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 2 }