{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# EXERCISE 1 - ML - Grundverfahren WS 21/22\n", "\n", "**Exercise 1**: Ge Li ge.li@kit.edu\n", "\n", "**Exercise 2 & 3**: Philipp Becker philipp.becker@kit.edu\n", "## Submission Instructions\n", "Please follow the instruction from Exercise ZERO!\n", "\n", "\n", "## 1.) Linear Regression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1) Matrix Vector Calculus (1 Point)\n", "Given the following element-wise expression of a matrix-vector product,\n", "rewrite it in matrix form:\n", "\n", "\\begin{align*}\n", " g &= \\alpha \\sum_i \\sum_j \\sum_k z_k x_{ij} q_i y_{jk}\\\\\n", " &= \\alpha \\sum_i^I \\sum_j^J \\sum_k^K z_k x_{ij} q_i y_{jk}\\qquad (\\text{mit }I,J,K\\in \\mathbb{N})\\\\\n", " &= Y\\cdot X \\cdot Q \\cdot Z \\cdot \\alpha\\\\\n", " &= \\begin{bmatrix} y_{11} & \\cdots & y_{1J}\\\\ \\vdots & \\ddots & \\vdots\\\\ y_{K1} & \\cdots & y_{KJ}\\end{bmatrix} \\cdot\n", " \\begin{bmatrix} x_{11} & \\cdots & x_{1I}\\\\ \\vdots & \\ddots & \\vdots\\\\ x_{J1} & \\cdots & x_{JI}\\end{bmatrix} \\cdot\n", " \\begin{bmatrix} q_1 \\\\ \\vdots \\\\ q_I\\end{bmatrix} \\cdot\n", " \\begin{bmatrix} z_1 & \\cdots & z_K\\end{bmatrix} \\cdot\n", " \\alpha\n", "\\end{align*}\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2) Derive Ridge Regression Weights (4 Points)\n", "Derive the optimal solution of weights in Ridge Regression using matrix form, i\n", ".e. $\\boldsymbol{w}= ?$\n", "\n", "Hint: You will need derivatives for vectors/matrices. Start\n", "from the matrix objective for ridge regression as stated here\n", "\n", "\\begin{align*}\n", "L &= (\\boldsymbol{y}-\\boldsymbol{\\Phi} \\boldsymbol{w})^T(\\boldsymbol{y}-\\boldsymbol{\\Phi} \\boldsymbol{w}) + \\lambda \\boldsymbol{w}^T \\boldsymbol{I} \\boldsymbol{w}. \\\\\n", "\\end{align*}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$ L = (y-\\Phi w)^T (y - \\Phi w) + \\lambda w^T I w $$\n", "$$ = w ^T \\Phi^T\\Phi w - y^T \\Phi w - w^T\\Phi^Ty + y^Ty + \\lambda w^T I w $$\n", "$$ = w^T\\Phi^T\\Phi w - 2 y^T\\Phi w + y^Ty + \\lambda w^T I w $$\n", "$$ = y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w $$\n", "$$ \\frac{\\delta x^TAx}{\\delta x} = (A + A^T)x: \\frac{\\delta w^T(\\Phi^T\\Phi + \\lambda I)w}{\\delta w} = ((\\Phi^T\\Phi + \\lambda I) + (\\Phi\\Phi^T + \\lambda I^T))w = 2(\\Phi^T\\Phi + \\lambda I)w $$\n", "$$ \\frac{\\delta y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w }{\\delta w} = -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w $$\n", "$$ -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w = 0, (\\text{to find optimum}) $$\n", "$$ (\\Phi^T\\Phi + \\lambda I)w = \\Phi^Ty $$\n", "$$ w^* = (\\Phi^T\\Phi + \\lambda I )^{-1} \\Phi^Ty $$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ridge Regression - Code\n", "Let's first get the data\n" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEGCAYAAACZ0MnKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhBElEQVR4nO3df3icZZ3v8fc34UdJ7TlC2hUOJZOuF+xFf2hrYxF7uopWqIAgHBFLqLBCoy519ZKDFsqxyLVhzzno4sURwdZFOW0sAtoVXV3a7rGsP4A1LeU3SEuTkBYhTQVKI4U23/PH80x4MplJJsnMPM/MfF7XNVdm7ueZmW+Sdr657+9934+5OyIiIvmoiTsAEREpH0oaIiKSNyUNERHJm5KGiIjkTUlDRETydljcARTb5MmTvbGxMe4wRETKypYtW/a4+5TM9opPGo2NjbS3t8cdhohIWTGzzmztGp4SEZG8KWmIiEjelDRERCRvFV/TyObNN9+ku7ub119/Pe5Qqt6ECROYOnUqhx9+eNyhiEgeqjJpdHd3M2nSJBobGzGzuMOpWu5Ob28v3d3dTJs2Le5wRCQPiRyeMrO/MrNtkdurZvYlM7vOzHZF2s8cy+u//vrr1NfXK2HEzMyor69Xj0+kkNraoLERamqCr21tBX35RPY03P0ZYDaAmdUCu4D1wN8AN7n7N8b7HkoYyaDfg0gBtbVBSwv09QWPOzuDxwDNzQV5i0T2NDJ8GNjh7lnnDIuISGjFircSRlpfX9BeIOWQND4FrIs8XmZmj5rZ7WZ2dLYnmFmLmbWbWXtPT09pohQRiVtX1+jaxyDRScPMjgDOAe4Om24F3kkwdPUC8M1sz3P3Ve7e5O5NU6YMWQU/egUeI3z55Zf5zne+M6bnfutb36Iv8y+JYfzgBz9g2bJlw56zefNmfve7340pHhFJkIaG0bWPQaKTBvBRYKu7vwjg7i+6+yF37wdWA/OKHkF6jLCzE9zfGiMcR+IoZdLIh5KGSIVobYW6usFtdXVBe4EkPWksJjI0ZWbHRY6dBzxe9AiKMEa4fPlyduzYwezZs7nqqqu48cYbee9738u73vUuVq5cCcD+/fs566yzePe7383MmTP50Y9+xM0338zu3bs57bTTOO2003K+/ve//31OOukk5s2bx29/+9uB9p/97GeccsopzJkzh4ULF/Liiy/S0dHBbbfdxk033cTs2bP59a9/nfU8EUmw9GjIkiVw1FFQXw9mkErBqlUFK4IDwVz5JN6AiUAv8J8jbWuAx4BHgXuB40Z6nblz53qmJ598ckhbTmbuQR9j8M0s/9fIsHPnTp8xY4a7u993332+dOlS7+/v90OHDvlZZ53l999/v99zzz1++eWXDzzn5Zdfdnf3VCrlPT09OV979+7dfsIJJ/hLL73kBw4c8Pe///1+xRVXuLv73r17vb+/393dV69e7V/+8pfd3X3lypV+4403DrxGrvOKZVS/DxEZbO1a97q6wZ9PdXVB+zgA7Z7lMzWRU24B3H0/UJ/RtqTkgTQ0BENS2doLYMOGDWzYsIE5c+YA8Nprr/Hss8+yYMECrrzySr761a9y9tlns2DBgrxe76GHHuKDH/wg6VrOhRdeyB/+8AcgWNR44YUX8sILL/DGG2/kXFCX73kikgDDjYYUsocRSvrwVPyKPEbo7lx99dVs27aNbdu2sX37di677DJOOukktm7dyqxZs7j22mu5/vrrx/1eX/jCF1i2bBmPPfYY3/3ud3Muqsv3PBFJgBLMmIpS0hhJc3MwJphKFWyMcNKkSezbtw+AM844g9tvv53XXnsNgF27dvHSSy+xe/du6urquPjii7nqqqvYunXrkOdmc8opp3D//ffT29vLm2++yd133z1w7JVXXuH4448H4I477sgaz3DniUgClWDGVJSSRj6am6GjA/r7g6/j7PLV19czf/58Zs6cycaNG7nooos49dRTmTVrFp/4xCfYt28fjz32GPPmzWP27Nl8/etf59prrwWgpaWFRYsW5SyEH3fccVx33XWceuqpzJ8/n5NPPnng2HXXXccFF1zA3LlzmTx58kD7xz72MdavXz9QCM91nogkUAlmTEVZUO+oXE1NTZ555b6nnnpq0IepxEu/D5ExaGsL6hZdXXDMMUHb3r1BD6O1ddx/3JrZFndvymxPbCFcRERyyNxjqrc36F2sWVOU4neUkkYZO+WUUzhw4MCgtjVr1jBr1qyYIhKRkijxjKkoJY0y9tBDD8UdgojEocQzpqJUCBcRKTclnjEVpaQhIlIu0tuFdHYGSwCiijhjKkpJQ0SkHEQ3T4Vgw5B04ijGHlM5KGnEYDy73J555pm8/PLLw57zta99jU2bNo3p9YejbdZFYpSt+O0eJIwCrB/Ll5JGHgp9yd3hksbBgweHfe4vfvEL3v72tw97zvXXX8/ChQvHGt64KGmIFEmMxe8oJY0RFOFyGkO2Rt+8eTMLFizgnHPOYfr06QB8/OMfZ+7cucyYMYNVq1YNPLexsZE9e/bQ0dHBySefzNKlS5kxYwann346f/7znwG49NJLueeeewbOX7lyJe95z3uYNWsWTz/9NAA9PT185CMfYcaMGVx++eWkUin27NkzJFZtsy6SEDEWvwfJtvVtJd3GuzV6KpV9Z/RUKu+XGCK6Nbq7+69+9Suvq6vz5557bqCtt7fX3d37+vp8xowZvmfPnjCeYGv0nTt3em1trT/88MPu7n7BBRf4mjVr3N39kksu8bvvvnvg/Jtvvtnd3W+55Ra/7LLL3N39iiuu8BtuuMHd3X/5y186MGTL9VJts66t0UWGsXbtWx9EmZdqKMAW6LlQblujJ0WpeoTz5s0btAX5zTffzPr16wF4/vnnefbZZ6mvH7RTPNOmTWP27NkAzJ07l46Ojqyvff755w+c85Of/ASA3/zmNwOvv2jRIo4+eujl1rXNukjMMld+p4vf6VpGAbYLGS0NT42gVD3CiRMnDtzfvHkzmzZt4oEHHuCRRx5hzpw5WbcnP/LIIwfu19bW5qyHpM8b7pzR0jbrIiWQkOJ3VGKThpl1mNljZrbNzNrDtmPMbKOZPRt+HfrncYEVYwPJkbY3f+WVVzj66KOpq6vj6aef5sEHHxz7m+Uwf/587rrrLiC4ENSf/vSnIedom3WRmCWk+B2V2KQROs3dZ/tbOy0uB/7N3U8E/i18XFRFuJzGoK3Rr7rqqiHHFy1axMGDBzn55JNZvnw573vf+8bxHWS3cuVKNmzYwMyZM7n77rs59thjmTRp0qBztM26SMySUvyOSOzW6GbWATS5+55I2zPAB939BTM7Dtjs7n813Otoa/TsDhw4QG1tLYcddhgPPPAAn//859m2bVsssej3IZIhve15euV39HO6rq4kC/nKcWt0BzaYmQPfdfdVwDvc/YXw+B+Bd2R7opm1AC0ADTFm5CTr6urik5/8JP39/RxxxBGsXr067pBEBBJZ/I5KctL4r+6+y8z+AthoZk9HD7q7hwlliDDBrIKgp1H8UMvPiSeeyMMPPxx3GCKSaaTid8wSW9Nw913h15eA9cA84MVwWIrw60vjeP1ChCnjpN+DSIYEFr+jEpk0zGyimU1K3wdOBx4H7gUuCU+7BPjpWF5/woQJ9Pb26gMrZu5Ob28vEyZMiDsUkfil9yvK9bmUkKH2pA5PvQNYb8EOjocBP3T3fzWz3wN3mdllQCfwybG8+NSpU+nu7qanp6dgAcvYTJgwgalTp8Ydhki8MusYmUq07Xk+Ejt7qlCyzZ4SEUmU9DUysomp+F2Os6dERKpDrnqFWSKK31GJrGmIiFSVBC7iy0VJQ0QkLgm4fOtoKWmIiMQhIZdvHS3VNERE4pDwRXy5qKchIhKHhC/iy0VJQ0SkwNKlipqa4Ougy0OXySK+XDQ8JSJSQJnr9Do7g8cAzZTPIr5ctLhPRKSAcq3TS6WggxwH0yfEvINtlBb3iYiUQM5SRWc/we5HWSRwEV8uqmmIiBRQznV6DFPgTngdI0pJQ0SkgM48M8s6PfbTyjXZn1AGdYwoJQ0RkQJpa4M77hg8Mcro5xK+TzPrhj4hwYv4clFNQ0SkQLKu16OGX3A28IXBBxK+iC8X9TRERAokZxGcjJpFmQ1JRSlpiIgUSF5F8DIckopKZNIwsxPM7Fdm9qSZPWFmXwzbrzOzXWa2LbydGXesIiIAtLXR+trfUcf+Qc2DiuDpqbVlmjAguTWNg8CV7r41vFb4FjPbGB67yd2/EWNsIiKDhcvAm/v6gD2s4Aa6aKCBLlq55q0ieBlNrc0lkUnD3V8AXgjv7zOzp4Dj441KRCSHSAW8mXXZZ0qVcR0jKpHDU1Fm1gjMAR4Km5aZ2aNmdruZHZ3jOS1m1m5m7T09PaUKVUSq1XA705qVfR0jKtFJw8zeBvwY+JK7vwrcCrwTmE3QE/lmtue5+yp3b3L3pilTppQqXBGpNiPtWJtKQX9/2dcxohI5PAVgZocTJIw2d/8JgLu/GDm+Gvh5TOGJSLXL3M42U4UMR2VKZE/DzAz4J+Apd//HSPtxkdPOAx4vdWwiUuXSvYuLL86dMCpoOCpTUnsa84ElwGNmti1suwZYbGazAQc6gM/GEZyIVKmRehdQVjvWjkUik4a7/wawLId+UepYREQGZNsnJFMFTKsdTiKHp0REEiU9JJXrAkppFVrHiFLSEBEZTnpIaqSEUcF1jKhEDk+JiCTGSENSdXVVkSzS1NMQEckmnyGpKuldRKmnISKSKZ9ZUmV6PYzxUk9DRCQtnzUYUBUF71yUNEREQAXvPGl4SkSqW1tbUOweKVlA1Q5JRSlpiEj1yqd2kVbFQ1JRGp4SkeqTb+0ircqHpKLU0xCR6jLa3oWSxSDqaYhIdVDvoiCUNKT6pD88ampg8uTglnm/sTE4r4ShlOgtq1O+M6Mg6F2sXVtRF04qJCUNKX/5JIH0fTNYsiT48HCH3t7glnm/szM4z2zk1xzjJ370cyz9li0tShwFpd5FwZnnukxhhWhqavL29va4w5DxSk+L7OqCY44J2vbuDe7v2wdvvBFvfBAkGHeorw8e790bbJPd2pr1QyjXDhWa1TlO0Sm06d/JSFS7GMLMtrh7U2Z72fU0zGyRmT1jZtvNbHnc8UgRpf9KHKl3kISEAW99OOXqsTQ2wt/+7UCvqKuzP+vLdHWVLOLKkzkMlU/CUO9iVMoqaZhZLXAL8FFgOsGV/KbHG5UUVLZEAfn950+qdOydnXDrrQPJr4Hs2aGhpltjVKM12mEoUO1ijMoqaQDzgO3u/py7vwHcCZwbc0wyXpWYKPLQyjXUsX9QWx37aT30lez1FFXKB8v17yYf6l2MWbkljeOB5yOPu8O2Qcysxczazay9p6enZMHJKMSdKCy8mnB9fXAze+t+9HgRNbOOVSwlRQdGPyk6WMVSmlk37FBXm11EY+3z1Fg/jdZB2+S/q45kkjnh4TOfGf2/G/Uuxs/dy+YGfAL4XuTxEuDbwz1n7ty5Lgmxdq17KuUO7mbB10LcDj/cvb4+eM36+pHvp1JBLPnEms9rFvr7Gea2lsVex2uDmut4zddyUfBgtN9nEmX72Y/nZ5x+Xrn+PGICtHu2z+FsjUm9AacC90UeXw1cPdxzlDQSYu1a97q68X1opv/zJ/GDsdAfdDluKXZmPZRiZ/n9zKKK9QdFEr/XMlEpSeMw4DlgGnAE8AgwY7jnKGnELPphUOZ/JUbzQt7hZD7p858f14ejcSj7j4lDY/u5ZiaTdHxj6Z3l8zMoVU+tri72fy/lriKSRvB9cCbwB2AHsGKk85U0YjDevxoTlCjSsnWUxv25NIbeyah7GoW8ZUs0CRq6S+K/m3JWMUljtDcljRIb6zBUwv/D5+ospVJFeLNhkknumsbi0nwwj/O2lsWeYqcbhzzFzvHFHa1lJfTfTTlT0pDiGsswVMITRVSuP5bNShhE+DNey0WequmKfPBeFHsyyDdhjDvhldG/mXKnpCHFM8rexVoWe6r2eTf6y+b/fkl7GqNVoiL8eG+jGlorlwJ+BVPSkMIbQ+9i7eGXet0Rbw7+a7MMapZFqWkUW8KSyYhFfPUiEkVJQwprNL2LyIdBqn5fcv9iH8GYZk8lUa7ZTLlmTxUi0ZgN39Mo6x9oZcqVNLTLrYxOdAfRfKRSg3Z5rakJPioymUF/9v37JAmG22V4pPvhTr9tNA+5YJ42l02uXLvc6nKvkr8CXCazoSF7vmloKFCMUhzNzeP+ZE8/O517htk1XhKs3PaekjgU8EI2ra1BPomqqwvapfI1NwfbPvX3a/uncqWehgyvAL2LqPQh/bUpUp5U05Dh5bq8XKaM2oWIlLeKuXKflEB0C+qREoa2mhapKhqeksFGMxyl3oVI1VHSkMFWrBg5YWiepEjV0vCUBMIhqbbO99PITmo4RCM7aWPxW+eY6TKZIlVOPQ0ZGJJq6zuXFlbTx0QAOmmkhdUANKd+F9QtRKSqqadRzTLWX6zghoGEkdbHRFbY/9RCChEB1NOoXlkK3l1kX5bd5SdoOEpEgAT2NMzsRjN72sweNbP1Zvb2sL3RzP5sZtvC220xh1reshS8G+jKempDykoRkYiUgcQlDWAjMNPd30VwWderI8d2uPvs8Pa5eMIrc+khqSzrL1q5hjr2D2rTFh8iEjVi0jCzL5jZ0aUIBsDdN7j7wfDhg8DUUr13xUsPSeVYsNfMOlaxlFRtN4ZropSIDJFPT+MdwO/N7C4zW2RmpRyr+Azwy8jjaWb2sJndb2YLcj3JzFrMrN3M2nt6eoofZbkYaQ1GXR3Na8+i4+BU+t20yFtEhshr76kwUZwO/A3QBNwF/JO77xjTm5ptAo7NcmiFu/80PGdF+F7nu7ub2ZHA29y918zmAv8MzHD3V4d7L+09FZHrYhag1d0iMsi4rqcRfmj/EfgjcBA4GrjHzDa6+1dGG4y7Lxwh2EuBs4EPh1eQwt0PAAfC+1vMbAdwEqCMMJL0BXSGSxhagyEieRgxaZjZF4FPA3uA7wFXufubZlYDPAuMOmmM8H6Lwtf8gLv3RdqnAHvd/ZCZ/SVwIvBcId+7Io20l5Qq3SIyCvn0NI4hGCIaVD11934zO7sIMX0bOBLYGJZPHgxnSv01cL2ZvQn0A59z971FeP/KMlwdQ0NSIjJKup5GpRrpWt66KLeIDEPXCK8m+Wxvrotyi8gYJHFxn4xXHlNrVccQkbFQ0qhEXdm3AwG0tbmIjIuSRiVJbxEy0tRaJQwRGSPVNCqFptaKSAmop1EpRppaqyEpESkA9TQqRa46hplWe4tIwainUe5GqmNoaq2IFJB6GuVMdQwRKTH1NMqZ6hgiUmLqaZQz1TFEpMTU0yhHqmOISEzU0yg3qmOISIzU0ygjbW3QeMkHqOnbRyM7aWPx4BNUxxCRIlNPo0wMdDAOTQWgk0ZaWA1AM+tUxxCRklBPo0xkmyjVx0RWcEPwQHUMESmBxCUNM7vOzHaZ2bbwdmbk2NVmtt3MnjGzM+KMs9S6OrMXvbtoUB1DREomqcNTN7n7N6INZjYd+BQwA/gvwCYzO8ndD8URYEm1tdFgC+j0ob2JhtrdqmOISMkkrqcxjHOBO939gLvvBLYD82KOqTRWrKDVl1PH/kHNddZH6x1TlTBEpGSSmjSWmdmjZna7mR0dth0PPB85pztsG8LMWsys3czae3p6ih1r8XV10cw6VrGUFB0Y/aToYJUvVb4QkZKKJWmY2SYzezzL7VzgVuCdwGzgBeCbo319d1/l7k3u3jRlypTCBl9KGYv4mllHB9Pop5YOptGc+m288YlI1YmlpuHuC/M5z8xWAz8PH+4CTogcnhq2VSYt4hORBErc8JSZHRd5eB7weHj/XuBTZnakmU0DTgT+o9TxlYw2IxSRBEri7Kn/bWazAQc6gM8CuPsTZnYX8CRwELiiomdOaTNCEUmgxCUNd18yzLFWoDrGZBoaoLMze7uISEwSNzxV9dLF787OoFcRpTqGiMRMSSNJ0sXvdA/D/a3EoTqGiCRA4oanqlq24rd7kDBUxxCRBFBPI0lyFb9ztYuIlJiSRpLkKnKr+C0iCaGkkQQqfotImVDSiJuK3yJSRlQIj5uK3yJSRtTTiJuK3yJSRpQ04qbit4iUESWNuLW2BsXuKBW/RSShlDTikp4xtWQJHHUU1NcHBXAVv0UkwVQIj0PmtTJ6e4PexZo1ShYikmjqacQh24ypvr6gXUQkwZQ04qAZUyJSphKXNMzsR2a2Lbx1mNm2sL3RzP4cOXZbzKGOnWZMiUiZSlxNw90vTN83s28Cr0QO73D32SUPqtBaW4de/1szpkSkDCSup5FmZgZ8ElgXdywFkZ4tVVMT1C4uuSSYKaUZUyJSRhLX04hYALzo7s9G2qaZ2cPAq8C17v7rbE80sxagBaAhCUM+mbOlOjvhjjuUKESk7Ji7l/5NzTYBx2Y5tMLdfxqecyuw3d2/GT4+Enibu/ea2Vzgn4EZ7v7qcO/V1NTk7e3tBY1/1NI72GbS/lIiklBmtsXdmzLbY+lpuPvC4Y6b2WHA+cDcyHMOAAfC+1vMbAdwEhBzRsiDZkuJSIVIak1jIfC0u3enG8xsipnVhvf/EjgReC6m+EZHs6VEpEIkNWl8iqEF8L8GHg2n4N4DfM7d95Y6sDHR/lIiUiESWQh390uztP0Y+HHpoxmHtrZgplRXFxxzTLDH1N69QQ+jtVVFcBEpO4lMGhVB+0uJSAVK6vBU+dP+UiJSgZQ0ikUzpkSkAilpFItmTIlIBVLSKBbNmBKRCqSkUWi6Ip+IVDDNniokzZgSkQqnnkYhhTOm2lhMIzup4RCNfU/Q9sWH4o5MRKQglDQKqauLNhbTwmo6acSpoZNGWnr/gba2uIMTERk/JY1CamhgBTfQx8RBzX1M1PIMEakIShqF1NpKF9mn1Gp5hohUAiWNQojMmGqo2ZX1FC3PEJFKoKQxXukZU52d4E5r/1epY/+gU7Q8Q0QqhZLGeGXsMdXMOlaxlFRtt5ZniEjF0TqN8cpSrGhmHc39d0J/fwwBiYgUj3oa46U9pkSkisSWNMzsAjN7wsz6zawp49jVZrbdzJ4xszMi7YvCtu1mtrz0UWehPaZEpIrE2dN4HDgf+Pdoo5lNJ7jc6wxgEfAdM6sNrw9+C/BRYDqwODw3Xs3NQdEildIeUyJS8WKrabj7UwBmlnnoXOBOdz8A7DSz7cC88Nh2d38ufN6d4blPlibiDNFLueryrSJSJZJY0zgeeD7yuDtsy9U+hJm1mFm7mbX39PQUPsKMabZ0dgaPtVeIiFS4oiYNM9tkZo9nuZ1bzPd191Xu3uTuTVOmTCn8G+hSriJSpYo6POXuC8fwtF3ACZHHU8M2hmkvLV3KVUSqVBKHp+4FPmVmR5rZNOBE4D+A3wMnmtk0MzuCoFh+bywRapqtiFSpOKfcnmdm3cCpwL+Y2X0A7v4EcBdBgftfgSvc/ZC7HwSWAfcBTwF3heeWnqbZikiVMnePO4aiampq8vb29sK/sGZPiUgFM7Mt7t6U2a5tRMaquVlJQkSqThJrGsmV3gK9pib4qim2IlJl1NPIV3ptRnqqbXptBqjHISJVQz2NfGlthoiIkkbetDZDRERJI29amyEioqSRN63NEBFR0sibtkAXEdHsqRFpEZ+IyAAljeFomq2IyCAanhqOptmKiAyipDEcTbMVERlESWM4mmYrIjKIksZwNM1WRGQQJY3haJqtiMggmj01Em2BLiIyIJaehpldYGZPmFm/mTVF2j9iZlvM7LHw64cixzab2TNmti28/UXRAtQW6CIiWcXV03gcOB/4bkb7HuBj7r7bzGYSXNr1+MjxZncvwmX4IrQ2Q0Qkp1h6Gu7+lLs/k6X9YXffHT58AjjKzI4saXBamyEiklOSC+H/Ddjq7gcibd8Ph6b+h5lZrieaWYuZtZtZe09Pz+jeVWszRERyKlrSMLNNZvZ4ltu5eTx3BvC/gM9GmpvdfRawILwtyfV8d1/l7k3u3jRlypTRBa61GSIiORWtpuHuC8fyPDObCqwHPu3uOyKvtyv8us/MfgjMA/5vIWIdpLV1cE0DtDZDRCSUqOEpM3s78C/Acnf/baT9MDObHN4/HDiboJheeFqbISKSk7l76d/U7Dzg/wBTgJeBbe5+hpldC1wNPBs5/XRgP/DvwOFALbAJ+LK7HxrpvZqamry9vbgTrkREKo2ZbXH3piHtcSSNUlLSEBEZvVxJI1HDUyIikmxKGiIikjclDRERyZuShoiI5K3iC+Fm1gN0xh3HKE0m2Iermuh7rg76nstHyt2HrI6u+KRRjsysPdushUqm77k66HsufxqeEhGRvClpiIhI3pQ0kmlV3AHEQN9zddD3XOZU0xARkbyppyEiInlT0hARkbwpaSScmV1pZp7eGr6SmdmNZva0mT1qZuvDrfIrkpktMrNnzGy7mS2PO55iM7MTzOxXZvakmT1hZl+MO6ZSMLNaM3vYzH4edyyFoqSRYGZ2AsHW8NVyrdmNwEx3fxfwB4Jt8iuOmdUCtwAfBaYDi81serxRFd1B4Ep3nw68D7iiCr5ngC8CT8UdRCEpaSTbTcBXgKqYreDuG9z9YPjwQWBqnPEU0Txgu7s/5+5vAHcCI14GuZy5+wvuvjW8v4/gg/T4eKMqrvAqpGcB34s7lkJS0kio8Frqu9z9kbhjiclngF/GHUSRHA88H3ncTYV/gEaZWSMwB3go5lCK7VsEf/T1xxxHQRXtGuEyMjPbBByb5dAK4BqCoamKMtz37O4/Dc9ZQTCc0VbK2KT4zOxtwI+BL7n7q3HHUyxmdjbwkrtvMbMPxhxOQSlpxMjdF2ZrN7NZwDTgETODYJhmq5nNc/c/ljDEgsv1PaeZ2aUE14D/sFfuIqJdwAmRx1PDtopmZocTJIw2d/9J3PEU2XzgHDM7E5gA/CczW+vuF8cc17hpcV8ZMLMOoMndy3GnzLyZ2SLgH4EPuHtP3PEUi5kdRlDo/zBBsvg9cJG7PxFrYEVkwV8/dwB73f1LMYdTUmFP47+7+9kxh1IQqmlIknwbmARsNLNtZnZb3AEVQ1jsXwbcR1AQvquSE0ZoPrAE+FD4u90W/hUuZUY9DRERyZt6GiIikjclDRERyZuShoiI5E1JQ0RE8qakISIieVPSEBGRvClpiIhI3pQ0RErIzN4bXi9kgplNDK8tMTPuuETypcV9IiVmZn9PsB/RUUC3u/9DzCGJ5E1JQ6TEzOwIgv2mXgfe7+6HYg5JJG8anhIpvXrgbQT7bE2IORaRUVFPQ6TEzOxegqv1TQOOc/dlMYckkjddT0OkhMzs08Cb7v7D8FrhvzOzD7n7/4s7NpF8qKchIiJ5U01DRETypqQhIiJ5U9IQEZG8KWmIiEjelDRERCRvShoiIpI3JQ0REcnb/wdwUQG2woCiYgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from typing import Tuple\n", "\n", "# Load data\n", "\n", "training_data = np.load('training_data.npy')\n", "test_data = np.load('test_data.npy')\n", "\n", "test_data_x = test_data[:, 0]\n", "test_data_y = test_data[:, 1]\n", "\n", "training_data_x = training_data[:, 0]\n", "training_data_y = training_data[:, 1]\n", "\n", "# Visualize data\n", "plt.plot(test_data_x, test_data_y, 'or')\n", "plt.plot(training_data_x, training_data_y, 'ob')\n", "plt.xlabel('x')\n", "plt.ylabel('y')\n", "plt.legend([\"test_data\", \"training data\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As in the lecture notebook, we will use polynomial-features here again.\n", "The following functions will be used for:\n", "- calculating polynomial features\n", "- computing the mean and std of the features (training data) as normalizer\n", "- normalize other data (test) features using the normalizer (mean and std)\n", "- evaluating the model\n", "- calculating the Mean Squarred Error for assigning a performance to each\n", "model.

\n", "\n", "Note we will use the mean and the standard deviation to normalize our features\n", "according to:\n", "\\begin{align*}\n", " \\boldsymbol{\\tilde{\\Phi}} = \\frac{\\boldsymbol{\\Phi}(\\boldsymbol{x}) - \\boldsymbol{\\mu}_{\\Phi}}{\\boldsymbol{\\sigma}_{\\Phi}}, \n", "\\end{align*}\n", "where $\\boldsymbol{\\tilde{\\Phi}}$ are the (approximately) normalized features to any input\n", "$\\boldsymbol{x}$ (not necessarily the training data), $\\boldsymbol{\\mu}_{\\Phi}$ is the mean of the features applied to the training data and $\\boldsymbol{\\sigma}_{\\Phi}$ is the standard deviation of the features applied to the training data for each dimension.
\n", "\n", "Normalization is a standard technique used in Regression to avoid numerical problems and to obtain better fits for the weight vectors $\\boldsymbol{w}$. Especially when the features transform the inputs to a very high value range, normalization is very useful. In this homework we will use features of degree 10. Since the input range of the data is roughly from -4 to 4 this will lead to very high values for higher order degrees. By normalizing each dimension of the feature matrix, we will map each dimension of the feature matrix applied to the training data to a zero mean unit variance distribution." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "def get_polynomial_features(data: np.ndarray,\n", " degree: int) ->np.ndarray:\n", " \"\"\"\n", " Function to create Feature Matrix. Extends the feature matrix according to\n", " the matrix form discussed in the lectures.\n", "\n", " :param data: data points you want to evaluate the polynomials,\n", " shape: [n_samples] (we have 1-dim data)\n", " :param degree: degree of your polynomial, shape: scalar\n", " :return polynomial_features: shape [n_samples x (degree+1)]\n", " \"\"\"\n", " polynomial_features = np.ones(data.shape)\n", " for i in range(degree):\n", " polynomial_features = np.column_stack((polynomial_features, data ** (i + 1)))\n", " return polynomial_features\n", "\n", "\n", "def get_mean_std_features(polynomial_features: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n", " \"\"\"\n", " Function for calculating the mean and standard deviation of the features\n", " :param polynomial_features: shape: [n_samples x (degree+1)]\n", " :return mean_feat: mean vector of the features,\n", " shape:[1 x (degrees+1)]\n", " :return std_feat: standard deviation (for each dimension in feature matrix),\n", " shape: [1 x (degrees+1)] \n", " \"\"\"\n", " mean_feat = np.mean(polynomial_features, axis=0, keepdims=True)\n", " mean_feat[:, 0] = 0.0 # we don't want to normalize the bias\n", " std_feat = np.std(polynomial_features, axis=0, keepdims=True)\n", " std_feat[:, 0] = 1.0 # we don't want to normalize the bias\n", " return mean_feat, std_feat\n", "\n", "\n", "def normalize_features(polynomial_features: np.ndarray,\n", " mean_train_features: np.ndarray,\n", " std_train_features: np.ndarray) ->np.ndarray:\n", " \"\"\"\n", " Normalize features\n", " :param polynomial_features: features to be normalized,\n", " shape: [n_samples x (degree+1)]\n", " :param mean_train_features: mean of the feature matrix of the training set,\n", " shape: [1 x (degrees+1)]\n", " :param std_train_features: std of the feature matrix of the training set,\n", " shape: [1 x (degrees+1)]\n", " :return norm_feat: normalized features, shape: [n_samples x (degree+1)]\n", " \"\"\"\n", "\n", " # note: features: (n_samples x n_dims),\n", " # mean_train_features: (1 x n_dims),\n", " # std_train_features: (1 x n_dims)\n", " # due to these dimensionalities we can do element-wise operations.\n", " # By this we normalize each dimension independently\n", " norm_feat = (polynomial_features - mean_train_features) / std_train_features\n", " return norm_feat\n", "\n", "\n", "def eval(Phi:np.ndarray, w:np.ndarray)->np.ndarray:\n", " \"\"\"\n", " Evaluate the models\n", "\n", " :param Phi: Feature matrix, shape: [n_samples x (degree+1)]\n", " :param w: weight vector, shape: [degree + 1]\n", " :return : predictions, shape [n_samples] (we have 1-dim data)\n", " Evaluates your model\n", " \"\"\"\n", " return np.dot(Phi, w)\n", "\n", "\n", "def mse(y_target:np.ndarray, y_pred:np.ndarray)->np.ndarray:\n", " \"\"\"\n", " :param y_target: the target outputs,\n", " shape: [n_samples] (here 1-dim data)\n", " :param y_pred: the predicted outputs,\n", " shape: [n_samples](we have 1-dim data)\n", " :return : The Mean Squared Error, shape: scalar\n", " \"\"\"\n", " diff = y_target - y_pred\n", " return np.sum(diff ** 2, axis=0) / y_pred.shape[0]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3) Implement Ridge Regression Weights (2 Point)\n", "The following function will calculate the weights for ridge regression. Fill in the missing code according to the formula for calculating the weight updates for ridge regression.
\n", "Recall that the formula is given by \n", "\\begin{align*}\n", " \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y}\\\\\n", " (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I}) \\boldsymbol{w} &= \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n", "\\end{align*}\n", "where $\\boldsymbol{\\Phi}$ is the feature matrix (the matrix storing the data points applied to the polynomial features).\n", "Hint: use np.linalg.solve for solving for the linear equation.\n", "If you got confused because of the normalization described before, don't worry, you do not need to consider it here :)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "def calc_weights_ridge(Phi:np.ndarray,\n", " y:np.ndarray,\n", " ridge_factor:float)->np.ndarray:\n", " \"\"\"\n", " :param Phi: Feature Matrix, shape: [n_samples x (degree+1)]\n", " :param y: Output Values, [n_samples] (we have 1-dim data)\n", " :param ridge_factor: lambda value, shape: scalar\n", " :return : The weight vector, calculated according to the equation shown before,\n", " shape: [degrees +1]\n", " \"\"\"\n", " ##################\n", " ##TODO\n", " #################\n", " identity_matrix = np.identity(Phi.shape[1])\n", " #for Ax=b\n", " A = (Phi.transpose().dot(Phi) + ridge_factor * identity_matrix)\n", " b = (Phi.transpose().dot(y))\n", " weights = np.linalg.solve(A,b)\n", " return weights" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For demonstrating ridge regression we will pick the polynomial degree of 10. In the lecture notebook we have seen that this model is highly overfitting to the data.\n", "We will investigate the role of the ridge factor $\\lambda$. For that purpose we first need to calculate the weights for different $\\lambda$ values.
\n", "We will pick $\\lambda = [1e-{6}, 1e-{3}, 1, 3, 5,10,20,30,40,50, 1e2, 1e3, 1e5] $ to see the differences of the values.

\n", "\n", "Practical note. We use here very high values for $\\lambda$ for demonstration\n", "purposes here. In practice we would not choose a model where we know from\n", "beginning that it is highly overfitting. When choosing an appropriate model, the value needed for $\\lambda$ automatically will be small (often in the range of $1e^{-6}$ or smaller)." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "# Let's do it on polynomial degree 10 and see the results\n", "\n", "# first we get the mean and the standard deviation of the training feature matrix, which we will use for normalization\n", "train_features = get_polynomial_features(training_data_x, 10)\n", "test_features = get_polynomial_features(test_data_x, 10)\n", "mean_train_feat, std_train_feat = get_mean_std_features(train_features)\n", "norm_train_features = normalize_features(train_features, mean_train_feat, std_train_feat)\n", "norm_test_features = normalize_features(test_features, mean_train_feat, std_train_feat)\n", "\n", "\n", "# now we can calculate the normalized features for degree 10\n", "ridge_factors = [1e-6, 1e-3, 1, 3, 5, 10,20,30,40, 50, 1e2, 1e3, 1e5]\n", "weights_ridge = []\n", "\n", "for lambda_val in ridge_factors:\n", " weights_ridge.append(calc_weights_ridge(norm_train_features, training_data_y, lambda_val))\n", "\n", "# We further have to perform the predictions based on the models we have calculated\n", "y_training_ridge = []\n", "y_test_ridge = []\n", "\n", "for w in weights_ridge:\n", " y_training_ridge.append(eval(norm_train_features, w))\n", " y_test_ridge.append(eval(norm_test_features, w))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are interested in the mean squarred error on the test and the training data. For that purpose we calculate them here and plot the errors for different $\\lambda$ values in log space. " ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEaCAYAAAD+E0veAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA38ElEQVR4nO3dd3hUZfbA8e8hVKkCQSlKEaRICRpQUZSAhQ4qIgj2VbHAigLqKqs/K6JYUBcXFFlXpCwsKwiuooIINoKigIC0sDQhgAQQkJL398eZCZOYnpncucn5PM88zNyZue+ZDHPPfct9X3HOYYwxxgCU8DoAY4wx0cOSgjHGmDSWFIwxxqSxpGCMMSaNJQVjjDFpLCkYY4xJY0nB+JKITBKRpwqprDNF5KCIxGTx/OMi8m5hxOInIvKGiIz0Og6TN5YUTKZEJElEDgcOhsHba4Ucw0IRORIoe7eI/FtEauZjP05EGmbz/M0iciJQzn4R+UFEugefd879zzlXwTl3Ir+fJT9EpIOIpAbiOiAia0XklsKMoSCcc4Occ096HYfJG0sKJjs9AgfD4O3ezF4kIiUz2ZbpWXVWsnn9vc65CsDZQBXgpbzsNw++CpRTBfgbMFVEqkSorLzYHoirEjAUmCAijcNdSGbfoSmeLCmYPAucWS8RkZdEZA/weKA5Z5yIzBOR34AEEWkaONvfJyKrRKRnyD7+8PrsynTO7QVmAs2ziOl2EVkvIntFZLaI1ApsXxR4yQ+BM+7rcignFfgnUB5oFNhHvUBto2TgcX0R+Txw9j4fqJ4hlhtFZLOI7BGRkYFa12WB50qIyEMisiHw/HQRqZpdTIG4nHNuHrAXaJmbfeUQx+MiMkNE3hWR/cDNIlJZRN4SkR0isk1EngomaxFpGPjMKYFa27TAdgn8P9gVqGWtEJHmgefSNfFl9R0FnnMiMkhE1gX+v7wuIpLT38WEnyUFk1/nAxuB04CnA9uuD9yvCHwDzAE+BmoAg4HJGc5yQ1+/OLvCRKQ6cA3wfSbPdQSeBfoCNYHNwFQA59wlgZe1CtR2puVQTgxwC3AssJ/MvAcsQ5PBk8BNIe9vhtY0BgRiqQzUDnnvYKA3cClQC/gVeD27mAL7LRFIqtWB9TntKxdxAPQCZqC1o8nAJOA40BBoDVwB/Cnw2ifR7/JUoA7wamD7FcAlaE2uMvod7Mkk/iy/oxDdgTZo0usLXJnDn8VEgnPObnb7ww1IAg4C+0Jutweeuxn4X4bXTwLeCXncHvgFKBGybQrweGavzyKGhcChQNnb0ANXbMj7nwrcfwsYHfK+CuhBvV7gsQMaZlPOzejBcF/gfYeBviHP1wvsoyRwZuC15UOefw94N3D/r8CUkOdOAY4ClwUerwY6hTxfM1BmyUzi6gCkBuL6HTgB3BfyfJb7ykUcjwOLQp4/LVBGuZBt/YEFgfvvAOOBOhli7Aj8DFwQ+l3n8zu6OOT56cBDXv8OiuPNagomO72dc1VCbhNCntuSyetDt9UCtjhtjgnaTPqz1cz2kdGQQNm1nXMDnHPJmbymFiFn9c65g+jZasYz4+x87Zyrgp4Jz0aTWmZqAb86534L2bY5w/Npn8s5d4j0Z851gVmBJpJ96IH9BHpQzsz2QFyVgLHoQTg3+8opDkj/968LlAJ2hOzv72gtD2AEIMC3gabAWwP7/Qx4Da2h7BKR8SJSKZPPkZvv6JeQ+4fQxGEKmSUFk1+ZTa8bum07cIaIhP4fOxM9489uH/mxHT2oASAi5YFqGcrKlcDB6i7gBhFpnclLdgCnBsoIOjPD83VCYikXiCVoC9AlQ7It65zLNlbn3O/Ag0ALEemdi33lFAek//tvQWsK1UP2Vck5d06g/F+cc7c752oBdwJ/k8CILufcWOfceUAztBlpeCYfIWzfkYksSwomUr5Bz/ZGiEgpEekA9OCP7cjhMAW4RUTiRKQM8AzwjXMuKfD8TqBBbnfmtFP7TbQJJuNzm4FE4P9EpLSIXIx+rqAZQA8RaScipdFmmtAO0zeAp0WkLoCIxIpIr1zGdRQYExJXdvvKKY6M+96B9hmMEZFKgT6Ms0Tk0sC+rxWRYJL5FU0oqSLSRkTOF5FSwG/AEbTJK6OcviMTJSwpmOzMkfTXKczK7RsDB7AeQBdgN9rpeaNzbk24g3TOfQKMREcn7QDOAvqFvORx4B+BZpG+udzty0BXEWmZyXPXox3te4HH0Pb2YCyr0A7gqYFYDgK70LNwgFfQ5qmPReQA8HVgX7k1EThTRHpkt69cxJGZG4HSwE/ogX8G2k8B2gH8jYgcDJT5Z+fcRrRZa0Lg9ZvRJqHnM+44F9+RiRLinC2yY0ykiEgFtKO4kXNuU3GPw0Q/qykYE2Yi0kNETgm0m78ArEBHcxXLOIy/WFIwJvx6oR2r29EL4Po5b6rk0RKH8RFrPjLGGJPGagrGGGPS+HoSrOrVq7t69ep5HYYxxvjKsmXLdjvnYjN7zpdJITAcr0fDhg1JTEz0OhxjjPEVEclqXi9/Nh855+Y45+6oXLmy16EYY0yR4sukEBhqNz4lJcXrUIwxpkjxZVKwmoIxxkSG7/sUjDHeOnbsGFu3buXIkSNeh2IyKFu2LHXq1KFUqVK5fo+vr1OIj4931tFsjLc2bdpExYoVqVatGrZYWvRwzrFnzx4OHDhA/fr10z0nIsucc/GZvc+XzUfGmOhx5MgRSwhRSESoVq1anmtwvkwK1tFsTHSxhBCd8vO9+DIpFLij+fvv4dFHwxuUMcYUAb5MCgX2ySfw9NPw3/96HYkxpoD27NlDXFwccXFxnH766dSuXTvt8dGjR7N9b2JiIkOGDMmxjHbt2oUl1oULF1K5cuW0+OLi4vjkk0/Csu9w8WVHc8joo9vXrVuX9x38/js0bw4lS8KPP0IeeuaNMemtXr2apk2beh0GAI8//jgVKlRg2LBhaduOHz9OyZLRMdBy4cKFvPDCC3zwwQdZvsY5h3OOEiVKZPo4K1l9zsy+nyLX0Vzg5qMyZeCll2DNGnjttfAGZ4zx3M0338ygQYM4//zzGTFiBN9++y0XXnghrVu3pl27dqxduxbQg3T37t0BTSi33norHTp0oEGDBowdOzZtfxUqVEh7fYcOHejTpw9NmjRhwIABBE+s582bR5MmTTjvvPMYMmRI2n5zIykpicaNG3PjjTfSvHlzvvjii3SPt2zZwvDhw2nevDktWrRg2rRpafG0b9+enj170qxZs7D87aIjfXqhWzfo3BkefxwGDIAaNbyOyBjfu+8+WL48vPuMi4OXX877+7Zu3cqXX35JTEwM+/fv54svvqBkyZJ88skn/OUvf2HmzJl/eM+aNWtYsGABBw4coHHjxtx1111/GOP//fffs2rVKmrVqsVFF13EkiVLiI+P584772TRokXUr1+f/v37ZxnXF198QVxcXNrjmTNnEhMTw7p16/jHP/7BBRdcQFJSUrrHM2fOZPny5fzwww/s3r2bNm3acMkllwDw3XffsXLlyj8MO82v4psURLS20KKFdjqPH+91RMaYMLr22muJiYkBICUlhZtuuol169YhIhw7dizT93Tr1o0yZcpQpkwZatSowc6dO6lTp06617Rt2zZtW1xcHElJSVSoUIEGDRqkHZj79+/P+CyOKe3bt/9D81FSUhJ169blggsuSNsW+njx4sX079+fmJgYTjvtNC699FKWLl1KpUqVaNu2bdgSAvg0KYTtiuYmTWDwYD0NGTQIzj03HOEZU2zl54w+UsqXL592f+TIkSQkJDBr1iySkpLo0KFDpu8pU6ZM2v2YmBiOHz+er9cUNN7MHuf2fQVVPPsUQv31r1C9OgwZAj7sdDfG5CwlJYXatWsDMGnSpLDvv3HjxmzcuJGkpCSAtDb/cGnfvj3Tpk3jxIkTJCcns2jRItq2bRvWMoJ8mRTCqkoVeOYZWLIEwvxFGmOiw4gRI3j44Ydp3bp12M7sQ5UrV46//e1vdO7cmfPOO4+KFSuS1UlrsE8heJsxY0aO+7/qqqto2bIlrVq1omPHjowePZrTTz893B8D8OmQ1KCwzX104gS0aQPJyToiKczVMWOKsmgakuqlgwcPUqFCBZxz3HPPPTRq1IihQ4d6HVbxGJIadjExMHYsbN0Kzz3ndTTGGB+aMGECcXFxnHPOOaSkpHDnnXd6HVK++LKmUOCL17Jy/fUwaxasXg229rMxuWI1hehWLGoKEVtk57nndKjq8OHh3a8xxviEL5NCxJxxBjz8MMyYAQsWeB2NMcYUOksKGQ0bBnXrwp//DBEYpWCMMdHMkkJG5crBCy/AihUwYYLX0RhjTKGypJCZa66BDh10+ou9e72OxhiTjYJMnQ06qdyXX36Z6XOTJk0iNjY23XUFP/30U7g/QlTx5TQXEScCr7wCrVvDY4/Bq696HZExJgvVqlVjeWAWvsymzs7JwoULqVChQpZrJlx33XW8ls1syhmnrM7tVN3RNKV3KKspZKVlS50Padw4WLnS62iMMXmwbNkyLr30Us477zyuvPJKduzYAcDYsWNp1qwZLVu2pF+/fiQlJfHGG2/w0ksvERcXxxdffJGr/Wecsjrj4yNHjnDLLbfQokULWrduzYLAwJVJkybRs2dPOnbsSKdOnSL2+Qsi+tJULoRtQrycPPEETJminc6ffKI1CGNM1qJg7mznHIMHD+b9998nNjaWadOm8cgjjzBx4kRGjRrFpk2bKFOmDPv27aNKlSoMGjQo29rFtGnTWLx4cdrjr776Ckg/ZfXChQvTPR4zZgwiwooVK1izZg1XXHEFP//8c9r7fvzxR6pWrZrvP0kk+bKmELHrFDKqVk0Tw2efwX/+E9myjDFh8fvvv7Ny5Uouv/xy4uLieOqpp9i6dSsALVu2ZMCAAbz77ru5brq57rrrWL58edqtXLlyAH+Ysjr08eLFixk4cCAATZo0oW7dumlJ4fLLL4/ahAA+rSkUqkGD4I034IEHoEsXKFvW64iMiV5RMHe2c45zzjkn7Yw+1Ny5c1m0aBFz5szh6aefZsWKFfkuJ1qmug43X9YUClXJktrpvGkTvPii19EYY3JQpkwZkpOT05LCsWPHWLVqFampqWzZsoWEhASee+45UlJSOHjwIBUrVuTAgQNhjaF9+/ZMnjwZgJ9//pn//e9/NG7cOKxlRIolhdzo1Amuukqn2N62zetojDHZKFGiBDNmzODBBx+kVatWxMXF8eWXX3LixAkGDhyY1vk7ZMgQqlSpQo8ePZg1a1aWHc3Tpk1LNyQ1q+Groe6++25SU1Np0aIF1113HZMmTUq3OE808+WEeEFhmzo7NzZtgqZNoU8fePfdwinTGB+wCfGiW7GYEM8T9evrFBiTJ0MuzhSMMcaPLCnkxUMPQa1aunRnaqrX0RhjTNhZUsiLChVg9GhYtgwisM6rMcZ4LaqSgoiUF5FEEenudSxZuv56aNdOp9hOSfE6GmOMCauIJgURmSgiu0RkZYbtnUVkrYisF5GHQp56EJgeyZgKLDgvUnIyPPWU19EYY0xYRbqmMAnoHLpBRGKA14EuQDOgv4g0E5HLgZ+AXRGOqeDi4+GWWzQ5BK5SNMaYoiCiScE5twjIOPd0W2C9c26jc+4oMBXoBXQALgCuB24XkUxjE5E7Ak1MicnJyZELPifPPKNrLwwd6l0MxhgARCRtWgnQGUhjY2Pp3l1bomfPns2oUaNyta+CTsUN/p6O24tpLmoDW0IebwXOd87dCyAiNwO7nXOZDu9xzo0HxoNepxDZULNx2mnw17/qMNV586BrV89CMcYvRi8ZTZtabUion5C2bcGmBSzdvpQRF43I937Lly/PypUrOXz4MOXKlWP+/PnUrl077fmePXvSs2fPXO2roFNxg7+n446qjmYA59wk59wH2b1GRHqIyPgUrzt6Bw+Gs8/W2kIuzyCMKc7a1GpD3xl9WbBJp5JesGkBfWf0pU2tNgXed9euXZk7dy4AU6ZMoX///mnPTZo0iXvvvReAm2++mSFDhtCuXTsaNGjAjBkzcrX/4jIdtxc1hW3AGSGP6wS25Zpzbg4wJz4+/vZwBpZnpUvDSy9Bt266EM8DD3gajjHRLqF+AtP7TKfvjL7cFX8X4xLHMb3P9HQ1h/zq168fTzzxBN27d+fHH3/k1ltvzfKAvGPHDhYvXsyaNWvo2bMnffr0yXbfx44dKzbTcXuRFJYCjUSkPpoM+qH9CLlWaOsp5EbXrnp74gkYOFCblYwxWUqon8Bd8Xfx5KInGXnJyLAkBNBpsZOSkpgyZQpdc2jO7d27NyVKlKBZs2bs3Lkzx32vXbs2bTpugBMnTlCzZs20cgcMGEDv3r3p3bt3rmLNqvkop+m4Bw8eDER2Ou5ID0mdAnwFNBaRrSJym3PuOHAv8BGwGpjunFuVl/0W2noKufXii3DoEDzyiNeRGBP1FmxawLjEcYy8ZCTjEselNSWFQ8+ePRk2bFi6pqPMhE5Ol5v534LTcQfXVFixYgUff/wxoNNx33PPPXz33Xe0adOG48eP5zv+aJiOO9Kjj/o752o650o55+o4594KbJ/nnDvbOXeWc+7pvO43avoUgho31tXZJk7Uq52NMZkK9iFM7zOdJxKeSGtKCldiuPXWW3nsscdo0aJFWPYX1Lhx42IzHXfUdTTnRtTVFABGjoTYWJ0XycczzxoTSUu3L03XhxDsY1i6fWlY9l+nTh2GDBkSln2FKl26dLGZjtumzg6nt96CP/1Jp9YeMMDraIwpFDZ1dnQrFlNnR13zUdAtt8B558GIEXDwoNfRGGNMnvkyKURl8xFAiRIwdixs3w65vHrSGGOiiS+TQlRr106bjl54ATZu9DoaYwqFn5uhi7L8fC++TApR23wUNGoUxMTA8OFeR2JMxJUtW5Y9e/ZYYogyzjn27NlD2bJl8/Q+62iOlKefhkcfhQULoEMHr6MxJmKOHTvG1q1bOXLkiNehmAzKli1LnTp1KFWqVLrt2XU0W1KIlCNHdF3nli3ho4+8jsYYY9IUudFHvlC2rF7Q9vHHEJhx0Rhjop0vk0LU9ykEDRqk6zo//7zXkRhjTK74MilE7ZDUjKpUgTvvhGnTICnJ62iMMSZHvkwKvnLffbqu80sveR2JMcbkyJJCpNWpo9ctvPkm7NnjdTTGGJMtSwqFYdgwnVp73DivIzHGmGz5Min4pqM5qHlzXYhn7Fg4fNjraIwxJku+TAq+6WgONWIEJCfDP/7hdSTGGJMlXyYFX7rkEmjbVudEOnHC62iMMSZTlhQKi4jWFjZsgFmzvI7GGGMyZUmhMPXuDQ0bwujRtjqbMSYqWVIoTDExOhJp6VL4/HOvozHGmD+wpFDYbrwRatSwqS+MMVHJkkJhK1cOBg+GefNgxQqvozHGmHR8mRR8d51CRnffDaecoiORjDEmivgyKfjyOoVQVavC7bfDe+/Bli1eR2OMMWl8mRSKhKFDdQTSyy97HYkxxqSxpOCVunWhXz8YPx5+/dXraIwxBrCk4K3hw+HgQXjjDa8jMcYYAEp6HUCx1qoVXHmlTpQ3dKgu4WmMKb4OH9Y50nbt0n+Dt8weP/88XHNN2EOwpOC14cPhssvg3XfhT3/yOhpjTDgdOpTrg7xLTkZ++y3T3RwvUYoD5WqQUjqWvSViSeYsKq2P5cIIhGxJwWsdO8K552rWv/VWKGEtesb40tGj8OqrnJg6HffLLmT3LmKOHMr0pcdKlObXUjXYUyKWXS6WHccbsf14DZKJTbvt4uTjA6mVqFJaOPVUHbxYtSo80DoyH8OSgteCE+X16wezZ+v8SMYY/3CO1P/M5vA9wyi/Yz3fcgHraJ/uAB88yKeUiuX4qbGUrlaRqtXSH+SrVoU6p0LLkMfB5ytX1llyCkPUJAURaQr8GagOfOqcKz7LlF1zDdSvD889B716aaIwxkS9Y8t+ZPcNQ6m5+jM205RR1efR4J4unHEGNKgK52U46JcrF/0/74gmBRGZCHQHdjnnmods7wy8AsQAbzrnRjnnVgODRKQE8A5QfJJCyZLwwANw772wZAlcfLHXERljsnFgwy42DhxJ86/fpDRVeKbmq9R95k7eGlCKUqW8jq5gIt2APQnoHLpBRGKA14EuQDOgv4g0CzzXE5gLzItwXNHnllugWjWdVtsYE5V2JP3OvI7Pk9qwEc2+nsh/6gzmh3+t4+Ft9zLgZv8nBIhwUnDOLQL2ZtjcFljvnNvonDsKTAV6BV4/2znXBRiQ1T5F5A4RSRSRxOTk5EiFXvhOOUUnypszB1av9joaY0yINasdr182i8P1m9F1wQjW12zP2n+t4JotL9OxT9WobxLKCy+GutQGQif82QrUFpEOIjJWRP5ONjUF59x451y8cy4+NjY20rEWrnvu0UZHmyjPmKjw5ZcwNGE5O5p15J5Pr6bcqWXZPvG/nLf9A5r3aeJ1eBERNeMfnXMLnXNDnHN3Oudez+61vp8lNSvVq+uw1H/+E7Zv9zoaY4ql1FQdCNij7U5WXXQ7YxaeS9tyKzgw6nVq7vqBWrdc6XWIEeVFUtgGnBHyuE5gW675fpbU7Nx/P5w4Aa+84nUkxhQrv/8OEydC66ZHWNLrOd5LbMStJSZxYvBQym9bR8UH79ZBIUWcF0lhKdBIROqLSGmgHzA7LzsosjUFgAYN4NprdT6kovj5jIkyKSk6vqN+Pcfc22YyN6kZz/EQ5bsnELN6FaXGjtELBoqJiCYFEZkCfAU0FpGtInKbc+44cC/wEbAamO6cW5WX/RbpmgLo1Bf79+sMqsaYiNi2TX9qZ5wBUx/8jv8e6cBM+lD77PIwfz4lZr8PZ5/tdZiFTpxzXseQZyLSA+jRsGHD29etW+d1OJFx2WU6CmnTJihd2utojCkyVq3SsRyTJ0Ps8R28W+9ROiS9jVSrBk89BbfdVuSbiURkmXMuPrPnoqajOS+KfE0BdOqL7dt1dTZjTIEcPw6ffAI9ekDz5vCfqUeYcd6zbDnlbBK2/hN54AFYtw7uvLPIJ4Sc+LKmEBQfH+8SExO9DiMynIPWreHYMVixwibKMyaPfv8dPv0U/v1veP992L0bqlV1jOs0g6u/Hk7Mls0619jzz0PDhl6HW6iKXE2hSHc0B4log+dPP8G84neBtzH5ceiQJoGBA6FGDejWDaZPhyuugM+e+ZqdTS7h2n/1JaZqFfjsM5g1q9glhJxYTSGaHTum/2Hr1oVFi7yOxpiotH8/zJ0LM2fChx9qYqhWTeeW7NP9CJftmUap8a/D0qWaKZ5+WqeVKaxpR6NQdjWF4t14Fu1KldLrFu67D776Ci6MxJIaxvjPnj16gdnMmTB/vi5lcPrpcNNNOunwpWduouSEcXD7RH1x06bw6qv6gooVvQ4/qvmyplAsRh8FHTwIZ54JHTpovdiYYmrHDvjPfzQRLFyo13jWrQtXX62J4MLzUykx/yN4/XVtci1RQvsM7rlHfz9FaYKiAsqupuDLpBBU5JuPgkaO1CrvmjXFcty0Kb42b9ZzoZkzdR4i56BxY00CV1+tixbKr3v1UuRx42DjRjjtNLjjDr3VqeP1R4hKlhT8btcurS3cdBP8/e9eR2NMRP38syaBmTNh2TLd1qrVyRpBs2aBk/7ERPjb32DKFDhyBNq311rBVVfZtT05sD4Fv6tRA26+GSZNgv/7P208NaaIOHxYr9N8/31NBKsC8xucf74uRnj11SEDhI4cgX9O1yaib7+F8uX1t3H33dCihVcfoUjxZU2hWPUpBK1bp/Xmhx/WpiRjfOLgQW0GSkr6479JSVoRBu0CaN9ek8BVV+n0E2k2bdL5wN56SzuOmzTRRHDjjbqAsckTaz4qKvr00atx/vc/G0FhokZKSvYH/T170r++dGntIK5bF+rV01v9+jqzS40aIS9MTYWPPtImorlzNWv06qVNRAkJ1nFcANZ8VFQMH6716zffhKFDvY7GFAPOwb59Jw/woQf74P19+9K/p2zZkwf7+Hj9NzQBnHZaDhfo790Lb7+tHccbNugbHn3UOo4LidUU/KZDBx1hsWEDRWJBWBM1fv1V2/NXrtRb8P7u3elfV778yQN86ME+eD82Np8n8cuWaV9BsOP44ou1VnD11dZxHGb5rimIyEDn3LuB+xc555aEPHevc+618IZqcjRihF67P22aXstvTB799pvOnhI8+AdvoYv9VawI55yjw/ybNNHmneBBv2rVArbcOKcXHaxercOsV6+Gr7/WpFC+vI6yu/tuaNmygJ/U5Ee2NQUR+c45d27G+5k9LkzFsqM5yDn9sYjADz9Yu6rJ0u+/6zE39Kx/5Urtsw0qW1aHeDZvrkmgeXO9nXFGGP5rHT+uha1effIWTAL79598XaVKWni/fpoQrOM44grSpyBZ3M/scaFxzs0B5sTHx9/uVQyeEYFhw3QY3kcfQefOXkdkPHb8OKxfn/6sf9UqHbB24oS+pmRJHbzWtq0uAx5MAg0ahGEKoEOHYO3a9Af91as1gKNHT76uZk2dbuKGG/Tfpk21GlKzpp3cRJGckoLL4n5mj01h6d8fHnlE1xC0pFDsbNmiE3wuWADff6/H4eCxV0TH9J9zjg5WC575N2oUhmb53bvTH/SDSWDzZq3BgvYgn3WWHuy7dUt/8LcagC/klBSaiMiPaK3grMB9Ao8bRDQyk7XSpXX00bBhOvNjmzZeR2QiKDlZ5/r59FNNBsEW0+rV9cy/c+eTTT9NmsAppxSwQOd0IMM33+ht+XJNAKE9zuXKadXjwgt1xtHgwb9RIyhTpoABGC/l1KdQN7s3O+c2hz2iPCiWo4+C9u/XqS+uuEInjDdFxv79OlP6Z5/p7YcfdHvFinDppdCpE3TsqEkgLGsv7d2rVwcHk8C33568uOCUUyAuTjsegmf8TZtqr7Mt/ORb+e5TyHjQF5FqwCXA/5xzy8IXosmzSpXgrru0CWnDBq2yG186ckQne/vsM60NLF2qfQFlysBFF+kF7B076pj/Aq8UefSonvkHE8A332iHBGjbU7NmeoHY+efr7Zxziv3ylMVNTjWFD4CHnHMrRaQm8B2QCJwFjHfOvVwoUWahWNcUQIf11aunV3e++qpW3U3UO35c53ILJoElS3SkUEyMtgQGawLt2unooHxzTq9pCU0A339/sgOiZs2TB//zz9esY1fKFwsFGX1U3zm3MnD/FmC+c+5GEakILAFeDl+YJs9q1tTTyEce0fbdq6/Wq57PP9/ryEyI1FQdERRMAp9/DgcO6HOtWumQ/I4d4ZJLtAKYb6HNQN9+q7dgP8App+hBf8iQk0mgTh0b9WP+IKeawnLnXFzg/qfABOfc1IzPFbZifZ1CZn75BcaO1WkB9u3To8uIEdCli7X7emTDBk0An36qo4SSk3V7o0aaADp10ovTY2PzWUBqqnY2LFlyshYQ/C0Em4FCawHWDGRC5HtCPBGZA3wMbAUmojWHfSJSDkh0zp0TiYBzq9g3H2V04IDOi/Tii7B1qx4Ihg/XIaw2TUDEOae1gWef1WQAUKuWJoBOnbSV78wzC1BASoquPTlvni5G/Msvuj1jM9B55xWwymGKuoIkhRrAE0BN4HXn3MeB7QnAec65FyIQb65ZUsjCsWMwdSo8/zysWAG1a+s6z3fcYQeLCEhN1bUAnn1WO4lPP13/3L1760J5+W6hcU6vQps3T29LlmiHxKmnwpVXak0wIcGagUye2dTZxZVzetXz6NHahlGpEgwaBH/+s57CmgI5dgzee08Xglm9Wq8OHjFCZ2rIdwfxwYNa3Qgmgi1bdHtcHHTtqrfzz7emIFMgBakpzM5ux865ngWMrUAsKeRBYqLWHGbM0GEuN9ygF781bep1ZL5z6JAuCfz887q0RcuW8NBDcO21+TxW//zzySTw+ec6OqhiRbj8ck0CnTtrbc+YMClIUkgGtgBTgG/IMN+Rc+7zMMaZZ5YU8mHDBu1zePttXQexRw89vb3oImuCyMG+fbrey8sva8fxRRfpQnhdu+bxT3fkiB78g4kgeJ1A06YnawMXX2z9QCZiCpIUYoDLgf5AS2AuMMU5tyoSgeaVJYUCSE7Wuetfe02vXr3wQk0OPXvaiKUMdu6El17SwV379+uJ+1/+oktH5lpSknYOz5unvdCHD+tUER07ahLo0kXnpzamEISlT0FEyqDJ4Xng/6JhLQVLCmHw229aa3jxRZ3m+OyztVnphhsKeOWU/23apE1EEydqi86112ozUevWuXjz0aPaMRysDfz0k25v0EAniuvaVeesKFcuop/BmMwUKCkEkkE3NCHUA2YDE51z28IcZ55ZUgij48d1qc/Ro+G773QJxCFDdCqNU0/1OrpCtWoVjBqlC4CVKKEdxyNG5OKC8W3bTtYGPvlEhwiXLq0H/2CzUKNG1kxnPFeQ5qN3gObAPGBqyNXNESEivdEEVAl4KzgENiuWFCLAOR2pNHq0jlwqX16Hsv75zzoJWhH29dc6rHT2bP3Yd94J99+fTR/vsWPw1VcnE8GPgUmEzzhDm4O6dtULFCpUKLTPYExuFCQppAK/BR6GvlAA55zLcdC7iEwEugO7nHPNQ7Z3Bl4BYoA3nXOjQp47FXjBOXdbdvu2pBBhP/yg7SdTp+oMbdWr6yyZTZrotBrB+/Xq+XaIpHN6Pdizz+r01FWragXp3nuhWrVM3rBjB/z3v5oE5s/XC8pKltSO4WDfwDnnWG3ARDVPr1MQkUuAg8A7waQQ6MD+Ge3E3gosBfo7534KPD8GmOyc+y67fVtSKCSbN+tQ1jVrdIWtNWtOztsA2kTSsOEfE0bjxlG7sEpqKsyapclg2TK9bGPYMLj99gwn9seP6xQSwauIv/9et9eqdTIJXHaZXRRofKUgE+IVmHNukYjUy7C5LbDeObcRQESmAr1EZDUwCvgwq4QgIncAdwCcWaA5A0yu1a0LDzyQftuePZoggklizRptjH///ZNrQIJOwRBaqwgmizPP9GSU09GjMHmyXnC2dq3msgkTtF89bW2YnTu16WzePPj4Y/j1V722o107zSJdupxcJ9uYIsarOn9t9PqHoK3A+cBg4DKgsog0dM69kfGNzrnxwHjQmkIhxGoyU62aHiTbtUu//ehRna45tFaxZo02Qe3bd/J15crpSKfQhNG4sXZwV66sjfphPOg++dlodiS2Yc6rCWzdqhcIj5y4gFMaLuVP7R7Q+SmCtYFg7fP003Wuii5d9EKyKlXCFo8x0SqqGoKdc2OBsTm9LmSW1MgHZfKmdOmTB/lQzmmTUzBJBBNGYqI2TaWmpn99TIwmh8qV9WAcvJ/xcXb3y5Rh506dQPaV2W34rUtfWsVPZ/z4BMqcPovrPriJ6UvawlWjteZTooRer/HUU9o01KqVXbNhih2vksI24IyQx3UC23LFOTcHmBMfH397uAMzESICNWro7ZJL0j935Ihe1bt2rc7/n5Jy8rZv38n7GzeevL9//8nF4rNwLKYMnKjMjVTh5lMrs2bFmdwa35mv3otlXM1tTP8XJBxacfK6gcsv155mY4oxr5LCUqCRiNRHk0E/4PrcvtlqCkVM2bK64HDz5jm/Nig1Va8DyJA8NnyfwoJ/72PT8hROJYU2TVM4t/4+Kqam0CglhbvWVuLJc7YxkktJ+NcLcO65VhswJkTEk4KITAE6ANVFZCvwmHPuLRG5F/gIHZI6MS9TZ1hNwVCiRFpTkXM6SnT0GB1WWqUK3P0XGDhYuwWCFmxawLgZfRkZP5JxieNIqHaABEsIxqRTGKOP+mexfR56UVyeWU3BgF47Nm2aXme3YoUuKzBmjA4rzbjU8IJNC+g7oy/T+0wnoX4CCfUS0j02xihfniY55+Y45+6oHKVj4E1k/fYbvPKKDie94QYdATtpkk4Ae//9ma89v3T70nQJIKF+AtP7TGfp9qWFG7wxUc4W2TG+sWuXTur6+uu6Rn379vDgg7YUtTF55enFa5FgzUfFy8aN2iw0cSL8/jv06qUT1F14odeRGVP0+PL8ypqPiodly6BfP51Y9M03YeBAnYF61ixLCMZEii9rCqbock5nnX7uOV2LplIlGD5cJ6mzZaWNiTxfJgVrPip69u2DOXN0hbPvv9cpk0aP1umrba45YwqPL5OCXafgf87pevUffKC3xYt1QtImTeCtt2DAgJAJ6owxhcaXScH409GjsGgRzJ2riSC4Xn2LFtpE1K2b9hXYSCJjvGNJwUTUzp068egHH+gs1AcOaA2gUye9pqBbN51F2xgTHXyZFKxPIXo5B8uXn2wWWrpUt9WuDddfr0mgY0edGdsYE33s4jVTYL/9piOF5s7V27ZtOilq27bQvbveWrWyNWmMiRZF7uI1473Nm0/2DXz2mV5UVrEiXHmlJoEuXXSWbGOMv1hSMLly4gR8/bUmgblzdQI60PmH7rpLE0H79rrGjjHGvywpmGzt3g0vvgjjx+viZCVL6sF/zBhNBGef7XWExphw8mVSsI7myEtOhhde0MnnDh2Cq6+Gvn3hiitsqWJjijJfjgi3uY8iZ+dOvWagXj1NCr16wcqVuoxy376WEIwp6nxZUzDh98svOq3EG29op/H118Ojj0Ljxl5HZowpTJYUirnt2zUZ/P3vupLZwIHwl79YX4ExxZUlhWJq61adiXTCBJ1z6MYbNRlYN40xxZslhWJmyxYYNUrXJ0hNhZtvhocfhgYNvI7MGBMNLCkUE5s3w7PP6uplALfeCg89pB3KxhgTZEmhiNu0SZPBpEk6zcSf/qTJwCahM8ZkxpdJwa5TyNmGDfDMM/DOOxATo4vVPPgg1KnjdWTGmGhm1ykUMevWaT9B48bw3ntw992aIF591RKCMSZnvqwpmD9auxaefhomT9b1CoYM0YvQatb0OjJjjJ9YUvC59evhscdg6lQoWxaGDtVkcNppXkdmjPEjSwo+9sMPkJCgy1wOGwYPPGDTVRtjCsaSgk/99BNcdhlUqACffw7163sdkTGmKPBlR3Nx9/PPusZxqVK64pklBGNMuFhNwWc2bNA1jlNTYeFCaNTI64iMMUWJJQUf2bxZE8Lhw5oQmjb1OiJjTFETNc1HItJARN4SkRlexxKNtm3ThLB/P8yfDy1aeB2RMaYoimhSEJGJIrJLRFZm2N5ZRNaKyHoReQjAObfROXdbJOPxq19+0YSQnAwffQTnnut1RMaYoirSNYVJQOfQDSISA7wOdAGaAf1FpFmE4/Ct5GQdZbRtG3z4IbRt63VExpiiLKJJwTm3CNibYXNbYH2gZnAUmAr0yu0+ReQOEUkUkcTk5OQwRht99u6Fyy/XzuU5c+Cii7yOyBhT1HnRp1Ab2BLyeCtQW0SqicgbQGsReTirNzvnxjvn4p1z8bGxsZGO1TMpKXDllbB6Nbz/vl6kZowxkRY1o4+cc3uAQbl5bVGfJfXAAejSRa9Y/ve/4YorvI7IGFNceFFT2AacEfK4TmBbrhXlWVJ/+w26d4dvv4Vp0/S+McYUFi+SwlKgkYjUF5HSQD9gdl52ICI9RGR8SkpKRAL0yuHD0KsXLF6ss51edZXXERljiptID0mdAnwFNBaRrSJym3PuOHAv8BGwGpjunFuVl/0WxZrC77/DNdfAZ5/B22/Dddd5HZExpjiKaJ+Cc65/FtvnAfPyu9+i1qdw7JgmgQ8/hAkT4MYbvY7IGFNcRc0VzXlRlGoKx4/DgAE6wui113QNZWOM8Yovk0JRceKELp35r3/BmDFwzz1eR2SMKe58mRSKQkdzaircfrt2KD/zDNx/v9cRGWOMT5OC35uPnNNawdtv61KaD2d5qZ4xxhQuXyYFP3NO11F+4w148EFNCsYYEy18mRT82nzkHDz0ELzyCtx3Hzz7LIh4HZUxxpzky6Tg1+ajxx+H0aPhrrvgxRctIRhjoo8vk4IfPfMMPPEE3HabDj21hGCMiUaWFArBmDHwyCMwcCD8/e9Qwv7qxpgo5cvDk5/6FF57DYYNg2uv1dFGMTFeR2SMMVnzZVLwS5/C+PEweLBOcjd5MpSMmonKjTEmc75MCn7wzjswaBB07apTYJcq5XVExhiTM0sKETB9OtxyC3TsCDNmQJkyXkdkjDG5Y0khzGbP1gnu2rXTSe7KlfM6ImOMyT1fJoVo7Wj++GPtUG7dGubOhfLlvY7IGGPyxpdJIRo7mj//HHr3hqZN4b//hUqVvI7IGGPyzpdJIdp89RV06wb16sH8+VC1qtcRGWNM/lhSKKDvvoMuXaBmTfj0U4iN9ToiY4zJP0sKBbByJVx+OVSpogmhZk2vIzLGmIKxpJBPa9fCZZdB2bKaEM480+uIjDGm4Cwp5MPGjdCpk06F/emncNZZXkdkjDHhYRMv5NGWLZoQDh+GBQugSROvIzLGmPDxZU3Bq+sUfvlFE8LevfDRR9CyZaEWb4wxEefLpODFdQq7d2sfwvbt8OGHEB9faEUbY0yhseajXPj1Vx1ltGEDzJunU1gYY0xRZEkhBwcO6HUIq1bpvEYJCV5HZIwxkWNJIRuHDkH37pCYqLOddu7sdUTGGBNZlhSycOSIzmW0eDG8957eN8aYos6SQiaOHtXZTufPh0mT4LrrvI7IGGMKhy9HH0XS8eO6HsIHH8C4cXDTTV5HZIwxhceSQojUVF0xbcYMePFFXU7TGGOKk6hpPhKR8sDfgKPAQufc5MIs3zlNAu++C089BUOHFmbpxhgTHSJaUxCRiSKyS0RWZtjeWUTWish6EXkosPlqYIZz7nagZyTjysg5uO8+mDABHnlEb8YYUxxFuvloEpBuIKeIxACvA12AZkB/EWkG1AG2BF52IsJxpXEOHn4Yxo7V2sGTTxZWycYYE30imhScc4uAvRk2twXWO+c2OueOAlOBXsBWNDFkG5eI3CEiiSKSmJycXOAYn3wSnntOm47GjAGRAu/SGGN8y4uO5tqcrBGAJoPawL+Ba0RkHDAnqzc758Y75+Kdc/GxBVzm7Pnn4bHHdITR669bQjDGmKjpaHbO/QbckpvXikgPoEfDhg3zXd5rr8GIEXoNwltvQQkbh2WMMZ7UFLYBZ4Q8rhPYlmsFnSV14kQYPBh69YJ//hNiYvK1G2OMKXK8SApLgUYiUl9ESgP9gNl52UFB11OoWxeuuQamTYNSpfK1C2OMKZIiPSR1CvAV0FhEtorIbc6548C9wEfAamC6c25VXvZb0JpCp056gVqZMvl6uzHGFFkR7VNwzvXPYvs8YF5+9xuOPgVjjDF/5MvuVS9WXjPGmOLAl0nBGGNMZPgyKRS0o9kYY0zmfJkUrPnIGGMiw5dJwRhjTGT4MilY85ExxkSGL5OCNR8ZY0xkiHPO6xjyTUSSgc35fHt1YHcYw/GynKJSRmGVU1TKKKxy7LNEXxkFLaeucy7TGUV9nRQKQkQSnXPxRaGcolJGYZVTVMoorHLss0RfGZEsx5fNR8YYYyLDkoIxxpg0xTkpjC9C5RSVMgqrnKJSRmGVY58l+sqIWDnFtk/BGGPMHxXnmoIxxpgMLCkYY4xJY0nBGGNMGksKxhhj0vg+KYjIRBHZJSIrI7T/KiKySUSOi8gREbkwzPsvKyLfisgPIrJKRP4vD++N6GcvaHx5LCdJRFaIyHIRSQzjfjP9G4lIZxFZKyLrReShAuz/DBFZICI/Bf4+fw53GYF9Zfk9hLOcwP5iROR7EfkggmVk+n0XtJy8/Cay++7CWEah/b7D9htyzvn6BlwCnAusjND+/wE8FyhjFVAlzPsXoELgfingG+CCaPjsBY0vj+UkAdUL4/8HEANsABoApYEfgGb53H9N4NzA/YrAz0CzcJaR3fcQ7nIC+78feA/4INx/r+y+73CUk5ffRFbfXZjLKLTfd7h+Q76vKTjnFgF7M24Xkfoi8r6IJAYydeO87ltEKqNfzEOBMpxzbl84y3DqYOBhqcAtV+OEs/rs4VSQ+KJBFn+jtsB659xG59xRYCrQK5/73+Gc+y5w/wCwGqgdzjIC+87qewhrOSJSB+gGvBmyOaxlZKPA5eTleJDNdxfOMsL++w7HcSc7vk8KmRGRUuh/6vudzg3yOHpgz6v6QDLwNjAXqCUi5cNcRrC6vhzYBcx3zn2Tn/1ESiHF54CPRWSZiNwRgf2Hqg1sCXm8lVwcDHIiIvWA1ujZYNjLyOJ7CHc5LwMjgNSQbZH4e2X2fUfqe8nxt5rhuwtrGeH8/eTwWcLyGyqZ3zdGud7AOcBMEQH9nF+EvkBEPgFOz+S9jzjn3g/cL4lW3wajf/xE9AsYGcYycM6dAOJEpAowS0SaO+ci1k+QV4UU38XOuW0iUgOYLyJrAmdKviAiFYCZwH3Ouf2B/xNhldn3EM79i0h3YJdzbpmIdAjnvjPxh+87gmX1JpvfasbvLtxlhPn3k2U5hOk3VFSTQiv0wPtWVi9wzl2Wi/1sBbY6574JnEnsR5NEOMsIff0+EVkAdAaiJikERTI+59y2wL+7RGQW2pQQqaSwDTgj5HGdwLZ8CZy9zQQmO+f+HYkyQmX4HpaEsZyLgJ4i0hUoC1QSkXeB18NYBpDl9x3OzxIqy99qFt9dWMsICtPvJ8tywvUbKpLNR8AO4EoRKQEgIi0kH6duzrlfgC0hbXYVgJ/CWYaIxAbOIBCRcsDlQCTPmvKkMOITkfIiUjF4H7iCyCbFpUCjQNtsaaAfMDs/Owp8528Bq51zL0aijEA5WX0PYSvHOfewc66Oc65eYD+fOecGRuCzZPV9h7WcEJn+VrP57sJZRrh/P1mVE77fUEF7qr2+AVMCf6hj6Jn9bUA5YAawFlgOvFuA/ccBewL7T0XPXMJWBtAS+B74MfAl/rUgnz0Cf998x5eHMhqgI01+QEd4PRLJ/x+B7V3R0SYbClIecDHalvtj4P/BcqBrOMvI6XsIZzkh++xAYPRRBD5Llt93QcvJy/Egu+8ujGWE9fedTTlh+w3ZhHjGGGPSFNXmI2OMMflgScEYY0waSwrGGGPSWFIwxhiTxpKCMcaYNJYUjDHGpLGkYIwxJo0lBWPCLHCV6WYRucvrWIzJK0sKxoSZc24FOkXDjV7HYkxeWVIwJjJ2obNZGuMrlhSMiYxRQBkRqet1IMbkhSUFY8JMRLoA5dGFmay2YHzFkoIxYSQiZdE1ve8GVgBhXQjHmEizpGBMeD0KvOOcS8KSgvEhSwrGhElgMabL0XWOwZKC8SFbT8EYY0waqykYY4xJY0nBGGNMGksKxhhj0lhSMMYYk8aSgjHGmDSWFIwxxqSxpGCMMSbN/wPQBe2UKBFgJgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "training_error_ridge = []\n", "test_error_ridge = []\n", "\n", "for i in range(len(y_training_ridge)):\n", " training_error_ridge.append(mse(training_data_y, y_training_ridge[i]))\n", " test_error_ridge.append(mse(test_data_y, y_test_ridge[i]))\n", "\n", "error_fig_ridge = plt.figure()\n", "plt.figure(error_fig_ridge.number)\n", "plt.title(\"Error Plot Ridge Regression\")\n", "plt.xlabel(\"$\\lambda$\")\n", "plt.ylabel(\"MSE\")\n", "x_axis = [\"$1e-{6}$\", \"$1e-{3}$\", \"$1$\", \"$3$\", \"$5$\",\"$10$\",\"$20$\",\"$30$\",\"$40$\",\"$50$\",\n", " \"$1e2$\", \"$1e3$\", \"$1e5$\"]\n", "plt.yscale('log')\n", "plt.plot(x_axis, training_error_ridge, 'b')\n", "plt.plot(x_axis, test_error_ridge, 'r')\n", "# let's find the index with the minimum training error\n", "min_error_idx = np.argmin(test_error_ridge)\n", "plt.plot(x_axis[min_error_idx], test_error_ridge[min_error_idx], 'xg')\n", "plt.legend(['Training Error', 'Test Error', 'Min Test Error'])\n" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuu0lEQVR4nO3deXiU1d3/8ffJnslCQkggEJKgArJJgLCJqFVUFIpiVdRotY8a24qXferPR2lUrIpdtK11qQrWluKIBbS4VCtFBUUQDJvsEMhCEpYsZF8myZzfHzMJk2QmC8lktu/runIlc99nZk5YPjn5nnOfW2mtEUII4Vv8XN0BIYQQfU/CXwghfJCEvxBC+CAJfyGE8EES/kII4YMCXN2BrhowYIBOTk52dTeEEMJjbN++vVhrHWvvnMeEf3JyMpmZma7uhhBCeAylVK6jc1L2EUIIHyThL4QQPkjCXwghfJDH1PztaWhoID8/n7q6Old3RViFhISQkJBAYGCgq7sihOiAR4d/fn4+ERERJCcno5RydXd8ntaakpIS8vPzGTZsmKu7I4TogEeXferq6oiJiZHgdxNKKWJiYuQ3MSE8gEeHPyDB72bk70MIz+Dx4S+EEN7qy4On+ds32Zgazb3+2hL+Qgjhpl7bcJTlm3MI8Ov936h9K/yNRkhOBj8/y2ejsUcvV1ZWxl/+8pdzeu6LL75ITU1Nl9v//e9/Z+HChR222bBhA5s3bz6n/ggh3MvhU5VsyynltimJ+En494DRCOnpkJsLWls+p6f36AdAX4Z/V0j4C+E93tmaR5C/HzdNSnDK63v0Us9uyciAtmFbU2M5npZ2Ti/52GOPcfToUVJSUrjqqquIi4tj1apV1NfXM3/+fH79619TXV3NLbfcQn5+Pk1NTTzxxBOcOnWKwsJCfvCDHzBgwAC+/PJLu6//t7/9jd/85jdERUUxfvx4goODAfjoo4949tlnMZlMxMTEYDQaqa2t5fXXX8ff35+3336bl19+mbKysnbtBg4ceE7fqxCi79Q1NPH+jnyuGTuImPBg57yJ1tojPiZNmqTb2r9/f7tjDimltWXM3/pDqa6/RhvZ2dl6zJgxWmutP/vsM33fffdps9msm5qa9Jw5c/TGjRv1mjVr9L333tvynLKyMq211klJSbqoqMjhaxcWFuqhQ4fq06dP6/r6en3xxRfrBx54QGutdWlpqTabzVprrZctW6Z/+ctfaq21Xrx4sX7++edbXsNRO2fr1t+LEKKd1ZnHddKjH+stR4t79DpApnaQqb4z8k9MtJR67B3vBevWrWPdunVMmDABgKqqKo4cOcLMmTN5+OGHefTRR5k7dy4zZ87s0utt3bqVyy+/nNhYy26sCxYs4PDhw4Dl4rYFCxZw4sQJTCaTwwuqutpOCOFejFtzOT82jKnD+jvtPXyn5r9kCRgMrY8ZDJbjvUBrzaJFi9i1axe7du0iKyuLe+65hxEjRrBjxw7GjRvH448/ztNPP93j93rwwQdZuHAhe/bs4Y033nB4UVVX2wkh3MeBExXszCvjtimJTr1uxnfCPy0Nli6FpCRQyvJ56dJzrvcDREREUFlZCcA111zDW2+9RVVVFQAFBQWcPn2awsJCDAYDd9xxB4888gg7duxo91x7pk6dysaNGykpKaGhoYHVq1e3nCsvL2fIkCEALF++3G5/OmonhHBfxq25BAX4cdOxb3t1dWJbvlP2AUvQ9yDs24qJiWHGjBmMHTuWa6+9lttvv53p06cDEB4ezttvv01WVhaPPPIIfn5+BAYG8tprrwGQnp7O7NmzGTx4sN0J3/j4eJ566immT59OVFQUKSkpLeeeeuopbr75ZqKjo7niiivIzs4G4Ic//CE33XQTH3zwAS+//LLDdkII93Sm2sSa7flcH1FL1M/vO7tIpXl1IvRahinLnID7S01N1W3v5HXgwAFGjRrloh4JR+TvRYhz88oXR3hh3WHWffIsI/Z8275BUhLk5HT59ZRS27XWqfbO+U7ZRwgh3FhdQxN/35zL5SNjGbF3q/1GeXm99n6+VfZxU1OnTqW+vr7VsRUrVjBu3DgX9UgI0dc+2FVAcVU96TPPc/rqRJDwdwtbtzr4KS+E8Alms2bZ19mMDmlk+qxUS/ArZbkaqVkvrk4EKfsIIYTLbTxcRNbpKtI/+AuqecSvteUHAPTK6sS2ZOQvhBAupLXmtY1Hia85w5zvP297stuTvF0lI38hhHChLUdL2JZdyk+/+SeB5qb2DXpxkteWhL8QQriI1po/rT/MoMgQFpw5YL9RL07y2vKp8O/l7fzPeUvn6667jrKysg7bPPnkk6xfv/4ce9a5p556ihdeeKHDNmvXrmX//v1O64MQvm5TVjHf5ZzhgS/+Tkj20bM1/ma9PMlry2fC3wnb+TsM/8bGxg6f98knnxAVFdVhm6effppZs2ade+d6gYS/EM6jteaP72xmcGUxt2x4t/mgUyd5bflM+He0nf+5st3Pf/LkycycOZN58+YxevRoAG644QYmTZrEmDFjWLp0acvzkpOTKS4uJicnh1GjRnHfffcxZswYrr76amprawG4++67WbNmTUv7xYsXM3HiRMaNG8fBgwcBKCoq4qqrrmLMmDHce++9JCUlUVxc7LC/S5YsYcSIEVxyySUcOnSo5fiyZcuYPHky48eP50c/+hE1NTVs3ryZDz/8kEceeYSUlBSOHj1qt50Q4txsPFzEztoAFn6zkuAmmwGj7SSvk4Lf+j6u36u/Kx893c/fCdv5t9rP/8svv9QGg0EfO3as5XxJSYnWWuuamho9ZswYXVxs2Zu7eS//7Oxs7e/vr3fu3Km11vrmm2/WK1as0Fprfdddd+nVq1e3tH/ppZe01lq/+uqr+p577tFaa/3AAw/o5557Tmut9aeffqoBh/cIyMzM1GPHjtXV1dW6vLxcn3/++S17/zf3S2utMzIyWt7Ltg8dtWtL9vMXomNNTWY996Wv9cU//auu9wvo3WCyQQf7+fvMyN/RnElvzqVMmTKl1Z75L730EuPHj2fatGkcP36cI0eOtHvOsGHDWjZtmzRpEjkOlnTdeOON7dps2rSJW2+9FYDZs2cTHR3tsG9ff/018+fPx2AwEBkZybx581rO7d27l5kzZzJu3DiMRiP79u2z+xpdbSeE6Njav6xmT0E5D3/9NkFmO2ViJ03y2vKZ8Hfydv4AhIWFtXy9YcMG1q9fz5YtW9i9ezcTJkywu59+860ZAfz9/R3OFzS366jNubr77rt55ZVX2LNnD4sXL3a4739X2wkhHKtZYeR3B+sZX3iYG/ZtaN/AiZO8tnwm/J2wnX+He/KXl5cTHR2NwWDg4MGDfPutnR36emjGjBmsWrUKsNxJ7MyZMw7bXnrppaxdu5ba2loqKyv56KOPWs5VVlYSHx9PQ0MDRpsZ8Lbfn6N2Qoiue+Of33AqvD9PfrEUP9rsquzkSV5bPnWFby9v599qP//Q0NBWN0efPXs2r7/+OqNGjWLkyJFMmzat997YavHixdx2222sWLGC6dOnM2jQICIiIuy2nThxIgsWLGD8+PHExcUxefLklnPPPPMMU6dOJTY2lqlTp7YE/q233sp9993HSy+9xJo1axy2E0J0TWFZLW+MnMXcA18xqeBg65NKOeVKXkdkP38PVl9fj7+/PwEBAWzZsoWf/exn7Nq1y9Xd8vm/FyHsMhr5xUeH+SQhhS+W3U9CRVHr807YxqGj/fx9auTvbfLy8rjlllswm80EBQWxbNkyV3dJCGGP0cimZ15m7Q1P8ODmd9sHfx/V+W1J+Huw4cOHs3PnzlbHSkpKuPLKK9u1/fzzz4mJiemrrgkhbNQ++RS/mvUow0oLeGDLqtYnk5Iswd8HdX5bTg9/pVQOUAk0AY1a61SlVH/gn0AykAPcorV2PFspuiwmJsYtSj9CiLNeTLyEvOh4Vr6ziJBG09kTfVznt9VXq31+oLVOsak9PQZ8rrUeDnxufSyEEN7FaGRvyiW8OXk+C3Z/xvTje1qf74P1/I64aqnn9cBy69fLgRtc1A8hhHAOo5GGn/6MReNvIrqmgl99+Vbr8y6o89vqi/DXwDql1HalVLr12ECt9Qnr1yeBgfaeqJRKV0plKqUyi4qK7DURQgj3lJHBnybOZ0/8cJ7971/oV1999lwfrud3pC/C/xKt9UTgWuABpdSltiet+0/YXW+qtV6qtU7VWqfGxsb2QVe77+KLL+60zddff82YMWNISUlp2bhNCOHdtuhIXpt2Ewt2f8bsw1vOnmiu87sw+KEPwl9rXWD9fBr4FzAFOKWUigewfj7t7H44y+bNmzttYzQaWbRoEbt27SI0NLTT9lprzGZzb3RPCNHXjEbKRozmf+c+zLDSQp78vM0SbBfW+W05NfyVUmFKqYjmr4Grgb3Ah8Bd1mZ3AR84sx/OFB4eDlj28rn88su56aabuPDCC0lLS0NrzZtvvsmqVat44oknSLP+pH/++eeZPHkyF110EYsXLwYgJyeHkSNH8uMf/5ixY8dy/Phxh+0cbQOdlZXFrFmzGD9+PBMnTuTo0aMO308I4QRGIzo9nUVjbqDE0I8/f/Q8YQ02e2C5uM5vy9lLPQcC/1KWmxMEAO9orf+jlPoOWKWUugfIBW7p6Rv9+qN97C+s6OnLtDJ6cCSLfzimy+137tzJvn37GDx4MDNmzOCbb77h3nvvZdOmTcydO5ebbrqJdevWceTIEbZt24bWmnnz5vHVV1+RmJjIkSNHWL58OdOmTeu03cqVK1m2bBm33HIL7733HnfccQdpaWk89thjzJ8/n7q6Osxms8PXufTSSzv/hoQQ3ZORwbKx1/DpyBks+vItxp06evaci9bzO+LU8NdaHwPG2zleArS/EsnDTZkyhYSEBABSUlLIycnhkksuadVm3bp1rFu3jgkTJgBQVVXFkSNHSExMJCkpqWUPoI7a2dsGurKykoKCAubPnw9ASEhIh68j4S9E79voF8NvL7ub6w5uIn3b+2dPuHA9vyNec4Vvd0boztKV7Zm11ixatIj777+/1fGcnJxWW0J31K7t+3Q0iezodYQQvchoJOc3L/LgvP9jRHEez3/yIq3uxusmdX5bPrOls7u45ppreOutt6iqqgKgoKCA06fbz3d3tV2ziIgIEhISWLt2LWDZ9K2mpqbbryOE6CajkYqFD5E+9Sf4aTNL33/Wbev8trxm5O8prr76ag4cOMD06dMBy4Tx22+/jb+//zm1s7VixQruv/9+nnzySQIDA1m9erXD14mLi3PSdyiEb6l7YjHpsx/mWP8h/H31YhLLT5096WZ1fluypbPodfL3IryF0QgZGZCXZ6nctMpxo5Gmxx9nYcptfDpyBi9+9AI37N9w9slKgYuXbMuWzkII0U1GI6SnQ02N5XFuruUxQBqWJZ2LZ9zFpyNn8Pjny1oHP7hlnd+WhL8QQtiRkXE2+JvV1FiO304GT09P4+2Jc7j/2zXcm9nmUiU3rfPb8vgJX08pW/kK+fsQ3iIvz8Hx3CaeHHEtf0u9np9kfsBjG//euoEb7NvTFR498g8JCaGkpISYmBisF5IJF9JaU1JS0nKNgRCeLDHRUuppTZN4zVZWpMwlfet7LNrwt9ZLOp1wK0Zn8ejwT0hIID8/H9nx032EhIS0XOgmhCe77jp4/XVo/mVWBTQxcG4mjDzDA5v/yf/7ekXr4PeAUo8tjw7/wMBAhg0b5upuCCG8jNEIy5efDX6/0HrifpRJ8OAzPP7FX7nnu7Vng18pO0uB3J9Hh78QQjiD7WRvQEwlcT/KxD+8DtYmcu/htWcbelCZpy0JfyGEaKN5stcwqpCY2d+jG/w59e40Ggr7nW3kYWWetiT8hRDCltFIYsBMKi6tJDI1h7r8aIo/mEhTVQhJ5FjauPGVu10l4S+EEM2MRg4ueob4tAwY2J+K74ZxZsOFYPbDQDVL+JVb7tB5Ljx+nb8QQvSGJrPm9RVfMu+W39IQrrjjvQ1EfmFAmSGJHJZyH2msdPsrd7tKRv5CCJ+3e9m7PP5dKXtS5nPtoW949rNXiamt4FleaN3Qw+v8tiT8hRA+60y1iedf/ZiVJWHEBpp4+YPfMffg19i9ZNQL6vy2JPyFED6nrqGJt175F68d19QEBPOT7R/yv5uMRJjs3BjJYPCI7Rq6S8JfCOEz6hqaWLM9n1c+/p6TjaFcmbeN/9u4nJHF7fZxsPCy0b4tCX8hhNerrm9k5bY8ln51jNOV9UwszuHPn73B1Px9jp/kwRdwdYWEvxDCa+WWVPOPLbmsyjxOZV0j08Ma+NOXr3HxtnX26/rNvGhi1xEJfyGEV6lvbGL9/tOs3n6cjYeL8FeK2WMH8T/Vh5n4i3vab9LflheXemxJ+AshPJ7ZrMnMPcPH3xfy4e5CymoaiO8XwoNXDCetIJOBv/q5vf2ZW/PSiV1HJPyFEB7J1GhmW3Yp6w+c4tO9JzhVUU9IoB9XjR7EzZMSmHHBAPxXvgMPpMto3w4JfyGExzheWsOmrGK+PlLE14eLqaxvJDjAj0tHxDL3onhmjRpIWHCAZU/mqzM6H+2D10/sOiLhL4RwS1prjpfWsi2nlG3ZJWzNLiW3xDKCHxgZzJyL4rly1EAuuWAAoUH+Z5/Y9s7rHfGBiV1HJPyFEG6huKqevQXl7C0oZ9fxMnbmlVFSbQIgyhBIalJ/7pqezMzhA7ggLrz9rVuNRstG/F0Z7YNPlnpsSfgLIfpUjamRo6erySqq5ODJSg6eqOTgyQpOVdS3tDk/NozLR8aRkhjFlOT+DI8Lx8/PzuJM28BX6uyttzriYxO7jkj4CyF6XY2pkeOlteSV1pBbUk1OSTU5xTUcK6qisLyupV2Qvx/nx4Uz/bwYxg7px9gh/Rg9OJLIkMDO36Rteacrwe/jo31bEv5C9ILmAWhenkfezrVb6hqaOF1Rz6nKOk6WWz4Ky2spLKulsKyOgrJaSq3lmmaRIQEMGxDGlGH9uSAunPNjw7kgLpzkAWEE+ndzZ/nulndARvt2SPgL92ebrP37W46Vlvbs615M6LYD0Nxcy2PwjKxpMmsqahsorTFRWm2ipMrEmRoTJVX1FFeZKKk2UVRZR1FlPUWV9VTUNbZ7DUOQP/H9QhgSbWDskH4kRIeS2N9AYn8DQ/sbiDYEtq/Rd8e5lHeayWjfLqW784foQqmpqTozM9PV3RC9qauhXlkJJpPj1zlXzSESE2P/vbv4AyI52f4gtC9XEDY2mamqb6SyrpHy2gYq6xqpqGugoraB8toGKuoaKa8xUV7bQFltA2U1luNnrMccxUBESAAxYUHERgQTGxHMgPBg4iKCiYsMYWBkCIMiQ4iPCiEiOKBn4d6R7qzesSWjfZRS27XWqXbPuSr8lVKzgT8D/sCbWuvfdtRewt+D2Qv5kpLuj+BcoQs/IPxys9F2dopRCsxmxy+ttaa2oYmq+kaq65uoqmu0ft1ItcnydVWd5XGl9euq+saWkK+sa2j5usbU1Om3EREcQJQhiH6hgUQZAokyBBEVGki0IZDosCCiDUFEhwURExZEf+tHSKB/h6/rVOdS3mn++5LRPtBx+Luk7KOU8gdeBa4C8oHvlFIfaq33u6I/wgkc/ZpeUnK2jbsHP7TqtwbqA4KorG2iOiiUqthhVDeFknjePoqC++MX1IgKasQvsAkV1Ei/4HIe+s0JqgcnUFVvCejmcK+pb6La1Ii5C38ESkF4UABhwQFEhAQQHmL5PCQqlPDgs48jQgKJCAkgMiSQyFDL536hgUSGBhIeHIC/vdUy7kbKO33GVTX/KUCW1voYgFLqXeB6QMLfkzn6j+sGIV/vH0BlcBgVwWGWzyGWz1VBBiqCDVQFG6gKMlBp/boyyPK5Oii05Xh1UChNfvZGwrnE0np0qhv8CDNVszOnjLDD+YTrBqIb6hlaXUFYkB9hkycRPn4MYcGWUA8P9icsKKAlzMOCA4iwnjME+TuvpOJqbX8rtC3xdfXfjZR3zomrwn8IcNzmcT4w1UV9ET3Rh4GvgcogA2cMkZSGRlIWGkFZSETL5/KQcMpDwikLtXxdYQ378pBw6gODO319g6mWiPoawk01hNfXEmGqJq6qlDDr8TBTLWGmWsKtny1fW45vMP2AP5se5rgpkaGmQp7TGZabfTtgXH07GUwkj0QS/QpYYn6U+TGfWU728oS0W+hK6c/2t8LOSHmnx1xS81dK3QTM1lrfa318JzBVa72wTbt0IB0gMTFxUm53an/CeXryq3kbjcqP4rBoisKiOB3en6KwaIrDoig2RFEcFkVpaD9Kw/pREhbFmeBwGv0dj1ci6quJqq2kX2Md/eqr6Vd5hn40EmGqoV9ZMRH+EGmqIaLkFBGmGiLqqomor7YGfi3+uoMCfS8ychvpLKOGsJZjBqpZyn2tf2DYm2/wpB8KvfjvpBUJ/C5zuwlfpdR04Cmt9TXWx4sAtNa/cfQcmfB1sXP4j9yo/DgRMYDCyFgK+sVxImIAJyIGcDJiACcjYjgV3p/isCi0ar/OO8JUQ0x1GTGNtfQ/P5H+I86jf7hlMjLKEET/MNsJyyAiQwO7XNM2GiHjoSrySgwkkseSsOdIC3m/9WSuEyekk8kml+R2x5PIIYdhnb9AZ5PQTlrS2kpnK7Wc8ecn5Z1uc8fwDwAOA1cCBcB3wO1aa4f3VJPwd4EuBH5NYDDHooeQGx1PTvRg8qIGkRsVT36/OE5ExrarkUfVVTGoupRBZacZRD1x0ycRd3EqcTZLCWMjgp22ysTeqkGHmeKkpah+NKFp/wNPYcaME75v2xLJddfBJ5/07JqJvlypJeWdHnG78AdQSl0HvIhlqedbWusOt9aT8O9jbVKyOjCEg7HJHBmQyOEBiRwZkMjRmAQKI+NaPW1A1RkSy08ytOwUQ8tPkVB+iiEhisH338XgO25pvfuiCzhlTX43l7L2eOTvBozcRgbPWeYsyGMJv+pwjqPLAgMhMtLzSlxuyi3Dv7sk/PuI0UjFU8+w1xTM94OGsyf+AvbHnUdOdHxLeSbUVMcFJce5oOQ455UWcF5pPslnCkkqO0l4Q51bj9T8/OwPWDtbk39OHPxQMHI76SztvObvpro8Z9FVMrp3Ggl/0aHjpTV8e6yEHZ9/x/asUxzpn9AS9AllJxl76iijTmdbPopyGFJ+Gj90+9FfzB9J+/NUt/7P6w5X42I0YnxoKxklv7RZ7fMYac2rfdz8Arhz/s3F0yewPZCEv2jlTLWJr7OK+epwEVuOllBQVgtAZF0VEwsOMrHwICmFhxh3Movoukq7r2F39OoB83Hdqvm7khtfFd2tOQsZ1buUhL+P01pztKiKz/ad4r/7T7E7vwytIaq+imk5u5met4epeXsYUZyHHx38e7D5j5xctZfckvB2TTzhjngevQNndza5c9IPiw5H/jGpZ/vhcX+43kfC30cdLarig50FfLznBMeKqgEYPzSKH9Sf4LJXnuGinL1dX9veZuTWp7Vzce7a/qTrhdU+xpJrSFfLqNGGlrdxy9+ehPvt7SOcp6KugbU7C1idmc+egnL8FEw7L4afBBZx1WvPMujQHktyN3W8EVgLB/+rExPt184TE3vhmxC9Jy2t1xM5DcCTf3sSgIS/19hbUM7yzTl89H0hdQ1mxgyO5PE5o5g3fjBxH70Hv7ApdHc1+Duo0y5ZYr927qP3wvY5TviZIvqYhL8HM5s1Gw8XsfSrY2w5VkJYkD/zJyRw+5RExiX0s/zKf2c3t8SFLv0O33xKRn9CeCYJfw+ktWb9gdP88b+HOXCigvh+IWRcN4oFU4aevfdpd2+AcQ6rMmT0J4TnkvD3MN9kFfP7/xxkd345yTEG/nDzeOalDG5/H9SMjM6D39/fMjsrw3YhfE4375wsXCWvpIb0f2SS9uZWiqtM/P6mi1j/y8v40aSE1sFvNEJyMsbci0kmGz+aSCYbI7e1fkGDAZYvt4R/To4EvxA+Rkb+bs7UaObVL7N4beNRAvwUj1wzknsuGWZ/4zNrqcdYc32ry+9zSSadZQCWy+/lghshfJ6Evxvbk1/OI2t2c/BkJdenDGbRtaMY1C+kfcM29zrN4LlWV94C1BBGhvotaSvmSOgLIST83VFjk5mXPj/CqxuOEhMWxF/vSuXKUQPtN7YzsZuH/cX2eXqoBL8QApDwdzunK+p4cOVOtmaXcuOEISz+4Rj6GQIdP8HOxG4ieXYvv09M8tL7wAohuk0mfN3I5qPFXPfSJnbnl/HCzeP544IUx8Fvndi1t4Z/Cb/CQHWrY3IBlhDCloz83cQ7W/N44oO9JMcYMN47lZGDIhw37mQNf/Oe6hn+vyevaQiJSUrmd4UQrUj4u5jZrHl+3SFe23CUy0bE8srtE4gI6WC0n9GFK3YNBtKWziEtLaH3OyyE8AoS/i5kajTz8OrdfLS7kNunJvL0vDEEtL1Yq1lXr9iVZZxCiC6Q8HcRU6OZhe/sYN3+Uzw6+0J+etl5KNXBhGxXrtj1hM30hRBuQSZ8XcDUaObnRkvwP/XD0fzs8vMdB38HE7utyIyuEKIbZOTfxxqazPzcuJ31B07z9PVj+PH0ZMeNpdQjhHASCf8+pLVm0ft7WH/gNM9cP4Y7Owp+6LzUI7dPEkKcIyn79KE/rT/Cmu35PHTl8I6DvyulnqQkCX4hxDmTkX8feXdbHi99foSbJyXwi1nDHTfsSqlHJnaFED0kI/8+8O2xEjLW7uXSEbE8d+O4nq3qkYldIUQvkPB3stMVdSx8ZydJ/Q28evuE9jddaSsvz/E5KfUIIXqJlH2cqLHJzMKVO6mub8R471THV+7C2at3tbZ/Xko9QoheJOHvRM+vO8S27FJeXJDSo716pNQjhOhtUvZxko2Hi3hj4zHumJbIDROGdNy4ozq/lHqEEE4g4e8E5bUNPLrme4bHhfP4nNGOG3a2pFMpub+uEMIppOzjBM9+vJ+iqnreuHOS/XvtQteWdCbavyOXEEL0lIz8e9kXB0+xens+P7vsfMYPjbLbxmiE5Lsuw6+mkmSyMXJb+0ZS5xdCOJGEfy8qr2lg0ft7uHBQBA9eeYHdNs0D/tymBDR+5JJMOsta/wCQOr8QwsmcFv5KqaeUUgVKqV3Wj+tszi1SSmUppQ4ppa5xVh/62p/WH6aosp4Xbh5PcID9ck/GQ1XtKj01hJHBc5YHzUs6JfiFEE7k7Jr/n7TWL9geUEqNBm4FxgCDgfVKqRFa6yYn98WpDp6sYMW3udwxLYmxQ/rZb2Q0kldip8QD5JEopR4hRJ9xRdnneuBdrXW91jobyAKmuKAfvUZrza8/3E9ESAC/vGqE44YZGSRi/wreRP9CKfUIIfqMs8N/oVLqe6XUW0qpaOuxIcBxmzb51mPtKKXSlVKZSqnMoqIiJ3f13H269yRbjpXw8NUjiTIEOW6Yl8cSfoWB6laHDVSzZHmCBL8Qos/0KPyVUuuVUnvtfFwPvAacD6QAJ4A/dPf1tdZLtdapWuvU2NjYnnTVaWpNTSz59wFGxUdy+xQHSzOb1/NrTRorWcp9JJGDwkwSOSyNWSS5L4ToUz2q+WutZ3WlnVJqGfCx9WEBMNTmdIL1mEd665tsCspq+cMt4/H3s7Nbp531/GmsJI2VlgcGA/x5aR/1VgghLJy52ife5uF8YK/16w+BW5VSwUqpYcBwYJuz+uFMlXUNLP3qGD8YGcu082LsN5KtG4QQbsiZq31+r5RKATSQA9wPoLXep5RaBewHGoEHPHWlz1ubciivbeCXV4103MjRFs3NWzcIIYQLOC38tdZ3dnBuCeDRaxrLaxp4c9Mxrh49kHEJdpZ2drZFs2zdIIRwIdnb5xy9uekYlXWN/K+9pZ2yRbMQws3J9g7noLTaxFubspkzLp5R8ZHtG0idXwjh5mTkfw7+9k02NQ1NPOToRuxS5xdCuDkZ+XdTramJt7/NZdaogYwYaHN3rua1/H5+lg97pM4vhHATMvLvpvd35nOmpoF7Lxl29mDbGn+TncVLUucXQrgRGfl3g9ms+eumbMYN6ceUYf3PnnBU4/f3t5R6pM4vhHAzMvLvho2HizhWVM2fb01BKZureR3V+M1my4cQQrgZGfl3w5ubjjEoMoTrxsW3PuGoli81fiGEm5Lw76L9hRV8k1XCXRcnE+hv/WOzvQG7arOvj9T4hRBuTMK/i/6xJYfQQP+zO3e23I8x1/JY67M/AKTGL4Rwc1Lz74Lq+kY+2l3InIvi6WcItBy0N8mr9dnbMAohhBuTkX8X/Pv7E1Sbmrh1ss1O1I4meR0dF0IINyLh3wX/zDzOebFhTEqKPntQJnmFEB5Mwr8TWacr2Z57hgWpQy3LO2WSVwjhBST8O/HP744T4Ke4cWKCTPIKIbyGTPh2wNRo5v0dBVw5Ko7YiGCZ5BVCeA0Z+Xfgi4OnKKk2cetkax1fJnmFEF5Cwr8D7+0oYGBkMJeOiLUckEleIYSXkPB3oKKugY2HipgzbjD+K9+RSV4hhFeR8Hfgv/tOYWoyM+fE9zLJK4TwOjLh68C/95xgSFQoE597QCZ5hRBeR0b+dpTXNPD1kSLmXBSPkkleIYQXkvC347N9J2lo0sy9KF4meYUQXknC346P95wgMaiJcZekyCSvEMIrSfi3UVpt4pvDp5mz+UOUTPIKIbyUhH8b/9l7kiYUfnuCSSYbP5pIJhujvvXsJK8EvxDCw8lqnzY+3XuCmNIKnjr9O2oIByCXZNJZBrnpSOwLIbyBjPxtVNc3svVYKSVHz28J/mY1hJHh/zsX9UwIIXqXhL+NTVnFmJrMnDpynt3zeU1D+rhHQgjhHBL+Nr74eDMRphoGFlTaPZ+YpOweF0IITyPhb2V+28gX+TVcejSTJeYMDFS3Oi+rO4UQ3kTC32rfH16nKCyaK45mksZKlnIfSeSgMMvqTiGE15HVPlafhw1FaTOXH8sEII2VpLHSsr4/x+zi3gkhRO/q0chfKXWzUmqfUsqslEptc26RUipLKXVIKXWNzfHZ1mNZSqnHevL+venLUTOYUHiImNqK1idkGwchhBfqadlnL3Aj8JXtQaXUaOBWYAwwG/iLUspfKeUPvApcC4wGbrO2danTlXXsjknmirxdrU9IoV8I4aV6FP5a6wNa60N2Tl0PvKu1rtdaZwNZwBTrR5bW+pjW2gS8a23rOkYjG+b9BIArTuyDmBhLqUcK/UIIL+asCd8hwHGbx/nWY46O26WUSldKZSqlMouKinq/l0YjpKfzRdQw4iuKGHVkF9TWwooVso2DEMKrdRr+Sqn1Sqm9dj6cPmLXWi/VWqdqrVNjY2N7/w0yMmiqrWNL4kXMzNmJAsuNWzIyev+9hBDCjXS62kdrPescXrcAGGrzOMF6jA6O9728PPbHnUd5aAQzcna3Oi6EEN7MWWWfD4FblVLBSqlhwHBgG/AdMFwpNUwpFYRlUvhDJ/Whc4mJbE4aD8D0vO9bHRdCCG/W06We85VS+cB04N9Kqc8AtNb7gFXAfuA/wANa6yatdSOwEPgMOACssrZ1jSVL2HzeBC4oziOu+ozlmKzwEUL4gJ6u9vmX1jpBax2stR6otb7G5twSrfX5WuuRWutPbY5/orUeYT3nmpQ1GiE5GdNdP+G7waO4+NRhWeEjhPApvneFr3WFDzU1fD9kFDWBIVycu8uywkdCXwjhI3xvb5+MDMuKHmBz0niUNjP1yHZZ4SOE8Cm+N/K3WcmzOfEiRp86RnRdJeRVubBTQgjRt3xv5G9dyVMXEMSOIaO4uHmVj6zwEUL4EN8L/yVLwGBg+5BRmAICuTj3e1nhI4TwOb4T/tYVPtx5J4SG8s2F0/A3NzFZVcgKHyGEz/GNmr/NCh8ASkrYMnAkF4VpwrPs7UsnhBDezTdG/jYrfMBS798bN4wpO75wYaeEEMJ1fCP82+zVs2fgBTT4BzLpwDYXdUgIIVzLN8K/zUqeHUMuBGCiqnRFb4QQwuV8I/ytK3yabR8yiuSyEwx4cpELOyWEEK7jG+GflmZZ0ZOUhFaKHYljmJgcIyt8hBA+y7vDv3l5p5+fZdJ3yRLyiiopDolk4uUTXd07IYRwGe9d6tl2eWduLqSns+N3gUAYk5KiXdo9IYRwJe8d+bdZ3glATQ3bv9xOeHAAIwZGuKZfQgjhBrw3/B3cinF7ZAITEqPw91N93CEhhHAf3hv+djZqqwoK5VBsMhMSpeQjhPBt3hv+bZZ3AuxOHofZz0/q/UIIn+e94W+zvLP5Fo3b738EpSBlaJSreyeEEC7lveEPlh8AOTlgNkNODtujEhkRF0G/0EBX90wIIVzKu8PfhtaaXcfLmJAY5equCCGEy/lM+OefqaW8toFxCf1c3RUhhHA5nwn/PQXlAIwbIuEvhBA+E/57C8oJ8FNycZcQQuBL4V9YwYiBEYQE+ru6K0II4XI+Ef5aa/YWlDN2SKSruyKEEG7BJ8L/RHkdpdUmqfcLIYSVT4R/82TvGAl/IYQAfCT89xWU4++nGB0vZR8hhAAfCf89BeVcEBsuk71CCGHlE+G/t7CCsVLyEUKIFl4f/qcq6iiqrJeVPkIIYaNH4a+UulkptU8pZVZKpdocT1ZK1Sqldlk/Xrc5N0kptUcplaWUekkp5dS7quyVK3uFEKKdno789wI3Al/ZOXdUa51i/fipzfHXgPuA4daP2T3sQ4f2FJSjFIweLCN/IYRo1qPw11of0Fof6mp7pVQ8EKm1/lZrrYF/ADf0pA+d2VtQwfmx4RiCvPde9UII0V3OrPkPU0rtVEptVErNtB4bAuTbtMm3HnOavQXlUvIRQog2Oh0OK6XWA4PsnMrQWn/g4GkngEStdYlSahKwVik1prudU0qlA+kAiXbuydsZU6OZS4YP4JILBnT7uUII4c06DX+t9azuvqjWuh6ot369XSl1FBgBFAAJNk0TrMccvc5SYClAamqq7m4/ggL8eOHm8d19mhBCeD2nlH2UUrFKKX/r1+dhmdg9prU+AVQopaZZV/n8GHD024MQQggn6elSz/lKqXxgOvBvpdRn1lOXAt8rpXYBa4Cfaq1Lred+DrwJZAFHgU970gchhBDdpyyLbtxfamqqzszMdHU3hBDCYyiltmutU+2d8/orfIUQQrQn4S+EED5Iwl8IIXyQhL8QQvggCX8hhPBBHrPaRylVBOS6uh/dNAAodnUn+ph8z75BvmfPkKS1jrV3wmPC3xMppTIdLbPyVvI9+wb5nj2flH2EEMIHSfgLIYQPkvB3rqWu7oALyPfsG+R79nBS8xdCCB8kI38hhPBBEv5CCOGDJPz7iFLqYaWUVkp5/W3FlFLPK6UOKqW+V0r9SykV5eo+OYNSarZS6pBSKksp9Zir++NsSqmhSqkvlVL7lVL7lFIPubpPfUUp5W+9Le3Hru5Lb5Hw7wNKqaHA1UCeq/vSR/4LjNVaXwQcBha5uD+9znqzoleBa4HRwG1KqdGu7ZXTNQIPa61HA9OAB3zge272EHDA1Z3oTRL+feNPwP8BPjG7rrVep7VutD78lta37vQWU4AsrfUxrbUJeBe43sV9ciqt9Qmt9Q7r15VYwnCIa3vlfEqpBGAOlptQeQ0JfydTSl0PFGitd7u6Ly7yP3jn3dqGAMdtHufjA0HYTCmVDEwAtrq4K33hRSyDN7OL+9GrOr2Bu+icUmo9MMjOqQzgV1hKPl6lo+9Za/2BtU0GllKBsS/7JpxLKRUOvAf8Qmtd4er+OJNSai5wWmu9XSl1uYu706sk/HuB1nqWveNKqXHAMGC35X71JAA7lFJTtNYn+7CLvc7R99xMKXU3MBe4UnvnxSQFwFCbxwnWY15NKRWIJfiNWuv3Xd2fPjADmKeUug4IASKVUm9rre9wcb96TC7y6kNKqRwgVWvtaTsDdotSajbwR+AyrXWRq/vjDEqpACyT2VdiCf3vgNu11vtc2jEnUpYRzHKgVGv9Cxd3p89ZR/7/T2s918Vd6RVS8xfO8AoQAfxXKbVLKfW6qzvU26wT2guBz7BMfK7y5uC3mgHcCVxh/XvdZR0RCw8kI38hhPBBMvIXQggfJOEvhBA+SMJfCCF8kIS/EEL4IAl/IYTwQRL+QgjhgyT8hRDCB/1/VprG48RWS0oAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Let us visualize the newly fitted model with the optimal lambda value here\n", "x = np.linspace(-5, 5, 100)\n", "new_features = get_polynomial_features(x, 10)\n", "new_norm_feat = normalize_features(new_features, mean_train_feat, std_train_feat)\n", "y_pred = eval(new_norm_feat, weights_ridge[min_error_idx])\n", "\n", "plt.plot()\n", "plt.plot(test_data_x, test_data_y, 'or')\n", "plt.plot(training_data_x, training_data_y, 'ob')\n", "plt.plot(x, y_pred)\n", "plt.legend([\"test_data\", \"training_data\", \"inference\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### 1.4) Error Plot (1 Point)\n", "In the lecture we have seen and analyzed the plot of polynomial degrees \n", "against the error (slide 47).\n", "Similarly, now please analyze the relationship between the error and the \n", "different values of $\\lambda$, as well as the reason behind it.\n", "\n", "Hint: Do not forget that we are in log space. Small changes in the y-axis mean high differences in the error values.

\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$\\lambda$ is regularization term such that the model cannot fit the training data perfectly anymore. Increasing $\\lambda$ strongly encourages the weight values to decay towards zero, unless supported by the data. In our case, a polynomial feature with degree 10 is highly overfitting to the training data and, therefore, with higher $\\lambda$ values (10-15), we can punish overfitting, which leads to lower MSE values. Increasing the $\\lambda$ value leads to an decrease in the error due to the higher punishment of high weight values and tackles though this the high polynomial degree. Of course, with low $\\lambda$ values, the MSE on the training data is low because there is no punishment of overfitting.
\n", "On the other hand, if $\\lambda$ is increased too far, the introduced penalty will force the regression to become to simple and underfit the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Probability Basics and Linear Classification\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## First Example (Two Moons)\n", "\n", "Let us start by loading a very simple toy dataset, the \"two moons\"." ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAi+0lEQVR4nO3df5Ac5X3n8fdXqxVafD6tfh2GXQnJgeBEWCCyBz5zhR3JNpiKJYXwy6lgkcMl+4LPZ6hTIYorhVPsszAuQ1z2hShALC4pQJKJELEdApIpV9mBaIlA/PBhBP7BrmVDJKQ7W0II8b0/pkeane2e6dnp6R/Tn1fV1s5098w+M2r108/3+T7PY+6OiIiU16SsCyAiItlSRSAiUnKqCERESk4VgYhIyakiEBEpuclZF2AiZs2a5fPmzcu6GCIihfLkk0/+q7vPrt9eyIpg3rx5DA8PZ10MEZFCMbOfhm1XaEhEpORUEYiIlJwqAhGRkitkH4GISKcdOXKEkZER3njjjayL0rKpU6cyODhIb29vrONVEYiIhBgZGeGd73wn8+bNw8yyLk5s7s7evXsZGRlh/vz5sV6jikBya8vOUW59+AV+vv8Qp/T3serCM1i+aCDrYklJvPHGG4WrBADMjJkzZ/Laa6/Ffo0qAsmlLTtHufGBZzh05CgAo/sPceMDzwCoMpDUFK0SqGq13Ooslly69eEXjlUCVYeOHOXWh1/IqEQi3UsVgeTSz/cfamm7SBkcPnyYK664gtNOO43zzjuPn/zkJ4m8ryoCSd2WnaOcv24781d/i/PXbWfLztFxx5zS3xf62qjtImVw1113MX36dHbv3s11113HDTfckMj7qiKQVFVj/6P7D+Ecj/3XVwarLjyDvt6eMdv6entYdeEZKZZWJL44Nzituueee1i4cCFnnXUWV111FQ8++CArVqwA4NJLL2Xbtm0kscqkOoslVY1i/7WdwNXHyhqSIuhEcsNzzz3H5z//eX7wgx8wa9Ys9u3bxwUXXMCcOXMAmDx5MtOmTWPv3r3MmjWrrfKrIpBUtRL7X75oQBd+KYS4Nzit2L59O5dddtmxi/yMGTPaLmcUVQSSqlP6+xgNuejXxv7zMn4gL+WQ/EsruWFgYIBXXnmFwcFB3nrrLQ4cOMDMmTPbfl/1EciETDQe2iz2H9aH8Ln7n2LR2n9MJOYaV9y+DBHoTHLD4sWL2bRpE3v37gVg3759LF26lA0bNgCwefNmFi9enMhYB7UIpGXtxEObxf7DmtgArx88kuqAsk409aV7rbrwjDH/J6D95IYFCxZw00038YEPfICenh4WLVrEHXfcwVVXXcVpp53GjBkzuO+++5IovioCaV27F8lGsf9GTek0L8QaxyCt6FRyw4oVK45lCVVt2rSprfcMo4pAWtbJi2RUH0KSf6NWVD9AnL6MZu8h5VLk5Ab1EUjLOjnYK6wPIem/UdWoHyDuOAb1JUg3SKQiMLO7zexVM3s2Yr+Z2VfNbLeZ7TKzc2r2rTCzF4OfFWGvl+zVdg7/+vBb9PaM7aBKarDX8kUDfPGS99LfN34e9aQHlDULcX3xkvcy0N+HAQP9fXzxkveOu+PTnEjSDZIKDX0D+BpwT8T+jwKnBz/nAX8BnGdmM4A/BYYAB540s63u/npC5ZKYGoU36juH9x86Qu8kY/qJvew/eCTxcEi1iZ1EyCXqPbbsHI0MQVXDT3Ga+upLkG6QSEXg7t8zs3kNDlkG3OOVsdCPm1m/mZ0MfBB4xN33AZjZI8BFwL1JlEviaZYFFHbXe+Rt58Qpk9m55iMdK1e7MdeozzX8031888no0E0r4adW+hJE8iqtPoIB4JWa5yPBtqjt45jZSjMbNrPhVhZckOaahTeKetcb9bnufeKV0BRVaD38pDmRpBsUprPY3de7+5C7D82ePTvr4nSVZhf6os4EGvW5jjaYpCusH6CRuH0JIkn43ve+xznnnMPkyZPZvHlzYu+bVvroKDCn5vlgsG2USniodvtjKZVJAs3CG50YLJOkVlNAe8xCK4OB/r4JXcCLnDYoxTJ37ly+8Y1v8OUvfznR902rRbAV+ESQPfQ+4IC77wEeBj5iZtPNbDrwkWCbpKhZeCPPd70TSQH9+HlzFM6R5O3aCLedCTf3V37v2tj2W9ZPQz1v3jwWLlzIpEnJXroTaRGY2b1U7uxnmdkIlUygXgB3vwP4NnAxsBs4CPxxsG+fmf0ZsCN4q7XVjmNJT5xRkXm9623Uv/H91YuPHVP/uYZOnaFBYJKcXRvhoc/CkaAFeuCVynOAhZdP6C3DpqHulKSyhj7eZL8D10bsuxu4O4lyyMTl9ULfTLP+jajPVdTPKzm1be3xSqDqyKHK9glWBJqGWgotzSkXlL4puXBgpLXtOVOYrCEphrSnXOiG9M1OLHEoKZs22Nr2GMKmoe4UVQSSqLSnXMhzR3YcmquoSyxZA711rdDevsr2Caqdhvqss87i+uuvZ8eOHQwODrJp0yY+9alPsWDBgjYLXqHQUAl1MnSTxeCzIsf7te5Bl6j2A2xbWwkHTRusVAIT7B+oCpuGemQk+XCTKoKS6cQi27UUs29NUUdtS4iFl7d94c+KQkMl0+nQTTfE7NNU1FHb0l1UEZRMp+9Aix6zT5sqznzzBtOR5Fmr5VZoqGTSCN0UOWaftk4tcSjtmzp1Knv37mXmzJmJLBCfFndn7969TJ06NfZrVBGUTN7nDSojVZz5NDg4yMjICEWc7Xjq1KkMDsZPXVVFUDK6AxWJp7e3l/nz52ddjFSoIigh3YGKSC1VBClKc+qFTumGzyAiY6kiSEmn8/fT0A2fQUTGU/poStKeeqETuuEziMh4ahGkpBtGkHbDZ0ibQmlSBGoRpKQbRpB2w2dIkyaUk6JQRZCSbhhB2g2fIU0KpUlRKDSUkm7I3++Gz5AmhdKkKFQRpChu/n6e48oagxCfZmKVolBFkKGwCz6gFM0uoek8pCgSqQjM7CLgz4Ee4E53X1e3/zbgd4OnJwL/zt37g31HgWeCfT9z96VJlCnvonLyT5g8SQuVdIlOhNLy3FqU4mq7IjCzHuDrwIeBEWCHmW119+erx7j7dTXH/xdgUc1bHHL3s9stR9FEdSTWb6tSXLmYkgylaUCfdEoSWUPnArvd/WV3fxO4D1jW4PiPA/cm8HcLrdULu+LKoiwk6ZQkKoIB4JWa5yPBtnHM7FRgPrC9ZvNUMxs2s8fNbHnUHzGzlcFxw0WcFrZe1IV9+om9StGUUMpCkk5JexzBlcBmd6+9rTnV3YeAPwRuN7PfCHuhu6939yF3H5o9e3YaZe2oqJz8P/3YAq3wJaE0oE86JYnO4lFgTs3zwWBbmCuBa2s3uPto8PtlM3uMSv/BSwmUK9eadSTqwl8OrXT+KgtJOiWJimAHcLqZzadSAVxJ5e5+DDN7DzAd+KeabdOBg+5+2MxmAecDX0qgTIWgnPxya7XzVwP6pFMsicWZzexi4HYq6aN3u/sXzGwtMOzuW4Njbgamuvvqmte9H/hL4G0qYarb3f2uZn9vaGjIh4eH2y63SFrC7vxvffiF0AFnUAkJ6iIvSTOzJ4NQ/NjtSVQEaVNFIEVSf+cPlZBOVKpw7THqH5IkRVUEmnROpMOi0j57zBq+TqmhkhZVBCIdFpXeedR9XOZY3NeKJEkVgUiHRaV3VlODBxqkfyo1VNKgikCkwxqt47B80QDfX72Y2684WwMJJTOafTTnNMlY8cVJ+1RqqGRJWUM5FpVtokwSEZkIZQ0VkCYZq7FrI9x2JtzcX/m9a2PWJRLpGgoNZSROyEeTjAV2bYSHPgtHgs994JXKc4CFl2dXLpEuoRZBBqohn9H9h3COTy2wZefYKZo0yVhg29rjlUDVkUOV7SLSNlUEGYgb8mmUbVIqB0Za2y4iLVFoKANxQz7KJAlMG6yEg8K2i0jbVBFk4JT+vtDJxsJCPpqhFFiyZmwfAUBvX2W7iLRNoaEMKOTTooWXw8e+CtPmAFb5/bGvqqNYJCFqEWSgWchHg8hCLLxcF36RDlFF0ET1ojy6/xA9Zhx1D50rvtWLd1TIp9XFSkSkBHZtrGTJHRip9I0tWZPojZFCQw3UpnlCZbZIGJ/uGTcdNA4NIhORMarjaA68AvjxcTQJDqpURdBA2EW5qvbinOTFW4PIRGSMFMbRqCJooNnFt7o/yYu3BpGJyBgpjKNRRdBAs4tvdX+SF29lFInIGFHjZRIcR6OKoIGwi3JV7cU5yYv38kUDxxYrMY4vXqKOYpGSWrKmMm6mVsLjaBLJGjKzi4A/B3qAO919Xd3+q4FbgWrv6dfc/c5g3wrgvwfbP+/uG5IoUxJq0zwbZQ3FGQHcSlaRBpGJyDHV7KAOZg21vR6BmfUAPwI+DIwAO4CPu/vzNcdcDQy5+2fqXjsDGAaGAAeeBH7H3V9v9DeLth6B1hUQkTzo5HoE5wK73f1ld38TuA9YFvO1FwKPuPu+4OL/CHBRAmXKFaWEikieJREaGgBqZwQbAc4LOe4PzOwCKq2H69z9lYjXht4im9lKYCXA3LlzEyh2epQSKlnYsnOU//HQc7x+8AgA/X293Lx0gVqhaerwQLCkpNVZ/BAwz90XUrnrb7kfwN3Xu/uQuw/Nnj078QJ2klJCJW1bdo6yavPTxyoBgP2HjrBq09MTGugoE5DCQLCkJFERjAJzap4PcrxTGAB33+vuh4OndwK/E/e13UApoZK2Wx9+gSNHx/f/HXnbFZJMS4EWVEoiNLQDON3M5lO5iF8J/GHtAWZ2srvvCZ4uBX4YPH4Y+J9mNj14/hHgxgTKlCtaV0DS1ijs2GifJjxMUIEWVGq7InD3t8zsM1Qu6j3A3e7+nJmtBYbdfSvwWTNbCrwF7AOuDl67z8z+jEplArDW3fe1W6Y8UkqopClqzYvqvjCa8DBhBVpQKZE+Anf/trv/prv/hrt/Idi2JqgEcPcb3X2Bu5/l7r/r7v+n5rV3u/tpwc9fJ1EekbJbdeEZ9PbYuO29kywyJKnstoSlMBAsKRpZLNKFli8a4NZLz2L6ib3HtvX39XLrZWdF3t0ruy1hBVpQSesRiHSpVsORrSyhKjEVZEEltQhEBFB2W5mpRSAigLLbykwVQUxKq5MyUHZbOakiiEFpdSLSzdRHEENUWt3n7n+K89dt15D9pO3aCLedCTf3V37ncEi+SCJycq6rRRBDo/Q5tQ4SVp2fpTo0vzo/CxQi+0Ikthyd62oRxNAsfU6DbhJUoPlZRNqSo3NdFUEMjZasrNKgm4Q0mp8lJ81okUTkaC4iVQQx1K4jHEWDbhISNQ9L3/TCTOkrJTPRG5QUFqWPSxVBTMsXDfD91Yu5/Yqzx7UOenuMXx9+i/mrv6XO43ZFzc8CuWlGd6MtO0c5f912ncOtamfNgRzNRaTO4hbVD7rpP7GXX73xFvsPVRYAUedxm6IW6n5gZfjxOZzSt2iUHt2C+hXH3vx19A1Ksw7fFBalj6vtxeuzkKfF689ftz10fpaB/j6+v3pxBiXqUredGTGl7xy47tn0y9NFdA7HVJ/l05DBzfs7XaKWdXLx+lLTjI0pyVEzutvoHI4pLMsnSg7XHGhEFUGbtB5xSgo0pW/R6ByOKW4YsoA3KOojaNOqC88YE18FzdjYMQWZ0jctSc1/pXM4pqgVx/pmwJR3ZB7nb4cqgjZpxkbJwpado6za/PSxBepH9x9i1eangdY7eHUOx7Rkzfg+gt4++OgtE7/w13c+q7M4vjx1FssE5OTkL7JFa/+R1w8eGbf9xN5JTH/HCbqgd0qS525Y53NvX0dDnlGdxWoRSLpyNL9KkYVVAgAHj7zNwaCTV2mgHZBkeLLRFBNFnGvIzC4ysxfMbLeZrQ7Zf72ZPW9mu8xsm5mdWrPvqJk9FfxsTaI87dDAmg5rNr+KppFIlObByrEcTTHRdovAzHqArwMfBkaAHWa21d2frzlsJzDk7gfN7D8DXwKuCPYdcvez2y1HEjSwJgXN5hJSayGW/r7eY4MYm1EaaE5FdT4XdIqJc4Hd7v6yu78J3Acsqz3A3b/r7geDp48DuUyyjVp3QHdUCWo0v0qOZmPMu5uXLqB3ksU6VmmgOZWjsTFJVAQDQG21NhJsi3IN8J2a51PNbNjMHjez5QmUZ8I0sCYFjU7+HDWV8275ogFuvewsBvr7MCqjgP/ofXO1+HyR5GhsTKqdxWb2R8AQ8IGazae6+6iZvRvYbmbPuPtLIa9dCawEmDt3bkfKd0p/X+hQe91RJajR/Crb1uamqVwEYesLD506Q2mgRZKTsTFJVASjwJya54PBtjHM7EPATcAH3P1wdbu7jwa/Xzazx4BFwLiKwN3XA+uhkj6aQLnH0cCalESd/FF52gUbpZklLT4vE5FEaGgHcLqZzTezKcCVwJjsHzNbBPwlsNTdX63ZPt3MTggezwLOB2o7mVNVu+5Atbn9xUveq/9YaclRU1mkTBIZUGZmFwO3Az3A3e7+BTNbCwy7+1YzexR4L7AneMnP3H2pmb2fSgXxNpVK6XZ3v6vZ39OAMpFkJDVNhRRD1IAyjSwWKan6dGmohELVCu5emoZaRMZQurRUqSIQKSmlS0uVKgKRktI6BAnokilRVBGIlNSqC8/QALR2tLNwfc6oIhApKaVLt6mLpkTRNNQiJaYBaG3ooilRVBEkSDnZkjc6JzsoR7OHtkuhoYRUc7JH9x/COT6FtdYzkKzonOywHM0e2i5VBAlRTrbkjc7JDuuiKVEUGkqIcrIlb1o9JxVGmoCczB7aLrUIEqKcbMmbVs5JhZHKTRVBQpSTLXnTyjmpMFK5KTSUkGoTWk3rCdq1MXyxGpmwVs5JhTbLTRVBgpSTPUFhi9Y/sBJ+9jj83leyLVvBxT0ntTpfuSk0JNkLG6GJw/DdhRyuX0QKbUbokrmEmlFFINmLHInphRyuX0SabiJEF80l1IxCQ5K9qBGaUMjh+kWl0GadRnMJdVn/lVoEkr0lawAL31fA4frSJbpoLqFmVBFI9hZeDkP/iXGVQbPh+iWJ30pGom5CuvDmRBWB5MPvfQUuWR9/uH6J4reSkS6aS6iZRCoCM7vIzF4ws91mtjpk/wlmdn+w/wkzm1ez78Zg+wtmdmES5ZGCWng5XPcs3Ly/8rtRHLaL5oKXnOqiuYSaabuz2Mx6gK8DHwZGgB1mttXdn6857BrgdXc/zcyuBG4BrjCz3wauBBYApwCPmtlvuvvYIY4i9eLGbzVQTdrRJXMJNZNEi+BcYLe7v+zubwL3AcvqjlkGbAgebwaWmJkF2+9z98Pu/mNgd/B+Io3Fid8qfCQSSxIVwQBQm/s3EmwLPcbd3wIOADNjvjYRW3aOcv667cxf/S3OX7ddk2kVXZz4rcJHkoUCJjEUZhyBma0EVgLMnTu3pddWZ1asTqpVnVkRUN50UVWb643CPiVK/5OcCJsu5aHPVh7nOMSURItgFJhT83ww2BZ6jJlNBqYBe2O+FgB3X+/uQ+4+NHv27JYKqJkVu1SzzuUSpf9JwiZ6V1/QVmgSFcEO4HQzm29mU6h0/m6tO2YrsCJ4fCmw3d092H5lkFU0Hzgd+OcEyjSGZlYsqRKl/0mCdm2EB68d27f04LXxKoOCtkLbrgiCmP9ngIeBHwIb3f05M1trZkuDw+4CZprZbuB6YHXw2ueAjcDzwD8A13YiY0iLxpRUidL/JEHfuQGOvjl229E3K9uhcWuhoK1Qq9yYF8vQ0JAPDw/HPr6+jwAqMyuWflItERnv5mnR+y75q7F9AFBpZVZvMOr7COr3Z8zMnnT3ofrtheksbkfcBTq0ZqsUmc7fFDSbiC5OEkMOlaJFEIdaDVJkOn8TdMt8OLRv/Pa+GXDodSDsmmmVpIWci2oRaK6hgDKLcqiA+dhZ0fmboI/eApN6x26b1FvZ3qwPoKDnbClCQ3EosyhnCpqPnRWdvwlqFt4J6wNYsqbQ56wqgoDWbM2ZEi0KkgSdvwmLmmOoUSVx25mFPWcVGgpozdacaTcfu6BN9InS+ZuiqIGMBR1DAGoRHBM3s0hSErV8ZZx87AI30SdK528OtHPOZkxZQ5JP7eRj33ZmxH/IOZU7OJFOyPkYAlDWkBRNO6OCJ9JEL1koSTqgwCPZFRqS/JrooiCtNtFLGEqSDinoQjZqEUj3aXWyuYLOGCmSFFUE0n1abaIXONtDJAkKDUl3qm+iV/sAwgYIFTjbQyQJahFI92u2drHWLZCSU0Ug3a9ZH0CBsz1EkqDQkHS/OH0ABc32EEmCKgLpTrs2Hp8PxiZB2MJ36gMQAVQRSDeqHxcQVgmoD0Aaqb2RKMjiMu1QRSDdJ6xPAMB6wN8uxX9saUMJBxiqs1i6T1SfgB8NUkVHKpWFppGQMCUcYKiKQLpPZOzfolNIRapKOMCwrYrAzGaY2SNm9mLwe3rIMWeb2T+Z2XNmtsvMrqjZ9w0z+7GZPRX8nN1OeUSA8HEBGOPWmu3yuzxpoNEkg82Wo+xC7bYIVgPb3P10YFvwvN5B4BPuvgC4CLjdzPpr9q9y97ODn6faLI9I+LiA0AXH6eq7PImgAYbjtFsRLAM2BI83AMvrD3D3H7n7i8HjnwOvArPb/LsijdWvIjVtTvhxze7yND1199EAw3HazRo6yd33BI9/AZzU6GAzOxeYArxUs/kLZraGoEXh7ocjXrsSWAkwd+7cNostpbNkTfSi41FKmD1SChpgOE7TFoGZPWpmz4b8LKs9zitLnUUud2ZmJwP/G/hjd3872Hwj8B7g3wMzgBuiXu/u6919yN2HZs9Wg0JaNJG7vBJmj6QuixZXCfsAmmnaInD3D0XtM7NfmtnJ7r4nuNC/GnHcvwW+Bdzk7o/XvHe1NXHYzP4a+G8tlV6kFa3e5ZUweyRVWbW4JtI67HLt9hFsBVYEj1cAD9YfYGZTgL8D7nH3zXX7Tg5+G5X+BS0oK/mhO8fOyqrFVcI+gGba7SNYB2w0s2uAnwKXA5jZEPBpd/9ksO0CYKaZXR287uogQ+hvzWw2ldy+p4BPt1kekeR0+s6xZNMYjJNli6tkfQDNtFURuPteYEnI9mHgk8HjvwH+JuL1i9v5+yIdVb1QdOJirY5oLQiUI5prSKSRTt05NgqLdHNFUNsK6psOPVPg6JvH95c8Vp8VTTEhkoU8dkR3OoOnfiDXoX3gDn0zUKw+W2oRiGQhb2GRNEJVYa2gt4/AlHfADT9O5m/IhKhFIJKFvE1jkEYGTx5bQQKoIhDprKhwS95SGNO4SCsdN7cUGhLplGbhljylMKYRqtJArtxSi0CkU4o0RUUaoao0W0GaLLAlahGIdEqRYuKdHDNR/3c63QqK2/Fd9gF9NVQRiHRK3jKDmslTqKodccZoaEDfGAoNiSSlPhxx+kfylRnUiiKHVuK0xIoUtkuBKgKRJIStejV8FzCpeAOmmq3glXdxspOKFLZLgUJDIkkIu8MEOPJr4G24ZH3+K4CqqLvlv/s0PLDyeDy9emzeYuxxspOKFrbrMLUIRJLQ6E6yaCGHqM/iRznWQnjwWtjyJ/lsNcTJTsrbgL6MqUUgkoSoO8yqIoUcmn0WGDtRXFWeJs1r1vGdVpZUQagiEElCWDiiVpFCDs0+SyN5rfCiUkVLeuGvp4pAJAnVC8p3bqjMqlmraCGH+rtlmxSEhWLIY4WnVNGm1EcgkpSFl1dm0bzkr/Izh9BELbwcrnsWbt4Pv3/H+Hh6zxSY1Dt2W14rPKWKNqUWgUjSui3kEBVPD9uWx8+tVNGmVBGIyHGNYulhF/k8XvjrKVW0KYWGRKSi6APJoihVtKm2KgIzm2Fmj5jZi8Hv6RHHHTWzp4KfrTXb55vZE2a228zuN7Mp7ZRHRNrQrbH0vK39kEPthoZWA9vcfZ2ZrQ6e3xBy3CF3Pztk+y3Abe5+n5ndAVwD/EWbZRLpDmnPjtnNsfRu67dJWLuhoWXAhuDxBmB53BeamQGLgc0Teb1I5jo5MVsWYRqtIFZa7VYEJ7n7nuDxL4CTIo6bambDZva4mS0Pts0E9rv7W8HzEWAg6g+Z2crgPYZfe+21Nost0qZOX6izCNMoll5aTUNDZvYo8K6QXTfVPnF3NzOPeJtT3X3UzN4NbDezZ4ADrRTU3dcD6wGGhoai/o5IOuLMed+OLMI0mnahtJpWBO7+oah9ZvZLMzvZ3feY2cnAqxHvMRr8ftnMHgMWAd8E+s1sctAqGARGJ/AZRNLX6Qt1VimPiqWXUruhoa3AiuDxCuDB+gPMbLqZnRA8ngWcDzzv7g58F7i00etFcqnT8fQ0wjRFXnxGEtVuRbAO+LCZvQh8KHiOmQ2Z2Z3BMb8FDJvZ01Qu/Ovc/flg3w3A9Wa2m0qfwV1tlkckHZ2+UHc65TFPYwZUIWXOKjfmxTI0NOTDw8NZF0PKrsiLn992ZkToaU5ljqG01E8IB5UKVXn+HWFmT7r7UP12TTEhMlFFjqdPpI+jExVfpzvdJRZNMSFSRq32cXQqlNTNg9gKRBWBSBm12sfRqXENGsSWC6oIRMqo1c7oTt25axBbLqiPQKSsWunj6NS4Bg1iywVVBCLSXNg6xknduRe5071LKDQkIs1pKueuphaBiMSjO/eupRaBiEjJqSIQESk5VQQiIiWnikBEpORUEYiIlFwhZx81s9eAn3b4z8wC/rXDf6MTilpuUNmzUNRyg8o+Eae6++z6jYWsCNJgZsNh07XmXVHLDSp7FopablDZk6TQkIhIyakiEBEpOVUE0dZnXYAJKmq5QWXPQlHLDSp7YtRHICJScmoRiIiUnCoCEZGSU0UQMLPLzOw5M3vbzCLTuszsIjN7wcx2m9nqNMsYUZ4ZZvaImb0Y/J4ecdxRM3sq+NmadjnrytLwOzSzE8zs/mD/E2Y2L4NijhOj3Feb2Ws13/MnsyhnGDO728xeNbNnI/abmX01+Gy7zOyctMsYJka5P2hmB2q+89wsbWZmc8zsu2b2fHBt+a8hx+Tje3d3/VT6SX4LOAN4DBiKOKYHeAl4NzAFeBr47YzL/SVgdfB4NXBLxHG/yvo7jvsdAn8C3BE8vhK4vyDlvhr4WtZljSj/BcA5wLMR+y8GvgMY8D7giazLHLPcHwT+PutyRpTtZOCc4PE7gR+FnDO5+N7VIgi4+w/d/YUmh50L7Hb3l939TeA+YFnnS9fQMmBD8HgDsDy7osQS5zus/UybgSVmZimWMUwe/+1jc/fvAfsaHLIMuMcrHgf6zezkdEoXLUa5c8vd97j7vwSP/x/wQ2Cg7rBcfO+qCFozANQu3DrC+H/YtJ3k7nuCx78AToo4bqqZDZvZ42a2PJ2ihYrzHR47xt3fAg4AM1MpXbS4//Z/EDTxN5vZnHSKlog8nttx/Qcze9rMvmNmC7IuTJggvLkIeKJuVy6+91KtUGZmjwLvCtl1k7s/mHZ54mpU7ton7u5mFpUPfKq7j5rZu4HtZvaMu7+UdFlL7iHgXnc/bGafotKqWZxxmbrdv1A5t39lZhcDW4DTsy3SWGb2b4BvAp9z9/+bdXnClKoicPcPtfkWo0DtXd5gsK2jGpXbzH5pZie7+56gSflqxHuMBr9fNrPHqNydZFERxPkOq8eMmNlkYBqwN53iRWpabnevLeOdVPpviiKTc7tdtRdWd/+2mf0vM5vl7rmYjM7MeqlUAn/r7g+EHJKL712hodbsAE43s/lmNoVKR2amGTjB318RPF4BjGvZmNl0MzsheDwLOB94PrUSjhXnO6z9TJcC2z3oWctQ03LXxXaXUokJF8VW4BNBFsv7gAM1IcfcMrN3VfuPzOxcKte0rG8agEpGEHAX8EN3/0rEYfn43rPuWc/LD/D7VOJzh4FfAg8H208Bvl1z3MVUev9fohJSyrrcM4FtwIvAo8CMYPsQcGfw+P3AM1QyXZ4Brsm4zOO+Q2AtsDR4PBXYBOwG/hl4d9bfc8xyfxF4Lvievwu8J+sy15T9XmAPcCQ4z68BPg18OthvwNeDz/YMEZlzOSz3Z2q+88eB92dd5pqy/0fAgV3AU8HPxXn83jXFhIhIySk0JCJScqoIRERKThWBiEjJqSIQESk5VQQiIiWnikBEpORUEYiIlNz/B+YEMR/KIohBAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from typing import Tuple, Callable\n", "\n", "data = dict(np.load(\"two_moons.npz\", allow_pickle=True))\n", "samples = data[\"samples\"]\n", "labels = data[\"labels\"]\n", "\n", "c0_samples = samples[labels == 0] # class 0: all samples with label 0\n", "c1_samples = samples[labels == 1] # class 1: all samples with labe 1 \n", "\n", "plt.figure(\"Data\")\n", "plt.scatter(x=c0_samples[:, 0], y=c0_samples[:, 1], label=\"c0\")\n", "plt.scatter(x=c1_samples[:, 0], y=c1_samples[:, 1], label=\"c1\")\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us also define some plotting utility" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def draw_2d_gaussian(mu: np.ndarray, sigma: np.ndarray, plt_std: float = 2, *args, **kwargs) -> None:\n", " (largest_eigval, smallest_eigval), eigvec = np.linalg.eig(sigma)\n", " phi = -np.arctan2(eigvec[0, 1], eigvec[0, 0])\n", "\n", " plt.scatter(mu[0:1], mu[1:2], marker=\"x\", *args, **kwargs)\n", "\n", " a = plt_std * np.sqrt(largest_eigval)\n", " b = plt_std * np.sqrt(smallest_eigval)\n", "\n", " ellipse_x_r = a * np.cos(np.linspace(0, 2 * np.pi, num=200))\n", " ellipse_y_r = b * np.sin(np.linspace(0, 2 * np.pi, num=200))\n", "\n", " R = np.array([[np.cos(phi), np.sin(phi)], [-np.sin(phi), np.cos(phi)]])\n", " r_ellipse = np.array([ellipse_x_r, ellipse_y_r]).T @ R\n", " plt.plot(mu[0] + r_ellipse[:, 0], mu[1] + r_ellipse[:, 1], *args, **kwargs)\n", "\n", "# plot grid for contour plots\n", "plt_range = np.arange(-1.5, 2.5, 0.01)\n", "plt_grid = np.stack(np.meshgrid(plt_range, plt_range), axis=-1)\n", "flat_plt_grid = np.reshape(plt_grid, [-1, 2])\n", "plt_grid_shape = plt_grid.shape[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2): Classification using Generative Models (Naive Bayes Classifier)\n", "\n", "We first try a generative approach, the Naive Bayes Classifier.\n", "We model the class conditional distributions $p(\\boldsymbol{x}|c)$ as Gaussians, the class prior $p(c)$ as\n", "Bernoulli and apply Bayes rule to compute the class posterior $p(c|\\boldsymbol{x})$.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a small recap, recall that the density of the Multivariate Normal Distribution is given by\n", "\n", "$$ p(\\boldsymbol{x}) = \\mathcal{N}\\left(\\boldsymbol{x} | \\boldsymbol{\\mu}, \\boldsymbol{\\Sigma} \\right) = \\dfrac{1}{\\sqrt{\\det \\left(2 \\pi \\boldsymbol{\\Sigma}\\right)}} \\exp\\left( - \\dfrac{(\\boldsymbol{x}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x}-\\boldsymbol{\\mu})}{2}\\right) $$\n", "\n", "and we already saw how to implement it in the python introduction" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def mvn_pdf(x: np.ndarray, mu: np.ndarray, sigma: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " Density of the Multivariate Normal Distribution\n", " :param x: samples, shape: [N x dimension]\n", " :param mu: mean, shape: [dimension]\n", " :param sigma: covariance, shape: [dimension x dimension]\n", " :return p(x) with p(x) = N(mu, sigma) , shape: [N] \n", " \"\"\"\n", " norm_term = 1 / np.sqrt(np.linalg.det(2 * np.pi * sigma))\n", " diff = x - np.atleast_2d(mu)\n", " exp_term = np.sum(np.linalg.solve(sigma, diff.T).T * diff, axis=-1)\n", " return norm_term * np.exp(-0.5 * exp_term)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Practical Aspect:** In practice you would never implement it like that, but stay\n", "in the log-domain. Also for numerically stable implementations of the multivariate normal density the symmetry and\n", "positive definitness of the covariance should be exploited by working with it's Cholesky decomposition." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The maximum likelihood estimator for a Multivariate Normal Distribution is given by\n", "$$ \\boldsymbol{\\mu} = \\dfrac{1}{N} \\sum_{i}^N \\boldsymbol{x}_i \\quad \\quad \\boldsymbol{\\Sigma} = \\dfrac{1}{N} \\sum_{i}^N (\\boldsymbol{x}_i - \\boldsymbol{\\mu}) (\\boldsymbol{x}_i - \\boldsymbol{\\mu})^T. $$\n", "\n", "This time, before we use it, we are going to derive it:\n", "\n", "### Exercise 2.1): Derivation of Maximum Likelihood Estimator (5 Points):\n", "\n", "Derive the maximum likelihood estimator for Multivariate Normal distributions, given above.\n", "This derivations involves some matrix calculus.\n", "Matrix calculus is a bit like programming, you google the stuff you need and then plug it together in the right order.\n", "Good resources for such rules are the \"matrix cookbook\" (https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf) and the Wikipdia article about matrix calculus\n", "(https://en.wikipedia.org/wiki/Matrix_calculus ). State all rules you use explicitly\n", "(except the ones given in the hints below). \n", "\n", "**Remark** There are different conventions of how to define a gradient (as column-vector or row-vector). This results in different ways to write the Jacobian and thus different, usually transposed, matrix calculus rules:\n", "- In the lecture we define the gradient as column-vector \n", "- In the Wikipedia article this convention is referred to as \"Denominator Layout\". It also contains a nice explanation of the different conventions for the gourmets among you ;) \n", "- The Matrix Cookbook uses the same convention (gradient as column vector)\n", "- Please also use it here\n", "\n", "**Hint** Here are two of those rules that might come in handy\n", "\n", "$\\dfrac{\\partial\\log\\det(\\boldsymbol{X})}{\\partial \\boldsymbol{X}} = \\boldsymbol{X}^{-1}$\n", "\n", "$\\dfrac{\\partial \\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x}}{\\partial \\boldsymbol{x}} = 2 \\boldsymbol{A}\\boldsymbol{x}$ for symmetric matrices $\\boldsymbol{A}$ (hint hint: covariance matrices are always\n", "symmetric)\n", "\n", "There is one missing to solve the exercise. You need to find it yourself. (Hint hint: Look in the matrix cookbook, chapter 2.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Solution\n", "\\begin{align}\n", " p(\\boldsymbol{x_i}) \n", " &= \\mathcal{N}\\left(\\boldsymbol{x_i} | \\boldsymbol{\\mu}, \\boldsymbol{\\Sigma} \\right)\\\\\n", " &= \\dfrac{1}{\\sqrt{\\det \\left(2 \\pi \\boldsymbol{\\Sigma}\\right)}} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\n", " &= \\dfrac{1}{\\sqrt{(2 \\pi)^{K} |\\boldsymbol{\\Sigma}}|} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n", "\\end{align}\n", "\n", "With K as the dimensions of the quadratic matrix $\\Sigma$\n", "\n", "##### calculate the likelihood\n", "\\begin{align}\n", " \\text{lik}(\\boldsymbol{\\theta};D) \n", " &= \\prod_i p(x_i)\\\\\n", " &= \\prod_i^{N} \\dfrac{1}{\\sqrt{(2 \\pi)^{K} |\\boldsymbol{\\Sigma}}|} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n", " &= \\prod_i^{N} 2\\pi^{-\\frac{K}{2}} |\\boldsymbol{\\Sigma}|^{-\\frac{1}{2}} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n", " &= (2\\pi)^{-\\frac{NK}{2}} |\\Sigma|^{-\\frac{N}{2}} \\exp\\left( - \\dfrac{\\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\\\\\n", " \\log\\text{lik}(\\boldsymbol{\\theta};D) \n", " &= \\sum_i \\log p(x_i)\\\\\n", " &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu}) \\\\\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### derive $\\mu$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\begin{align}\n", " \\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial\\mu}\n", " &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n", " &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}^T-\\boldsymbol{\\mu}^T) \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n", " &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} - \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu} - \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} + \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n", " &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}+ \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n", " &= -\\dfrac{1}{2}\\sum_i^N -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + 2\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n", " &= -\\sum_i^N -\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + \\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n", " &= -\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "for Optimum: $-\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu} = 0$\n", "\\begin{align} \n", " 0 &= -\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n", " 0 &= \\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n", " 0 &= -N\\boldsymbol{\\mu}\\sum_i^N \\boldsymbol{x_i}^T \\\\\n", " \\boldsymbol{\\mu} &= \\frac{1}{N}\\sum_i^N \\boldsymbol{x_i}^T \\\\\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### derive $\\Sigma$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\begin{align}\n", "\\frac{\\partial\\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial \\boldsymbol{\\Sigma}} &= \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\left(\\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right) \\\\\n", "&= -\\frac{N}{2}\\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\log(|\\Sigma|) - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n", "\\end{align}\n", "Because $(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})$ is scalar, we can take its trace and obtain the following form \n", "\\begin{align}\n", " &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\text{Tr}\\left((\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right)\\\\\n", "\\end{align}\n", "The rule on page 10 (formula 63) of the matrix cookbook allows us to derive the trace\n", "\\begin{align}\n", " &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T\\\\\n", " &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T = 0 \\\\\n", " &= -N \\Sigma^{-1} + \\Sigma^{-1}(\\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T) \\Sigma^{-1} = 0\\\\\n", "\\end{align}\n", "Multiply from both sides with $\\Sigma$\n", "\\begin{align}\n", "-\\Sigma N + \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T &= 0 \\\\\n", "\\Sigma = \\frac{1}{N} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### **Implementation**\n", "\n", "Lets reuse one of the implementations from the zeroth-exercise for that " ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def mvn_mle(x: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n", " \"\"\"\n", " Maximum Likelihood Estimation of parameters for Multivariate Normal Distribution\n", " :param x: samples shape: [N x dimension]\n", " :return mean (shape: [dimension]) und covariance (shape: [dimension x dimension]) that maximize likelihood of data.\n", " \"\"\"\n", " mean = 1 / x.shape[0] * np.sum(x, axis=0)\n", " diff = x - mean\n", " cov = 1 / x.shape[0] * diff.T @ diff\n", " return mean, cov\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now use this maximum likelihood estimator to fit generative models to the samples of both classes. Using those models and some basic rules of probability we can obtain the class posterior distribution $p(c|\\boldsymbol{x})$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2.2) Generative Classifier (2 Points)\n", "\n", "Given a way to fit the class conditional using our Maximum Likelihood estimator, we can implement the generative classifier" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 0.83\n" ] } ], "source": [ "# Fit Gaussian Distributions using the maximum likelihood estimator to samples from both classes\n", "mu_c0, sigma_c0 = mvn_mle(c0_samples)\n", "mu_c1, sigma_c1 = mvn_mle(c1_samples)\n", "\n", "# Prior obtained by \"counting\" samples in each class\n", "p_c0 = c0_samples.shape[0] / samples.shape[0]\n", "# LEAVE AS EXERCISE\n", "p_c1 = c1_samples.shape[0] / samples.shape[0]\n", "\n", "def compute_posterior(\n", " samples: np.ndarray,\n", " p_c0: float, mu_c0: np.ndarray, sigma_c0: np.ndarray,\n", " p_c1: float, mu_c1: np.ndarray, sigma_c1: np.ndarray) \\\n", " -> Tuple[np.ndarray, np.ndarray]:\n", " \"\"\"\n", " computes the posteroir distribution p(c|x) given samples x, the prior p(c) and the\n", " class conditional likelihood p(x|c)\n", " :param samples: samples x to classify, shape: [N x dimension]\n", " :param p_c0: prior probability of class 0, p(c=0) \n", " :param mu_c0: mean of class conditional likelihood of class 0, p(x|c=0) shape: [dimension]\n", " :param sigma_c0: covariance of class conditional likelihood of class 0, p(x|c=0) shape: [dimension x dimension]\n", " :param p_c1: prior probability of class 1 p(c=1) \n", " :param mu_c1: mean of class conditional likelihood of class 1 p(x|c=1) shape: [dimension]\n", " :param sigma_c1: covariance of class conditional likelihood of class 1, p(x|c=1) shape: [dimension x dimension]\n", " :return two arrays, p(c=0|x) and p(c=1|x), both shape [N]\n", " \"\"\"\n", " # TODO: compute class likelihoods \n", " # TODO: compute normalization using marginalization\n", " # TODO: compute class posterior using Bayes rule\n", " p_x_c0 = mvn_pdf(samples, mu_c0, sigma_c0)\n", " p_x_c1 = mvn_pdf(samples, mu_c1, sigma_c1)\n", " p_x = 1 / samples.shape[0]\n", " p_c0_given_x = (p_x_c0*p_c0)/p_x\n", " p_c1_given_x = (p_x_c1*p_c1)/p_x\n", " return p_c0_given_x, p_c1_given_x\n", "\n", "\n", "p_c0_given_x, p_c1_given_x = compute_posterior(samples, p_c0, mu_c0, sigma_c0, p_c1, mu_c1, sigma_c1)\n", "# Prediction\n", "predicted_labels = np.zeros(labels.shape)\n", "# break at 0.5 arbitrary\n", "predicted_labels[p_c0_given_x >= 0.5] = 0.0 # is not strictly necessary since whole array already zero.\n", "predicted_labels[p_c1_given_x > 0.5] = 1.0\n", "\n", "# Evaluate\n", "acc = (np.count_nonzero(predicted_labels == labels)) / labels.shape[0]\n", "print(\"Accuracy:\", acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets look at the class likelihoods" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABHU0lEQVR4nO2dd3hUVfrHPyeNEHpCbwlVmoCAKKgI2BVx7QpiW3/YUMR1Leuuy6rs6uquYMHeQbErNmygKODSpPdOqCEQekiZ8/vjnTGTZGYySe7MvXfmfJ7nPHdm7p1739zc+d73vuc971FaawwGg8EQ+yTYbYDBYDAYooMRfIPBYIgTjOAbDAZDnGAE32AwGOIEI/gGg8EQJxjBNxgMhjjBCL4h5lFK/UUp9YrddhgMdqNMHr7B6SilNgFpQBut9WHvZzcB12itB9po14/AyUARUAwsBm7XWi+1yyaDIRTGwze4hURgtN1GBGCU1ro2kA78CLxtrzkGQ3CM4BvcwhPAPUqp+oFWKqUmKKW2KqUOKKUWKKVO81s3Vik1yfv6a6XUqDLfXayUusT7upNS6jul1F6l1Gql1BXhGKe1LgamAF389ttXKTVHKZWnlNqhlHpWKZXiXfecUuo/ZeyYqpQa433dXCn1kVIqRym1USl1Z5n9zvf+rbuUUv8Nx0aDwQi+wS3MRzzoe4Ksnwf0RDztd4APlFKpAbZ7F7ja90Yp1QXIBL5UStUCvvN+vzFwFTDRu01IvEI+HPjV7+NiYAzQEOgHnAHc5l33JnC1UirB+/2GwJnAO97PPkdCRC2837tLKXWO97sTgAla67pAO+D9iuwzGMAIvsFdPATcoZRqVHaF1nqS1jpXa12ktf4PUAM4LsA+PgF6KqUyve+HAx9rrY8BQ4BNWuvXvfv5DfgIuDyETU8rpfKAg8Ao4B9+Ni3QWv/q3dcm4EXgdO+6ucB+RMxBbi4/aq13AScCjbTWD2utC7TWG4CXvdsAFALtlVINtdaHtNb+NxmDIShG8A2uQWu9DPgCuL/sOqXUPUqplUqp/V4Brod41mX3cRD4khLxvBqY7H2dCZzkDcHkefczHGgawqw7tdb1gZrIDeNDpVR3r00dlVJfKKV2KqUOAP8sY9ObwDXe19dQEv/PBJqXseMvQBPv+j8CHYFVSql5SqkhIewzGH4nyW4DDIZK8ndgIfB7/Nsbr78X8ZaXa609Sql9gAqyj3eBvyulZgKpwAzv51uBn7TWZ1XWKK21B/hZKbUOOBtYAjwP/AZcrbU+qJS6C7jM72uTgGVKqR5AZ+BTPzs2aq07BDnWWkrCQZcgN5kMXwaTwRAM4+EbXIXWeh3wHnCn38d1kNTIHCBJKfUQUDfEbr5CvOiHgfe8Yg3y9NBRKTVCKZXsbScqpTqHY5tSqh/Sabvcz64DwCGlVCfg1jJ/SzbS9/A28JHW+qh31VzgoFLqPqVUTaVUolKqm1LqRO9xrlFKNfLanef9jgeDoQKM4BvcyMNALb/33wDTgDXAZiAf8ZID4o3Xf4y3k9Tv84OId34VsB3YCTyO9AcE41ml1CGl1CFEuP+qtf7au+4eYBgS338ZuVGV5U3gePzSOb0ZP0OQTuiNwB7gFSRMBXAusNx7zAnAVX43C4MhKGbglcFgI0qpAUhoJ1ObH6MhwhgP32CwCaVUMjKY7BUj9oZoYATfYLABb79AHtAMGG+rMYa4wYR0DAaDIU4wHr7BYDDECY7Nw2/YsKHOysqy2wyDwWBwFQsWLNijtS43Gh0cLPhZWVnMnz/fbjMMBoPBVSilNgdbZ0I6BoPBECcYwTcYDIY4wQi+wWAwxAmOjeEbDAZDpCksLCQ7O5v8/Hy7Tak0qamptGzZkuTk5LC/YwTfYDDELdnZ2dSpU4esrCyUClZc1XlorcnNzSU7O5s2bdqE/T0T0jEYDHFLfn4+GRkZrhJ7AKUUGRkZlX4yMYJvMBjiGreJvY+q2G1COoaIU1wMeXnS9u2T5YEDkJ8Px46Vb4WFkJAgLTFRmv/rtDSoXbt0q1VLlg0bymuDwVAeI/iGKrN/P2zaBNu3B267d4vAHzwYXbtq1oRGjaBxY1n6XrdsCZmZ0rKyoH59cKlzZ4hxjh07xrXXXsuCBQvIyMjgvffew4rKA0bwDSHZvx9WroR160q39ethz57y2zdsCM2bS+vaFRo0kFa/fklr0ADq1IHUVKhRo3xL8l6VHo88Hfgvi4rgyBE4dEja4cMlrw8ehNxcudHk5EjbvRuWL5dl2XBnnTolN4COHaFz55KWkRHhE2swhODVV1+lQYMGrFu3jilTpnDffffx3nuB5s+pHEbwDYAI6Zo1sGQJLF0qyyVLYMuWkm2UgtatoX17uPRSaNcO2rSBFi1E4Js2FcG2Cl8IpywNGlR+X1rLDWrz5vJt40aYPh2O+s0Z1aiRCH+XLnDCCdCnD3TrBikpVf97DIZgvPXWWzz55JMopejevTu7d+9m7NixAFx22WWMGjUKrXW1+xuM4MchWsPWrfDrryVt4UKJn4N42J06wamnwvHHi9B16CBhECsFPZooVRLe6dOn/HqPR8R/5crS7d134YUXZJuUFDkfffpA795w4onyPtBNyeA+7roLFi2ydp89e8L48aG3Wb58OY8++iizZ8+mYcOG7N27lwEDBtCqVSsAkpKSqFevHrm5uTRs2LBa9hjBjwM8Hli8WLzY2bNF4Ldvl3WpqSJet98uF2f37iL2bhX2qpKQIE8rbdrA+eeXfK41bNgACxZImz8fpkyBF1+U9fXqyY1xwABpvXtDJcbBGAxMnz6dyy+//HcxT09Pj9ixjODHIFpLjP2HH6RNny6xbYC2bWHgQOjXD04+WQTehCmCo5SErtq1gyuukM985/fXX2HmTGlffinr0tLk3J55ptw4jj/edAy7hYo88WjSokULtm7dSsuWLSkqKmL//v1kWNCxZAQ/Rjh2DGbMgM8+g6+/lvAESHx9yBA44wwYPFjeG6qHUtKP0b49XHONfLZrF/z8s4j/jz/CAw9Ia9ECzjtPxP+MM6BuXVtNNziQwYMHc/HFF3P33XeTkZHB3r17GTp0KG+++Sb9+vXjww8/ZPDgwZaMFzCC72L27oWvvoKpU2HaNMlSqVULzjoL7r1XBKZjR+NhRoMmTeCyy6SBhMymTZP/z/vvwyuvSN/I6afD5ZfDJZdIf4LB0LVrVx588EFOP/10EhMTOeGEE3jhhRcYMWIE7du3Jz09nSlTplhyLMfOadunTx9tJkApz/798PHH8M474tEXF0t2zNChcNFF4sWnptptpcGfwkLpO/nqK/j0U8mGSkiAQYMkTHTxxUb87WLlypV07tzZbjOqTCD7lVILtNYBUhNMaQVXUFAgoZorrhBxv/FG6Ui8916JI2/bJp2I559vxN6JJCeLZ//447BqlWSCPPCApLzefDM0awZnny0ZQS4s2mhwESak42CWLJGUwClTZMRqo0Zw000wfDicdJIJ1bgRpaBHD2mPPCLZUx98AJMnw7BhMjBt+HD44x8l/99gsBLj4TuM/Hx4+23o319E4fXXpdPvyy/Fk3/mGcmuMWLvfpSSVNhx4+SJ7bvv5H/9yivQq5e0iRNlFLHBYAVG8B3Chg1wzz2S1XHttZJG+d//ishPnizhGpPfHbskJEgq5zvvSIfvM89I+uftt0OrVhK+27rVbisNbscIvs0sXAhXXikjWSdMkE7X77+XWO+YMRDBMRgGh5KeDqNGwW+/SWfvWWfBf/4jg8KuugrmzrXbQoNbMYJvA1rLgKizzpKRmdOmiXe/ebPEc884w4RsDEK/fpLWuWGDDP3/+mvpvzn9dMnSMhgqgxH8KKK1xOL79pXH92XL4LHHJFvj8celAJmhNJMnSw2fhARZTp5st0X2kJkJTz4J2dkyInTdOnkaHDgQfvrJbusMVjNz5kx69epFUlISH374oWX7NYIfJX75RWqtDBkiA6ZeekmqNN53n9RjMZRn8mQYOVKefLSW5ciR8Sv6ICWdR4+W0g4TJkhO/8CBIv4//2y3dQaraN26NW+88QbDhg2zdL9G8CPM4sVwwQVw2mnyI33+eYnP/9//mZz5injwQal978+RI/J5vJOaCnfeKdfU+PFS2XPAABnBu3693dYZKstbb71F9+7d6dGjByNGjCArK4vu3buTkGCtRJs8/Aixa5d472++KbnVjz0Gd9whxbUM4eFfiz+cz+ORmjXF4/+//xPh/+c/JWx4111yYzS1eyrBgrtg3yJr99mgJ/QeH3KTQOWRI4Xx8C2mqAieflpq2LzzjqTTbdgg4m/EvjyhYvStWwf+TrDP45m0NPjLXyTEM2wY/Pvfkvn18stSHtvgXEx5ZJfyyy+SN71kiQyVf/ppOO44u61yLr4YvS9s44vRg4w2HTeu9HoQYRs3Lvq2uoXmzWWw3qhR4uWPHCkD+V5+2VyLFVKBJx4LGA/fAg4ckEfq006TEggffSSpluYHFpqKYvTDh0vndmampKlmZsr74cOjb6vb6N1bSjW//rpMWdmjh4QVCwvttsxQlsGDB/PBBx+Q6520IpIhHbTWjmy9e/fWbmD6dK0zM7VOSND6z3/W+tAhuy2yj0mT5FwoJctJk0Jvr5TWkn9TuilVsr+MjJLPMzIq3mc07XcLO3Zofemlcg5POEHr336z2yLnsGLFCrtN0Fpr/cYbb+iuXbvq7t276+uuu07PnTtXt2jRQqelpen09HTdpUuXgN8LZD8wXwfRVduFPVhzuuAfPqz16NFyBjt00Hr2bLstspdJk7ROSyst3GlpoUUzMzOw4PvENjm5/LqUlMgIcVXsdxsffaR106ZyXp96SmuPx26L7Mcpgl9VjOBHgUWLtD7uODl7d9wR3169j1DiHYxQIhtsfxXtM5r2u5E9e7QeOlT+tgsvlPfxTLwJvonhV5I335RqlQcPSs2bp5+WWabinaqkUIaK0Yf6XnXTMgNlBoVjfyyM+s3IkElYJkyAb76Rap1mwFYcEexOYHdzmod/9KjWI0eKZzRokNa7dtltkbOw2kOOlIcf7KnCv68g0LFiMeQzf77W7dtL/9PTT8dniGfFihXa49I/3OPx2BPSAV4DdgPLgqxXwNPAOmAJ0KuifTpJ8Ddv1rp3bzlb99+vdWGh3RY5B//wS9lO2OoIYiRi+JMmaZ2YGFjYMzJCC3qshnz27y8J8dx2W/xd2xs2bNA5OTmuE32Px6NzcnL0hg0byq0LJfhW5eG/ATwLvBVk/XlAB287CXjeu3Q8ixbJpBRHjsij8EUX2W2RPUyeLOmSW7bIwCdfLrx/nrzWEprRWsIz48ZVPYXS973Ro2VuAJBwxIQJ4e/T3+b0dAnDFRcH3nbvXslXL/s3+o4Vq6N+69aVOZIfeACeeEKKsr33nowOjwdatmxJdnY2OTk5dptSaVJTU2nZsmXlvhTsTlDZBmQR3MN/Ebja7/1qoFmo/TnBw//2W61r19a6VSutly2z2xr7qGoYxE4C2RyqVWRzrHr4/rz6qtZJSVp36iRPtQZ3ggM6bVsA/vP1ZHs/K4VSaqRSar5Sar7dd9y33pJZptq2hTlzoGtXW82xlWADpHyed1mi6fUG60gNZHMwwhm9O25c+dIYsTbq98YbZZrFHTukENuGDXZbZLCcYHeCyjZCe/hfAKf6vf8B6BNqf3Z6+OPHi/d2xhla5+XZZoZjCDZAKpppk4EI1ZEars2JieH3CcTqwKyyzJ+vdXq61i1aaL1qld3WGCoLDvDwtwGt/N639H7mOJ57TmqQXHopfPWVqVUPwYuVZWTY6/WGKs0QToG1tDRJsw23T2D4cNi0SYqRbdoUuyUeeveW2bQKCmRmreXL7bbIYBXREvypwLVKOBnYr7XeEaVjh82LL0rRqYsugnffhZQUuy1yBsHCGRMm2FvrJlRHaiCbk5PlJmXq8lRM9+4yk1ZCAgwaZGrsxwzBXP/KNOBdYAdQiMTn/wjcAtziXa+A54D1wFIqCOdoG0I6r74qj/gXXKB1fn5UD+0KqhPOiFQopKKO1HgJwUSSlSslvNO+vda7d9ttjSEcMKUVQvPVVzL45JxzZICVwToiOWDJDYOhYuGmM2uW1qmpWvfta8qIuAEj+CFYulTrOnWkiqC5mAUrRSrS6YxOFlQ33JDC5ZNPxCkaMkTroiK7rTGEIpTgK1nvPPr06aPnz58f0WPs3g19+0rn1Ny5UNkxDLFI2UlJQGLhVY13JySI1JVFqdifiSkrSyZ1KUtmpnT6uo3nn4fbboO//Q0efthuawzBUEot0Fr3CbQubounFRTAxReL6E+dasTeh9UTh8fzNIWxNjr31lvhhhvg0Ufh22/ttsZQFeJW8B96CGbPlrS8PgHvhfGJ1SIVDwOWghGLN7tnn5VBiMOHwzZHJlYbQhGXgj9jhkzyPHIkXH653dY4C6tFKp6nKYzFm11aGnzwARw9CldfHbw2kcGZxJ3g790LI0ZAx47w3//abY3ziIRIxcuApbLE6s2uUyeYOFHq6L/wgt3WGCpD3HXaXnWVVAf89Vfo1cvy3ccEgSpjul2kDNaiNZx7rtSZWrkSWpSrjGWwC9Np6+WHH6T069/+5i6xj/ZMS1XxyGNhNihD+CglXn5hIdx5p93WGMIlbgTfd2G2bQt//rPd1oSPL01y82bxqjZvlvdOElQ32Giwnnbt4O9/lyfmL76w2xpDOMRNSGf8eBgzBj77DIYOtWy3EccNudxusNEQGQoLoVs3SE2F336TJzyDvcR9SCc3VzyRc8+FCy+025rK4YZcbjfYGEniOZyVnCy/rSVLxNM3OJu4EPxnn4UDB2QKN6XstqZyuCGX2w02RgoTzoIrr4TOnWHs2NgfPe12Yl7wDx2Cp5+WME63bnZbU3nckMvtBhsjhdUjk91IYqJ4+cuXw0cf2W2NIRQxL/ivvCK59/ffb7clVSOcXG67Qwqxmm8eDvEezvJx+eWSEDFxot2WGEIR0522xcUigG3bymQOsUDZHPnzz5fyEFYVOzNUDtNhXcI//ynX5po10KGD3dbEL3HbaTt9OmRnwx132G2JNQSKF7/wggkp2IlV4Sy7n9Ks4Prrxf7XXrPbEkMwYlrwJ0+GunVhyBC7LbGGQPHiYA9o8RZSsAsrwlmx0vHbvDlccAG88YapseNUYjakc/QoNGkCl10WOx5HsNrygYjHkIJbiaWw0JQpUlRtzhw4+WS7rYlP4jKk8+WXcPBgbMWxg6U5lk01jZcMmVghljp+zzpLrsdvvrHbEkMgYlbwv/tOwjkDB9ptiXUEixffckt8Zsi4gXBi87E0jiEjQ+aXMBOkOJOYFfyZM+HUUyVHOFYIFi+eODE+yw87jbLifttt5WPzI0bI/85f/GNtHMM550g12rw8uy0xlCUmBX/3bli1Ck4/3W5LrCdea8s7nXAzqHx9MP4ds7E2jmHgQLk+582z2xJDWWJS8H/+WZYDBthrhyF+qEwGlQ//9NlYupH7So8vWGCvHYbyxKTgL1wooZwTTrDbEkO8UNUOVjd2zFZEgwbQsqVMjGJwFjEp+CtXQvv2UKOG3ZYY4oVwM6jC/Z7bad8e1q2z2wpDWWJS8DdskAsuHomFEZtupKIMKoiv9NnMzNh8enE7MSn427bJI2W8EQsjNsvGvR06LrAcFWVQaQ1vvx07HbMV0bgx5OS45/8XL8Sc4Hs8MuFJ48Z2W2I9FXnvbi/VO3aszErmEwmt5f3Ysd4NNk6GT7PgnQRZbnTWnayijtdY6pitiPR0OHYM8vPttsTgT8wJ/uHDIhR169ptibWE4727ecSm1pK3PWFCieiPGSPv8/JAb5wMc0fCkc2AluXckY4TfYOwbJksc3PttcNQmpgTfJ9HkZpqrx1WE4737uYRm0rBU0/B6NEi8gkJshw9Wj5Xix+E4jInoPgILHbJ40uc8dtvsty3z147DKWJOcGPVcLx3t0+YtMn+v489ZS3s/NIkBMQ7HODrVx3nSx9HdYGZxBzgp+cLMvCQnvtsJpwvHe3j9j0hXH8+T2mnxbkBAT73GArBQWyjLUnbbcTc4Jfp44sDxyw1w6rCdd7d2vHoH/MfvRosd8X3hkzBnSPcZBY5gQkpkEPlzy+xBm5ufJbTEmx2xKDPzEn+ImJ0mEba51Fobz3WMi9Vwrq1/eL2fvF9OvXB9VmOPR9CdIyASXLvi9BG5fc0eKMnTuhUSO7rTCUQ2vtyNa7d29dVTp31vqSS6r8dUuZNEnrzEytQevERFlmZsrnZbdRqvy6cPaflib79bW0tMrtw0l4PKHfG9zBSSdpfcYZdlsRATZM0vqTTK0nK1lucN4PDZivg+hqkt03nEiQmQkbN9ptRUkqpS+7xjftmy+l0of/Nv7rwgnHhMrecUs4x5+yo1ErKk1gcB5aw9q1MttcTOFLDfZli/lSg8E1T5oxF9IB6NxZyiN7PPbaEUiMffhEubqDpdyce2+ITbZtg717oXt3uy2xmBhIDY5JD79bN5nTds0a6NTJPjsqEt1Q68MV7NatA8+H6obce4NFFBdA0SHwFICnsGSpC73vi+RRSSVBQhKoxNKvE9MguQ4k1LDkkcpXB79nz2rvylnEQGpwTAr+qafKcuZMewU/mBj7r4fQgj15snj7W7bIZ+PGlQ7VjBtXOiQE7sq9N/ihNRzLhfxdgduxPVB4oHQrOgjFFtUvUImQVEfEP6m2LFPSoUYjSG0szf91zeZQsxmo0oGC77+HWrXgxBOtMcsxpLX2jvQO8LlLsETwlVLnAhOAROAVrfVjZdZfDzwBbPN+9KzW+hUrjh2IDh2gWTP48cfSsfJoE0iMffiLcjDBLtsHECi+71uGuikYHET+Hji4Fg5v8mubS5aBxFsllYhtSn1IawnJdb3iXNf7ujYk1oCEZFDJsvQ1lQRo0MXi7esiv9eFUHRUbhxFh6DwoLwuPCTLY7lwYCXk74bio+VtS0iGtFZQK1OEr1YmdXMyuf3yDqR4OgMNI3gyo0yPcaVj+OC61GClq1nOTimVCKwBzgKygXnA1VrrFX7bXA/00VqPCne/ffr00fPnz6+yXcOGieBv22Zvx5/PQ9+8WVJGi4ulU9lflIN58VlZgb3/zEzJsTc4FK3hyFbYv1LE8sBK7+sVIqD+1GgItbJEMGtliXjWbAqpTUpaSgNn9F4XHRbh97Wj2d6bVUnTR3eg8NOUGhlQtzPU7STLep2hQU95MnAjGydLzP7IFrnB9RjnuA5bpdQCrXWfgOssEPx+wFit9Tne9w8AaK3/5bfN9URZ8F9+WbzhJUvg+OOrvBtbSUgIXF5WKfs7pA1etAcOroe9C2DfQlnuXQiFeSXbpKSL0NX1tY5Qq42IfHLtsA81ebKMS/CNMcnIkIFpTnqae/65Ap58eCszPltD6/qrvDe6VXLTO7anZMOazaBBL0jv7W29oGaLyN/YXCDY1SWU4FsR0mkBbPV7nw2cFGC7S5VSA5CngTFa661lN1BKjQRGArSuZq/jRRfBrbfKj+Sxxyre3omYDlkHUpAHObMh5xfYMxv2/SaxdICEFKjfHTKvFC+2XhfxbGs0qraQTZ4MN95YUrIARPhvuEFeO0X0X3sjhdpN29HqpHagziu9Mn+PPOXs/a3kBrnja7lpAqQ2hcanQaPTZFnveEhItM64GEirrC5WePiXAedqrW/yvh8BnOTvzSulMoBDWutjSqmbgSu11oND7be6Hj7ABReIh795s3jLbqNsDB8kvu+m+jiu50g27P5ZBD7nZ8hbBmiJi6f3gvQ+JR5qva4S044AwcJ74JwQ35Il0KMHjB8vTyJhUXQY9i2RG0Dur3KufVkvyfWgYX9oPACanQUNTijXQVwpPs0K0umaCX/YVPX9OoxIe/jbgFZ+71tS0jkLgNbaP3D5CvBvC45bISNGwNVXw08/waBB0TiitZgOWRsoOgK7f4Id38KObyQUAdIp2rA/HH+5eJ8ZfSEpLfS+LKSyKbwVZXdFgtdfl+KFlTpOUi1o1E8aXh/x8BbvTdbbFj8grUYjaHYOND8Pmp4NqZXsEI6BtMrqYoWHn4SEac5AhH4eMExrvdxvm2Za6x3e1xcD92mtTw61Xys8/CNHoGlTGDIE3nmnWrsyxDL7V8G2z2HntyI0nmOQmAqNBkCzs6HJIAnVJNiXxVwZD9+OJ8PDh8WOQYPggw8s3nn+bu8NeJrchI/tAZQ8XbUcCq0vh7rHVbwf4+FXX/C9BzgfGI+kZb6mtR6nlHoYqekwVSn1L2AoUATsBW7VWq8KtU8rBB/gnnukCNfatdC2bbV3Z4gFtJb48daPpR3wXor1uooH2ewciSMn1bTXTj8CxfBBPOrXXy8t5HZkdz3xBNx7L8yeDf36ReYYgMT79y6A7dNg+1cSBgKo102Ev/Vl0ncSiLIxfJC0yhgrwhdxwY8EVgn+9u3Qpo10br3wggWGGdyJ1tLJuuVDEfkjW2SgUePTodUl0GIo1GpV8X5sJNwsnWhndx0+LDeZ3r1h2jTr9x+SI9tg60fyf835BdCSCZU1HNpeJ2MW/InzLJ2YF3yQbJ3XXpOCas2bW7JLg1s4uA42vg2bJsGhDVI+oNnZXpG/UPLEY4xoe/hR8+4r4sh2yP4EtrwPu2cCSv7XbW+ElhfJwLQ4IJTguzB3pfLce68MeHrkEftsiIWa9a7h2F5Y+zx82x8+7wDLHoHabaHfW3BpDpw+FdpeH5NiD9Gd6nLvXnj8cTj7bJvFHiCtOXS8Hc78CYauh25/hf0rYNaV8ElzmH+n9NfEM8HqJtvdqlMPPxB33CH15hcssHS3YRFrNesdicejdc7/tJ59ndbv1tB6Mlp/0VXr5Y9rfXir3dZFnerMsVAZbrpJ5nlYvDgy+682xUVab/9W61+u0vrdFLkupp+r9bZpMTvZAiHq4cdFSAcgLw+OO046bmfNim5efkUZFibVMgDhxlqLjsDmKbB2onTmJdWGNiOg3U3evG0HlCSIUWbNkkKF99wjYR3Hk78b1r4o10r+Ton1d7oLsq6RFNsYie/HfQzfx5tvwvXXSzzfN0IxGgTrRPNhBlOVIZxsiiPbYfV4WPeylDGo1wU63CZin1zXDqvjisJC6NVL5o5esUKqY7qG4gKJ8696SrK1UhtDkzNh6yfg8SsQ59IMHiP4XjweOO00mRxl8WJo2bLi71hBKA/fh1NGSzqCYPnSyRnS8Za/3fuBglaXQsdRMhrTePNR46GHpE/s00+ljIkr0VoGdi17FHZ+F3gbF+box32nrY+EBMlZPnZMRuH6phyMNIE60cpiZqjyI9jIx8JcP7FHMm5a/gGanG7EPop8/z08+qg8JTtW7DdOFsfhnQRZbgyQJaGUOAqDvw2+nxgbhRtXgg/QsSM884yUTn788egcc/hwCdlkZgbfRmuTvfM74U4o4cl31fRy0SDS2WA7d8I118g0os88Y+2+LcMXEjyyGdAlRdICib6PtCA/zpSGoeOxLiPuBB8kjn/llfJYOmdOdI45fLiEbCZNCu7t+yY4iXvR7zEOElLD2zbGPLDq4CupsHmzaJTV11NxsYj9gQPw/vsOi9v7e/S/Xlf5uWd7jJOYfSkUFOTAjLMhb6nVFttCXAq+UjLqtnVruPRS2FquUHPkqMjbr8wE5jFJfg7k/k9mYsIbpqnZElKC5My7aHq5SPPgg+VnV7PqetIaxoyBH34Qz75r1+rv0zLKevQ6SKw2lHPQZrh00KZlAkqWJ70OvZ+W7K+ve0oef+GhCPwB0SOuOm3Lsnw59O8vj76//AJ16kT0cOUwE5z44SmCtS/Akr/J1Hptb4TjHyoZGh8ndVCqQySvp//+F/70J7j7bvjPf6q3L8sJ1slflqp2wB7LhSUPyWC+Wq2h78tSrtmhmE7bIHTtKpX9li+XEE9RUXSPH2wik7ib4CRnFkzrAwvukNry5y+Fk14qXQclkAcWg2JfnRh8pK6nDz4Qsb/sMofm24cT1qvO3LM1MuDE5+Csn6WK6oyz4dcboWBf1fZnJ8FGZNndrB5pG4oXXpDRryNHRnfwXdyPwD26S+tZI2T04ycttd78fsyOfgyHQNeDUlrfemvVv1/d62nGDK1r1ND6lFO0Pnq06vuJKJ9kyjVUtr2TqPVkJes3hHESNkzy7ivEd4qOav3b/bLvj5tpvXO6tX+LBRBipK3twh6sRVPwtdb6/vvlbNx+e/RFPxpD4B3H5g+0/rCh/HDeq+sV/czwfpgxSmZmabH2bxkZ4V0jVl5P33+vdc2aWnfpovWePVXfT8TZMEnrKWmlxX5KWuWupcruI3e+1p930vqdBK2X/ENKODiEUIIf1zF8f7SGP/9Z4pOjRsHTT5vU7ohwLBfm3Q5b3pOJvI/ukPRKH764PMTEMPfKUNGIbB/RGJn97beSY9++vXTUNm4cuWNZQnXLIlRlcpTCQzDvVqnE2mQw9J8MNZtWxXpLMTH8MFBK4pN33w3PPit1x6t6LzSVMYOQPRW+7ArZH0P3R8BTXFrsQTplF4yufB51DBBurD3SmVxffw1Dh0rtqRkzXCD2IOL+h00wzCPLyjoHVZn+MLm2VGA96VXYM0cyeXKilOddRYzg+6EUPPmkpJ8980zgGYYqItK50K6kuAAW3AUzL4LUpnDOPCldezRIPmxBbuXzqGOAcePCf6qM1Mjst94Sz75LF/HsG1Zy2ljXEiy9t6K0X6Wg3Y1wzlxIqgM/DIItVs/xaB1G8MuglIR1xo6FN96ACy6A/fvD/34kc6FdyeEt8P3psHoCHDdafhgNesi6yubQx/ggq+HD4ZZbwhN9qzO5tJaBiNddJxUwf/hBZtSKGwINvKpMZk/9bnD2HMky++UKWPFvR47QNYIfAKXg738Xwf/xR/kBhDs4K5jnFZe1crZPg2m9YP9yOPV96D0eElNK1gf7kSXH7yCriRPh7bdlYJ5SIropKaW3sXoyk/x8udk88og81U6bBg0aWLd/V2BF2m9qQzjjB2h9JSy6T+L7nigV7AqXYL25drdoZ+kE47vvtK5bV+tmzbSeM6fi7YNlWmRmRtpSB+HxaL3iCUlv+/J4rfevDr5toFQ4K7IuYohIZnJt26Z1//5yjf7rX3GdFWsdnmJJ3ZyM1nNukPdRBJOWWT2WLNG6TRutk5K0Hj8+9I8i7nPriwu1nnurXOw/X6514eGq7SecnGhDKSp7Y5g2TetGjeT6fP/9aFgYZyz+uy2ibwTfAvbu1XroUDljl16qdV5e8G3jNre+8JDWM4bIRb7wz1H3bOKZyjgahYVa/+Uvsk23blqvWBF9e+OGxQ95Rf/GqP0ejOBbhMej9b//LXN4tm9vz/y4juVojtZf95aBKGsm2m1N3BFuKHHLFq1PO03W3XST1oer+ABmCBOPR+vFfytxgqJAKME3nbaVQCkZnDVjhmTenHSSdO5WNnUz5sjfA9PPkM7ZAZ9Bh1vttijuqChZQGt4+WWpH7VwoXQMv/xyxRPzGKqJUnD8P+Q3sfIJWP+areYYwa8Cp50GS5fCVVfBww9D376waJHdVtmET+wProEBU6HFELstiktCFU7btAnOOkvGg5x4IixZInXtY5pwZryKFkpJmeWmZ8Pcm2HXj7aZYgS/iqSni5f02Wewa5f8kMaOlekT44ayYu/gkrGxTqBpNGvWhIEDoVs3mDtX5oD4/nto29YWE6NHVWa8ijQJSXDqe1CnA/x8CRzaaI8Zthw1hhg6tKS88j/+IT+uL7+026ooUHQEfjzPiL1D8J9YRylo0kRKIrz5JpxyCixbBjffHCf1oRY/6MyR2in1YeAXMkHLnBEyB0SUMYJvAenpMnXhtGmQmAhDhkhbty78fbiq/o6nGGZfIzMBnfKeEfsoEuo6GT4cfv5ZnI9du2RKwnfflesyruZYqEpdnGhRuy30mShzQKyI0qTafhjBt5BzzpH46BNPwE8/SQfZX/5ScWkG19XfWXQfZH8iI2dbDrXbmrgh1HVy+LCMlO3UCT79VMokrFol/Uxx4dX7U9W6ONEiaxhkXgVLx0LuvKge2pRHjhA7dsB990mcPz0d7r1Xyi4Hmvg5K0t+vGXJzJQON0ex7iXpeOp4B/R52m5r4opg10mDBpCcDLt3y6xUp5wC48dLhk7r1hLfj2QpZcfhhukwC/bBV90huR6ct0hi/BZhyiPbQLNmUnlwwQJJ37z/fmjXTqpwlu3YdU39nX2LYP4d0Ow86PWU3dbEHcGuh3374PjjYdYs+MMfpFCfa54WI4EbpsNMaQC9J0gq8/qXo3ZY4+FHiVmz4K9/lWJsrVqJ93/DDZJZ4QoPv+gwTOsNhQfhvMVSKMoQVYJdJ02awM6dobdx1LVkELSWcsr7l8GF66RT1wKMh+8ATjkFpk+H776Dli0lvNO6tQzcuvfe8il1VldErDYLRsOBNdB/UmTF3kn50w5i1y7o2bP85zVrSjlvH655WjRI50qvp+DYXlj2SFQOaQQ/iigFZ54p3v7PP8tN4OGH4U9/gpNPhubNZZvMzMhPYVcptn4C61+Frg9Ak0GRO06g/Ok5I2DubZE7psNZsQJuukmcg6lToXdv8ehBrpOXXy59nYQagBVzxIJzkH4CtL0O1k6UcS0Rxgi+DSglNfY/+0wyKUaMkJvA9u0weLBk+Vx+ud1Weik6DAvuhPo94PixkT1WoPxpNKx7wZ0/5iqSny8x94EDJdPrnXfgj3+E1ath/nwJ32gtIZqyTkGgAViOe1q0AicOrqoqnf8Mxfmw7sWIH8oIvs0cd5x485s3S1rd2rVwxRUS9rnvvsrl8keEZY/CkWw48TlISI7ssYLmSWv7B81EgWXLZC7l5s2l9MHWrfDPf0o4ZuJE6NCh4n2UHYDluKdFq3Dq4KqqUK8LNDsH1j4n04FGENNp6zCKi+Hbb+VH+vnn8n7gQBg2DC69VFI8o8aB1fDV8ZA5DPq9EfnjfZrl9dgCoWSCan82TpYf+JEtkmPdY5yzMjHCYOdO+OAD8eJ//VXSKy+5BP7v/2DQIBlgZQjAOwlAIO0KcJ24ge3fwI/nQr+3oU31Ch2ZTlsXkZgI550Hn3wint0jj8C2bZJa17QpXHihiMOhQ1Ew5rd7IbEm9IzSiMAe44Ago4TKDppx8SN9bq7E3s84A1q0gDvvlP/nk0/K/3rKFFlnxD4ETh9cVVmanQ212sDmKRE9jLmkHEzz5pLK6Yvd3nmnVOUcPlzqpFxxhZR0yM2NwMHzlsK2qdDpT1CzSQQOEIA2w6H9LZQT/UCTSbvskX7bNnlqu+ACuXGPHCkhmwcflFDO0qXSed+okd2WuoTqTjruNJSCVhfDzu+g8EDEDmOJ4CulzlVKrVZKrVNK3R9gfQ2l1Hve9f9TSmVZcdx4QSnJznjySYn1z5wJ118vyxEjRPxPOw0ef1yyOiyJ0i1/DJJqQ8dRFuysEvSdKI+1FQ2aqaheis0ZHB6PVKh86CHo1Uv6ZG6+Wf4/Y8bIgLzVqyVLq2vXqJoWG7hhcFVlaXkxeApg+9cRO0S1Y/hKqURgDXAWkA3MA67WWq/w2+Y2oLvW+hal1FXAxVrrK0Pt18kx/MmTxTOze+i6xyOe/xdfSLzfV5O/TRtJ/xw8WOL/TZtWcscH18MXHaHT3XDCExZbbRHB4v1pmeLlRXlovdawcaNMjvPjj9IPs3u3hGX69y8pqNelSxzWtjGEh6cYPm0hqc+nvFvl3YSK4Vsh+P2AsVrrc7zvHwDQWv/Lb5tvvNvMUUolATuBRjrEwZ0q+L4CVkf8tCQtzRmZENnZIv5ffine/wHvk2HnztIBOGgQDBggTwQhWfgnWPMMXLQZajaLuN1VIlS9lMUPBr8Z/GGTJYf3CfxPP5WI/Natsq5xY4nBX3ABnHsuZGRYckhDVXFT5/6sYbD7J/hDdpU9g0gL/mXAuVrrm7zvRwAnaa1H+W2zzLtNtvf9eu82e8rsayQwEqB169a9NwcaI24zbhm6XlQEv/0mYjRjhgz0OnxY1mVlySxdvtarl19RN08xfNYKMvrCgE9tsj5Mgv2QI5DBsWcPzJsnYZq5c+V1To6sa9RInqQGDpSbaqdOxot3DG4opObPmudg/ihxtmpVrQM6lOBbV6LNArTWLwEvgXj4NpsTELcMXU9Kklm4TjxRSjcUFkr4Z/bsEtF6/33ZNiFB4sg9e8IFvWdwZaMd5NQaTkPtcOFqMzzwjzatdRAPv+IfUGGhjH1Ytkwmtlm+XOLtG70TFCklYZkhQ+TcDhhgwjRRZ+5tsP4lmUhEJUK7kdL3E4hQnftOFPwGPWWZt7TKgh8KKwR/G9DK731L72eBtsn2hnTqAZHILYk4rVsH9vCdPnQ9ORn69ZPmY/fu0l7r9OkwqOZkDvStQ+t+Q0iqITeCrl2hfXuZGq9NG1lmZDhY5ILF8L0ZHB6PZM1s3FjSVq0ScV+9WkQf5O9r1046zG+9teRpqE4dG/4mgzD3Nlj3fMl7XVzyvtEp5Z/4nDwZSiDqdZHl/hXQ4gLLd29FSCcJ6bQ9AxH2ecAwrfVyv21uB47367S9RGt9Raj9mhi+DWgPng8bsyflPD7Z+TbLl5d4urt3l960Th0R/zZtJH20SRPpHPa1Jk2k1awZRfO19Fvs2gU7f/uOXfM/YefuJHYd6cDOhPPYmteejRvlhl3gN6DRNyK1W7eSG1y3bhKaiab9hjB4N0lEvhwJkJha/iafWBMKAviWFvbnWM4H9aDNdVWebyKiIR2tdZFSahTwDZAIvKa1Xq6UehiYr7WeCrwKvK2UWgfsBa6q7nHtwifqobJ0nJLFU2n2ryShMJfGvQZx80WlVx06VOINb9ggbeNGCX/88kvwsQA1akDdutLq1St5XbeurEtOlpaUVPI6OVm88MJC6YsoLCz9+sgREfZArej3aULP8jYZzNa4sQxy6tlTRrL6blZt2sj/qEaNCJ3TauLaaylSBBR7AE/g0E1CTRH+IE97jqRmCzhaNkhiDaa0gsW4+glg7Qsw71a4cC3UaV+prxYUSCfmzp3Sdu2SlpdXIsb795cW52PHyot6YaGIPXhvAklFJKmjJCcWkJzkISm1Fml10krdOPxvKA0alH7KaNpUwk9uHLXq6mspUgT18IOhZDBfuDF/J/BeLXlauaxqUe+IZulECrcKvluyeAIyazjsmg4Xb7c1QO/xyOHVJpdlWFhMpa8lXaaXvez7WKBsDN9HQi3wHC7/eXIGeI666xp6x/s/G1Y1bTa1dKKIW7J4ArJvITQ8yXaRSEjwmuCy8glWU6lraclYWDimZJi11vJ+ydiI2GYbfSdC+1vFUwdZtr8VTnoxcKkFhfuuoRqNxMOPAEbwLca1E1B4iuDQeqjbyW5LSqhKhkUsTIrhJexrSWsozIPVE0pEf+EYeV+YZ1GtDQfRdyJcXSQe8NVF8j5YqYWCvYH34dQsHYD63SA9oINebYzgW4xrJ6A4ugM8hVC7rd2WlFDZiogurqAZiLCvJd9UeceNFpF/N0GWx42Wz2MtrBOMNsMl82aYR5ZthruzqmZBHiTXj8iujeBbjGsnoMj3zoKd6qBSCpWtiFjZEJDDnwYqdS35RN+feBL7YLixquaxPVAjMhNfOGqkbawwfLgLBL4sBftkGaELrUr4OtXCrYNSmRBQ2SH3vqcB/+M6gLCvJV8Yx5+FY4zoV/YashtPoaRk1sqMyO6N4BuE4qOyTHTYSKNg5RMCUZmSCm4bch8K/5i9L4zjew9G9CtzDdnNwfWgPRELrZqQjkH4vWPPxcJQmcd3tw25D4VSEvP1j9n7YvrJ9eNb7N3G3nmyjFCnrfHwDUJCiiw9hfbaUR0CPb43P1/ezxlR+nG+GgXWHEn3saXz7n2ib8TeXeTOg6RaULdzRHZvBN8gJHsrghVFbnq1qOD/+B4qTl9BgTVXUlbcjdi7j9y54t0nJEZk9yakYxBqNJRl/u7Q27mJiuL0sTZFnsHdHNsLexdAw/4RO4Tx8A1CWktZHnZhDLssv0+MEmQCHV+c3k2deYbYZ9tU0EUymXmEMB6+QUiuA6mN4eBauy2pHqUGXwXBrXF6QwkOH0NRJbZ8JNdmhDpswQi+wZ963SBvsd1WVI9AYZxSKOnINbiXGBtRDUDBftj5LbS+LKJ9L0bwDSWk95ap1Yrz7bak6lSYVqlh45vuFod4JxaL6m14AzwFkDUsoocxgm8oodGpctHt+Z/dllSdcMI1bheHeCFY2CaWxlCAFC5cPR4anSZOVwQxgm8oofEAUAnyaOlWAg2+CkSwcguxFhd2K6HCNm4siBaK7E/h8CbodHfED2UE31BCSn1oMhg2v+fekrpl0y1VkHzmsuIQi3FhNxMqbOPGgmjB0BpWPgG120GLCyN+OCP4htJkXi118XPn2W1J1fEvk3vym+GJQyzGhSNFNJ6EQoVtYmkMxdYPZbBVl/sjNtjKH5OHbyhNq0tkXtvN70LDvnZbU33CrZYYa3HhSBGtKqMVlb6IhTEUxfnw271Qvwe0vSEqhzQevqE0KfWh+QWwaRIUhUpvdBGBJsYoSyTiwrHYJxCtJ6FYCtsEY9V4id33fioq3j0YwTcEotMYmYRh/Wt2WxI9rBaYWO0TiNaTUCyFbQJxeDMsHwctL4Img6J2WCP4hvI0Pg0anSKdSW6unlkZrBaYWOsT8D2tEKQzPxIZMuE8mbkRTzHMuRZQ0Gt8VA9tYviGwHR5AH4aApvegbbX2W1NdLAyLmxnn8DvtYQsmuGpbNy+LLEWaok0q5+C3TPh5NehdlZUD208fENgmp8PDXrBkr9C4SG7rXEfduWKRyKUFKpcRayFWiLNviVyPlteDG2i70gZwTcERino8wwcyYZlj9htjfuwq9MxEqGkoE8lKrZCLZGmYB/8chmkNIC+L9oyX4ERfENwGvWXdLFV/4X9K+22xrkEysaxq9MxEqGkWBvZageeIvjlCsnKOe0jSG1kixlG8A2h6fkYJNWGuTdLZ5OhNKFCKHZ0OkZCnCP5tBKLqauBWHg37PweTnxBEiJswgi+ITSpjaH3eMj5GZY/arc1zsNp2TiREOdIPa1U1N8QKzeDtc/DmmfguDHQ7kZbTTFZOoaKaXOteCfLHobGA6HJ6XZb5BycNkI33JHFVdmv1U8oFd0sozGiN9JseAvm3S6DGU/4t93WoLRDi2T16dNHz58/324zDD4KD8K03lB0GM5bZFsM0hFsnAzzR0NhbvBt0jIljGM3VqdoWsk7CQTO61chSis45LyGw+b3YfbV0HgQDPwCElOjclil1AKtdcBps0xIxxAeyXXg1PfhWC78fDEUHbXbInvYOBl+vSG02DslL93pIZNQ/Q1Oe3KqLNlTYfZwmZD89M+iJvYVYQTfED4NekL/SZAzG+aMAO2x26Los/hB0CFGHzspLz1YyGTONfBhQ7lx2Vn6IVR/g5szgza9K+mX6b1g4JeQVMtui37HCL6hcrS+DHr9B7Z+BAv/ZLc10Sekh+mwvPRQthbklr9xRbuzOVRnsFuLp616CmYPE89+0DeQXNdui0phOm0NlafTGDi8RaZlq5EB3f5qt0XRI1hs2bfOSYSyNRjRDpmU7Qz2hZmObIHkdEisCQV7ndf/UBbtgUX3wconodVl0P9tx4Rx/DEevqFq9PoPZI2AJX+DxX9z7wxZlaXHOFDJ5T9PSHGe9xnudI/+2HnTKtvnUJgLxUeh39vOenIqS9ERmH2NiH2H2+GUKY4UezAevqGqqATo9wYk1pD8/OKjcMITtgwXjyo+0fHP0knJgN4TnCdIpVI0w/D07Q6ZhErTdNq59XFoA8y8GPKWQo9/QZf7HP0bMIJvqDoqQWqCJKbCqv9IymafZyAhxi8rN8y2VDYds/2tsPHN0oKakAKJdaDQISETt2XmbP8aZg0TgR/4FTQ/126LKiTGf5mGiKMSoPfTkomw4nE4vFEeaVPq221Z/BJoGsKNb0p1xu1fOTMnHyqe1tApeIpk8pKl/4D63WHAx1C7rd1WhUW1BF8plQ68B2QBm4ArtNb7AmxXDCz1vt2itR5aneMaHIZSUnOndnuZD/fbfnD651Cnvd2WxSfBQiPbv3L2oKUe48rX3bc7zFSWg+slJXnPHMi6Rp5wkyrZT2Ij1e20vR/4QWvdAfjB+z4QR7XWPb3NiH2s0v4mGPw9HMuBb06CnT/YbZF1WD1IKZKDntwWGvHh5GkNtYZ1L8PXPWD/Cug/Gfq95Sqxh2qWVlBKrQYGaq13KKWaAT9qrY8LsN0hrXXtyuzblFZwMYc2wE8XSknlrg/A8WMhIUBmi1sINONTYlrVxcjq/ZXl0yz3lyVwEkd3SLXYbZ9DkzNkpqparey2KiiRLK3QRGu9w/t6J9AkyHapSqn5SqlflVJ/CLYzpdRI73bzc3JyqmmawTZqt4Vz5kplwOX/hO9Ok5uAW7G6ImakK2y6ddCS0/AUw+pn4YtOsONbmX928LeOFvuKqDCGr5T6HmgaYFWpq1NrrZVSwR4XMrXW25RSbYHpSqmlWuv1ZTfSWr8EvATi4VdovcG5JNWCk16BpmeLN/tVTzhxImQNd3TaWkCsDpFEIuRSNivH6R20Tid3Psy7BfYukGu4z7NQt4PdVlWbCgVfa31msHVKqV1KqWZ+IZ3dQfaxzbvcoJT6ETgBKCf4hhgk8wrI6CuFpOaMkEnRT5wY9cmbq4XV2SNW7y9YVk4k499OrsJZHY7thaV/hzXPQc2mknHW+gr3OSlBqG5IZyrgm4n3OuCzshsopRoopWp4XzcETgFWVPO4BjdROwvOnCmPxDkz4cuusPK/kt7mBqwOkVi9v2hPwhKJidLtpjgfVjwBU9vB2onQcRRcsBIyr4wZsYfqC/5jwFlKqbXAmd73KKX6KKVe8W7TGZivlFoMzAAe01obwY83EhKh02i4YAU0GQS//Ukyefb8ardlFWN19ojV+6tMiMiK7CCnzfJVHTzFMknJ5x1h0b0y/eB5i6HP05BSz27rLMdMgGKIPlrD1g9h/p2Qv1MemXv+yzWDVxxHuFk5VmUHhZq4ZJhLSmZrD2z7ApY8BHmLIb2PlAZpMtBuy6qNmQDF4CyUgtaXw4VrodtD8sP7ohMsuFtiqIbKEW6IyCrP3M216j3FsPk9+LonzLwIig5JnP6c/8WE2FeEEXyDfSTXhu7/EOFvcy2smSAx1KWPQEGe3da5h3BDRFZlB7kx7dNTCBvegC+7wKyrpP+o39swZJU3Th8fUmhCOgbnkLdUvM1tn8vEER1HwXF3xff8uVZi5YAst2TpHMuF9a9I1s2RrTJrW9e/QquLY1bkQ4V0jOAbnMe+RTJga8uHMgFGh1ug092Q1sJuy9xNpEf4Ool9i2HNM7BpsmTgNBkk11DzC2Iq6yYQoQTfVMs0OI8GPWXC9P0rYfm/YPUEaa0uEa+/0Wkx/6ONCKXq4zvcM68Kxfmw9VNY9zzsninOQpvr5Jqp381u6xyB8fANzufQRlj7vDyaF+yD+sfLjzhruKMmiDbYgNaQO1fi85unQGEe1MqS66PdjZDSwGYDo48J6Rhig6IjsPldWP2MpNIl15OUzrbXyaTRxuuPH45sk3DNhjfgwErx5ltdCm2vl/BNjMbnw8EIviG20Br2zIa1L0L2xzLTVu120GaENJPPH5sc3gpbP5IxHDmz5LOG/aHtDZLmG4MDpaqCEXxD7FJ4CLZ+DBvfgl3TAS0i0OoSycQw4u9uDm+GLR/Blg8g1zsqu353EfjWV0Ddjvba50CM4Bvig8NbYdMk2PK+ZPoA1O8hwt/qEqjXzYR9nE5xAeT8IpU+d3wtk42AdOS3vhxaXWZEvgKM4Bvij0MbYesnkP2J9/FfQ6020OxsaU0Gm3l3nYDWcGi9zI6242tZFh2SCdYbD4Bm50HLoWa6zEpgBN8Q3xzdBds+g21fwq4ZUHRQOvXS+0Kzs6DpWZBxIiSm2m1p7KM1HFgFu3/ytplwdLusq5UpAt/8PLkhJ1dqkjyDFyP4BoMPTyHs+R/s/E5mMdo7VwppJaRAem+pltiwv7SawSZwM4RNwX6ZRGTvPEmfzPkF8r3TZtRsBo1P97aBUPc4E3KzACP4BkMwCvLE08yZJZk/ufPAUyDrareTyVsa9CxpqY3ts9XpFOyDvOWwb6Gcx73z4MDqkvW120LDU6DJ6dBogIRpjMBbjhlpazAEI6U+tLxIGkDxMdi7EPbMkptAzizJ/fdRsxnU7wkNekDdztKBWKcj1Ei3w3p7KNgH+1fB/uXetkyWR3eUbFOzuYTJsq6B9BMhow/UyLDPZgNgBN9gKE1iDWjUT1rne+SzY3tloNe+RSVt53eg/WbsqtFQhL9uR6jdHmq1hrRWkNZSmlv6B7SWPo6jO+DwFpl8vmwrzCvZPrEm1Osi877W6yqtQQ9T98ihGME3GCqiRrqM3mwyqOSz4gI4vBEOrIGD3nZgjfQLHH0jwD4aifDXbAGpDeUGkZIhXm+NDO/7dCkVkZgGSWmyTEisut2eIvAcg8IDEroqyBPvvND7ujBPOrTzd4jA+1rZmvkJyZLhVLstNDxZlnU6SH2aWllxParVbRjBNxiqQmKKdDLWPa78uqIjcCRbyvGWW26FvEVwbI8U+6qIhBpe8a8JKhFQXoH1Ln2vPQWyP88xCUt58qUzuiKS6kiYqmYzCcGkNit5n9ZS+jFqNq/ejcfgGIzgGwxWk5QmoZ2KBggVHZF67QW5cgMo2CdlIoqOiJdddETeFx+B4qMi4NoD6NJL7ZFQVEINCR35v06oIXMLpNSH5Pqy/P11PfeEmgyWYATfYLCLJG/oplYruy0xxAkm+GYwGAxxghF8g8FgiBOM4BsMBkOcYATfYDAY4gQj+AaDwRAnGME3GAyGOMEIvsFgMMQJRvANBoMhTnBseWSlVA6wOYxNGwJ7ImxOVTG2VQ1jW9UwtlWNWLMtU2vdKNAKxwp+uCil5ger/Ww3xraqYWyrGsa2qhFPtpmQjsFgMMQJRvANBoMhTogFwX/JbgNCYGyrGsa2qmFsqxpxY5vrY/gGg8FgCI9Y8PANBoPBEAZG8A0GgyFOcJ3gK6UuV0otV0p5lFJB05WUUpuUUkuVUouUUvMdZtu5SqnVSql1Sqn7o2RbulLqO6XUWu+yQZDtir3nbJFSamqEbQp5HpRSNZRS73nX/08plRVJeypp2/VKqRy/c3VTlOx6TSm1Wym1LMh6pZR62mv3EqVUr2jYFaZtA5VS+/3O2UNRtK2VUmqGUmqF9zc6OsA2tpy7MG2z5txprV3VgM7AccCPQJ8Q220CGjrNNiARWA+0BVKAxUCXKNj2b+B+7+v7gceDbHcoSueqwvMA3Aa84H19FfCeg2y7Hng2mteX97gDgF7AsiDrzwe+BhRwMvA/B9k2EPgi2ufMe+xmQC/v6zrAmgD/U1vOXZi2WXLuXOfha61Xaq1X221HIMK0rS+wTmu9QWtdAEwBLoq8dVwEvOl9/SbwhygcMxThnAd/mz8EzlBKKYfYZgta65nA3hCbXAS8pYVfgfpKqWYOsc02tNY7tNYLva8PAiuBFmU2s+XchWmbJbhO8CuBBr5VSi1QSo202xg/WgBb/d5nE6F/bhmaaK13eF/vBJoE2S5VKTVfKfWrUuoPEbQnnPPw+zZa6yJgP5ARQZsqYxvApd5H/w+VUk6ZmNau6ytc+imlFiulvlZKdbXDAG9o8ATgf2VW2X7uQtgGFpw7R05irpT6HmgaYNWDWuvPwtzNqVrrbUqpxsB3SqlVXg/ECbZFhFC2+b/RWmulVLB83EzveWsLTFdKLdVar7fa1hjgc+BdrfUxpdTNyJPIYJttcjoLkevrkFLqfOBToEM0DVBK1QY+Au7SWh+I5rErogLbLDl3jhR8rfWZFuxjm3e5Wyn1CfKYXm3Bt8C2bYC/N9jS+1m1CWWbUmqXUqqZ1nqH9zF1d5B9+M7bBqXUj4i3EQnBD+c8+LbJVkolAfWA3AjYUmnbtNb+dryC9JE4gYhdX9XFX8S01l8ppSYqpRpqraNSuEwplYwI6mSt9ccBNrHt3FVkm1XnLiZDOkqpWkqpOr7XwNlAwMwBG5gHdFBKtVFKpSCdkRHNhvEyFbjO+/o6oNzTiFKqgVKqhvd1Q+AUYEWE7AnnPPjbfBkwXXt7sCJMhbaVie0OReKuTmAqcK034+RkYL9fKM9WlFJNfX0wSqm+iP5E4waO97ivAiu11v8Nspkt5y4c2yw7d9HohbayARcjsbVjwC7gG+/nzYGvvK/bIpkVi4HlSLjFEbbpkmyANYjnHC3bMoAfgLXA90C69/M+wCve1/2Bpd7zthT4Y4RtKncegIeBod7XqcAHwDpgLtA2itdZRbb9y3ttLQZmAJ2iZNe7wA6g0Hut/RG4BbjFu14Bz3ntXkqITDYbbBvld85+BfpH0bZTkX69JcAibzvfCecuTNssOXemtILBYDDECTEZ0jEYDAZDeYzgGwwGQ5xgBN9gMBjiBCP4BoPBECcYwTcYDIY4wQi+wWAwxAlG8A0GgyFO+H8oXENEGzt+CgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.title(\"Naive Bayes\")\n", "plt.scatter(x=samples[labels == 0, 0], y=samples[labels == 0, 1], c=\"blue\")\n", "draw_2d_gaussian(mu_c0, sigma_c0, c=\"blue\")\n", "plt.scatter(x=samples[labels == 1, 0], y=samples[labels == 1, 1], c=\"orange\")\n", "draw_2d_gaussian(mu_c1, sigma_c1, c=\"orange\")\n", "plt.legend([\"c0\", \"c1\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the final posterior distribution for the case $p(c=1|\\boldsymbol{x})$" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "(-1.5, 2.5)" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAEKCAYAAAC48pk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABusklEQVR4nO29eZxkZXX//z639t57emZ6NjZhGBZBlBE0LI4L2yjbT1SMUSQQY4I/QxQVl0RjjEvka4hLTAii4s+vyFe/6CiDgOjIEEUZkH1xhm2Y6Vl736q7qu7z++Peqq7l3tq7q7r7vOdV01W3nrr3udVdpz73nPOcI8YYFEVRFEVRlMZjNXoCiqIoiqIoioMKM0VRFEVRlCZBhZmiKIqiKEqToMJMURRFURSlSVBhpiiKoiiK0iSoMFMURVEURWkSVJgpiqIoiqI0CSrMFEVRFEVRmgQVZosUEfmCiFxdx/19RkQ+U+Vr/yAix9drLoqiLGzUfikLGRVmixARWQa8B/ivRs/F5Trgs42ehKIozY/aL2Who8JscfJeYLMxZrLRE3HZBLxeRFY0eiKKojQ970Xtl7KAUWG2QBGRF0Tk4yLypIgMisi3RSTqPn0e8BuP17xTRJ4QkQkReVZENtRpLkeKyICIvMp9vEpEDqT3b4yJAw8C59TjeIqizG/UfimLGRVmC5t34RiLI4GjgU+5208AnskeKCIfdp9/N9AGXAS8UI9JGGOeBT4G/H8i0gJ8G/iuMWZL1rCngFfU43iKoiwI1H4pixIVZgubrxtjXjLGDAD/ArzT3d4FjKYHuTkbnwb+3BjzkDHGNsY8Zox5QUS+JCJbReR7IhKqdiLGmP8GdgC/B1YCn8wbMurOS1EUBWq0X8Cgm5g/JiIvr2Uiar+UuUSF2cLmpaz7LwKr3PuDQHvWc28CHjPGPJL9YhF5BbDaGHMG8DRwSY3z+W/g5cDXjDFTec+1A0M17l9RlIVDTfYLmADeDPyoTvNR+6XMCSrMFjaHZN0/FOhz7z+KExpIswRvo/JnwF3u/V8Ap1U7ERFpA64HvgV8RkSW5A05Fsg3rIqiLF5qsl/GmIQx5kA9JqL2S5lLVJgtbK4SkTWuEfkk8EN3+2bgdVnj/gicLiKvEIe1InIs0A2MuGOGcQxgtfw7sM0YcyVwO/Cf6SfcpN6Tgbtr2L+iKAuLWu1XPVH7pcwZKswWNv8bx+P1HPAs8Dl3+83ARhGJARhjfus+93OcXInbgBjOVWiH+5pOYKCaSYjIhcC5wN+4mz4EvEpE3uU+Ph/YYozp83q9oiiLklrtV11Q+6XMNWKMafQclFlARF4ArjTG/NLn+c8D+40x1xfZx0nAh4wx7xGRTwDPG2N+4DP2MwDGmM9UMdffA1cYYx6v9LWKoiw86mG/ssZ+B7iumH1R+6U0E8FGT0BpDMaYT5Qx5mER2SciW4GdOBWuZ2Mup87GfhVFWZiUY78ARGQzcBKwTkT+yxjznVmYi9ovpa6oMFOKYoz5SJlDt8zmPBRFUSrFGLOxzKFbZnMeilIJGspUFEVRFEVpEjT5X1EURVEUpUlo6lBm2IqamNVeeqCiLGJGUgcPGmOWVfKaDRsiZmDQLmvsY48m7zTGnFvV5BYxar8UpTxm04bNR/vV1MIsZrXz2o6LGj0NRWlq7hy88cVKXzMwaHP75qVljT10zd7yBio5qP1SlPKYTRs2H+2XhjIVRVEURVGaBBVmiqIoiqIoTYIKM0VRFEVRlCahqXPMFGU2aO2K8raPvJHeI3oQSxo9nbIwtmHf8/38ny/fw/hQvNHTURSlQcxH+wVqwypBhZmy6HjbR97IceuPJhKIITI/DJsxhp6eHt72EfjOJ29v9HQURWkQ89F+gdqwStBQprLo6D2iZ94ZNREhEojRe0RPo6eiKEoDmY/2C9SGVYIKM2XRIZbMO6MGjmGbT6ELRVHqz3y1X6A2rFxUmCmKoiiKojQJKswUpQmYnp7i6o99gLMu2MDb3nMRu/p2NXpKiqIoZaH2q75o8n8TY4zBJoVtUlk/bQw2tnF+GvcfGMjuRy/Ofzn/xMIigIWFJQHnvvtzvrrGFwr/5ye30tHRyd2btnD7nT/jun//Itd/6euNnpai1IRtbGyS2CZFyrVhM/bLYLAhbcNM3ovFsWAgWFi5NizLfgUIIKI+hkai9qu+qDCbRWyTYtrEmbbjJMyUe4uTsJ37STPt3hLOT5KkTMK9JUmRnLO5BggSkCABCRGQIEFCBCREUMIEJUxIwoQkQsiKOD8lSkgihK0oYYliSWDO5jrX/GxzK1/5Rjd79gZYuSLFh64a5PyN4zXt8yc//zHfuvm/ERHWrT2GgYF+PvDXVwNwzhvP47Nf+jTGGBXMSsMwxmRs1nSW3UqYOAkzTcJMkzRTGfuVMo79SpoEKdeWmQK1NTs4Ii1IkLQNCxHM3PJsmEQIWa79kihhK0qA0IL9rKn9mn+oMKsC26SI2+NM2RNMmjGm7ImZm5lk2v2ZNNO++7AIZIxFQMIEJUSLxBxxRMgVSUECBLEkiCUWAYLOtaNYSNYVpHNVSeYnkDGIBoMxxvWuzXjabFKkTArbJEmRyhKDzs+0sY3bYyRNgoSZwiblez5BCRORGGErRkRaCFsxolYLEWklajm3iNVCQObXn9zPNrfyqc/1EI87V+R9e4J86nPOqqJqjdv2Z//EN2/8Oj/49o9Z0r2EoeEh/uLKd7ByxUoAgsEg7W3tDA4NsqR7SX1ORFFcjDFMm0ni9gRxe8yxZcaxX9P2pGvDJpk2k0WElRByBU/QtV9hiRG02jPCKG2/AhLAwrFhluvdcjxglvvFPePVz5lnVjTAiR6kbZgTOUhHEhxBmMyIwWyBOGEmHVtmT5Ek4fueWARc2zVjwyJWjIg1Y7+i0kpQwvNKbKj9mp/Mr2/JOcI2NnF7jAl7hAl7hMnUGHF7lEl7jEl7jGkzWfCaAEEiVgthidEW6GaJtdr9kEdnvEtZV2rzTaAApExyxvNnx5lOX03bcabNJFP2JFNmgpHUQaYSk6Q8DGFYYsSsNmJWG1GrnVigjRargxarg6jVhtVkIYmvfKM7Y9TSxOMWX/lGd9WG7f4Hfsu5b9qYMVpdnV21TlNRMjjCK+7arhEm7FHiru2atEeJ2+NuCHEGQQhLjIjVQtRqoSPQQ8SKEZaYY78sx7uU9jYF56GHyRjb9fSl7daMNzAtRKfsSSbtMYbt/UybwiKoAUJErVZaAu1ErTZiVjsxq92xYYF2ghJuwJn5o/ZrfjL/1EGdSF81jqUGGU8NM24PM+H+jNtjOVeKguV+CNtYFjqEqNU2cxXl3prtAzkbpL14UVqhjMil43Ebz3gX4/Y4k7YjckdS/exLvJjzBSEIUaudVquD1kAXLe7PtkAXYWlM3Z49e71P1G97tfQu62XP3j2s6F1JMplkdGyU7q7uuh5DWVgkTYLx1JBzs0cy9msiNVJwUeRcELXTGVhGb+iILNvVRtS9oJxvQqtSRCwndEkU6Cw53jY2U/Y4cZO2YeNM2uOuyB1lMLmvICoSligtVietgU5aAp20Wp20BbqIWR0NuehU+zU/WRTCbMqeZCw1wFhqkNHUoCPG7KGcD1WAIC2BTjoDS1kZfhkx14vTEuggIi0L3mjNBkEJ0xYI0xbw/oAaY5gyE0ykXM+kPcJEaoRxe4TBqb05OXZBCdNmddEWWEJboJu2QDftgSWEreisnsPKFSn69hR+TFau8A/rluI1r/4zPvDhv+a9f3El3V3dDA0P8YbXvYnbfv5jXvmKV3HnPXfwmle/Vv/mFMDxVI+lhnJs2HhqkLjJ9ngIMauNVquT7kgvLVaHY8MCHcSstnnpoW80lljEAu3EaPcdkzDTrlfS8UxOpIaZsEc4kHiJ6ek/ZcYJFi1WR8ZutQUcW9Zidczq51zt1/xkQX1ajTFM2COMpvoZSfUzkuxnLDXAlJnIjAlJhLZANyvDR9JqOd6Y1kCXiq8GICJE3Ry0JazMeS4t2sZTQ65Xc4gxe4h9iefZNf10ZlxEWmgP9NAR7KEj0EN7YCkxq61uv8sPXTWYk6MBEI3afOiqwar3ufbIo3n/FR/g3X91KZZlcdwxx/NPn/gXPvIPf89ZF2ygs7OTf/vC1+ox/TlDRF4ARoEUkDTGrBeRJcAPgcOBF4C3G2MGxfnl/DuwEZgA3muMecjdz2XAp9zdfs4Y8113+8nAd4AYsBn4O2PM3GSWzyHTdjzHfo2m+pmwhzMefIsArYEuukMrcuxXi9WxoBfgNCshCRMKLqWDpQXPJc0046nhHPs1kjrIvsTzmTEBQrQHumnPsl/tge66/S7VfpVHs9mveSvMjDHEzTgjyQMMJQ8wkjrISOpgxgsmCG2BbnpCq50//EAPbYHuReGyXwhki7ae0OrM9nQIejQ14N76GU0O0B/flfnyCkmEjsBSOoNL6QwspzO4jIjVUtU80nkY9V7VdPH5b+Xi89+as+2r//ofNe2zCXi9MeZg1uNrgXuMMV8UkWvdxx8DzgPWurdTgW8Cp7qG8NPAepziCQ+KyCZjzKA75q+A3+MYtnOBO+bmtGaHpEkwkjzIcOoAw64Nm7RHM89HrTbaA0voDR9Oe2AJ7YElDQuJKZUTlDCdwWV0BpflbE+HoDP2KzXAnqkdvMRTgONdaw8soSOwlK6gY79arc6qSoKo/aqIprFf80aY2SbFSKqfoeRehpL7GUruz3jC0n/IK0IvozO4lI7AUloDXeq+X4CIiLtiqoWloTWZ7SmTzBi69Jfc8/FHM2ItZrXRGeilK7gcYypz45+/cbxmQ7ZIuRDY4N7/LrAFx7BdCNzsXjHeLyJdIrLSHXu3MWYAQETuBs4VkS1AhzHmfnf7zcBFzCNhZoxh0h5lKLnPuaX2M5oahKy/z47AMtZEjqEzsJT2QM+sh+mVxhCUUIFgy472OParn72J5zLRgQAhV6QtxzZ/gcEUrGL1Q+1X1TTMfjW1ckmaabZPbmMwuZfh5IFMuYaY1U53cGXmaqIj0KNu/EVOQIJ0BZfTFVzOIZFjAUesjaQOMuwK+cHkHvYmnmXKTDKS6icoIQIEnZ8SavAZzC1jdpitk4eWOXrvUhHZlrXhBmPMDXmDDHCXiBjgv9zne40xe9I7AXrd+6uBl7Jeu8vdVmz7Lo/tTU3KJHkx/gSDyb0MJvdmVnMHcb6YXxY9ia7gMjoDywhbsQbPVmkkIkJrwFk0sCL8MsARa+P2EMPJA44NS+3nufjDbsSgHytju5yfssga+ZRvw+af/WpqYTZhj/B8/BHaAz0cEjmWrmAv3cHeqsNSyuIiIEG6gyvoDq7IbJtMjRKSKGGJOPWNmGbKjfQ7Qs0pSKne1hwOGmPWlxhzujFmt4gsB+4WkaeznzTGGNfoLRrG7SGenvwdUWmlJ7SK7uAKugLLaQss0XQKpSQiklnotDpyNOCEQcPi1FlLkXDEvvupStfGTF9olutRWwTMO/vV1N8+LVYHr+24aFGUolDmhlig3Sn5YbUBYLBzCuommWDKTCBIVueDEFY59UEWMcaY3e7P/SJyG3AKsE9EVhpj9riu/v3u8N3AIVkvX+Nu281M6CC9fYu7fY3H+KYmZrXz6raNxAL+q/oUpRKCEsKSQI5zYqbbgiPU0p5ZJxoQXpQRgUppNvvV1L7PdFVpRZktBIughIlarZml7DGrnaBESJkkcXuMMbfEStweJ2n8q4cvVkSkVUTa0/eBs4HHgU3AZe6wy4Cfuvc3Ae8Rh9cAw27I4E7gbBHpFpFudz93us+NiMhr3BVR78naV9MSkoiKMmXWCUiIiNVCi9Xp5FdbnUTEEW5TZoJxe5jRVD+T9igJM1VQXHix04z2q2aPmYgcAtyME381OPHbf88b47u8VFGaCcHK9NMDsElleprOXI1mt6NZfLkdHvQCt7nhuSDwv40xvxCRB4BbReQK4EXg7e74zTi2YAeOPbgcwBgzICL/DDzgjvtsOpEW+FtmlpvfQR0T/9WGKQuJdEusCC0YbDca4EQEEmbKGeN600JWRKMBTWi/6hHKTAIfNsY85KrOB0XkbmPMk1ljPJeX1uHYijKrWAQIi9OaxmDcsMF0prUL4DZKjhCScNUi7YEHf8/n/9c/88z2p/nKF77KuW/aWM/TmFWMMc8Br/DY3g+80WO7Aa7y2ddNwE0e27cBL695st6oDVMWJPkXmk7ahnOhOWUmmEpNZPVtrr5VoNqvzHN1sV81X+obY/akrxyNMaPAUxSuOMgsL3WXjKaXlyrKvEEQN+zp1JdqtbqISCzTW3U0NcB4atht/lxZuGDlytV84TNf5i3nXjBLs1f8UBumLBYC4vR0bg100R5YkulPPG0mGbed2mpxe5yUSZbeWRZqv+pLXZP/ReRw4JU4RdSy8VtGuidvHCLyPuB9QCZBW1EaSfvwT1i6/zqCyT6SwVUcXH4No50XZXqHRmjFJkXCdhq8x+1x4owTkJB7tVroSfvJz3/Mt27+b0SEdWuP4cuf+zcALGvRh0UbSq02TO2X0mz42S/B7R0qUQwmE+pMp2xYaW+bFcHKkwpqv2aXugkzEWkDfgxcbYwZqXY/bv2QGwA6g8sW1fJ6pfloH/4JvXs+geWudAold9O75xMAjHZelBln4ayUitCSK9LMGHHIhApCEmH7s3/imzd+nR98+8cs6V7C0PDQ3J+YUkA9bJjaL6WZKNd+CZKxT45ImyJhppkyk0ylJp2UDitKSCLseHaH2q9Zpi7yVkRCOAbt+8aY/+sxxG95qaI0NUv3X5cxamksM8nS/df5viYt0toC3bRaXYQlRsokmbRHGU31s/X3v+bsN53Hku4lAHR1ds3mKShloDZMWYhUY78ckRalxepww52tCELcHmc0NcC9v/8VZ73pHLVfs0jNwsxdrfQt4CljzFd8hvktL1WUpiaY7Ktoez5OzbRW2gNLaLE6CEqYFEkSJs5YaohpE8+0jVIag9owZaFSq/1ywp0xWgNdtAW6nZxabJJmmtFUP3F7PNORR6kf9fCYnQa8G3iDiDzs3jaKyPtF5P3umM3AczjLS/8bZ+moojQ9yeCqirYXIyhhYlY7rzv1Tfzqnl8xNDRE3B5j18DzTNpjFSfcKnVDbZiyIKmn/XIiAa1sOPVN/OqeXzM2PMG0mWT3wAuM28OZVepK7dScY2aMuQ+K934otrxUUZqZg8uvycnRALAlxsHl11S9z6OPXMffXPH/8v73/zWWZXH0uqN56yVv5aMf/QgjIyP86t5f8rX/vJ7bf3RXPU5BKYHaMGWhMhv2a+2RR/M3V3yAv/rrK7Esi3XrjuGSS97KNR/9sNqvOtHULZkUpdGkE2S9VjXVwsXnv5WLz39r5rHBcM8dW5i249ikECym7AnCVlQL2CqKUhVzZb8Afn3HVqZNnKSZBmDSHiUsMe07XAX6jilKCUY7L6rZkJVCEKeQbSDmdhmIZwpAhiVK2IpphW5FUSpmLuwXzLRQtEkxbcdJmDgJM+V0IZCYtlesABVmitJk5Bq4SaZNnOlUnKCEMUYTbRVFaV4sAkStViK0kDBxpuxJJsyIm6MWa/T05gUaI1EWHcY2OClDzY1j4JwuAxFpIWUSTKbG2TZ6BwMJXRCoKIuR+WK/0lGA9sASYlY7IEzaY0wmR9kZf1IXOxVBhZmy6Nj3fD9Tqcl5YdxgZsl60ETZ+/xBRlMDPDB2O38YvV0FmqIsMuab/QIISYRWq5OAHeaFZ3fy1ORv2Tp8Ky/GH1eB5oGGMpVFx//58j287SPQe0QPYhVdjNc0GNuw7/l+bv9fD3Bm5zvYNfU0z8Uf4YGx2xs9NUVR5pD5aL9gxobd/ZXHWd+2kWfjD/H05P0qzDxQYaYsOsaH4nznk/NX0AQkyGHRl7MmcgwDyT08NHZno6ekKMocMd/tF0BPaBU9oVX0J/poDyxhe3xbo6fUVKgwU5R5SkCCLAsdUnqgB2OpKPeNrC1z9B+qOoaiKEoxekKVF7pNU74Nm3/2S3PMFEVRFEVRmgQVZoqiKIqiKE2CCjNFURRFUZQmQYWZoiiKoihKk6DCTFEURVEUpUlQYaYoiqIoitIkqDBTFEVRFEVpElSYKYqiKIqiNAkqzBRFURRFUZoEFWaKoiiKoihNQl2EmYjcJCL7ReRxn+c3iMiwiDzs3v6xHsdVFEWpFbVfiqI0E/Xqlfkd4OvAzUXGbDXGvKVOx1MURakX30Htl6IoTUJdPGbGmHuBgXrsS1EUZS5R+6UoSjMxlzlmrxWRR0TkDhE53m+QiLxPRLaJyLZpE5/D6SmKovii9ktRlDmhXqHMUjwEHGaMGRORjcBPgLVeA40xNwA3AHQGl5k5mp+iKIofar8URZkz5sRjZowZMcaMufc3AyERWToXx1YURakFtV+KoswlcyLMRGSFiIh7/xT3uP1zcWxFUWYfEQmIyB9F5Ofu4yNE5PciskNEfigiYXd7xH28w33+8Kx9fNzd/oyInJO1/Vx32w4RubYB56b2S1EWOM1kw+pVLuMHwO+AdSKyS0SuEJH3i8j73SGXAI+LyCPAV4FLjTHq5leUhcPfAU9lPf4S8G/GmKOAQeAKd/sVwKC7/d/ccYjIccClwPHAucB/uIYyAHwDOA84DninO7ZuqP1SFIUmsmF1yTEzxryzxPNfx1mOrijKAkNE1gBvBv4F+JDrXXoD8OfukO8CnwG+CVzo3gf4EfB1d/yFwC3GmCngeRHZAZzijtthjHnOPdYt7tgn6zV/tV+KsrhpNhs2V8n/iqI0ERPJEI8OrCp3+FIR2Zb1+AY3yT3N9cBHgXb3cQ8wZIxJuo93Aavd+6uBlwCMMUkRGXbHrwbuz9pn9mteytt+arkTVxRlYVKBDStlv6DJbJgKs/nEoSvrs5+de+qzH2WxcNAYs97rCRF5C7DfGPOgiGyY01kpiqKUxtd+QXPaMBVmjaZeYmu2jqkiTinOacAFbhmJKNAB/DvQJSJB94pzDbDbHb8bOATYJSJBoBMnkT69PU32a/y2K4qi1ErT2TBtYj4XHLrS/9bszNd5K3OCMebjxpg1xpjDcRJff2WMeRfwa5ykeYDLgJ+69ze5j3Gf/5WbSL8JuNRd8XQETp2wPwAPAGvdFVJh9xib5uDUFEVZBDSjDVOPWb1ZLKLF7zzVw6Y4fAy4RUQ+B/wR+Ja7/VvA99zE2AEcI4Ux5gkRuRUnITYJXGWMSQGIyAeAO4EAcJMx5ok5PRNFURYjDbNhKsxqZbEIsXLJfz9UqC0ajDFbgC3u/eeYWZGUPSYOvM3n9f+Csyoqf/tmYHMdp6ooilJAs9gwFWaVokKsMrLfLxVpiqIoilIUFWbloGKsPqg3TVEURVGKosLMjwaLsclD2ksPqiOxl0bn9HiAetMURVEUJQ8VZtnMoRiba+FVimLzmRPRln7vVaApiqIoixgVZjDrgqzZRFil+M1/VgSbetEURVGURcziFmazJMjmuxArl/zzrLtQUy+aoiiKsshYnMKszoJsroTY2IrSv67+Hevo23Y6ibF2Qm2jrFp/Hz1HPZN5vm1vssirayP7fairSFOBpiiLgr6pU9gev5i43UPU6mdt9DZWRf7Q6GkpypyyuIRZHQXZbIixcoRXMfp3rGPn1rMwqRAAibEOdm49CyAjzso5Rj3E26yINBVoirJg6Zs6hScm3oNNBIC4vZQnJt4DoOJMWVQsDmHWZIKsVgGWJt87lkoEM6IsjUmF6Nt2eo7XrNr5VSvY6i7SVKApyrwn3zuWtMMZUZbGJsL2+MUqzJRFxcIWZnUSZLWKsXoJsWy8vGNgPMcmxurj3cs/j2qEWl1Fmgo0RZmXeHnH/OxX3O6Zw5kpSuNZmMKswYKsHkIs2xsW7Bhl+Yb76Hz5jNdr962nF3jHQDz3FewYZXK593Fi+4sf2ytPLU32edYi0uoi0FScKUpTUSxfbHv84gLvmJ/9ilr9dT22ojQ7C0+Y1UGUVSPIahVj2cJp+PF17LnvLEzSEV7JkQ72bHZyxdLiLDlS7hwNbUc9W/S4w4+vY/+W00mOtBOITZKKh8E455Odpwb4CrZaRFpdBJp6zxSlaSiVL1a+F8ywNPBoWcdLC7GQjJEwUSCUc+zBxJEcTJ2oYk1peqx67EREbhKR/SLyuM/zIiJfFZEdIvKoiLyqHsfN4dCVNYuyyUPaKxZlYyuCFYuyyeWFt2z2bzk9I8rSmGSI/VtOzzwOdpQrYoSxHUf6Pjv8+Dr2bD6L5EgHIKQmWzKiLHPsVIid929g59az3JCpZARb/451BftMvycVvy9VvP8FaPsspUKawn4tMLw8Yul8MajECyYcTJ1YdERaBDrhUCFh2kmLsuxj70q8PjMmLdb6pgp6VCtKw6mXx+w7wNeBm32ePw9Y695OBb7p/qwPdRBklVCNEPMj21sV7Bil7ahnfb1hyZF24r02AF0XbqX/lrMxifxwZvHXAUzdc2zmmE5eR2l9buIx8kMN5SwsSL9XlXjRavagqfesJIlkgL4DXY2eRrPwHRppv+YxXiFD8M8LS29fG70tx6NWDK99ZR+3XBuWb790YcH8ZiHbsLoIM2PMvSJyeJEhFwI3G2MMcL+IdInISmNM7d+cNYiy2RRkxcRYmrS3KjtkOfTQSfjlWgS6RzL329c/DcDgz08nNdgBlgHb2zhlv2502zH033FWlqDzPla5lLuwoFqBVnN4U8WZUoKG2q95jFe48rGJ97oWpXi+WHaumSOubCBQ9DV+x63FhunCAqUZmascs9XAS1mPd7nbajNscyTK6i3I0uzbelpByNLPyEgoQfdb7svZ1r7+6YxAG912DAe/fw7YeXMNJIkd9xw7P3NlSQHnhwQTWKGEE+bMI39hgddigmwqFWh18Z6pOFNqY3bs1zzHO4E/5LO2EiymMh41cMRZWqD1TZ3CYxPvJT8EKSRYGniU3wx/IeOVS5lIWZ62XAxetrWahQWKMts0XfK/iLwPeB9A1GrzH1ilKJsNQVaJGMsOKaYGO8p8laHn0rtoP+8R3xHt5z0C7Qn6bz4bMxYDwGqbpOU1TzB+7ysw02FnoF3O1aVBWiYxEzEC3SMZQZgfOpVgguUbcsVi+r2YDYGm4kxpdsq2XwuAyrxNhuNbbvYNG6a3PzXxDpI4NjokY/QGH6AvcVpZZTXyjxdklCTtRK1+lgYezdkPFApFRWkW5kqY7QYOyXq8xt1WgDHmBuAGgM7gssJPYBN5ycoVZNliLJtA9wipwc6Srw8sHYH2BDs/eBWpgx0Elo7Q/fZf0376kznj2k9/smDbzg9eNSPKysRqneSwz3/T87l06NSrhEd+vtzyDfexYnnxwraVCDQVZ0qDqJ/9WkBErX5XKJU3FsjxfOWvisz2oKX5zfAXyi6rkU1IxnhD14dztnVPPVuyhIaW2VCagbkSZpuAD4jILThJs8NznZ9RriirlyDzE2PghB0zuWEFLvbcxxKeJnbSn+i/cWNGYKUOdtJ/40aAAiGWT+pguV65meMtuewuWDYFB3INYnboNE0ciO6zPPPl9mw+CzY6JT7K8aCVK86gytCmijOlOhpuv5qJwsT7bPuVQACTFZK0mGJp4NGq2i35e+W8Q5Pp4x0Tu6Vgu5fwyz8vbQmlNAN1EWYi8gNgA7BURHYBn8ZNFjDG/CewGdgI7AAmgMurOtAshy/rIcqKCTJwk+8LVlM6F9aBpSPETvoTkw8fTaq/g0CP4xkbvPX1BV4vMx1m9EevY93Fvy16vL7lQ0zt7y58wrLBCNI6iQD2eCxzvIzYWzY1M/6Af05HvNdm3zcL8+XSJT46X/5MWSHOcsUZ1OA9U3Gm5DFn9msBUJh4D2n7lb0qM9/rVKx8RjHR4+eVCzJK0JombvcQZBQRIWHaavJyVTtHRak39VqV+c4SzxvgqpoO0mBRVqsgA2DZFIN3nO5R4kIILB3m0K9+w318N6uWDWWe/Z9vXui5u6kylgof+pd38Oy/XYI9NSPsrMg0R/79j1j+xocByltynBZpPgLNL18uv/TH5PLS4gxmObSp4kzJYk7s1wLBr2J/1DrI6zo/ntmSL2Qem7jSc3+l8tS8ympYTHFsyw/rLpZKlfhQlLmi6ZL/PZnvoizL8+QXWkz1d+SIsWwiy7y9XhGf8dmkxdfOm85j6kAXkWVDHPqXd2S2AwXHLSrUfLxofvlyXoVwS4kzqCy0WZfG6IqilKRa8eLn+Sq1KjK/rMZs5n1VO0dFqTfNL8yaSJTlJ7d3Xbi1IOcqh2wR4xJYOkLqYKGAyRZZ++85KUdIdZ36JAfuenWO1ysYneL0D/5vjl7SV3T+ALytj0ezhFgpsoVaSZHmirPut9xXuGozVLhqM029Q5sVo14zZZFRj8T2csVL/rGWBh6lz65uVWSp3LB64eed05WbylzT/MKsCmZLlOUnt/ffcjZAoTjzEGRput/+awa+tbEgtHjoX94BOKIsO/Q4tb+b/rvXc8z5W9h53ysZ29dDW28/p37gVo4+73dlnSfAiR4C7tGBVSVflxZpXgJt9L7jGLz19c5K0e4RWk95nMknX0ZqsCNTZiOy/mnY5183rZzQZilxpiFNRSlOvRLbyxEvXsfqs09jVeh/mqpXpZdQPb7lZl2VqTSc5hZm4dLthvKZrfClZ//KRIjBn5+eK8yKiLJVy4bg4t+yv2PCN7S45ztn5Yg2gGQ8ws77Xsm7b//7kudVCflirZhQyxdoo/cdl7tSdLCT8QeOp+cddxeu3HRDvVEfgVZOaLMUGtJUFH/qldheTmjR71gHUyfm5KE1Ej+henzLzU0zR2Xx0tzCbJaoJqfMr39larAjU1Xfr74Y5IYHl7/x4ZwcL5gRSf+zz7su0Ni+2U9ATc+hHIF2v89K0cE7TvcN78Z77arEmYY0FaU2iuWGFast5kWp0OJ8SKLXFZhKM7OghFk53rJqE/2DHaMkR3wS992kd6/6Yn4J/WnyvVZtvQcZ27usYFxbr5PDcXrH9qL7K8V9I2tLjilHoKX6/Rcx5NdAy67b5lWYthxmNaSpKAucYsVg09vrVbdrPiTRVyoetfCsMpdU1jRxgVNs9WXXhVuRUMLjmdwih2Y6zOCtrweKi7ITl/R55n2d+oFbCUZzw6Hh6CTv+chXaxZl4Ai79K0UfnME/xWhgR63Ybob0k3XbXPEq2QKzw4/vq7gtZW0tvKj0sb0QE3dJBRlPrA2ehsWXmkWufYr7TWq97GaLYneTyR6bU+HPR2xKRkB2zd1yizPUlmsLBhhVg9vmR/xXpv29U/Tc+ldTmHWEhQrfQHeifhpjj7vd7zuUzeydNUeEJulq/Zw5b98ntMuuKuaqRelEoGWz6F/eQdWZDp3YzCJHQ/x/J9/nJ0fvIrRZ45k8OeFddvShWe9KCbOqv39KcpiZ1XkDxzfcjOQKjm21pBj+lhR6yBgiFoHi/bJbARe4lFIkLTD3Dl4A78Z/kJGeBULeyrKbKDfdC7leGva1z/Nwe9tLDmuWH2xYqIMXLH0ju3wjptLT6hOpMVZsTDniUv6ckKb6Ry55//jQpIjLc7GZAAz5txPHezk4H94F8YF/5w9RVFmh1WRP/gWes2mHiHHuSpxUS3puT09eSkJ4zSbNwRJ4qRoxO2lPDZxZU5T9XyaKWdOWVgsCI/ZbHvLsgl0jxQdn136Ip9yRFkjKeVB85q/PRXCCYekb9l4bXMo9j7W6jXTcObcIiJREfmDiDwiIk+IyD+5248Qkd+LyA4R+aGIhN3tEffxDvf5w7P29XF3+zMick7W9nPdbTtE5No5P8kFQinR1Wwhx9kmZcIUs19poeZFM+XMKdXTjPZLPWZUltvkVUg1u9fly668vWDFJdRflJ0R21n22K2Th1a079M7tvt6z7I9ZztvOq+gtEc5SChB91u8C88q85Ip4A3GmDERCQH3icgdwIeAfzPG3CIi/wlcAXzT/TlojDlKRC4FvgS8Q0SOAy4FjgdWAb8UkaPdY3wDOAvYBTwgIpuMMYXLn5WieNUhy+912cyernri3V7KC6cte7ZwW2wCdoHTdPZrQXjMZguvVkvt65+m569uJ7B0GMQQWDrM0r/9KUv/9qcELZvtX3on2971cfbfc1LZx6lElJ0R21mRKKv2NcVIi8xyenXOYABDoHuYnkvvon390+X1F1WaHuMw5j4MuTcDvAH4kbv9u8BF7v0L3ce4z79RRMTdfosxZsoY8zxO0/BT3NsOY8xzxphp4BZ3rFIhXvlfJ7TcyAktNwJOT8vs/KqFTKWhyGbOmVOqpxntl3rMKmR02zEM3nG6U+3erVsG5BRbndrfzbP/dgng5GIV85ZVKsrKpWXfBJ3PjxGYsklFLIaPaOOM3p1le8+Kec3S+PXw9CLQPcKhn7mxrLHKHJC0fBvSe7BURLZlPb7BGHND9gARCQAPAkfhXB0+CwwZY9I1TnYBq937q4GXAIwxSREZBnrc7fdn7Tb7NS/lbT+13MkrM3iVfQDq0hWgnmxYv4PLz9/GsiVjHBho49s/W8+WbUfV9RjFSoh4jdXCs01G+TZs3tmvRSHM6rWaL136IR3GzNQtCycLiq3aU2F23nSeZ1izGioVZd1/GsksIA1O2XT/ycnpqkScleLQv7wjp32Ug3F/zrj9JTxd19DlrBacVbw4aIxZX2yAMSYFnCQiXcBtwDFzMTGlfPyq3VtMNVWx1Q3rd3D1O7cSjTgrSHt7xrj6nVsB6irOioV1NWy5oJh39mtRCLN64Vn6YToM096toyoL9dWPzufHCqp6WLazfaK3pW7HSYvO7PZS7Zf8BsDpodnfQaDH7Yaw7tm6HVdpXowxQyLya+C1QJeIBN2rzjXAbnfYbuAQYJeIBIFOoD9re5rs1/htV8rEr+yDjXeeaKNWHV5+/raMKEsTjaS4/PxtdRVmfu2lvLZp2HJx0Cz2S4VZBaQG/VfoeFGsbMZsEpjyzt3y214L+e2l0r00C9pSlR82U+YZIrIMSLhGLYaT5Pol4NfAJTg5FZcBP3Vfssl9/Dv3+V8ZY4yIbAL+t4h8BSd5di3wBxz3xVoROQLHoF0K/Plcnd9CofKcqsasOly2ZKyi7bXgV9ZDhdjioRnt16IXZv071rHr+xtITcYAkOgkK8/eAsC+b57m9MDsHqH7LfcR6B7JtF/KpbAkRLGyGbNNKmIR9BBhqYiu9WgW5iKHZg5ZCXzXzdOwgFuNMT8XkSeBW0Tkc8AfgW+5478FfE9EdgADOIYKY8wTInIr8CSQBK5yQwyIyAeAO4EAcJMx5om5O73mJTtnDGzAImr1szTwKAdTJ+Z4ffxzqjzsVwPDdwcG2ujtKRRhBwbaGjCbEsxGmZ150LtX7dfs2q9FIcza9iY988z6d6xj571nY+yZ50y8hb5N5yABg0k521ODnfTfcjatpzzO+B9eXhDOLMBKceTf/6hu+WXglLwoN89s+Ii2nBwzANtyttcrv6yRzPf8srnKoZkrjDGPAq/02P4czoqk/O1x4G0++/oX4F88tm8GNtc82QVEfs6YY/Od3LFd9utJC650Ltmq0P/QZ59WRomIVENXHX77Z+tzPh8A8akA3/5Z0TSh+tHomoaVHL8BIk7t1+zbr0XtQunbdnqOKJshkBFlaUwixOSTL6Pn0rsypTJmEkXzMFaOKCvWDLycpuJpyhVVE70tDB7dQTJiYYBkxGLw6A7u7Cg/n7HYvIqdj1KaYjk0ilIuxetwFfbAPJg6MadUhq/9wmpoKG/LtqO4/gdnsK+/DdvAvv42rv/BGfX90j90pf9tPtGAc1D7NfvUxWMmIucC/45zyXajMeaLec+/F/gyMwlvXzfGNLx2QmKssgrxqcEODn5vI4HuEZb+zU+dBPeDhaHNSnPL7htZW3bZjLQ4K+U9m+htyST6V+olq0QsZtPnt9jBJ78sum9xXhfMZQ6NUh7z0YZVmjMWt3t4bOJKolY/J7Tc6IZAC0ObzVDRfsu2o+onxOab2KoHXudcJ++a2q/Zp2Zh5sZly6lq+0NjzAdqPV49CbWNkhirJKHfuQpNDTplMlrPfITxe1+RUyrDL7fs0YFVReuZVSLOoHKxVS6lRFk9vWXDj69j392F+X0rlj9Tt2OUTZ2MVjm5F/Mqh2YRMF9tWCV1uBxKhzYXRGmIORRiT7y0kV0Dr8YJPtmsWfIAxx/SpBH3MsSa2q/moB4ui4ZX5Y69NFpyjFde0qr19yGWV75SCgkUz2My02EmHz6anis353QBWHLFZt/cslKiplpPVb2oRZRV6i2buudY9tx+NqnJFtJ96ky8hb6fnU3/jnWAkwP42C1X8NCNV/PYLVfQv2NdWfll5fw9zAbp3IvenjEsmcm92LB+R864b/9sPfGpQM62Oc2hUfJpuA2rhrXR27CY8nnWL0zp4BXanLcV7RsUinRE2ak4TlYBAuwaOJUnXtoIQN/ACfzmyQ9x5yP/xG+e/BB9AyfM2dzKJut9U/vVPNQjlJmpguviV9X2rSJyJvAn4O+NMS95jEFE3ge8DyAa8loBWT96jnI8M7t+t4HUVOGqzP1bTic5kg53Fq5cSvV30H76kwWlIfoOdLHKJ5xZjucM5raheTmCsJ6iDJz3Nj+PDwATpG/b6QDs3HoWJuUstEiMdbBz61lET0jRu+bpkvNtBOXWX0rfX0CrmuY7dbNhOfbLml0PQn4dLr9Vme7MCl4ft3t8y0U0NU0SmnQ8ZYWNz3cNnkLroQfY3ncutmu/4okunui7iOmeqK/9atQFZZrLL/6j2q8mYa5WZf4M+IExZkpE/hqnz9QbvAa6rRJuAOhsWVX8si+L2EujTB5SPGfMa3Vmz1HPZARafjPzzpc727d//QqSI4Uhz0DXiO+xahFnMDcCrVwPXVWirAjRfVaW4C0kMdbuLMxI5RXzTYV44ZkziwqzqoxbA3Iv6ppDo8wFZdmwHPsVXFa2/aqW4sLqBwD8ZvgLTZtLVjYNEGOlvk94xCfgZIQXnjkzI8rS2CXsV8njucyWgFvWMey9Xe3XnFMPYVas2i0AxphsC3Aj8K91OG7die0vFGcAyzfcx57NZ2GSMx80CSWcNkMHIrDMO5xQSpwBZQu0NLUItUpDpaVCr0VFWYmE/2DHqKfYhXTun7eRmpqsrMjvXKK5F/OWBWPDvPBqPTQvcsnmQIyVK4Y8EQOm0BOJGF87VQ/75TfnWgXbgZFOejsLxdmBkc6Z38U8qLG2EKiHMHuAElVtRWSlMSb9G70AeKoOxy2gWq9Z/4519G07ncRYO6G2UZa94b6MtwxmPGfp0Ga64Gz7evfKp0pxBuULtDRzkYc2G4IMcldhLt9wH3tuP7swnClJVq2/z/19FBqxSMzfS9noUIBX/aVE0iIaTrD5azeqy795aRobVi1ezcnTnjS/1kNNG8KcJUFWkwjzYMUhf2TvzleRG840rDjkjwweOIqpSY8V+0XsV614nV8lNvHb976eq8+9nWgokdkWT4S4f8dRfPevv8qyjmEOjHTy7Xtfz5anTlCRNovULMzc7uoFVW1F5LPANmPMJuCDInIBTjXcAeC9tR63FrLFWf+OdQW5TH2bzmPf3RvoPWtLRpR1vvyZzP14r0dro7Qg8RBoaSFTjkCD8kVaPuuHn+fCAw+zJDnBQLCFny47iW2dR5T12nJXW9ZDlMGM2M1elWlFJjnktVsyoeXs3wuAFUhw+Lp7y5pn2dTRuOTnXoyOR4hFpulsd/4m5nshxoXKfLRh2Xg1J39s4koGE0dyfJsTzmz6XLJDV7Lh2Me4/MxfFwqAKqm3EMvn6BN/BcDel17peM7E0HPMI6w67V4iO/YV2C8JJFhx6v8UOAZms2B2/ntQTKil3+vs38H9O47inBMfzYi13s5hrj73dmc87u9GBVrdEWNmPQ2iajpbVpnXHn1lxa8r5wOZ/nA8dssVviUzJJhg5ca7c7xnaTzFWRof71maYgItn3JF2vrh53nX3t8TMTPemvRv1k+kVVL6omQuWQWiLJvYfv9d9u9Yx97fn8bUZAeR2AiHr7u3vomzdTAoxZaXf/efbvEMbe7rb+OyT19a87HT3Dl444PGmIqWREUOPcSsvubqssY+/3fXVLx/xckxe23HRbN+HL8cMjCc0HJj0wsygA3HPlbgrbGN44vaX4FIq7cY8+oYUy75kZhV6+/LXHRWQ70FXOyl0ZJi+Lt//VXP8Oa+4U4u+68P5m6swZ7Opg2bj/aruVsyTSdKj/GgkpBmsSKzJhmi72fnAhSIs+g+y1+cFfGeQXketDR+4ilfsF144OEcUQYzDvae5ATv3PMHdo5188vYcSWPmT/PopRoTl6NKMs2aJHYCOtO+nn9E/7rQKnWJFqIUZkL/AvNCo9N/CXQhE2588KVl5/56xxRBmC5BizHS+MhzuohxmoRYF7UW5SB9xxrEWt/dvp2rj7ldqJBD2+Y+z77Lgjw2q55aHWjuYUZOL/kKnIOyhVnJYvMGos9m88CvMUZFPGelSnQoDIvGhQKtu7kRNHxMZK8f3xrSWFW9irLGgQZFBdl2SGAqclOtj/miGMvcVa1KKvReGxYv4OPvPs3BAK5Hufs5eW6GECZC4oXmg3wxMR7gCYRZz623E8ApImGElx+5q8zgqFWMVZvIZaNV3rMzq3Od0it4iyfasXaGw97iE+85ocErTz7lfc+F10Q4IcKtJppfmEGsyrOXnbUb9j+2LkFS5uzMckQ+7ec7hnShBLeMygp0KBQEFUq1PZbHaywiyeWLvd5vqKSFzUIsuHH12UWUPhdRXqVyfBbZt5IUXb1O7cWiLI0aY9Yw5sxK4uCtdHbeGziSrxqlYFTTHZ7/OLGCrMS9ttPAGSzrGO4akFWDyG2d/+M/Qp2jLJ8w32e3wm7b/Uu9bP7odNp+bPyhVmxNI9iZJ+rl0h742EP8dFTflwgytKk3+fYS6O+CwK+fe/rS0/k0JUqzqpkfgizGiglztJf9jueeCOpRAw/45YcafctpwFleM8gV9SUyEPzE0vZgu1Nk0/y/vGtLLdHGJEY0wjhIhW/++iqqu5YKTGWppQo2/PzszNN4xNjHby45Txe3HJejkgrt0xGo0QZeBeSzSbtEduy7SiOPWIfbzn9aSzLYNvCnfcfrYn/Sl1ZFfkDg4kj2ZV4PX72q9K+mnWjiCDLzm8anYwxnbQIB/3t5/6JrooOXa0Y87Lxw4+vyymZlBxxFon1bTqvQKT51WgsVrux3HlAZYLNa6HB+17xi0z40ov0+zx5SDt3jP0Zxz76Em955UNYYrCNcOejJ5a/KEO9Z1Uxf4RZlV4z8BZn+3YdwwvPnJlJLD/q+HsAeOaRN4MpFBihNkcIFBNnUKZAg4pEWjZpYXWBeZiPchctOB+wLjOJIbcRS7aJniDEdZxV9nHqIcbSxPbDn+7ckBFl+TPMVPUfShGJjZRcZt5IUQbFc8SyPWIb1u/gnNf8KeNZCwQM57zmTzz1fG/DxZmVWLxN5BcCXuUxukPPujllgYLxDSkmW0KUZXtiOlsmSaYE2wZxDZdkGbB4MsQNj5xb8pCVirFitjyb/VtOz6lj6eBMMDnSkZPu4lejMdhRn1xYrzmXK9bGVgRZ3jLk+3z++/zGwx7inFc8SsD1rgXEcM6Jj/JU3yGVrZidBe/ZQrZh8+usavjFZn+Z79t1DNsfO9cVAJKTx7TuFbdjBXKvJiSQYNX6+2b2tb/0ByG6z8rcSnIgknsrg2u4OyPKMvPMuk1jMUALNrCLLj7BRWySk+o2h3LPLf0+pVte+ZGu6n/4unsL3v/sMhmNFmXgnyOWSgnX/+CMnLIZfi1OFKVa0uUxnLwyyTQlBzih5aaC/plzXky2jJ6VXsn+wYDBshxBJuKsyjQG9o538a9/eCv3vPgqz32NrQhmbuUwuXzmVi6lvF0mGWLf1tOI99p0XbgVyTs3CSXounAr8V67rFulVHJOe+PdntuTthS8z17etXQeWsU0SSut+cD88ZilqYPnrFi7jFPfeANAxptWbEVNWnSU+jBkC5iyPnTFhJHrXVvFUNFdRLA5QJj18sny9lsG5V6dVJsbMTXZkQktZ3szD193L4ebB3K7GVZCna/U/HLHskUZVNaiSVHKZXv84pwq/jCTR/a6zo9nxjSkmGyZtrlUsj84qzL3jnfxjk2f8Hy+EiFWKfl2OtA9QmqweO/m1KDjJUsXHh/8+emkBjsKC5JXcfw05djg7PP1ssXfeG4jnzrmVmJZF8CTqRBf/n2h+PXzrpXz+/NEQ5tlMf+EGdQszkq1y+hd83ROonkpA5D9x193kZaPK676urtYExgqOnSVGYKD1YuxSt3ExQSZFZnEnmop+vp0uDL//a+pHMYsGIBym/jqqkxlNvDLF0tvb0gx2QrtcTnJ/uAtDOotyMqxw91vuY+D39uIXw4fQGDpSObCuf28R2g/75HiO63iQjl/rqVstJdI+8X+kwG46mWbWREdZG+8m288t5FfTJ0MK3IXDOyf6GJF61DBfvdPdGUWCFSFLgwoyvwUZlCTOIuGhoknugq2+7XLSP+hlmMQyvWigfeHqlyxdt34eXy+/Ue0iH8SZ5/dVda+is2nFOV6xw557RZe/M3ZYLzfw+xwZXb+XzQ0zNoVd7NqyWMVz202P/jlNPHVVZnKbOBXHqNhTcmrsMNeq/28yE74L8f+1lOMZdN+3iPE9yxn7Jfr8RJnEp6m++1OeG/0vuMYvPX1pA52EFg6Qvfbf0376U8W7rRUbnEZwi37PMoVabH9jjhLC7R8xlYEM995NzxyLh895cc54czsPDQVZ7PD/BVmMPNLrdAwrF1xN0/suhDbhDPbymn3U41Ag8pc6cU+XNkfwk3TJ8MoXNN6B6utIQwzBRkBJkyI68bPK3vflVBNqPKwtieIviKVEVyB0CQCJBOxnKr+6fy/dKg5nujiiV0XAlQmzuboA1+s8n+5njVFqYSmakpe5cXxHWN/xvQDUd73il+wvGWIkakWWkJxwoEZG5cWAPUSZGWLMR/BtOwv7yZ69O6M6MIyYEuO+Bq97zj6b9yImXa+W1IHO+m/cSOAtzirZB4lhFq5Ii1boJ27/MFCz9n+kzPveTq0mf497Z/o4oZHzs0Jeao4qz/zW5ilqdB7lv6C3773LOKJTqKhYQ47fmvR6vLZVCLQoFDIVJPzAIUftrt4NXfxagDe3PEgV/duZmVokD2Jbq7ft5G7Rk4mWt2hCqhGjOXX0MkPUXrx4hNnFOb/mTDb955VvjCbQ1FWrPJ/+qcKMaWeNE1T8ipFWXqF/D0vvirnC/6Nhz2UIwC+9sKbnfBasX3VS5CVuTK+/fQnMwLLq9bkth+9LiPK0pjpMKM/eh3rLv6t5z7LLmFUgVBLn3MxgfaGox7kU6tvJWY53rBVsUE+dcytwEy4c2xFsOD35EX6d1qVQDt0JQxW/rKFzMIQZlCVOMv5ojfASzN/YPnlNLz6NGYLj0qWaXuJnGrFWprbR07m9pHiRqxcqk3eh8pbhOSHLeMJ7wRbv+05zPGVV7FVlyrGlNmknDwyr5IadRNvNYoyL7IFQCl7Wspe1lOMFSv2vf+ek9h503lMHegismyIqf1dnuOmiogvv/2XFGzZ8/cRacUE2tW9mzOiLE0skOCql23OCXOmfxfl2PaavGdKhoUjzKDq0GY2sZdG6Rs4ge19M+G0Um2BoHqRljmujxiqVbBVerxqqLZf275dx7D9kXMyIWUn78+7QG4oULzlVCPc4brqUmlW0iU10uHO7JIaNYuzWRBlaWoNW5YUZGWIsXK7ruy/5ySe/bdLsKcc+zW1vxs/+xVsL2G/yphHUaGWPq8KBNrKkLebakXUe3t27lkxVJzVzsISZmlqFGjb957lW06jnHBn/h9vLe1A6img6kktzXPBEcAvPnlGTp6fg+AYt9wE20QqQt/ACYXhzAbmJ+iqS6VZKVZSoyZh1kBRVpOXrIQgq7QF3olL+vjedz6SEWUzeNuv1ESY6O8P4+jzfue5v/zex6Xm6CvSyhBoaXG2J9HN6nChCPOrcwblizOlNhamMEtT5cpNv7CZX5mNUtRTqHnhl8BZT+rxYcy/iiorPJkhmJtn1gQJo7rqUmlWSpXUqIpZEmVvPOwhrnzlnUXtV9VesjoIshOX9HluH9vn1zi+EJMM8fuvv91XmHkdo5hYS8+7GoGWfr+u37eRz2blmAFM2iH+7eBG3+NCeeJMvWa1sbCFGVTlPfMrpxENDWf+2Kptpgv+IqcawXbu8gdzigWuig3yz8d9n1d0PM+XdlxSt7lVS7EPp9/77Ec80dlQQea1AvP6H5yhqy6VpqPuJTVmUZR95NQfF7VfVYmyGgWZnxjLpq33IGN7l5Ucl2ZsX2WiOHsOfiKtLIGWJc4uCD/INa13sMoaos/u4rbBV/O69qdyFo3dPnIyLC8erVFxNrssfGGWpgKB5llOQ6ZZu+LuzOPsP7haRFo25YiifPF21cs251RwBqdsxiVrfssjI0dkrjzn0v1c7ofR7322JEHSbi0Y37A6TfivwLz+B2dw2acvbdi8FMWLZiipUY5dvPKVdxa1X7fh7fmv1ktWTJCVI8bSnN6xHfnIV7nxk59gOj7Tbi4cnSQcjTM2VBgOXLpyH6d3bC/Yft/I2pLHS8+tKoHmirMLwg/m1L5cExjioiUP8Oldb/dcODZZQpyVg4qz6lg8wixNGQLNq5xGsSKn+X949RJqXuQLLL9ETUvg/z38diLDqaI1aOpFNR8+v/cZKBRsjarT5KIrMJX5RF1LalThLSs3p6yY/frbtZu5bfvJBaWAvhw/16nj6IWPKKtVkHkJqtMuuAuAW//X33JwTy9LV+7j7R/+DwBPwZZ+rtS+iwm1cgSanzi7xtxR0F+5RRL83arbfVf0FxNnmm82e9RFmInIucC/AwHgRmPMF/OejwA3AycD/cA7jDEv1OPYVVNCoBWU06iAuRRqfi0zwGlnkl21eUWr8/j4pS9w2uqnqxZr9bwC8n2f+4caX6cpC12BubCZlzasBHVpzTSLogycRPNVMW9xtjI0yJs7HszJg1odHuRfQ7fyD+YndMskfXYX142f5wi1CkVZtYIsm9MuuCsj0PLJF2x+4/yOWa1A8xNnfv2VV1lDOYsCKkFDmrNDzcJMRALAN4CzgF3AAyKyyRiTXeb4CmDQGHOUiFwKfAl4R63HLpei9Xyyc5ZqKLNRDK8/ynqJtRseOZdPvfaWnKr/aWwjOa00AKLBBBevvT8zPi3WAE9xNqcfqKzfxarInoYKsXxKrcAs1gFAaW6a3YbNaj2yBvON5zbyz8d939N+7Ul0e9baCkuKHpkEnHDc59t/BJJkEycV7KMaUVZKjHlxRmxn7uN33Mi177jRc+zWyUPL2me5Aq1ccdZHF2s8xFm6dZ9XkfLbR06uS0hTqYx6eMxOAXYYY54DEJFbgAuBbKN2IfAZ9/6PgK+LiBhjvIu+1JGK6vnMgUhLU0rwlCvc7nnxVRy/9IUcsQVOO5NIwLsPXb4RjAYT/PXxd/Db+0rnOswKTbDCshT3P34I55/5VO577K7ALKcDgNLUNK0Nm9V6ZKWYZW8ZOBXmX9HxPJes+W3OZ2vSDnH9vo18ac33S+6vRRJcw90Fwmw2RVm+EKuE/NeWEmqlBFo54uwC8zAtTBcU8ki37rsg/CD/1DOTf7Y6PMhnVztdAOpVuFwpn3o0T1wNvJT1eJe7zXOMMSYJDAOeS1RE5H0isk1Etk2beM2TK1bPpyg798zcGkDspdGyb//9k7P4159fxL7hTmwD+4Y7uf6ON7N/pPxyFMs6hmfxbDxo8PtbCRvW72Djac/kfHEYA7aBj162hY+8+ze++WeLARE5RER+LSJPisgTIvJ37vYlInK3iGx3f3a720VEvioiO0TkURF5Vda+LnPHbxeRy7K2nywij7mv+aqIePhYqqZuNqxp7Nc84ks7LuEfnnwXfZPd2AZ2T3fzj7udhPQ9Cf+aWtnkh+kqFWWnd2wvKcrOiO3M3OpJufssNj+/81q1bIgLzMN8np+whImMKDPApG0RNyG+0v4Drmu/JSPK0sSsBFf3bgaKr4wtqyjwLKbz1Eoz2q+mS/43xtwA3ADQGVxW89VosXo+vxn+QnnhgTn0pFXLlqdOYMtTJxRsv/rc24mGZj5wtin0mAEcqEDEVUXWe+iEZj44b0Izf3PJ7wgFc1eBiUBL1BVjAe8/00WUf5YEPmyMeUhE2oEHReRu4L3APcaYL4rItcC1wMeA84C17u1U4JvAqSKyBPg0sB7nu+NBN6Q46I75K+D3wGbgXOCOOTzHsmhK+zVH1FJE9hf7T/ZcgXn9vo3805pbC0RDPn10Ze5XI8pKkS+cbrvtYv71ix+nr281q1bt5qPXfoGLL65tcVL6GMU8aKd3bC9rFWc213B3QdK/AFGxiYnTkcDy6Vjg1x1ggdF09qsewmw3cEjW4zXuNq8xu0QkCHTiJNDOOn71fIDM9orCA/keniYTahuOfYzLz/w1yzqGOTDSyZ2PnshrjtqReXz/jqM458RHc8RaPBHi2/e+vv6T8fCGNTQ0UyUdreX11MtnsXQAMMbsAfa490dF5CkcD9OFwAZ32HeBLTiG7ULgZjcMeL+IdInISnfs3caYAQDXOJ4rIluADmPM/e72m4GLqJ8wa1obVnf7VS4NsmteeU6fGL0kU3tryLTQKnEiMnOhNEGI6zgLqC3R3wsvT9Ztt13MtR+9jsnJFgB27z6Eaz96HUDN4ix9zHLz0LLxDWn6JP2X43Mu12M5n2lG+1WPUOYDwFoROUJEwsClwKa8MZuAtFvvEuBXc5FfBk49H4v8L9bClhlVhweyQ3INDsttOPYxrj73dno7h7EEejuHOefER/n2va9n45f/gcv+64M81XcIU4kgxjjhuOGJGNf/4s2e3raKyH8ffN6LxRCaAee9vf/xQ0oPbBBW0knoLecGLE2H59zb+/z2KyKHA6/EuTLsdY0ewF6g173vFzostn2Xx/Z60bQ2bNbtVxORXoG5OjyIJbl5TmcOfoqj+q/js2MXMmYiGJx3YYAWPsFFbJKTqjpmNYn+//rFj2dEWZrJyRb+9Ysfr2oOlVLpnPdb1XWsmbYtrt9XvAtAIynXhjEP7VfNHjNjTFJEPgDcibPU/CZjzBMi8llgmzFmE/At4HsisgMYwDF8c4JXPZ9ZaVeSxkuQzNHV5+Vn/jrHEwYQDSW4/Mxfs+WpEzLCLXtMJFRFHZoaBOisvvezxMhYhM72yrxmIvCal7/EN//PLE1qbjlojCnZZ0pE2oAfA1cbY0ay0yiMMUZE5uRirFKa2YbNuf2aRYrlIk0ux3MFZsxKcE3rHWyaPrmgQCpAlOIhTqivtwygr8/7O9Vve7XHrsZr5sV/tp7BP47eXrEXZtyOLpTE/3lnv+qSY2aM2YwTN83e9o9Z9+PA2+pxrGrIr+fj5GbUsV1JKYoJmTqKNr8E/vT2coRbJgw6SyUf6t4qZg745o9fy4fe9RvCoZnPpTGlQwGLKMcMEQnhGLXvG2P+r7t5n4isNMbscV396UX3fqHD3cyEDtLbt7jb13iMrxvNbMMabr/mCL98plXWEADXtN5RkGvWwsyKzAvMw1x78E6W2yPstzr4z9Yz+GXsON/jVeMtA1i1aje7dxd6w1etquufZN34Zew4XjZ6gL/gDznirJQN6wxMzPrcmoVms1/1CGXOO7zCAw2rLO8VAix2K4JfAv+BEafHZDHhtmH5Vq4++2czYVC35MOG9TtqPsVsmuq9L5Mt247iK99/Hfv625xVr/1tbLr32MzjVMrbui2WHDN3hdG3gKeMMV/Jeio7/HcZ8NOs7e9xVze9Bhh2QwZ3AmeLSLe7Aups4E73uREReY17rPdk7WvR0YyfoXqsuvPLZ0rX2UoLtHxW4aw8/AK3scIewQJW2CN8bPRO3jT5pOdrauGj136BWCxXtMRiE3z02i/U/Vj14jNyIR/ibeyiCxvYRRffm3wtu1Jd+AXkF0N+GTSn/Wq6VZlzQV3blcw1RcTZt297ZU49LXBrbd32SsC/SKqxhb+55Hdz0nJovr73W7YdVfA+pMOU+XXMYKbG2SLhNODdwGMi8rC77RPAF4FbReQK4EXg7e5zm4GNwA5gArgcwBgzICL/jJPzBfDZdCIt8LfAd4AYTtJs063InCvm62eoFNfv25hT5R9m6myBI9DWBIYKXmcj/CO3EyM3LSNGkvePb+WznF/XeaYT/Ou9KjObeoUxs9kkJ+XWepuI8JkJPEPE6Tpyi4Sms1+LUphBndqVNBlp4eBXgf7bP1vPh//i3oLSD4GA8V15OBvhuIX23pd63xc6xpj7yM9Gn+GNHuMNcJXPvm4CbvLYvg14eQ3TXFAstM8QzBQy9euLed34eQUCAiCIoRvvsNtye8T3ePeNrK06nHnxxbfVVYhVQqXlMjx7Z2axafpkGCWz8jW76v9ioBnt16IVZgsVL89ONn4LyfxyDRZLOK4cirVdKvW+K4pSmttHTs4RBPHemYvITdMn86rJF3h37HcF9srvW3W/1cGjA6uqWgCwdfLQuheTLfe41eDX2DzNBeZhruFuVjFEH11cFz4vI3o3TZ/MpumTq+qXWQ7aK7MyFmWO2WLl8vO35SSw55Ov2RZZOK4o6XBlb8/YrObgKcp8ZLa+ePOFwhsiT/leROZbtkmC/GfrGUX3X8r7NBshxVqOV623LF39fw1DWMAanB6jF4QfLGs/xXpllmpirlSOeswWEaXCkganNER769SiC8flk+8di0YSc5KDpygLlba9Sd+SGbH9xdv+pPFbAACO/RqkhS4mClZlVus1g7nxnJUjAIuJsnxv2Zsmn+T941tZbkboo4sWpguq/7fITCkSKBTBSuNQYbaI8Ev+T2MJxKdCvOPad2eEyUcv27LoRJpXU3K/lUuLqSSGojQavwUA4IR/JghzwfKrMsLkH0dvnxFpA8d5irO04CmWb5YtnOol0irxxlUqyj42emdmMcQahnwaLhUXumlq9ZZpGLNyVJgtIr79s/UFqwfzWbZkzFOYXP3OrQDzRpwVywcrxeXnbyt4jzQHT1lU7Nwz522Z/Lxm0X1WJtfMbwFAmlUMsX7/C3yMGWGSLp0B+IozKH8xQL6gKleoVSLEDjt4kJN27aJlepqBYAvxZUG2dR6RM8Yvp+z941sLVqj65eClS5H4ecuKiTJl9lBhtojIXj24fMmYp9g4MNDmKUzKCdvVIobqSa3C0s8Lll+QUXPwFGWG2EujJeuZFQtnFiMtztIrCK9rv4WgRyH2Prq4hrt9S2f8Mnacb1hz/fDznLPjcZYkJ5gIh3l4zRpeXOrdpzSbeuehHXbwIOuff5GIcexXT3KCd+39PUBGnBVL9PdbiZpvv9KlSKoNYaq3bPZQYbbISK8eLFZ766OXbfF8bbGwXb29bNWIvPRrvERnJflgfiHfkbEI8elQw4WnoixUysk1S4uzfM9Zupn5V/Dug5YtWPLF2frh53nX3t9nxFDr9DTrn38RoCxxlk+2x6tckXffyFrWDz/Pa/Y8TyAv+BgxKS488DA3pU7zfX060b+PLtZ4NC4fMDEm7QirrCH67C6uGz+Pu156te/+NITZOFSYLVKK1d66/PxtnsKkWNiuWi+bF9WIPC+hmU+5+WBeId/4VIBv/vi1KsSUxUMV4cx6eM3KCWnm197qs7u4LnAWm+QkrjF3ewqT/Gbeaa/TiUv6uPDAwxlRliZiUhyzcy/fD782s62cUOdhBw9y6gsvELSdubZOT3PqCy8AuSIvP28sLQ7zRVma7qR/i6TsWmXXcRaf5yc5yf4TJsQ/j12USfSH4sn+KsoaiwqzRYxf7S0/YZIftsv2avnlMFSTHF+NyPN6TT7l5oMt9oKxijLb1BrShJnaWzksm/IWJoQKSmdkVi7uH/G1X0vyxFA55SrO2fl4RpSlCdp2gcjLx0scZpMvLMG7eOwmOQkMXJO6O8c7NleiTKkdFWZKAeUIk3I8VFBdcryfmCsm8koJwFL5YPmh0/sfL2xSrCiLjlnymkH15TPSoiK7+GyGAxE2cSosJbeYKmexaewkGINVy4YKVi76MRBsKXke+eSLuVLby3k+vyZbviDLKR6bcoTYdZyX8She0+p0AKo2dAnlizL1ltWOCjPFEz9v2ob1O/ibt/6OjrYp35WKaapNji/W03PD+h2e8/J9jYH9JTxeXqHTC86cKWQ5H1elKkojqZc4g+ICzVOcAZsOnsqmZScVbL/APMw/7P85S5j09ZKlmSTI12Kvzwl5lsNAsIUeD5FlI6wffj5ndWV2Ev8+q4MVHon7SYQvtZ/DzWN/Bh7Xn+nisWkP4ZrAEF9q/yGCEJZUZtsX2n5EqMPybLXUUFFWpP/zYkWFmVI2G9bv4EPv+k3J7gEGagr/+ZX1CAQMH7tsCx98x1ai0VTOMfzCr9f/4IyScyinPIYWk1UWLVWWzqiHOIPavGcALHP6AF9gHuZL/JgI3mIOHNtloKBALZRueZTma7HXe3rjAhgu3/Nb3rHnARIE6CCec5z/bD2j4HUThPgEFznePh+uSd1NSyC3fEhECs8xZiW4undzgTBTUdZ8qDBTyqZUSydwvFOXffrSmo6TFj8fefdvCARyjycCLTH/RQHV5IWVmwenxWSVRUsTizMo7j1LC7Rruu8mEvAXZQC76eJM+Qirlg6VmrIvaTH3qdHNBPMS+QVoJQGudyunxlrsOIZGWwpDsHKS94Hc81rVU/5cV4YGM/fLqVGm4cvGoMJMKZta87gqYcu2o3zLdmST7cmqtpF4qY4I2eMWClbCaCKvUhlNIM6gSu8Zpavcp8ttgHdSPTj5aeXwy9hx/OPo7WWNjZHkytH/4eaxP2OTnMQmTvIfnPYCZlGsG0I+exLdzSHI6uApW8g2TJtjKWVTTJikUlJW2LBex8umVk/Wt3+2nvhUoOgYLSarKFT9hVruF3hZpRj2F/f2RPdZmVs26Sr3XiQRJ2To551K7+NAV/k3/I+XzyqP8h6AI8Sybx5cN34ek3ao5DEmUyH+Y/vGomPa9ibnhShb6NQkzERkiYjcLSLb3Z/dPuNSIvKwe9tUyzGVxvHtn61nOlGYMptIWnz5e6+rqyjbsH4H0UjCt0dlNrV6srZsO4rrf3AGqZR3OvBsiE6lOVAbVgVzIM7qIdCAHIF23fh5TJnCr7xpE+CakUvZdPDUkiKoEn7F0UWy2XLpo6tQhJWYQ/rc7nrp1fzj7reT9Dg3cPJ++ya7+dzTb+cX+wsT/9NUIshUlM0utXrMrgXuMcasBe5xH3sxaYw5yb1dUOMxlQaQLicRDBps203yNzA8FuF//X9n+q7g/O4/3cLmr93Id//pFjas31H2sa5+51Y6Z3HlZz5bth2FWN4qUCyjomzhojasGmoQZ40QaG+d/CMfif6CEDYpM2O/BuwWPjr69sJ6aMAFw49zb/LL7LA/yb3JL3PB8ONli6YLzMNcwh/L+oKdMCGuGzmvjJH+3sDbR07G8pGBBjj//n/wFWVz4iUDFWUVUGuO2YXABvf+d4EtwMdq3KfSZBTULJPSKx5radHkVyzWGBgZj4CB9rapuhd+9cs1W0i5ZUoBasOqpYZG5+XmncGMJ6dUQVq/HLQ3dzzIZ1ffSsyaWbk4aYf4x91vn1mhmJeXdkH4wZyWT2sCQ3y+/UcwyoyIKyLOrukuXCkJjg0bNC0YDN0y6Vn8NZtK+ljuSXSzOjxYsH1v3NMJXFF+Vs3J/SrKKqJWYdZrjEm/43uBXp9xURHZBiSBLxpjfuK3QxF5H/A+gKilX4jNQL0q8ZfbCH25XxNx4B3XvrusOVfTa7PcjgfNxETKu2GxUjZ1tWGLzn7VKM6AWRNo4Ii0q3s354gyKCwdkS+APrL2Fzl9OAFaJME1rXf4iihwBN01rXew2mehgQHWD3y2YLuXAHtzx4NcvXYzK0OD7El0c/2+jUVrkP0HG/nUMbcSyxKEk6kQ33huJq+s0mT52RZkcXuckNQeNl5olBRmIvJLYIXHU5/MfmCMMSLilxF0mDFmt4i8DPiViDxmjHnWa6Ax5gbgBoDO4LIyMoyU2aaelfjLaYTuF74s13NVrbduPrViGkjsYUf8IYaTZSyxWuTMpQ1blParBnEGlXnPoHyBBo5oWXlcoRcJcktHlPvcKmvI14v15o4H+eySHxWIwGz2JLrL8oLle/lWhwf57OpbAfjVDm9hmA5VXvWyzayIDrI33s03ntvIL/afPPeCDIqKsil7gufij7Br6mmOjp1S+7EWGCX/so0xb/J7TkT2ichKY8weEVkJeH5LGGN2uz+fE5EtwCsBT2GmNB/VhPiqeU2xfpfTCSEaSbD5azeWFEy1NFSvtuTGXGCMYSC5h2fjf2QwuYewxDg69mqenry/0VNratSGzQHpL+E58p5BrvenmEjbG+9mVaz8EB/4hwX3JPxf4+WZy8Y2sCo0yN1r/9nX+1VsXzErwd8v3ewrzMARZ2mBln5/2kq0nco5xiwLsrg9zvPxR9k19TQGm1XhtSwPHcrTk7+r/bgLiFqT/zcBl7n3LwN+mj9ARLpFHF+liCwFTgOerPG4yhziVU6iVIivmtf4edOMARGhs20KS2Y8YH6LCarx1jUzxhj2T+/kD6M/Y9vYZiZSw6yLvYYzO9/BYdGXN3p68x21YfWkxlyi9OKASgVCOoHdyzP0jec2MpnKLSeRDvGlFw7k367ft7GgBMWkHeL6ff7lJvy8bMY4oswSp0B22vv15o4HM2Pyj++3rxVRfy8fFH8filHTSss0O/f4/v7HU8M8Mb6Ve4d/yEtTT7IyfCSnd7yNl7eeSSxQvhhfLNSaY/ZF4FYRuQJ4EXg7gIisB95vjLkSOBb4LxGxcYTgF40xatTmEdWE+Kp5jZ+XzbaFUDA3ObeYB2yhJPHbJsWe6Wd5If4YY/YgUauNY2N/xurI0QREa0PXCbVh9aZG71maarxoUJhH9Qv8Q3x+/GrHyYRHCl/zq/0nE/N5zd6XeXvmUsYiaOXar1Ler3K9fLUUWK1btf4iYnw4eYAX4o+yN/ECFhZrIus4InKiirES1GTdjTH9wBs9tm8DrnTv/xY4oZbjKI2nmhBfpa/xS76PhL3Dm34esPmYxJ/NtB1n19TT7Jx6kikzQVugmxNaXseK8JFYojWh64nasFmkzgINKhdp4IiX+/a+gvsefQVQXm4a5IYFy+Ebz3kn30d9wpvFvF9e+4onQ9z4x3NqrnY/24LMGJv9iZ28OPU4g8m9BCXMEZETOCz6ciJWS32OvcDRy26lafDzsl1+/raKPGDzKYk/m9HUADvjT9I3vR2bFD3BVbw8eiY9wdVIqYJuitKs1EmgQfVetGz8hE25gs0Pv+T7q162uaIct7SQ/PJwive94hcsbxli/0QXNzxyLve8+KqK51X3PpY+gixhT7F7+k/snHqSSXuUqNXGutiprImsIyjh+s5hgaPCTGkq/LxslXrAmjmJPxvbpNifeJGdU08ymNyLRYCV4aM4LHo87YEljZ6eotSPWRBoaWoRamnK9UQVE3B+XrZqvF/3vPiqqoQYzFJTcR9BNpLs56Wpp9gzvYMUSboCvRzdegrLQ4eph79KVJgpTc989YAVYyI1wq7pZ9g99QzTJk7MauPo2KtZHV5H2Io2enqKMnvUUaClmQ2h5kelocR6er/8mBUhlsZDkCVNgr3Tz7Fr6mmGUwfcC8ojOTRyHB3BpbM3l0WCCjNlXjBfPGDFSJkk+6ZfYPf0Mwwk9wDCstAhHBI5lqXB1YheXSqLiVkQaGnmUqiVQy3er3xmVYSl8RBjxhiGUwfYPfUMe6afI0WCVquLY2KvYVV4LSFLC8XWCxVmijKLGGMYSu1j99R29k0/R5IEMaudo6InszpyNFGrtdFTVJTGki0CZkGkgb+YabRgK8acCLBsfEKVk/YYe6Z20De9nXF7mABBesNHsCayjq5Ar+a/zgIqzBRlFhhLDbJn+ln2TO9g0h7LGLNV4bUsCa5UY6YoXsyBSMumXPFTTwE354KrGEUS+fclnmfP9LOudx+6gr0cHz2RFeEjNJl/llFhpih1YiI1wt7p59iTeJax1CAg9ARXcVT0ZJaHDycooZL7mI+IyE3AW4D9xpiXu9uWAD8EDgdeAN5ujBkUR5H+O7ARmADea4x5yH3NZcCn3N1+zhjzXXf7ycB3gBiwGfg7Y8ziaHe0mJljkVaMphJTteIjxpJmmv3TO9mbeI6DiV0YbFqsDo6MvopV4aNoCXTM8UTnjmazYSrMFKUGxlKD7J9+kb2J5xlN9QPQFVjOMbHXsiJ8xGKp2/Md4OvAzVnbrgXuMcZ8UUSudR9/DDgPWOveTgW+CZzqGsFPA+txej0/KCKbjDGD7pi/An6PY9TOBe6Yg/NSmoV8MdFgoTavKFIANmFPcSCxk32JFziY2IVNioi0cGjkOFaGj6QjsHSxePe/QxPZMBVmilIB6QTY/YkX2T/9AuP2MACdgeWsi51Cb/hlxKz51WGgVowx94rI4XmbLwQ2uPe/C2zBMWoXAje7V4v3i0iX26NyA3C3MWYAQETuBs51+1J2GGPud7ffDFyECrPFjQo1b8poiTVpj3EgsZP90y8ykNyDwSYiLayJHMOK8BGLMm+s2WyYCjNFKUHSTNOf2M2BxEscSLzEtJlEELqDKzkkchy94cPnXRK/NW1XEp5ZKiLbsh7fYIy5ocRreo0x6W+JvUCve3818FLWuF3utmLbd3lsV5QZvATJQhdrZfYlNcZmOHXQtV87M579FquTwyLH0xs+gs7AsnknxiqwYdXYL2igDVNhpih5GGMYTQ1wMLmLg4ldDCX3YjAEJczS4BqWhQ9lWfCQxbQ8/KAxpup+VsYYIyKaE6bMLcWEy3wRbVU2hZ+yJ+hP7HZt2G4SJg4IXcHlrI29muWhw2i1OuedGKuSmuwXzL0NU2GmKDju/YFEH/3J3fQndjNt4gC0BZZwWOQEloUOoSvYq5Wsy2efiKw0xuxx3fz73e27gUOyxq1xt+1mJmyQ3r7F3b7GY7yiVE+5gme2BFyVgsuPpEkwmNzLQKKPg8ndjKUGAAhLlKWh1SwLHUpPcLUWr66MhtkwFWbKomTSHmMwsYeB5F4Gkn1M2o5LPCxRekKr6Qmupie0et6FKJuITcBlwBfdnz/N2v4BEbkFJ3F22DV8dwKfF5F0A8GzgY8bYwZEZEREXoOTOPse4GtzeSLVYlAn4bynzgKqXiRNgqHkvowYG04dwGAQLLqCvayNvZqe4Go6Aj2LxSs2GzTMhjW1MBtJ9fPQ2J10B1fQFeylI7CUgDT1lJUmxBjDmD3oGjLHmMVtpyl6UMJ0B1dwWOR4loRW0WZ1qyGrEBH5Ac6V4lIR2YWzMumLwK0icgXwIvB2d/hmnGXmO3CWml8O4BqvfwYecMd9Np1EC/wtM0vN72CeJP6Ppvq5f+SnGfvVFewlYsUaPS1lHjJlTzCY3JcRY6OpfleICR2BpRwePZElwZV0B1fod2QVNJsNa+rfYEjCjKeGOZBw8ukEi45AD13BXjqDy+gMLCNmtesXqZLDlD3BcPIgw6kDDCf3MZw8QBKngXBYYnQHV3B45OV0B1fSHliifz81Yox5p89Tb/QYa4CrfPZzE3CTx/ZtwMtrmWMjCEsMEYsXp57ghanHAIhZHXQFl9EV6KUjuJSOQA+WBBo8U6WZSJkko6l+hpMHGEruZzh1IOPRtwjQGVzGEdET6Q6upCvYu2DrI84lzWbDmlqYxax2XttxEVP2JEPu1cJw6gAvTT3Fi1OPAxCSKB2BpXQGl9IRcAxd1GrTL9tFwpQ9wWhqgBFXiI2k+jPeMBDaA0tYGT6SzmAv3cFeFfLKnBG1Wjm1/Xxsk2I4dTBjw/oTfeyZfhZwLjbbAz10BnroCC6jI9BDW6BbxdoiwRFhA64QO8hI6iBjqYFMGDwiLXQFezk0cixdwRUq5BcJTS3M0kSsGL3hw+kNHw6AbWzGUgOuR+QAI6mDPB9/JPPHHJQI7YEltAeWZAxda6BLryzmMbZJMW4PM5YaZCTZz1hqgNHUAFNmIjOmxeqgK7CczshxdAaW0x7s0d+50nAsCdDtXhiAE1qPm3GGkwcYTu5nJHWQPYnneGn6acARa22BLteG9dAeWEJboNv1wOlFxXwk/TsfSw0ymhpw7FdygHF7KPO9FZIIHYEeDo+eSGdgGZ3BZZrjukiZF8IsH0ssJwwQXMohkWOB3CuPkeRBRlMD7J56hp0kM6+LWe2OSLO6aAt00RrootXqXExlD5qepEkwkRrOiLDxlPNzwh7OGDDBojXQxZLQKjoCPXQEemgP9OjvUZkXiAgxaSMWbmNF+AjA+eKesEcy9mskNUB/oo++6R2Z14UkUmi/Al1EpVUFW5NgG5u4Pca4PcRYaohx9zaWGsykUwBErTbaA90sDx/m2q+lxDTSo7jUJMxE5G3AZ4BjgVPcOKrXuHNxeksFgBuNMV+s5bheBCRIV3A5XcHl4H4/zxi7AcZTQ+6VymCmD1iakERptTppCXTQYnUQC7TTYnXQYrUTkqh+WOqIMYakmWbSHmXCHmHCHmUyNcK4PcJEajjHAwZCi9VOa6CL3vBhtAa6abcc76e685V60Cw2TERoDXTSGuhkRfhlme3T9mTGbo2lhhhLDbA38TzJ6anMGIuAa7s6Hbvl2q+Y1UHUatUSL3XGNikm7THHfqVGZmxZapgJezTnuyUsUVoDXayMHEVboJs2q5v2wBK9iFSKUqvH7HHg/wH+y2+AiASAbwBn4VS8fcDtH/VkjccuSbaxy8Y2NpP2qHM1Yw9nPDT9id30me05YwMEiVqtxKx2YlYbUauNiNVC1GolYrUStVoISni2T2XekDQJpuwJ4vY4U/Y4cePcj9ujTKbGiNtjOVeO4BivFquDntBqWqwO1xPQSavVqQJMmW2a2oaFrRg9llO6JY0xhmkzmWW/Rhi3hxhPDXIgsTNHGIAQlRZigbT9cu2WtLr3WzREmoVtbKbN5Iz9sieI22PE7XEmbcd+5V48usLYtVvLQ4fR4n7ntFpdWjdMqYqahJkx5img1If6FGCHMeY5d+wtOL2mZt2o+WGJ5SnYwAmJTtqjmSuhSXvM/UCOMpw4QMJMFbwmQJCw1UJEooStGBGJEbZihCRK2IoSkkjOLSjheWEI096thJkmYaZImLjz055i2sSZNpNM25NMuT+n7ckC0QVOSYqo1UYs0M6S0EpiVpsrdJ2rexW2SqOYjzZMRIhICxGrhSWsynnOGJu4mciyX6PEXRs2kNzLlD1eUF9NEMISI2LFCLu2y3kcJSSu/bJybdh8uGAyxmCTcm2XY7fSdmzaTJGw467tmmDKxB0bZiYL9iNYjv2y2lgaWpO5n46wqLBV6s1c5Jh59Y86dQ6OWxUBCTou50C35/Mpk3SvplyvkBlnyp5kyp5g2sSZSI0waPZ6CrhsgoQISpiAhAhKiICECEiQAEHnpwSxCGR+ilhYWIhYCBaS9c9BwDW4acNrY2OwMcb5aRsbmxQpk8ImScqkSJkkKRLYJknSJEmZBEkSJM00STNd/Bwk4orRFjoCPYRDzpdFRByPYtRqIWK1agK+Mt+ZNzZMxHLy16w2z+eNsZkykxn75diwCecCyxUmY4lBpsxknuctF4sAQQkTlLQdCxIglLFd+fbLEgshkGXD0rZrxoJl5uj+7/xz7JbBdv6ZFDYpbNd22Th2yzZJkiRImSRJkyDlXlAWO4cAQUJWlIjEiFltdAWWEbZaHLslLZnoSFjTWZQ5pqQwE5FfAis8nvqkMeanHttrQkTeB7wPnATJZiMgQV9vWza2sQu8TOkrt7QXyhE/CVLuLWHijlAySVIkM0ao3jiGMpARgZYECRIkKCEiVizH4OZ7+2aunKOau6LMC+bShjW7/RKxMmHMTpb5jjPGkCRBwo7PeJyy7Fgyy34lTYIUCabNJCk7QQpXNJkkKVJQ5w4IgmARJCBp4RfMXNhG3QvB9IVv0IoQ8rRjUb1gVJqWksLMGPOmGo/h11fK73g3ADcAdAaXzdueJpZYRCRGhNoqfRvjXjWSylw5GozrBTOebV8yvrTMlamFJZbjecPSqz9lUTGXNmyh2C8RIUSYUKD2NIO0p94Ye8aLj8nYtoJjp71pkrFerg2b8bgpykJmLkKZDwBrReQIHGN2KfDnc3DcBYGIuCGAAAU+f0VR5gK1YTVguWkYar8UpTxquvQQkYvdvlKvBW53m3giIqtEZDOAMSYJfAC4E3gKuNUY80Rt01YURakdtWGKojQbta7KvA24zWN7H06Tz/TjzTiNPxVFUZoGtWGKojQbGqxXFEVRFEVpElSYKYqiKIqiNAnzslemoig1Mp2AnXsaPQtFUZTqWMA2TD1miqIoiqIoTYIKM0VRFEVRlCZBhZmiKIqiKEqToMJMURRFURSlSVBhpiiKoiiK0iSoMFMURVEURWkSVJgpiqIoiqI0CSrMFEVRFEVRmgQVZoqiKIqiKE2CCjNFURRFUZQmQYWZoiiKoihKk6DCTFEURVEUpUlQYaYoiqIoitIkqDBTFEVRFEVpElSYKYqiKIqiNAkqzBRFURRFUZqEmoSZiLxNRJ4QEVtE1hcZ94KIPCYiD4vItlqOqShK8yEi54rIMyKyQ0SubfR8ykVtmKIozWa/gjW+/nHg/wH+q4yxrzfGHKzxeIqiNBkiEgC+AZwF7AIeEJFNxpgnGzuzslAbpiiLmGa0XzUJM2PMUwAiUp/ZKIoyHzkF2GGMeQ5ARG4BLgSaXpipDVOURU/T2a9aPWblYoC7RMQA/2WMucFvoIi8D3if+3DszsEbn5nluS0FFspV8EI5l4VyHjA353JYpS8YSR28887BG5eWOTyaF767Ie8zvBp4KevxLuDUSufU5JRlw9R+1YSeS/MxV+cxmzZs3tmvksJMRH4JrPB46pPGmJ+WeZzTjTG7RWQ5cLeIPG2MuddroPuG+Qq3eiMi24wxvrkl84mFci4L5Tygec/FGHNuo+cwV8ylDVP7VT16Ls1HM5/HQrZhJYWZMeZNtR7EGLPb/blfRG7DcR16CjNFUeYdu4FDsh6vcbc1BWrDFEUpQtPZr1kvlyEirSLSnr4PnI2TcKsoysLgAWCtiBwhImHgUmBTg+dUN9SGKcqCpunsV63lMi4WkV3Aa4HbReROd/sqEdnsDusF7hORR4A/ALcbY35Ry3HrzJyFHeaAhXIuC+U8YGGdiyfGmCTwAeBO4CngVmPME42dVXksABu2kP6+9Fyaj4VyHr40o/0SY0wjj68oiqIoiqK4aOV/RVEURVGUJkGFmaIoiqIoSpOgwozy27I0K83WTqJaROQmEdkvIvM+sVpEDhGRX4vIk+7f1t81ek7KwmS+2y9QG9ZsqP1qLCrMHNJtWebd8vesdhLnAccB7xSR4xo7q6r5DrBQatMkgQ8bY44DXgNcNY9/L0pzM2/tF6gNa1LUfjUQFWY4bVmMMbNdoXu2yLSTMMZMA+l2EvMOt2DnQKPnUQ+MMXuMMQ+590dxVvusbuyslIXIPLdfoDas6VD71VhUmM1/vNpJ6AeoiRCRw4FXAr9v8FQUpRlRG9bEqP2ae+aqV2bDqVNbFkWpCBFpA34MXG2MGWn0fJT5idovpRGo/WoMi0aY1aMtS5PSdO0kFAcRCeEYte8bY/5vo+ejzF8WsP0CtWFNidqvxqGhzPlP07WTUEBEBPgW8JQx5iuNno+iNDFqw5oMtV+NRYUZ/m1Z5gPN2E6iWkTkB8DvgHUisktErmj0nGrgNODdwBtE5GH3trHRk1IWHvPZfoHasCZF7VcD0ZZMiqIoiqIoTYJ6zBRFURRFUZoEFWaKoiiKoihNggozRVEURVGUJkGFmaIoiqIoSpOgwkxRFEVRFKVJUGGmKIqiKIrSJKgwUxRFURRFaRL+f/u8AZIarEb/AAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "p_c0_given_x, p_c1_given_x = compute_posterior(flat_plt_grid, p_c0, mu_c0, sigma_c0, p_c1, mu_c1, sigma_c1)\n", "p_c0_given_x = np.reshape(p_c0_given_x, plt_grid_shape)\n", "p_c1_given_x = np.reshape(p_c1_given_x, plt_grid_shape)\n", "\n", "plt.figure(figsize=(10, 4))\n", "plt.subplot(1, 2, 1)\n", "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=10)\n", "plt.colorbar()\n", "# plot decision boundary \n", "plt.contour(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=[0.0, 0.5], colors=[\"k\", \"k\"])\n", "\n", "plt.title(\"p($c_0$ | x)\")\n", "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n", "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n", "plt.legend([s0, s1], [\"c0\", \"c1\"])\n", "plt.xlim(-1.5, 2.5)\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], p_c1_given_x, levels=10)\n", "plt.colorbar()\n", "# plot decision boundary \n", "plt.contour(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=[0.0, 0.5], colors=[\"k\", \"k\"])\n", "plt.title(\"p($c_1$ | x)\")\n", "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n", "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n", "plt.legend([s0, s1], [\"c0\", \"c1\"])\n", "\n", "plt.xlim(-1.5, 2.5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The color indicates the posterior likelihood for the respective call and the black line indicates the decision boundary. \n", "We achieve a train accuracy of 87%.\n", "For such a simple task that is clearly not great, but it nicely illustrates a\n", "problem with generative approaches:\n", "They usually depend on quite a lot of assumptions.\n", "\n", "### 2.3) Wrong Assumptions? (1 Point):\n", "Which untrue assumption did we make?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "data is independent and identically distributed\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3) Stochastic and Batch Gradients\n", "\n", "In the recap sessions with Prof. Neumann we already saw (or will see) an implementation of a Discriminative Classifier using Logistic Regression. Here we are going to extend this to stochastic and batch gradient descent. \n", "\n", "We start by implementing a few helper functions for affine mappings, the sigmoid function, and the negative Bernoulli log-likelihood. - Those are the same as used for the full gradient case." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "def affine_features(x: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " implements affine feature function\n", " :param x: inputs, shape: [N x sample_dim]\n", " :return inputs with additional bias dimension, shape: [N x feature_dim]\n", " \"\"\"\n", " return np.concatenate([x, np.ones((x.shape[0], 1))], axis=-1)\n", "\n", "def quad_features(x: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " implements quadratic feature function\n", " :param x: inputs, shape: [N x sample_dim]\n", " :return squared features of x, shape: [N x feature_dim]\n", " \"\"\"\n", " sq = np.stack([x[:, 0] ** 2, x[:, 1]**2, x[:, 0] * x[:, 1]], axis=-1)\n", " return np.concatenate([sq, affine_features(x)], axis=-1)\n", "\n", "def cubic_features(x: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " implements cubic feature function\n", " :param x: inputs, shape: [N x sample_dim]\n", " :return cubic features of x, shape: [N x feature_dim]\n", " \"\"\"\n", " cubic = np.stack([x[:, 0]**3, x[:, 0]**2 * x[:, 1], x[:, 0] * x[:, 1]**2, x[:, 1]**3], axis=-1)\n", " return np.concatenate([cubic, quad_features(x)], axis=-1)\n", "\n", "def sigmoid(x: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " the sigmoid function\n", " :param x: inputs \n", " :return sigma(x)\n", " \"\"\"\n", " return 1 / (1 + np.exp(-x))\n", "\n", "def bernoulli_nll(predictions: np.ndarray, labels: np.ndarray, epsilon: float = 1e-12) -> np.ndarray:\n", " \"\"\"\n", " :param predictions: output of the classifier, shape: [N]\n", " :param labels: true labels of the samples, shape: [N]\n", " :param epsilon: small offset to avoid numerical instabilities (i.e log(0))\n", " :return negative log-likelihood of the labels given the predictions\n", " \"\"\"\n", " return - (labels * np.log(predictions + epsilon) + (1 - labels) * np.log(1 - predictions + epsilon))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are also using the same bernoulli objective and its gradient as before" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "def objective_bern(weights: np.ndarray, features: np.ndarray, labels: np.ndarray) -> float:\n", " \"\"\"\n", " bernoulli log-likelihood objective \n", " :param weights: current weights to evaluate, shape: [feature_dim]\n", " :param features: train samples, shape: [N x feature_dim]\n", " :param labels: class labels corresponding to train samples, shape: [N]\n", " :return average negative log-likelihood \n", " \"\"\"\n", " predictions = sigmoid(features @ weights)\n", " return np.mean(bernoulli_nll(predictions, labels))\n", "\n", "def d_objective_bern(weights: np.ndarray, features: np.ndarray, labels: np.ndarray) -> np.ndarray:\n", " \"\"\"\n", " gradient of the bernoulli log-likelihood objective\n", " :param weights: current weights to evaluate, shape: [feature_dim]\n", " :param features: train samples, shape: [N x feature_dim]\n", " :param labels: class labels corresponding to train samples, shape [N]\n", " \"\"\"\n", " res = np.expand_dims(sigmoid(features @ weights) - labels, -1)\n", " grad = features.T @ res / res.shape[0]\n", " return np.squeeze(grad)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1) Implementation (3 Points)\n", "\n", "Finally, we can implement our batch gradient descent optimizer. When setting the batch_size to 1 it will become a stochastic gradient descent optimizer.\n" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def minimize_with_sgd(features: np.ndarray, labels: np.ndarray, initial_weights: np.ndarray, schedule: Callable,\n", " num_iterations: int, batch_size: int):\n", " \"\"\"\n", " :param features: all samples, shape: [N x feature_dim] \n", " :param labels: all labels, shape: [N]\n", " :param initial_weights: initial weights of the classifier, shape: [feature_dim * K]\n", " :param schedule: learning rate schedule (a callable function returning the learning rate, given the iteration\n", " :param num_iterations: number of times to loop over the whole dataset\n", " :param batch_size: size of each batch, should be between 1 and size of data\n", " return \"argmin\", \"min\", logging info\n", " \"\"\"\n", "\n", " assert 1 <= batch_size <= features.shape[0]\n", " # This is a somewhat simplifying assumption but for the exercise its ok\n", " assert features.shape[0] % batch_size == 0, \"Batch Size does not evenly divide number of samples\"\n", " batches_per_iter = int(features.shape[0] / batch_size)\n", "\n", " # setup\n", " weights = np.zeros([batches_per_iter * num_iterations + 1, initial_weights.shape[0]])\n", " loss = np.zeros(batches_per_iter * num_iterations + 1)\n", " weights[0] = initial_weights\n", " loss[0]= objective_bern(weights[0], features, labels)\n", "\n", " for i in range(num_iterations):\n", " #--------------------------------------------------\n", " # shuffle data\n", " idx = np.random.permutation(len(labels))\n", " features, labels = features[idx,:], labels[idx]\n", " #--------------------------------------------------\n", " for j in range(batches_per_iter):\n", " global_idx = i * batches_per_iter + j\n", "\n", " #--------------------------------------------------\n", " # TODO: do stochastic gradient descent update!\n", " #--------------------------------------------------\n", " start = j * batch_size\n", " stop = start + batch_size\n", " batch_features = features[start:stop,:]\n", " batch_labels = labels[start:stop]\n", " gradient = d_objective_bern(weights[global_idx+1],batch_features, batch_labels)\n", " diff = -schedule(i)*gradient\n", " weights += diff\n", "\n", " # log loss (on all samples, usually you should not use all samples to evaluate after each stochastic\n", " # update step)\n", " loss[global_idx + 1] = objective_bern(weights[global_idx + 1], features, labels)\n", " return weights[-1], loss[-1], (weights, loss)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The loss curve is expected to look a bit jerky due to the stochastic nature of stochastic gradient descent.\n", "If it goes down asymptotically its fine." ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Final loss 0.048020512453803184\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA6KElEQVR4nO3dd3zV9fX48de52ZCEFfbeSwUBWSLiFhVxVcVV6/5Vq62tFlcdbdVqtXW24vg6i3UVQUTBgQsBQZmy94aEkT3v+f3x+dybm5Dc3ITc3OTmPB+P+8i9n/v53HveuXBP3ltUFWOMMaYynkgHYIwxpn6zRGGMMSYoSxTGGGOCskRhjDEmKEsUxhhjgrJEYYwxJihLFMYYY4KyRGHqHRG5TEQWiUi2iOwSkVkiMiaC8WwWkTw3Ht/t2RCvnSsi14U7xlCIyNUi8m2k4zANT2ykAzAmkIjcDkwGbgI+BQqBM4GJwGFfciISq6rFdRDaBFX9rLZftA7jN6bGrEZh6g0RaQY8BNysqh+oao6qFqnqDFW9wz3nARF5T0TeFJFM4GoR6SAi00Vkv4isF5HrA15zuFs7yRSRPSLypHs80X2NDBE5KCI/iEjbGsR8tYh8KyJ/F5EDIrJJRMa7z/0VOAF4NrAWIiIqIjeLyDpgnXvsejf2/W5ZOgS8h4rIrSKyUUTSReRxEfGISLx7/tEB57YRkVwRaV3Ncox2fweH3J+jy5Vxo4hkueW73D3eS0S+cq9JF5H/Vvf3ZxoIVbWb3erFDafmUAzEBjnnAaAIOA/nD50k4GvgeSARGAzsA052z/8euNK9nwyMdO/fCMwAmgAxwFAgtZL33AycWslzV7vxXO++zv8DdgLiPj8XuK7cNQrMAVq68Z8MpANDgATgGeDrcud/6Z7fBVjre0233H8LOPc2YEaQWL+t4HhL4ABwJU4rwyT3cSugKZAJ9HXPbQ8MdO9PBe5xP4dEYEyk/w3ZLTw3q1GY+qQVkK5VN8V8r6rTVNULpAHHA39U1XxVXQK8BFzlnlsE9BKRNFXNVtX5AcdbAb1UtURVF6tqZpD3nObWPHy36wOe26KqL6pqCfAazpdpVbWTR1R1v6rmAZcDr6jqj6paANwFjBKRbgHn/809fyvwT5wvc9z3myQi4j6+Enijivcu72xgnaq+oarFqjoVWA1McJ/3AkeJSJKq7lLVle7xIqAr0MH93Vv/R5SyRGHqkwwgTUSq6jvbFnC/A7BfVbMCjm0BOrr3rwX6AKvdJpVz3ONv4PSBvC0iO0XkMRGJC/Ke56lq84DbiwHP7fbdUdVc925yNcuwJeA1snF+Fx0rOX+Lew2qugDIBcaJSD+gFzC9ivcur8z7B7xHR1XNAS7B6TPaJSIz3fcBuBMQYKGIrBSRa6r5vqaBsERh6pPvgQKcZqVgApc83gm0FJGUgGNdgB0AqrpOVScBbYC/Ae+JSFN1+j4eVNUBwGjgHEprIbWpsuWZy5ehq++BiDTFqe3sCDinc8D9Lu41Pq8BV+DUJt5T1fxqxljm/QPew/c7/FRVT8OpKa0GXnSP71bV61W1A05T3vMi0qua720aAEsUpt5Q1UPAn4DnROQ8EWkiInEiMl5EHqvkmm3APOARt4P6GJxaxJsAInKFiLR2m6kOupd5ReQkETlaRGJw2uCLcJpYatseoEcV50wFfiUig0UkAXgYWKCqmwPOuUNEWohIZ5x+iMCO4zeB83GSxetVvJe4vyf/DfgY6CPOsORYEbkEGAB8JCJtRWSim7wKgGzc35OI/EJEOrmvewAn+YXjd2gizBKFqVdU9QngduBenE7pbcAtwLQgl00CuuH8Zfw/4H4tHcp6JrBSRLKBp4BL3X6BdsB7OEliFfAVwdv2Z0jZeRT/C7FITwEXuSOinq7oBDfW+4D3gV1AT+DScqd9CCwGlgAzgZcDrt8G/IjzRf1NFfGMBvLK3Q7h1Kh+j9PkdSdwjqqm43xH3I7zu90PnIjTYQ9wHLDA/d1OB25T1Y1VvL9pgHwjM4wx9ZSIKNBbVdcHOecVYKeq3lt3kZnGwibcGdPAuaOjLgCOjXAoJkpZ05MxDZiI/BlYATyuqpsiHY+JTtb0ZIwxJiirURhjjAkqKvso0tLStFu3bjW6Nicnh6ZNm9ZuQPWclblxaGxlbmzlhSMr8+LFi9NVtcI1wqIyUXTr1o1FixbV6Nq5c+cybty42g2onrMyNw6NrcyNrbxwZGUWkfKz8/2s6ckYY0xQliiMMcYEZYnCGGNMUJYojDHGBFXvO7Pdxciex9kSc66qvhXhkIwxplGJSI1CRF4Rkb0isqLc8TNFZI27JeRk9/AFOEsnXw+cW+fBGmNMIxeppqdXcVb19HOXe34OGI+zxPEkERkAdKJ005aSOozRGGMMEWp6UtWvy23zCDAcWO9bplhE3gYmAttxksUSgiQ2EbkBuAGgbdu2zJ07t9pxeVX5fEMOJd4vifFI1RdEiezs7Br9vhoyK3P0a2zlhfCVuT71UXSk7HaP24ERwNPAsyJyNjCjsotVdQowBWDYsGFak0kn7yzaxlvrl9GxV2d+Pa7xbNRlE5Mah8ZW5sZWXghfmetToqiQu2fvr+rivXILigHYfai6O0kaY0z0qk/DY3dQdl/gTpTdMzjsEuJiACgost0cjTHGpz4lih+A3iLSXUTicbaCnF6XAcTHOL+OwhJLFMYY4xOp4bFTge+BviKyXUSuVdVinL2RP8XZw/gdVV1Zl3F53N+G7dFhjDGlIjXqaVIlxz8GPq7jcPwEZ6ST1/KEMcb41aempyMmIhNEZMqhQ4dqeL3z0/KEMcaUiqpEoaozVPWGZs2a1eh6j/hqFJYqjDHGJ6oSxZHy1ygsURhjjJ8ligC+GoXlCWOMKWWJIsCPWw4AUGK92cYY42eJIsBL326KdAjGGFPvWKIwxhgTlCWKCvj6KowxxkRZojjSeRRdWjZxfrZqUpthGWNMgxZVieJI51G0SUmo5YiMMabhi6pEcaR8E+28NurJGGP8LFEEKHHzg+UJY4wpVemigCJyQbALVfWD2g8nsnwzsm0JD2OMKRVs9dgJ7s82wGjgC/fxScA8IOoShW+inS3hYYwxpSpNFKr6KwARmQ0MUNVd7uP2wKt1El0d81rTkzHGHCaUPorOviTh2gN0CVM8EeXrxLamJ2OMKRXKxkWfi8inwFT38SXAZ+ELKXL8o54sTxhjjF+ViUJVbxGR84Gx7qEpqvq/8IZVMyIyAZjQq1evGl1fotZHYYwx5YU6PHYeTmf258B34QvnyBzphLt/XzEUsKYnY4wJVGWiEJGLgYXARcDFwAIRuSjcgUVCn7YptEwUa3oyxpgAofRR3AMcp6p7AUSkNU4fxXvhDCxSBKtRGGNMoFCanjy+JOHKCPG6BskjgOUJY4zxC6VG8UkFo54+Dl9IkSUCuYUlZOYXkZoYF+lwjDEm4qqsGajqHcALwDHubYqq/jHcgUWKAJ+s3M0xD8yOdCjGGFMvhFKjAGekUxFOo8zC8IUTeSXW7GSMMWXYqKdybG87Y4wpy0Y9GWOMCSqqRj0d6VaoxhhjDhfKF/4nIvKpiFwtIlcDM6mno56OdGa2McaYw4Wy1tMdInIhcLx7qN6u9WSMMab2hTTqSVXfB94PcyzGGGPqoVBGPV0gIutE5JCIZIpIlohk1kVwxhhjIi+UGsVjwARVXRXuYIwxxtQ/oXRm77EkYYwxjVelNQoRucC9u0hE/gtMAwp8z6vqB+ENLTL25ZVOzVZVRGwKnjGmcQvW9DQh4H4ucHrAYwWiMlEE8irEWJ4wxjRylSYKVf1VXQZSH3lVibFFPYwxjVywpqc7VfUxEXmGCnZoUNVbwxpZPWAbGBljTPCmJ18H9qK6CKQ+sjxhjDHBm55muD9fq7twjoyITAAm9OrVq1Zez2oUxhgTvOlpBkE2BVXVc8MS0RFwk9uMYcOGXV8br+e1PGGMMUGbnv5eZ1HUU1ajMMaY4E1PX/nui0gS0EVV19RJVPWEeiMdgTHGRF4oaz1NAJYAn7iPB4vI9DDHVS9YjcIYY0JbwuMBYDhwEEBVlwDdwxZRhPVqXvorsURhjDGhJYoiVS2/ZVzUfoNOHp7IvWf3B6wz2xhjILTVY1eKyGVAjIj0Bm4F5oU3rMiJ9QhJ8TGAs9aTMcY0dqHUKH4DDMRZEPA/QCZwWziDijSPuxCg1SiMMSa0RDFJVe9R1ePc2z3Ag+EOLJI87vJO1kdhjDGhNT1dKCL5qvoWgIg8CySFN6zIEn+NwhKFMcaElCiA6SLiBc4EDqrqteENK7J8TU+WJ4wxJvgSHi0DHl6Hs3HRd8CDItJSVfeHObaIsaYnY4wpFaxGsRhnGKwE/DzbvSnQI+zRRYh1ZhtjTKlgS3g0uEl1tbV6rFiNwhhj/II1PZ2sql8E7J1dRn3cM7u2Vo8t7aOwRGGMMcGank4EvqDs3tk+Ub1ntjU9GWNMqWBNT/e7Pw/bO1tELgxnUJFmndnGGFMqlAl3FflHrUZRz/jnUdgy48YYU+NEIbUaRT1jNQpjjClV00QR1d+gNuHOGGNKBRv1tJyKE4IAbcMWUT3gcdOn1SiMMSb4qKdz6iyKeibGzRRFJdZJYYwxlTY9qeqW8jfg6ID7USs10cmfF/37e9KzC4Keu2TbQb5dl14XYRljTERUt4/iobBEUc+kJsX576/ZnRX03POe+44rXl4Q7pCMMSZiqpsoonq0k0+zgESRV1gSwUiMMSbyqpsobgxLFPVMiybx/vt5RZYojDGNW5X7UZRf60lEOgGHgOWqujdcgUVSjKe04mSJwhjT2IVSo7gWeAm43L29CPwR+E5ErgxjbBE1rm9rAH7emcnezPwIR2OMMZETSqKIBfqr6oWqeiEwAGd+xQichBGVnpl0LACvztvM8Ic/r/J8r60gaIyJUqEkis6quifg8V732H6gKDxhRV5iXEy1zp+7Nipb4YwxJqQ9s+eKyEfAu+7ji9xjTYGD4Qos0uJiqtfPX1hsk/OMMdEplERxM3ABMMZ9/Brwvjq7+pwUrsDqA5HQ13sqKrGmJ2NMdKoyUaiqisi3QCFO38RCradbv9XWVqg+gaXsNnkmbVISePLiwYzpnXbYucW2JrkxJkpV2b4iIhcDC3GanC4GFojIReEOrCZUdYaq3tCsWbNaeb0/nN6nzOO9WQWVzsK2GoUxJlqF0vR0D3Ccb86EiLQGPgPeC2dg9cHNJ/Vi7Z5spi/dWeW5xZYojDFRKpQeW0+5iXUZIV7X4IkI+7KcRQEfPHcg5w3uQJeWTSo89+7/LSffJucZY6JQKF/4n4jIpyJytYhcDcwEZoU3rPrj/nMHcHyvVlw0tBPxsR627s9l3npntdjyXTX3TVsRiRCNMSasqkwUqnoH8AJwjHuboqp3hjuw+qJfu1Teum4kTRNiyXUXCLzsJaefwjfH7uaTegKw42BeRGI0xphwCqWPAlX9APjA91hEtqpql7BFVU91aJ7kv7/jYB5tUhIASIqLYVDn5sRWc+6FMcY0BDX9ZmsUy42Xd/tpfbj0uM4AzFuf7t8qVURoEhdDXmFxJMMzxpiwqGmiaJRDfBLjYrj5JGeOxh3vLfPPs/CI0CQ+hpwC68w2xkSfSpueROT2yp4CksMTTv3XqYXT/NQmJYEl2w4CkJlfRHysh593ZVJY7CU+1pqgjDHRI9g3Wkolt2TgqfCHVj+JCGcMbMv+nEIunTIfgHV7smmV7Gx2tHzHoUiGZ4wxta7SGoWqPliXgTQkCbExFAcsK96nbTKThnfhzflbWbsni6FdW0QwOmOMqV0hjXoyZbVsGl/m8TVjutOiSTxxMcKWjNwIRWWMMeFhjek1cMcZfZl6/Uj/47TkBGI8QucWTdi6PyeCkRljTO2zRFEDTRNiGdWzFQBpyaW1i84tm/Dx8t2s3p0ZqdCMMabWBRv1dIWqvlnZ6CdVfTJ8YTUMs383tsxS5GN6pfHV2n2c+c9veO+mUQzr1jJywRljTC0JVqNo6v6sbPRTo9enbQp925X+Kq4Z091//88f/RyJkIwxptYFG/X0gvvTRj+FKMYjLHvgdI55YDYDOtTOnhjGGBNpwZqeng52oareWvvhNHypiXG0b5ZISQ12vCss9jJz+U7GH9WexLiYMERnjDHVF2x47OI6iyLKxMV4KCyufqJ44asNPDFnLXExHs45pkMYIjPGmOoL1vT0Wl0GEk08AtOW7OTYLi345ehuIV/3xJy1AGRkF4YpMmOMqb5gTU8zCLL4n6qeG5aIokC/dqlszsjl/ukr6dsuhZE9WlXrelsryhhTnwRrevp7nUURZZ657FjW7M7inGe+Zd3e7GonCk+jXMTdGFNfBWt6+qouA4kmcTEeuqU5o4s37cshu6CY5ITQV0tplhQXrtCMMabaqmzjEJFNIrKx/K0ugmvIktxRS698t4lTnph72P7aFfHtmBfCqcYYU2dCaQwfBhzn3k4AngbeDGdQ0SDGIzx+0TEA7MksYPuBqvfT9uUHryUKY0w9UmWiUNWMgNsOVf0ncHb4Q2v4fjGsM1OuHArACY99yUvfhFYR08a5gaAxpp6qsuFcRIYEPPTg1DDq5fLkIjIBmNCrV69Ih+I3tk9r4mKEohLlLzNXkRDr4cpR3YJeYzUKY0x9EkrT0xMBt0eAocDF4QyqplR1hqre0KxZ/Vk+IzEuhnV/Pcv/+L4PV5JfFHxv7cD+jB+3HqDb5Jl0mzyTAzk2v8IYU/eqrBmo6kl1EUi0G9a1BYu2HABgS0ZumcUEfXz5wRuQKN5fvN1//+ddmRzfK+2IY1FVRGwMrjEmNKGMemomIk+KyCL39oSI1J8/2RuI164ZzkMTBwLw67cWl6k1eL3KV2v34evODhz1VBLQDlVSC21Sm9Jz6H7Xxwz40ydH/FrGmMYhlKanV4AsnOami4FM4P/CGVQ0apoQy5UjuwKwYZ/zZb37UD4AL36zkV++spB0d+kObxgTxQ+b9gOQWxi8+csYY3xC6ZTuqaoXBjx+UESWhCmeqCYi3H1WPx7+eDUAIx/5nPOP7cj/ftpR5rzApqeSgPvFtZAokhPr5TgEY0w9FkqNIk9ExvgeiMjxQNWTAkyFbhjbk0X3nup/XD5JQNnO7MqaoWoqKb50+fJQJgEaY0woieIm4DkR2SwiW4Bn3WOmhtKSE7hrfL9Kn5++dCeDH5rN1ozcWm96CuzCLiyp/lLoxpjGJ5QJd0tVdRBwDHC0qh6rqkvDH1p0u/HEnmx8+CxO7tfmsOe+W5/Bwdwivlq7t1zT05F/sQemmprsmWGMaXxCmXCXAFwIdANifcMqVfWhsEbWCHg8witXH8ewv8zxd2QHuu/DlQDEx3goLPHy2CdrGNu7NS2axtf4PQObmyxRGGNCEUrT04fARKAYyAm4mVryxrUj6Bcwr+Kkvq3LPN/STQw7Dubx5vwtR/Regd0SBZYojDEhCGUITCdVPTPskTRi/dunMuu2E5i/cT/d0pqw+1A+X67Z53++ZdN4Pv3dWAY9OJuPV+zmN6f0rvF7BXZzWI3CGBOKUGoU80Tk6LBH0siJCKN6tqJ9sySO7dKCpycdy/BuLQFnJVrfHhWrdmUe0WilwGtzCouPLGhjTKMQSqIYAywWkTUiskxElovIsnAH1tidO6gDWQXOF/meTGdi3jXHdwfgpW82AbA3M585P++p1usG1iju+d+KWojUGBPtgiYKcXqubwJ6A6cDE4Bz3J8mzCYMag/A3qwCAO45uz8Aj8xaBcDp//ya619fREZ2QcivGVijyMgJ/TpjTOMVtI9CVVVEnlNVa3qKgP93Yk8y84o5fWBbwGmC6ts2hTV7sug2eab/vMz8YlolJ4T0moGNVtv21968yYO5hcTFeGhajS1fjTENQyhNTz+KyHFhj8QcRkSYPL4fQ7q08B+788y+h5136ZTvKSguu3ZTVn5RhX0ZvuVBOjRLBCAzv+iI41y1K5PBD81h4P2fHvFrGWPqn1ASxQhgvohssD6KyOvaqulhx/ZkFtD33k8oLPbi9SrTl+7k6Adm83/fbT7sXF/uOG2AU0s5lHvkieK1eYe/jzEmeoSSKM4AegAnY30UEdc9rSmXj+jif3xq/7b++wdzC7ngX/O4depPgLPpUXm+GkWbVKdGcd5z35FTVP1RVMUlXpZuOwhAz9bJFZ6TXWCjqoyJBqEs4bEF6Ayc7N7PDeU6Ex4xHuGv5x/NeYM7ADC2Txr/vGQwAFe9spAl7pc3OPMzABZszOBQbhEHcwt5dJazcm2Ku4psRk4hC3ZV/wv9yTlrmfjcd6zceajMare5hcXkFZZwwmNfcNT9n7LjoK0faUxDF8oSHvfj7JPdF2cfijjgTeD48IZmgnni4sHccWY/2qcmsj/XWf5j9e4sAP5xySB+99+lPP7pGh7/dA0AZw5sR9OEWHa5e2CkBCw3HjhkNiO7gClfb+SOM/oSG1P53wMrdmYCzoiswOsH/KlsP8W+rAI6Nk+qeUGNMREXSs3gfOBc3GU7VHUncPg+nqZOxXiEjs2T8HiEtOQEerR2+i6uHdOdiYM6HnZ+y+R42qaWjozyTeAr788f/cwLX2/kRXeuRmU8vmVoFZTKm65s9rcxDV8oYxkL3WGyCiAih/emmoib+ZsT+GHzfsb2aV3h8y2axNGpRRP/42ZJpQsL5peUftGv35cNwN8+Wc2IHi3LjLgK5MsTXlV/B/nav4wnPtbDQzN+5tV5m/AqfLx8F8O7tzyCkhljIi2UGsU7IvIC0FxErgc+A14Mb1imupLiYypMEucN7kB8rIfiEqU4YP+JTi2SmDTc6RR/b23pyKfuaaUd0xc8P481bnNWeR7/KsLOnt8A7iH+NGEA8yafAsBPFXSoG2MallA6s/8OvAe8j9NP8SdVfSbcgZna8fvT+5LgLlPu20r1+cuH0DY1kUcuKJ1HmV/kzMM4oXcaAKe7w2cP5B6+/DmUJgWvljY8+ZIHQLtmifRtm0I7d76GMabhCmkararOAeaISBqQEd6QTG2Y/buxpGcX0LllE+JiPczfuN8/r8KXDAB+MbQT7y7eTn5RCYlxMf7awTmDOjD75z0Ul1Tc/+Dfl4TSIbdS7hyPR7BN9Ixp+CqtUYjISBGZKyIfiMixIrICWAHsERFbdrye69M2hdE9nYRw4ZCOrNqV6X8u1lP6sR/r9kHkFznf6L4d9RJinXOKKtlVz9+XraWjpqRcpojxUGborDGmYQrW9PQs8DAwFfgCuE5V2wFjgUfqIDZTS+4+qz/jj2oHQJP4GBLjSj/2pHjnfp7b9OT70o93E0XlNQrfPQVVREprGT4eEUsUxkSBYE1Psao6G0BEHlLV+QCqurr8F4Kp30SEf10xlAM5hTRvElfmCz0xNgaAC57/jucuG+JvevLVKKYt2eFf7iOQrz/C69YoKvoX4RGhxGuJwpiGLliNIrDNofz0Wvvf3wC1aBp/2F/9ifFOojiQW8QVLy/wf7EnxTnHZy7bxTnPfMPxj37BG/O3+BcaDOzM9qqW6cj2ifFYjcKYaBCsRjFIRDJx/lhMcu/jPrahLFEitdwM7cXucNYerZNJS44nPbuQFTucj/6+aSs4pV8bOjRPKu3MVuevhgoThdUojIkKldYoVDVGVVNVNUVVY937vscVT+s1Dc7gzi34f4MSGNKlOeDUIABiPcK8yacw9fqRZc73DaPd6+6656tRVNT25PFAJX3hxpgGxBb3a+RiPMKI9rG8e9NorhrV1X88LsZDfKyHUT1bcf0J3f3Hi9zO7R82l06kUw1Y0iOAdWYbEx0sURjASRgPTTyKDQ+fxeJ7T/WPegK45+wBvHjVMACKSrw8/fk6/3NeVQ7kFPqH1wbKKyph0ZYDfPDj9vAXwBgTNpYoTBkxHqlwW9W4GKfKUFDs5ck5a/3H9+cU8e7iihPB2N7OkiK3v7M0DJEaY+pKSIlCRLqKyKnu/SQRsdVjG5km8U6n94X/mlfm+Atfbaj0mt+d1od+7VLonmbrSBrTkFWZKNyFAN8DXnAPdQKmhTEmUw8N6dK8zNIfPknu8NrKdGnZxD8no7q8NmLKmHohlLWebgaGAwsAVHWdiLQJa1Sm3omN8fDGtSMAWLhpPwM6pDL0z3PYkpELUOGkPHBmeNdkT4o/fbiCqQu3MveOk2zjI2MiLJQ/9QpU1b+EqIjEYhPuGrXh3VuSnBDLCb1LlzV/aOLACs+Nj/WwMT2HbpNnsjPEbVFnLN3J699voahEWbun4mXOjTF1J5RE8ZWI3I0z6e404F1gRnjDMg3BQxMHMrJHSy4Y0pF2qRXPwRzZvZX//prdWRSXeFmx41DQ1w0cJZVbUFI7wRpjaiyURDEZ2AcsB24EPgbuDWdQpmHo0DyJt28YxZMXDz5saRCfi4/rzOzfjQXggRkreeHrjZzzzLfc/s6SMuf987O1zPl5D+AMq23V1NmBL6egOHwFMMaEJJQ+ivOA11XVdrUzNdK7TTIJsR4Kirz833fOXtwf/LiDWct306F5Iv3bp/KROyN886NnU1yitGuWSEZOIXe+v4yXvt3If64fSVoFw3aNMeEXSo1iArBWRN4QkXPcPoo6IyI9RORlEXmvLt/X1B4RYdLwLuzOzCc92+nuGt6tJXlFJWzYl+NPEkd1TAWcSX3Nm8QR4073Xrsnm+XbgzdXGWPCp8ovfVX9lYjEAeOBScBzIjJHVa+r6loReQU4B9irqkcFHD8TeAqIAV5S1UeDvP9G4FpLFA3bpcM7A85w2TOPakeH5kn8uPUAh3KLOLFPa8555lvapSYyf2MG2w7kMaRLC966bgRLth3k0VmrybYmKGMiJtStUItEZBbOaKcknOaoKhMF8CrOBkiv+w6ISAzwHHAasB34QUSm4ySN8hsiXaOqe0OJ0dRv/dql8sC5ZUdGDXF31wNndNRnq/by2Srn446LEUb2aEXH5kk8Oms1BTUYYmuMqR2iVSzaJiLjgUuAccBc4B1gtqqG9CeeiHQDPvLVKERkFPCAqp7hPr4LQFWD7ponIu+p6kVBnr8BuAGgbdu2Q99+++1QwjtMdnY2ycnJNbq2oaoPZZ61qYj/rvGPwmZsp1iuOSqBg/lefjvXGVZ7/dHxjO4QW2nHeXUcSZm9qizbV8LGQ14u6B1/xLHUlfrwOdelxlZeOLIyn3TSSYtVdVhFz4VSo7gK+C9wo6oW1CiCsjoC2wIebwdGVHayiLQC/gocKyJ3VZZQVHUKMAVg2LBhOm7cuBoFN3fuXGp6bUNVH8o8bhw84lVOeOxLdhzMY1j/Howb15us/CKYOxuAF5cX8uJyJ5n8/rQ+dGnVhPFHtS+zgGGojqTM4x7/ks0Zzn+FmyeMom+7hrGiTX34nOtSYysvhK/MVf4PU9VJqjqtlpJEtalqhqrepKo9q6p1mIbN4xFm/fYErhrVlYmDOwCQkhjHoxcc7R8u6/PEnLXc9vYSjv/bF/69MerCwdxCNruz0QF++cpCW2rERL1KaxQi8q2qjhGRLMrOxBZAVTW1hu+5A+gc8LiTe8wYUhPjeGjiUWWOXTq8C5cO7wKAqrL9QB65hSX8Z8EWXvt+C6Me/YL1fx1fK01SVXnPXSn3mUnH8uis1ew4mEdeUQlNE+p0MKAxdSrYDndj3J8pAbvb+Xa4q2mSAPgB6C0i3UUkHrgUmH4Er2caERGhc8sm9G2Xwn3nDACgxKuc93zpqrYzlu5k8vvLQl4ypDp+2naQGI9w2oC2/g2drKPdRLtQVo99I5RjlVw7Ffge6Csi20XkWrcT/BbgU2AV8I6qrqxe2MY4CxXee3Z/AJZuO8iTc9by27d/4jdTf+LtH7Zx7rPf1vp75hWW0L99ColxMSTEOSvnFhTbMiMmuoVSXy4zptGdcDc0lBdX1UmVHP8YZykQY47IdSf0wKvKwx+vLrPzHkBWvjMwT1Wd7Vo9wvq9Wbw5fytjU2rWr5CZV0STOOe/zbo92QD8/dO1PHHxoCMohTH1W6U1ChG5y+2fOEZEMt1bFrAH+LDOIqwGEZkgIlMOHbJZvI3JxcM6c+lxnfnzxIH0bpNMu9REzj+2IwXFXr5dl073uz6mx90fk5VfxKlPfs2r8zaz+VD1m4uWbjvIoi0HGNS5GQCXHOd0tdVB14gxERWsj+IRVU0BHi/XP9FKVe+qwxhDpqozVPWGZs2aRToUU4eaN4nn0QuP4cpR3Zhz+4l8f9fJHNPJ+TdwxcsL/Of9ZupP/vtztxczacp83llUOlL7gx+3c9T9n/L695srfJ9l2w8CcOHQTgD0bZdCckIsqYlxtVwiY+qXUJbwuEtEWgC9gcSA41+HMzBjakpEuHp0N1Thi9V7SYqPoaDYy9w1+/znfL29GMjg+40ZNImP4ZxjOvj39v7Thyu5alS3Mq85a/ku7vvQ6Upr36x0I6W4GKGoxDqzTXSrMlGIyHXAbTjDWJcAI3E6qE8Oa2TGHAER4Zox3blmjDMyKT27gGtf/YGlAYsLpiTEklVQzC3/+YkFG/eXuX793ix6pCXjcRcmDEwyTQK2f42N8VDs9VJc4iXGI3UyRNeYuhZKZ/ZtwHHAfFU9SUT6AQ+HNyxjaldacgIf3jIGVeXHrQeZ8fUibjl/LEUlXkY98gVvzN9S5vxTn3QqzC9dNYy2qYn8N6CJKi6mtMVWVZm6cBtTFzrPf3/XyWVqHMZEg1DWPshX1XwAEUlQ1dVA3/CGZUx4iAhDu7ZgXOc40pITaN8sic9/f6L/+evGdOfso9uXPn59ES9+s9H/eMYtY8q8Xp+2ZZfvWLipbM3EmGgQSo1iu4g0B6YBc0TkALAl6BXGNCA9Wyez9i/jKSguIcXtmD5x0TbufG8ZANOX7vSf2y2tSZlr37h2BCVeZcO+bMY/9Q3frktnwjEd/E1WxkSDUNZ6Ol9VD6rqA8B9wMs4y4wbEzXiYz3+JAHOkNvNj57N2D6tSUl0/p4a1Lk5yeWW6ojxCPGxHvq5CwO+u3g7gx6c7SxmGEZfr93H7e8socTWmTJ1IJTO7JYBD5e7P+vlv04RmQBM6NWrV6RDMVHi9WuGh3Sebxe/qQu3klVQzB/eXcrGfTn85pTenDuoQ5lzfYsIHkmt46Y3F5NbWMIfz+xH29TEqi8w5giE0kfxI7APWAusc+9vFpEfRSSkGdp1xeZRmEh65IKj+ez2sQB8unIP6/ZmM3vlbvbnFHLXB8u55T8/8vHyXfS425kAmF/kLP3h9SoPzfiZH7ceCPm9cgtLyvw0JpxCSRRzgLNUNU1VW+FsifoR8Gvg+XAGZ0xD0z0tmV8d342BHZx1Mz9atoshf57D1IVb+WjZLn791o/+c/vd9wl3fbCMPVn5vPLdJi54fp4/eYQqJ8gWsXsy81m/N6tmBQnRrkN5PPP5uqBxmIYvlEQxUlU/9T1Q1dnAKFWdDySELTJjGqAYj3D/hIHMvPUEmgbMtwjULKm0L2Tqwm3MXLbL//jeaSuY+Nx3rN+bxbSfdvC+u6x5ZfKCJJYRD3/OqU9+zYZ92dUsRej+s2ArT8xZy4dLdlZ9smmwQhn1tEtE/gj49ha9BNjj7n1tU1KNqcTKh84EYF9WAaqKxyNM+2kHV4zsymer9rDjQB6PzFrNX2au8l/j2+/CN48DSpcMqUhGdmGFxwM7uf89dwN/Of+oCs87UjkFJe5Pq1FEs1BqFJfhzMqeBvwPZ9Ohy4AY4OKwRWZMlGidkkCb1ETSkhO47oQeJMY5S4ZcNqILg92RVFeP7kbH5qFP1EtylzhfvuPgYc/lFZZw0t/n+h+/u3g7z3y+nrUHSrjutR/Ytj/3sGtqyiaiNw6hrPWUDvxGRJqqak65p9eHJyxjol9KYhzTbj7e/7htaiJ/+2Q1PVo3ZeO+0v9qeYUlJJVrxmrfLJGN6Tk89+UG7jijX5nnth/IZev+XAZ3bs7Y3mk8/cV6nv3S9181n9E90/xLm9QWrZ8DIU0tCWXjotEi8jPOJkOIyCARsU5sY2rZTSf24Pu7Tuaz353IQxMHcmKf1gD0/9Mn3PzWj7zzwza2ZuSiqmU6vbtNnklewOinQ3nOHI7bT+vD7af3ZdLwLiQnxDKuk/N3Yfm5ILVBLU9EtVCanv4BnAFkAKjqUmBsOIOqKduPwjRkIkL7Zkl4PMJVo7rx1KWDGde3NWnJ8Xyxei93vr+MsY9/Sfe7PmbnoXzOG1w6P+PCf81jb2Y+a3ZnsfNQPlDaaf7IBUez4sEzOK+387jQXe128Zb9bMko30hQtdve/ok73nVW2vW1PJVYpohqIf1poarbyq2KWS8Hb6vqDGDGsGHDro90LMYcqeZN4nn1V86EP69Xmb8xg7V7sli9O4vkhFguOa4zd53VnxMe+5Kfd2Uy/OHPy1xffiJerPt/+NOVu9l+II9/f7UBgM2Pnh1yTNN+2uEf4TSiRyv/ca/NEI9qoSSKbSIyGlARicNZTXZVFdcYY2qRxyOM7pXG6F5phz235s9n8sis1Uz5emOZ4+2alUsUbvvBN+vS+WZduv/4odwiUpNiKfEq//5qA6pwy8m9yiyZvjk9h+tfX8SmdKcGkpIQy3uLt3F0R2dyq23JEd1CSRQ3AU8BHYEdwGzg5nAGZYwJnYhw91n9uf20PsR6hI+W7SIh9vBW5YQYmDS8C4XFXi45rjNXvryAgmIvgx6aDUDzJnEczHX6NwZ2TOXkfm0BeOGrDTwya7X/dfq1S6F9s0T2ZRf4j1nTU3QLZVHAdFW9XFXbqmobVb1CVTPqIjhjTOgS42KIjfFw3rEdGR+wVLqPiPDIBUfzxMWDGN69JVeO7Ao4tQOAdqmJ3HNWfwCm/VQ6gS4wSQDceGIPkhPjyCkowdfiZE1P0a3SGoWI/CnIdaqqfw5DPMaYOnL3Wf25/fQ+NIkv+zXw149XMX3pTjZn5LDM3RHwjIFt+dOEgXy+ag/nDurIwk372ZSew8vfbgLg2S/X8/vT+9gOf1EqWI0ip4IbwLXAH8MclzEmzDweOSxJAP7Vbn1JIiUhlvMGd6Rj8ySuGtWNGI8QU8HKt5e8MJ+tGbU3mc/UH5XWKFT1Cd99EUnB6cT+Fc5SHk9Udp0xpmF7etKx3HpKL5ZsO0SnFkmM6N7ysJrClSO7sTk9l3MHd+C1eZtZuTOThZv38/fZa3h60rHO63y+ju/Wp3P5yK6HLbV+JHYfymfkI5/TND6GpfefTmxMKKP8zZEI2pnt7kVxO3A58BowRFVDXwvZGNMg9WqTQq82KZU+37ddCm9eNwKAiYM7sHjLAS57cQHTl+5k9s+76d8+lZ+2HgRgwab9jO7ZihKvkpacUGFtpDo2pjuLHOYUlrB6dxZHdbRtBcKt0lQsIo8DPwBZwNGq+oAlCWNMeQmxMYzumcb0W47ntAFtaZ2SwKpdmWXOGfaXzxjx8Oc888U6SryKqnIot4jtB3J5YvYaDuRUvLhhRYpLSjvOdxzMq7VymMoFq1H8HigA7gXuCah6Ck5ndmqYY6s22+HOmMg5plNzXrxqGACqioiwfPshLpnyvX+DpX9+to5nv1hPcblRUs98sZ7Pbj+RXm2SeX/xdtbvy+aqUV3JKyxh+4E8BnVu7p9pXuwtnbRx4xuLee2a4f7lTkx4BOujaHANfzYz25j6wfeH5dGdmvHDPaeSU1hMYbGXdxZtJz27AFX4YvUeju+Zxgc/7QDg1Ce/KvMa/5q7wX+/a6smdE9rSkpiHH3bJgPgEfAqPDprda0mivyiEm58YzE3ntiD0T0Pn+DYGNX+6mDGGBOgaUIsTd25Gref1ifgmaMBePKSwXy9dh9/eHcpe7OcSXx3n9WPohLF61WemLOWLRm5bHFHVM1wr55121j+MvNnvlmXzpA/z+HNa0cwwN1ZMK+whM2HnFqMqqIKBcVe5q7Zy4HcIi49rnOle5bvOpTPV2v3cTCviA9vtkQBliiMMfXA2D6tWXjPqSzecgCvKsd1a+l/bkjXFlz+0gIA0pITSHdnhBd7vfz7iqHc+f4yZi7bxVlPf0NKYiwpCbH+hRHj223l319tYGu5PTgGdEhlcOfmFcZS7K5Hku4mrR+3HmDBxv0UFnu57oTu/qTXmDS+Ehtj6q2hXVscduz4XmlsePgsvKrExXjYuC+bJ2avpXebFOJjPTx32RDSmq5g1e4sFm7aT1Z+6W57d/9veYXvc//0lTx32bF0atHksOeK3M7yHQfz6DZ5Zpnn8opKmDy+32HXRDtLFMaYei/GI8S4i5r3aJ3Mc5cPKfP8gxOdrV6z8osoKlE8Ave/NZf+fXpy/Qk9iPFImS/9pdsOMuZvX/LmtSMY07ts81JRuRUOT+3flocmDmT0o1/w09ayAz+LS7xl5nGkZxfww6b9jOmdRkpiHNHCEoUxJmoEfjmf3zuecSf29D++6cSezFi6kylXDWXW8t08++V6rnh5Af+5bgQ9WifTvEkcCbEe/8ZPFw3txO9P70O71EREhNE9WzFvQwaPf7oar5Z2tr941TAO5RWRGOfhqc/WsW5vNmcf057nLiubzBoySxTGmEZh8vh+/majgR2akZVfxGvfb+Eyt/+jvAuO7Uj7ZqX7mJ/Svy3zNmTw3Jcbypx3/euL/Pdj3Q7y3W4fSTAlXuU/C7dy1lHtaJWcUO3y1CVLFMaYRunBiUdx80m9WLc3m437ssnML2ZvZj6vz9/C6J6t6Nuu7Mz0a8d055LjOvPavM1k5hdx2ym9+XZdOpszcvh4+W4ycgp489oRPDprNev3Zlf5/gs2ZXDftBWs35PlbzqrryxRGGMarTapibRJTeT4gA2hgn1pJyfEcvNJpRN6Tx/YDoAbxpY2cSXFxbBubzZnPfUNT08aTMumCew8mOdfaqS4xMv+nEL/AopHOrvcN7kxnCxRGGNMLerZxpkQ+POuTE598mv/8ZSEWPq2S2HRlrId4pn5xczfmMHIgK1lQ6GqnPLkV8THeJh+yxjiK9isqrY0uNnXxhhTn/16XE8W3n0KL1w5lJtO7Mmp/dsAkJoUR5FX6dM2mbTkBM4b7Kyou3DTfi6dMp///rCV/KIS9ley7pWqsj+nkIJiZyLhloxcNu7LYfXuLM5++puwlimqahS21pMxJtJEhDapiZwxsB1nuE1Tlbl4WGf+8dlafth8gD++v5w/vu/M++jVJpl9WQW0S00kI6eATi2akJ5dwPYDTjPVFSO7cNnwrv7XWbc3m/s/XEGLgmLGhaFMUZUobK0nY0xDMrpXGqN7pfHNun1sP5DHzoN5fLs+HYBN6Tms2ZMFQLdWTendJpnxR7XjxW828eb8rbw5fysAI7q3ZMGm/bz2/RbiY+C3F9d+nFGVKIwxpiE6oXfpooa/P70v4OxDLsJhHdVXH9+dB6avZOGm/RzKK+J3p/Vhwcb9/OOztRSWONdVto5VTVmiMMaYeqiyL/uOzZN48apheL1Kek4BbVISGdmjFSN7tOTpjxZRUOwlKT6mdmOp1VczxhhTJzweoU1Kov/xiB6tuP6YhFpPEmCJwhhjTBUsURhjjAnKEoUxxpigLFEYY4wJyhKFMcaYoCxRGGOMCcoShTHGmKAsURhjjAlKVDXSMdQ6EdkHbKnh5WlAei2G0xBYmRuHxlbmxlZeOLIyd1XV1hU9EZWJ4kiIyCJVHRbpOOqSlblxaGxlbmzlhfCV2ZqejDHGBGWJwhhjTFCWKA43JdIBRICVuXFobGVubOWFMJXZ+iiMMcYEZTUKY4wxQVmiMMYYE5QlCpeInCkia0RkvYhMjnQ8tUlENovIchFZIiKL3GMtRWSOiKxzf7Zwj4uIPO3+HpaJyJDIRh8aEXlFRPaKyIqAY9Uuo4j80j1/nYj8MhJlCVUlZX5ARHa4n/USETkr4Lm73DKvEZEzAo43mH/7ItJZRL4UkZ9FZKWI3OYej8rPOkh56/ZzVtVGfwNigA1ADyAeWAoMiHRctVi+zUBauWOPAZPd+5OBv7n3zwJmAQKMBBZEOv4QyzgWGAKsqGkZgZbARvdnC/d+i0iXrZplfgD4QwXnDnD/XScA3d1/7zEN7d8+0B4Y4t5PAda6ZYvKzzpIeev0c7YahWM4sF5VN6pqIfA2MDHCMYXbROA19/5rwHkBx19Xx3yguYi0j0B81aKqXwP7yx2ubhnPAOao6n5VPQDMAc4Me/A1VEmZKzMReFtVC1R1E7Ae5999g/q3r6q7VPVH934WsAroSJR+1kHKW5mwfM6WKBwdgW0Bj7cT/MNoaBSYLSKLReQG91hbVd3l3t8NtHXvR9PvorpljJay3+I2s7zia4IhCsssIt2AY4EFNILPulx5oQ4/Z0sUjcMYVR0CjAduFpGxgU+qU2eN6nHSjaGMrn8BPYHBwC7giYhGEyYikgy8D/xWVTMDn4vGz7qC8tbp52yJwrED6BzwuJN7LCqo6g73517gfzjV0D2+JiX351739Gj6XVS3jA2+7Kq6R1VLVNULvIjzWUMUlVlE4nC+NN9S1Q/cw1H7WVdU3rr+nC1ROH4AeotIdxGJBy4Fpkc4plohIk1FJMV3HzgdWIFTPt9Ij18CH7r3pwNXuaNFRgKHAqr0DU11y/gpcLqItHCr8qe7xxqMcv1J5+N81uCU+VIRSRCR7kBvYCEN7N++iAjwMrBKVZ8MeCoqP+vKylvnn3Oke/Xryw1ndMRanJEB90Q6nlosVw+cEQ5LgZW+sgGtgM+BdcBnQEv3uADPub+H5cCwSJchxHJOxamCF+G0v15bkzIC1+B0AK4HfhXpctWgzG+4ZVrmfhG0Dzj/HrfMa4DxAccbzL99YAxOs9IyYIl7OytaP+sg5a3Tz9mW8DDGGBOUNT0ZY4wJyhKFMcaYoCxRGGOMCcoShTHGmKAsURhjjAnKEoVpNESkW+BKq+6xB0TkD9V8nc0iklbFOXfXIL5zROQnEVnqrhZ6o3v8PBEZUN3XM6a2WKIwJjyqlSjc2bdTgAmqOghnTZ+57tPn4awKakxEWKIwxiUic0XkKXd9/xUiMtw93kpEZrv7AbyEM4nLd800d7HFlb4FF0XkUSDJfZ233GNXiMhC99gLIhJT7u1TgFggA0Cd1T/XiMho4Fzgcffanu7tE/d9vxGRfu57vCoi/xaRRSKyVkTOcY8PDHjvZSLSO6y/SBN1LFEYU1YTVR0M/Bp4xT12P/Ctqg7EWSurS8D516jqUGAYcKuItFLVyUCeqg5W1ctFpD9wCXC8+9olwOWBb6qq+3Fm2G4RkakicrmIeFR1nnv8Dvf1NuDUPH7jvu8fgOcDXqobzro/ZwP/FpFE4CbgKfe9h+HM4jYmZLGRDsCYOlTZMgSBx6eCs9eDiKSKSHOcDYIucI/PFJEDAeffKiLnu/c746ytk1Hu9U8BhgI/OEv3kETponWlQaheJyJHA6fiJIDTgKsDz3FXER0NvOu+Fjib1Pi8o85CcetEZCPQD/geuEdEOgEfqOq6Sn4PxlTIEoVpTDJwdjML1BLYFPC4fDKpdI0bERmH86U+SlVzRWQukFjRqcBrqnpXVQGq6nJguYi84cZ1dblTPMBBt3ZQ4Usc/pL6HxFZgFPL+FhEblTVL6qKxRgfa3oyjYaqZgO7RORkcPZZxtnV7NuA0y5xnxuDs9LoIeBr4DL3+HhKk00z4ICbJPrhbLXpU+R2UIOzWN1FItLG974i0jUwNhFJdhOPz2Bgi3s/C6cPA3X2ItgkIr9wrxMRGRRw3S9ExCMiPXEWhFwjIj2Ajar6NM6qqseE8vsyxsdqFKaxuQp4TkR8SzY/6Lb7++SLyE9AHM7qogAPAlNFZCUwD9jqHv8EuElEVuGs1Dk/4HWmAMtE5Ee3n+JenF0GPTirvd5MaSIAp9Zxp4i8AOQBOZTWJt4GXhSRW4GLcPo3/uW+Zpz7/FL33K04y0qnAjepar6IXAxcKSJFOLu/PVy9X5lp7Gz1WGNcbtPRH1R1UaRjqQkReRX4SFXfi3QsJrpY05MxxpigrEZhjDEmKKtRGGOMCcoShTHGmKAsURhjjAnKEoUxxpigLFEYY4wJ6v8D2WcSJnpNCcIAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAEICAYAAAB7+s71AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABGB0lEQVR4nO2dd3hc1bW33zWqVrGqJVkuuAO2KQbjQjEGm2J6DS2UhITw3Tg3vdxcbupNQm5yk5sEkuDQCaEmgAmmhGKMCTYYgnED29jYlmRJtnpvs78/zow8MzpnimZGM7LW+zx6NHPOnn32jEa/WfPba68txhgURVGU5MeV6AEoiqIo4aGCrSiKMkxQwVYURRkmqGAriqIME1SwFUVRhgkq2IqiKMMEFWxlyBARIyLTPLfvF5H/9tw+TUQ+Suzo7BGRSZ5xpw7y8deJyEuxHpcyMlHBHuaIyCci0iEirSLSICLPiciERI8rEowxbxhjjnQ67yv0AcdvEpG14VxDRE4VkX+KSJOI1IvImyJyUjTjtrnGAHE3xjxsjDk7ltdRRi4q2IcHFxpjcoCxQA3wu8F0MtgoMtkRkdHA37Fel0JgHPBDoCuR41KUSFHBPowwxnQCTwIzvcdEJENEfikie0WkRkT+KCKjPOcWi0iFiHxbRKqB+0TkByLyuIg8KCItIrJFROb69He0iKwWkUbPuYt8zq0Wkc/53A8rAvaOI0Yvgx0zAIwxjxhj+owxHcaYl4wxH3iu7xKR20Rkj4jUep57nsNYPxGRpT73fyAif/bcXeP53ej5xrMw8DUQkZNF5B1PpP+OiJzsc261iPzYE/23iMhLIlIc6xdDGb6oYB9GiEgWcBWwzufw7ViCdTwwDSu6/J7P+TKsqPMI4BbPsYuAR4F8YCVwh6f/NOBZ4CWgBPgS8LCIONoZScJ2oE9EHhCRZSJSEHD+Js/PGcAUIAfPc46QRZ7f+caYHGPMW74nRaQQeA74LVAE/Ap4TkSKfJpdC3wG6/VNB74xiHEohykq2IcHT4tII9AEnAX8AkBEBEuEv2qMqTfGtAA/Ba72eawb+L4xpssY0+E5ttYYs8oY0wc8BBznOb4AS8xuN8Z0G2NexbIaronv04sOY0wzcCpggD8BB0RkpYiUeppcB/zKGLPLGNMK/AdwdRwsovOBHcaYh4wxvcaYR4APgQt92txnjNnu+Vs8jvVBqyiACvbhwiXGmHwgE1gOvC4iZcAYIAt412NhNAIveI57OeCxUnyp9rndDmR6xKsc2GeMcfuc34MVtSc1xphtxpibjDHjgdlYz+X/PKfLsZ6Hlz1AKlBKbAm8jvdavq9f4GufE+MxKMMYFezDCI8/+zegDyuiPAh0ALOMMfmenzzPBGX/wyK4RBUwQUR83zcTgUrP7TasDwgvZRE/iSHAGPMhcD+WcIP1vI7waTIR6MWawA0k2HMM9VoGXsd7rUqbtooyABXswwixuBgoALZ5IuE/Ab8WkRJPm3Eics4gL7EeK+r7loikichirK/zj3rOvw9cJiJZnjS8mwf9ZAaSLiKZPj8pnuMScDwz8IEicpSIfF1ExnvuT8Cycbxe/yPAV0VksojkYNlGjxljem3G8T6WXZLmmYy9wufcASyLaYrDc1gFzBCRa0UkVUSuwpog/nskL4QyclHBPjx4VkRagWbgJ8CNxpgtnnPfBnYC60SkGXgZGNQkoTGmG0ugl2FF778HbvBErAC/BrqxItMHgIcH93Rs2YL1bcH78xnP8ZMDjnfYeM8twHxgvYi0YQn1ZuDrnvP3Ynn1a4DdQCfWhKod/wVMBRqwUgP/4j1hjGnHev3f9FhQC3wfaIypAy7wXLcO+BZwgTHmYNivgjKiEd3AQFEUZXigEbaiKMowQQVbURQlxojIvZ5FWJsdzouI/FZEdorIByJyQjj9qmAriqLEnvuBc4OcXwZM9/zcAvwhnE5VsBVFUWKMMWYNUB+kycXAg8ZiHZAvImND9ZvUxX7SXZlmlCs30cNQFCXJae47eNAYMyZ0S2dOX5xhGurdoRsCmzb1bsHKJvKywhizIoLLjQP2+dyv8BzbH+xBSS3Yo1y5LMy/NNHDUBQlyXmx7k+BK0gjpqHezcpV4dXamjyhutMYMzd0y9iiloiiKMrQUwn41q0fTxgrXlWwFUVRhp6VwA2ebJEFQJMxJqgdAkluiSiKogxHROQRYDFQ7Kn1/n0gDcAY80esMgXnYa1CbufQyt2gqGAryggkOy+TK7+5hNLJhVhVeJMfYww1u+t54hev0NYUWGAyuTDGBC05bKwl5l+MtF8VbEUZgVz5zSXMPHE6GamjhpVgFxUWceU34f7bnkv0cBKCetiKMgIpnVw4rMQaQETISB1F6eTCRA8lYahgK8oIRESGlVh7Ga7jjhUq2IqiKMMEFWxFUZKC7u4uvvLt5Zx10WKuvP4SKqoqEj2kpEMFW1GUpOCJpx9ndG4e/1i5mpuuu5lf/ub2RA8p6dAsEUVRQvLsqmx+dUcB+6tTGFvWx9eWN3DheW1R9fn0s3/lnof+hCAcOeMo6uvrWP6FrwBwztJl/Ojn38cYM6I960BUsBVFCcqzq7K57cdFdHZaX8ir9qdy24+LAAYt2js+3s4f7r6DR+7/K4UFhTQ2NfLpm69ibJlVsC41NZXcnFwaGhsoLBi5WSGBqCWiKEpQfnVHQb9Ye+nsdPGrOwoG3ee6t//JuWed1y/G+Xn50QxxxKCCrShKUPZXp0R0fLCUlpSyv9oqp9Hb20tLawsF+YP/UDgcUcFWFCUoY8v6IjoeDgvmncwL/1hFQ2MDAI1NjZx5+lKeevavALz48vMsOGmh+tcBqIetKEpQvra8wc/DBsjMdPO15Q2D7nP61BncevNyrv/c1bhcLmYeNYsf/udP+OZtX+WsixaTNzqPX9/+u1gM/7BCBVtRlKB4JxZjnSVy6UWXc+lFl/sd++0vfh9Vn4c7KtiKooTkwvPaohZoJXrUw1YURRkmqGAriqIME9QSURRFAZrdmbzcPi3M1tVxHYsTGmEriqIME6IWbBGZICKvichWEdkiIl+2aSMi8lsR2SkiH4jICdFeV1EUZaQRiwi7F/i6MWYmsAD4oojMDGizDJju+bkF+EMMrqsoymHEO++u59JrLmDm3Gm88I9ViR5OUhK1YBtj9htj3vPcbgG2AeMCml0MPGgs1gH5IjI22msrinL4MHbsOH72w19wwbkXJXooSUtMJx1FZBIwB1gfcGocsM/nfoXn2H6bPm7BisLJdOXEcniKogyS3ManKa79Jam9VfSmlnOw5Bu05F8SVZ+B5VV/8d+/BsDl0qk1J2Im2CKSA/wV+Ioxpnmw/RhjVgArAPJSx5gYDU9RlEGS2/g0pfu/i8t0AJDWW0np/u8CDFq07cqrKqGJyUeZiKRhifXDxpi/2TSpBCb43B/vOaYoSpJTXPvLfrH24jIdFNf+ctB9annVwRGLLBEB7gG2GWN+5dBsJXCDJ1tkAdBkjBlghyiKknyk9lZFdFyJH7GIsE8BrgfOFJH3PT/nicitInKrp80qYBewE/gT8G8xuK6iKENAb2p5RMfDwa68qhKaqD1sY8xaIGjRWmOMAb4Y7bUURRl6DpZ8w8/DBnDLKA6WfGPQfdqVV732qutZ/rVbaW5u4rU1r/C7P/4fz/31pVg8hcMGXZquKEpQvBOLsc4SsSuvuubFt6Lq83BHBVtRlJC05F8StUAr0aMJj4qiKMMEFWxFGYEYY7CmloYXw3XcsUIFW1FGIDW76+nq7RhW4meMoau3g5rd9YkeSsJQD1tRRiBP/OIVrvwmlE4uHDY7kxtjqNldzxO/eCXRQ0kYKtiKMgJpa+rk/tueS/QwlAhRS0RRFGWYoIKtKIoSB0TkXBH5yLNxy3dszk/0bP7yL8/GLueF6lMFW1EUJcaISApwJ9bmLTOBa2w2drkNeNwYMwe4Gvh9qH5VsBVFUWLPPGCnMWaXMaYbeBRrIxdfDDDaczsPCFlNSycdFUVRgNa+DN5onBFm67XFIrLB58AKTy1/L3abtswP6OQHwEsi8iUgG1ga6qoq2IqiKJFz0BgzN8o+rgHuN8b8r4gsBB4SkdnGGLfTA9QSURRFiT3hbNpyM/A4gDHmLSATKA7WqQq2oihK7HkHmC4ik0UkHWtScWVAm73AEgARORpLsA8E61QFW1EUJcYYY3qB5cCLwDasbJAtIvIjEfFuC/914PMishF4BLjJhKgVoB62oihKHDDGrMLabcv32Pd8bm/F2rErbDTCVhRFGSbEatf0e0WkVkQ2O5xfLCJNPns+fs+unaIoiuJMrCyR+4E7gAeDtHnDGHNBjK6nKIoy4ohJhG2MWQOM3CK1iqIoQ8BQetgLRWSjiDwvIrOcGonILSKyQUQ2dJvOIRyeoihKcjNUgv0ecIQx5jjgd8DTTg2NMSuMMXONMXPTJXOIhqcoynDFTB6X6CEMGUMi2MaYZmNMq+f2KiBNRIKu6FEURQmFmTyOtonZiR7GkDEkgi0iZeLZh0hE5nmuWzcU11YU5fDEK9at5SmJHsqQEZMsERF5BFgMFItIBfB9IA3AGPNH4Arg/4lIL9ABXB1qRY+iKIoTvmK9b/PziR7OkBETwTbGXBPi/B1YaX+KoihR4SvWlbvWUL/6H4ke0pChKx0VRRk2eCcYW8tTqKn9FwdXPc3ExUckeFRDhwq2oijDAq9Y1ywYzYHWj6j+618onVPGd36rWSKKoihJg1es2yZmU9+3j/2P3Ef+5NF8d8UklhXsSvDohg4VbEVRkhpfsa6YWE/ln1eQXZjOf947jazRI6vgqAq2oihJi69YV01vo+LPdyFi+Pa9M8gvzWBp1s4Ej3BoUcFWFCUp8RXrpqJuKv6yAtPawjm/WULZlKwRJ9agGxgoipKE+Ip1c6lhx2v30lNVzZL/XcqlJzfERazbe9PZeKA85v3GEo2wFUVJKnzFuqVM+Pidh+j68GNO/a9TuXpZ+4iMrL2oYCuKkjT4ifVYF3vefYr2DZs48d9P4sZr3CNarEEFW1GUJMFXrFvLU6j88GWa1r/JzOtmc+sXR94Eox0q2IqiJA1esd5fsY66V55nyrKpfPm2vEQPK2lQwVYUJeH41gepbdhM7TNPUL5wHN/8RSkul2h07UGzRJSIqeqcx46OS+l0F5Lpqmf6qKcoz3w70cNShim+Yl3XuYvqxx8ka/pYTv/pmaSm7Vax9kEFW4mIqs55bGm7HjcZAHS6i9jSdj2AirYSMb5i3SD7qfzLPaSX5nHB7xZzZrmKdSAq2IojdpH0jo5L+8Xai5sMdnRcqoKtRISvWDdl1FN5z12kZKZw/u/O5KxJe1WsbVDBVmxxiqTdpNu273QXDuXwlGGO79ZeLbmtVNx3F6ani/NWLGPZzP0q1g6oYI8wwvWfnSJpcAMyoH0qrUM+RmV44ivW1bN7qfjzn+irq+fs353DRXMPJnh0yY0K9ggiEv/ZOWIeKNYAfWRS1TkvImG1E2Yg6BhVzIc3fpF1KVT+7T66P6lg8e1ncsUZzQAaXQchJml9InKviNSKyGaH8yIivxWRnSLygYicEIvrKpERzH8OJNNV79CLvWAb0mz7ccL74dHpLgKkX5i3tV3lOEanx1R1zgv7ukri8C6MAWgZK+z+56N0bt7Owu+czHUXdwEq1qGIVYR9P9aejQ86nF8GTPf8zAf+4PmtxBCn6NP3uB12x6ePesov0rUwOAm2Uz9OY+t2Z9kKczCPXCc8hy+BS873vr+Slo3vcvytJ/DZG633lIp1aGK1Ce8aEZkUpMnFwIOendLXiUi+iIw1xuyPxfUVZ7ujoWcKVd2nDBA6X+yiaa8A+opsrzudXnIj6sdpbJb4h0+mqz6iDxwleRiw5HzHqzS++TpHfWomx9x0HLBDxTpMhsrDHgfs87lf4Tmmgh0jnKLPiu7TgRTHx7no6veOAynPfNsvcg0UXrt+wk0FdIrU06SVPpPu197bt9Vv0YDHONs3SqIJFOvqqrepe/Hv5C+ayUlfnc+iAhXrSEi6SUcRuQW4BSDTlZPg0QwfnKNMp2kKE/GkXWDUbWWMuEJOGDrZHHZjOirrMb9rBI4x8AMj2AeOklgCxbq2cQs1Tz9OzpzJnP/Tk1hUqGIdKUMl2JXABJ/74z3HBmCMWQGsAMhLHRPZ9+YRjGUZDIw+LVEdGGFnuuo5veA/gMjS6AKjbl9eb/iZQypgn+0YAkmT1v6+7a5hZ9MEG6tmlCSOQLGu69xF9WMPMGpqGRf+76ksHvNxgkc4PBkqwV4JLBeRR7EmG5vUv44eX0FKpRWhB0Na/3kXXZSnvznAw/aNSmO51DxYlO+iK+gEpouu/ug6GME+MHzRJfSJI1CsG6ii8uG7SS/N48I7FnPmuN3A4T/JKCLnAr/BilbuNsbcbtPmU8APsP4hNhpjrg3WZ6zS+h4B3gKOFJEKEblZRG4VkVs9TVYBu4CdwJ+Af4vFdUcygSluveRigDRpwbI76piV/RCzch9lVvZDZLrq/I77RqvhpvqFwslLznTVDxjD+PTXHMcUC2L5vJTwCRTrpvQ6Kh+6i9SsNM6/YwlnTdoLjAixTgHuxMqQmwlcIyIzA9pMB/4DOMUYMwv4Sqh+Y5Ulck2I8wb4YiyupVjYT+SlkSLNnFn4Bb+jwaLSWGZe2KUCeqN5+zE8GvE1wkUzSoaeQLFuzmmh4p4/Ql8Py/64jGVHVwGHv1h7mAfsNMbsAvC4CxcDW33afB640xjTAGCMqQ3VadJNOirhEStBcvK+B5N5EanHHE9i+byU0ASKdUt+BxUP3IW7uZlz7jyHC084ACS3WPf1uGiodk5bDaBYRDb43F/hmX/zYpcZF7j2ZAaAiLyJZZv8wBjzQrCLqmDHkXhOekUqSH5jSW9i2rhXKC/azLTRq9m650Lc7kOZHC5XN9OOWI0pGofstp0bdiRcjzneBIv2ldgSKNatRd1UPnI3vfurOfN/lzLmmBKgManFehAcNMbMjbKPVKzFhIuxEjHWiMgxxpjGYA9Q4kC8J73CESTvP1JV3Wy2NhwS5c7ufLbuudAaS5FVTWBn5RI6u/P8xNy3D18iFfF4E+yDMRmi/cOZQLFuG9NH5V8foGvHbk770WLGLRjPafnbEzzKhBBOZlwFsN4Y0wPsFpHtWAL+jlOnKthxIt7LqJ0EaezRlRj8RXZn5RK/CBrA7U5nZ+USyos29/+ES6CIJ1LAQ30wqkDHjwFiXeKm5m+P0fnBNsb92zImnz22X6wPs+g6HN4BpovIZCyhvhoIzAB5GrgGuE9EirEskl3BOlXBjhOhPOZY2CVeQbKLgv2u2W2/ianT8UjxXj8Rwq31RRJDoFgbYzj4/DO0vL+Bsk+fzpIbR7RYY4zpFZHlwItY/vS9xpgtIvIjYIMxZqXn3NkishVrscI3jTF1wfpVwY4TwRaybGm52i83erB2SSih7h9LehOd3fkDjmeMau4vdWlH9t62sMcSOJ6hEu9oJl91YU3k+P6NvWINULH1RRrfeoPii+dz1vIpI1qsvRhjVmGlNPse+57PbQN8zfMTFrprepyYPuopXHTZnEmhovuMqHKEzeRxYYl128Rs2iZmM+HYtbhSevzOuVJ6mDjrjbAe7/sTLuGOMVqC5X4HQ0u1Ro6TWFfuWkP9ay9RsPQ4ln1nJosKdgAjW6zjhUbYccIbqW1q+wwDl2XbFz4KJyoMJYJ2oloy8UMA9m45ja6O0WSMambirDf6j0eCb//hRODxtkuCTb6GmoxUKyV8nMR6f+XbHFz1NHknH8X5P5zDokIV63iigh1HyjPfZlPbzWG3DxYVDkaofSmZ+CElEz/s/0cDaA1R3yOnqi/sa4YS73gJt1dcP2y/ih5jFQtz0T2grGynu4hNbTezre0qjs5+TBfWRICTWNcc3Ejt04+RM2cyF/x8PqcXWyKtYh0/VLDjjLOXPbCWhlOOcDCxDibUvuI8GOwe7yTi4Yp3vIS7z6TjfT17yaWi+wwGfpOxlvBvabueVFpta3vrwhp/nMT6QPM2qp/4M1lHjePCX59KSnp07zUlPNTDjjN2XraLrrBraTiJtZOn3Fqe0v8TD8LpPxy/O5YedyT1tsGyPkSw/bvowppDOIn1wY6d7H/kPkZNGsNFvzudtFFpOsk4RGiEHWeCL+Cwaml4vdZNbTf75VPb4SSEwQS0vSy655BVbX/c95p2kbd3rKEi7mij7cHYGD0mh2Oy79EsEQecxLq+dw9Vf/ZU3rvzDNJzM1SshxAV7ARju/Cj4wZM3bMDFrM4RdR2hBJpYwympwfT14e4BElNQ1LC68tXwJcVvcuXJq6ibEED1V0FrNh4Li9X+O+xHEq4o7VJwrWdAh+jC2vscRLrBlNJ5YMrSCvI4sI/LCWzYJSK9RCjgh1nQq3Es81W8FmF6CVQrO2E2k6k3T3ddO7bS2flXrqrq+hqqqW3oQl3Sxu43X5tJTODlLxcUosLyMgtJaOsnMzxE0kvKUNch9wz73UuyH6X7xU9zihPymB5ZgPfnvskXQUu3th0/ICxtE3Mjku07ZQpUp7+JjU9J3kmI8ObLxjpOIq1q5qKe/9ISnYaU3/yabJK2lWsE4AKdpwJlT7mmK3gWYUYTlQdKNR97W20bvmA5o830vnhLujtBSClMJ+0sjGMmjCWlNwcXKMyISUFjBvT3YO7rZ3ehmb6DtbTvPNtTFc3AJI1iqxJ08iecTTZR88mNdvKxvh6wap+sfYyKqWHL01cxfN1JwIDrZJ4RNvBbKdZPKoLZMLESawbUw9Qcd8fcKUJU//705w0W8U6Uahgx5lg6WOvN/zM8XGZ6U0ho+pAoe6s2EvjW2to2bIRevtIHVNE7uIFZB49jfSpE0nJiWDhi9tN74F6unftpfOjXXRu2U7b1k3wzBNkzpxG3tHzyC9vsH1sWcah494xD0a4IxXtwWxrpgwUajj0d2tKr6Pivt/joo+pP7meecd3qlgnEBXsMPCN0Ow2ng0WvTn7qzged7m6mXDsWr9jwcS6s2IPB9asonPrDmRUBjmL5pNzylzSJpYjYlkBBWUtQEtEz7vBVUxaaTHZC0/AGEPPvv20v7uJtvX/ombLw4x/QbhlseFLZ8N4n6eyv7dgQF+t5SmOE5Px8raV0AQV64x69t3/e+jtZspPPs38E3sGXXkvu6adgk9aSely05fhomFSDm2lWdE/gRGGCnYIAj1o76pFayHGjQj076NoVxPEzl91ngwzZKY3MeHYtX6rEH3F2leoe1uaOfjCSlo2vocrN5v8K84j5/T5uEZlegS6NarnbvVxiAYpJ31iOXkXn0XXR7vIff15fvlcBb96Hm44Fb57MYwdk8b/NpzXP07fCcpg0Xa8M0mUgQQV68wGS6w7O5jyk+tYML/PT6wjia6za9op2tGMyzNlktrlpmhHs3VdFe2IUMEOgX2Or5c0Ard1D1zebOevBktDO/GCu/3u24m1MYaWf22g9oWnMd09jD7/TEYvW4wrM8Mjsv6+cqzwFfAG1zR6jv4S1ze8gbz0En9Z3c0Db8DM+UfQumg6qbn+Yw4U7qHwthVngor1qEYq7r8Td1sb035yHQsX+r/LI7VCCj5p7RdrLy63dVwFOzJiItihdgcWkZuAX3CogPcdxhh/ZUpSBpPj2+kuDDrR9XrDz+x3i0lv8rtvJ9Z9nZ1Uv/Ao7Rs+IGP6ZApvvJy0sjEeMe0OOq7jxlSF/Rw2HigPet4r3q9xGlx1GoXnNNP83KtsWrMeee+nFC5aSv4pi3GlpvaPX6Pt5CAcse5rbWPqf1/LwlOsdtH41ild7oiOK85ELdg+uwOfhbWDwjsistIYszWg6WPGmOXRXm+oCeZBO5EmrUFT+WzT0AJ8azux7j5YS+Wj99BbW0feZecy+tzTKSxvw8mbjkSgQz3WScC9wt3AaAqvu4TcpafS+MRz1P1jFU0fvE3peVeSNXW63/OIRbQ9FKK9peVqKrpPx1oQ7GZ8+uvMyo3fxsHxJnBlqZ0N0i/WP76Gk0+1bLtAsY7Uj+7LcJFqI859GbrQOlJiEWGHszvwsMXeg/bS4+dhg5XjawxBU/nKM9/GlBT4bcvl61vbiXXHJ7uo+ss94HJR8vXPk3nklAEes5dohNoJ3z7txPuQcBczZvmNdGz+iIa/PEPlfX9g9AnzKF52MSmjRgGxibbjbZFYYu1bj8Qqi0sLA0R7OKQNhhTrjHr23X8n7rYOpv73tUHFOlI/umFSjt9jANwu67gSGbEQ7HB2Bwa4XEQWAduBrxpj9tm0QURuAW4ByHQl/g8a6EGHkyXiVKHPa6+YyeMo59C2XL7pe3Zi3bZ9G/sfuY+UogJKvvwZxhyTjl1UHQ+htsN7nWDCDUeS8YOv0vzsyzS/uIa23R9SevE1ZE87Ehgo2uAcbSfCIrEi64HFoyq6T2cWhwQ73nt3xoJQYt2YdpCK+36Pu7OLaT+9loUnW+3sbJDB+NHe48meJSI9QkZ1WuiGCWSoJh2fBR4xxnSJyBeAB4Az7Rp6topfAZCXOiZwTi8hhMrjDTxnCXh4O5qHFOuPtlL1yH2klZdR8tWbKZ4+8KtlJEIdKi3rjcYZYfcVjnC7Ll/GqBOPoe6ex6i6/y7yT15E0dkX4EpNTXKLxOnruv/xZK6r7STU4CPWKTVU3PcHTG8v0352HQsXWO8vJ896sH50W2lW0gn0cCQWgh1yd+CAfcruBv4nBtdNWoIV1Q+nQp1XyNp372T/o/eTVl5G6dc/R9FUfyELR6gjzZu1ax9KxEMJdwPjKfuvf6fxyVU0vrqGtoqdlF92I+nFY4BktUjcDNx4wnv8EMlaVzuUWLeXQdf+Siru/SOIMO32T7PgJGtFbLD3jPrRiSUWgh1yd2ARGWuM2e+5exGwLQbXHVIi8SmD7Wjui13E4xXrrtpqqv5yr2WDfPXmiMR6sIsbwukvmHg7CXd/tH3txWTOnE79fU+w94+/ovSSq8mdfRyQfBbJ+PTXbWpqG8anv+7XzmlSOpF1tUNZIO1l0LHvEyof+hOujHSm334t8463Ss2GyrV28qPbC9IZv742qS2Pw4GoBTvM3YH/XUQuAnqBeuCmaK87lNj5lJvabvYrhxoo3nY2imFgOhUMXMXY195G1SN3I2mptjaIk1jHWqjtuLFvLSdV7iGnu4u61GyeKJzLutFTB4zP0SY5fibp3/t3Dt71F6offYDOUxZTfPb5SEpKRBZJvDdJmJX7KLQQMksk2Lepocbu25udWLfv3E7VI/eSMjqX6T+/hpNmtQOhxRrs/ej2gnRyajt1YcwQINbGvclJXuoYszA/vI1p44lT3rQXF12OGxB4CcdPbC+zanhUPHoXndt3UfqtWyk72X+ZdyLFekpdLaft2UmaT5W/HoROVzo5bnsBtxPuhupcTG8vDY/9ndbX3iLjqKmMu/RGUrIPTTIHRttOO92E2ppsKNL/kiFLJNz3V8vmjVQ/+Wcyxxcx5cfXMneGJaxOYh1OCt/49bW2NklvhouK+SXRP7kwmDyh+l1jzNxo+sgcN8FM/LfwNjDfcdvXor7eYNCVjmEQyo90k+HZbDe8zIBgVkj96n/QuXUHhTdcFpZYD4VQezmpco+fWIO11jPNbX2dLu5t47MHrFxyr2jbRdsFZS00VOdSeN0lpE+aQP1Df2PPn/6PcVd/lowyq+1wWGgTKNTHZN+TcKEG5/dX4/o3OfD3v5F11Dgmf/8qTpxsFekKJtbhpPDpwpihQ2cKwiA8PzKFLW3XU9U5b8AZp4nGQCukY+9u6l97iaz5c8g+zb+fRIs1QE53V8g2GaaPK+s3+B07bkzVgPEXlLVQUNZCziknUvqtW6Gnh31/+i2tH27pb2NX39uuDvhQbkfmxWuTWd+8pD+dz+7vHw+cnpOdWLeVGupefp4Dz/6V0XOnMfXH1zmKdXZNO+PX13LEmmqKP2p2TOHzxWnCUSciY4++omFgty+jHd50rmDYiUt7Gbi7u6h+6i+kFOVT+OlLKBx76J8iHmK9NGun3084tKY71VTxp6jXPuK1ex4FZS1kTJlA6W1fIq1sDPsfvpeGtavxWnWRiPZghbuqcx6vN/yMF+vu4vWGn4UlusHS+eKNk1D7+tX9Yj2mj9qnHqN+9T8oPOs4Jt12JXMmHADsxbpoRzOpXW4E510xAyPnhkk5uAOURBfGxAe1RMJg4OIZcHo7B9onoaJrryDVvfwCvQfqKPnGLRRNPlS8KRZiHY4g27XZtW90/wRja3oGe0cXMKO+doAtEkhrurXPn11GiaNFQh4l37qVunsf5+ALK+k+UEPJRVdENBkJoS0S8LdJnCaUt7ZdRx+Zjp50ItL5gm3I7MX3fdXX2UHVY/fTuXUHpdecRum1izi+xErWsrNB7BbF2BEYObeVZpHR1E1udeehcZRk6oRjHFDBdsBuIun0gv/oP2d51gMjvWD2iVME2FlVQeNba8g5fQFjFx+apIlGrKMtLp9d087iPdX9/8C53V3MqK+lOiuHca3NniX5Fr7/vj0uF++MO6J/rBGJdnUuxV+4lqanX6J51Wt0tdcx7rLPBF3SHo1oA+x4z37H9T6sazqtXBzKdL5whBr8xbqnoZ7KR++mp7qWCV++gMKzju9/Pzl51uF4znaRc3ZNOzm1nX4hTE5tJ1157SraMUYtERtC+ZPlmW9zTPZ9A2ySwHSucKJr43ZT88ITVj3ry88NOq5wxDoSiyMYdtFWmtvN+NZmXFjfL6xkN+hIScUALekZvHHENHYVHfrQcRqzk68tLhf5l51L4WeupGv7bvbe+xt66g+tuwq0SHy//vsSjkUC4UXEdlaHnU0W63S+YN57YFTtl2O9dzd7V/wfvfWNTPnhNWGJNTh7zsbz05vhom766AEiHGy5uhJbNMK2IZzlxsH2EbTDSTxaNr5H9659FH7mUxRN6e0/Hihm4Yp1rHCKtgKNoFSgIyWFe+YscOzLKdKGgdF2fy2SU+aSWlTAwd8/xN4//R/l197MqAmTgPAX2UDoaDszvYnO7nzH814ChT3Sv38kBJsgDbZtXHsZNL/3NjUrnyC1MJ+pP/w0mROKwxJrcF4UYyfSvmiWyNChgm1DuP7kYPYK9I2E3D3dHHz176RPGs/4S6f3t4lUrOOxt57TEmQ7wske8T6HyHztqZR+94sc+M19VN77e0ovu5bcY44HIhdtsM/ZnjbuFTbvvgznKTYLO6sjlntFhpPFEswCMX19HPj7MzStW0vG0dOY9r2LSc0dFbZYw+CLNOly9aFDBduGWPiTdkXiA2lct5a+hmaKPncN4rIXjESINdhHW04bm0Xyjxn5ZOQYSr/7RQ7e+SDVjz1I98FaChefhYhEtA0Z2Efb5UWbaWwZT8XBeQ7Pzt7qiMVimXBTDUNF1b0tzVT97QG6duym+OL5lH92CZLiikis+681iCJNWj516FDBtiFey439Z/A7aVj7KpmzZzD29DH9bSKpvBfPXavDWYIMh/4xl2bt5OX2aWH1Hblo51Ly9c9T98CT1L/yAp2t1Yw992pcaelA9NH2zEkvkJ9b0V+fPDWlHQF6+rLITG9i2rhXGFtU2V9aYP+2cYMqqRppLngwoQbPMvNdO9j/5EOYzi4mfuMSChbPBhiUWAcS7kYFw6V86uGACrYN0fqT4UTXTevW4m5rJ++Sc/qPRWKFxFOsvdhFW115zv/E3jGFI9yDySApuvkq0saW0PTUi+yrOci4T32W1NH5gLNog3O0DYeEu7zoUH3yUOzoucJ+jqPnigEFvgaD3XsmMKo2bjf1r/6D+tdeIrW0mCk/vZZRk0r83kPRinUkGxVo+dShQQXbgUj8ycCvx9NGrx7wz+/nXXd30bBuNZnHHEXZgjzbPhMt1k6E848ZbrQdTLSBAZORDdW55J1/JmnlpdTd/Sh77vo15VfdxKiJkwH7LcjAOdqG0LW2q+pm++0MNG3cK3R22//NnI6HSyihBk/KXmMD+1f+ma7tu8laMIcpXz+LlFHpMRNr0I1zkxWdFYgSuxTArXsupKputuNjmt9dj7u1nbzzzug/Fq4VkkixjoRwxxnsg8lpOXvWnFmUfveLuNLTqbjn9zSuewPfImZOqyPt0v+82KUBVtXNZuueCz1ZJEJndz5b91xIWkq7Qy/uoH/3YNe1sz8Co+q2UkPz+++y585f0L2nkglfvYjpt50fc7EGzfxIVlSwo8Q2BdCdzs7KJbalLU1fH/XrXydj2iTKTrOvAOgkYsNFrL2EmxN+Wv72oPnagRSUtZA+royy25aTOWsGB/7+FFXPPkhf56GVdnaiDfbL2n3xFdCdlUtwu9P9zrvd6RisTZMHkhLywzrwGnbjCxRq78Ri5d/upebJh0krL+XIOz9P4ZJjAWIu1qD1QWKBiJwrIh+JyE4R+U6QdpeLiBGRkNX/1BKJEscUQIevx60fbqbvYAMFn7qg/5jTP1wyEelO2b7EwiJx8rXHLL+B5udX0/T0S+z9pIKxl99A5jhrA6RgFgk4l2z14vQ37O3LYvbkv7F59yUErnb1flj7WmLhLOCx+yBpLwNjDM3vvcPBF5/B3d1D/uXLmHjDHL8sEBj4von2w10zP6JDRFKAO4GzsPa5fUdEVhpjtga0ywW+DKwPp1/9uIwSp1S/zPQm2+MN764hpbiA8nMm2J53IpHRdWBRoNQuN8UfNVO4vTHsPqK1SJwibXG5yDv/TEq++QVMby/7VvyG+tX/wPQdEmNvlBqIN5p1irozRjU7Hs+bsxunf5/O7rygUbTdGHzxjrerZj/7HrqD2qceJW1cKUf+7nMc8ZkTYy7WvhX6xq+vJbvGWlJeN300vRmuoKscFUfmATuNMbuMMd3Ao8DFNu1+DPwc6LQ5NwCNsKPENgXQ1c2EY6260L52SFdNNV3bd5N/+TIkxfpnDye6TrQVYjcBJUBudWT1IuIVaQM0MJmx3/8K9Q8/Td3Lz9OycxNlF1zdX18b7DNJvNhF3RNnvcHH/zoHd9+hnbRdKT1MnPUGYAl3V8fAKNxJ6AOvFYj3Q6WvvY36116icf2buEZlUnjDZYy/4uj+XP1Yi3WwbJCRJNCuHuf3hw3FIuJbR3iFZwNxL+OAfT73K4D5vh2IyAnABGPMcyLyzXAuqoIdJQNSAD2ZBHkTdw9o2/TOPyE1hexTTyJwM9dkJtgydd+sgXBsk3hkkIBP8ahbrqVtziwaHn6GvX/4FQWnLKZw8Vm4PKVhnWwSL75iWsKHAOzdchpdHaPJGNXMxFlvUDLROh5K0O36tMM7JndPN03r36RuzcuYjk5yFs3jiFtOITUvy++5e4mFDaLZIIPmYDQ7zoiIC/gVEW6XGBPBFpFzgd9gGXp3G2NuDzifATwInAjUAVcZYz6JxbWTAW8KoDf/OvBrsHcZevMH75J1wjH9ezQOh+gagi9T94p5qEjNV8xvyqhlTbl/kSg7IqlBAodEO/uk48g8ahqNTzxHw5pXad70LsVLzif32BMQl/XNxtciCSbeWeU7OGrBDv8Tnj+bV7i9gp6W20LZgrVkHbmDVtsd1w/he313TzfNG9ZTv/Zl+ppayJx9JPlXLKN07iG/ONR7RbNBko5KwNf3HO855iUXmA2sFhGAMmCliFxkjPHfAcSHqAU7THP9ZqDBGDNNRK7G8myuivba0RLLvfjsVrH5RlatWzdh2jvIOe2ksPtMBrEGa6Kp+KPmoMvSQ1VsCxTzxXu2c0x6Nc/kHhv02oMRbYAGcin67KfIOW0e9Y+spObJv1C/fjXFi84l+6hZeP5JgNBRdyC+f1dbQXcg0Efva2+j6e1/0rD+DdwtrWTMmEzRLdcGXfkaS7GG8OuARDPpPEJ5B5guIpOxhPpq4FrvSWNME1DsvS8iq4FvBBNriE2E3W+uey7sNdd9Bfti4Aee208Cd4iImATuAGxXuD6c5cWDpWnzelKKCyg73Yoqh0NmiBffAvW+ou2bNRAsUgsm5kvnh7ZIIi0cBYei7Yzpkyi7bTntb2+k6ZmX2P/wvaSNH0vh/MXkHDMHV+qhf4FAQY3Az3QksE9jDJ0Ve2h+Zx0tm97D9PSSOXsGo5edQdmiYr8PknhYIAPGV5Ae9O8Kka96VMAY0ysiy4EXsZyHe40xW0TkR8AGY8zKwfQbC8EOaa77tvE8kSagCDgY2JmI3ALcApDpil8KUTglVGNBexn0NjfS+eHHjL7gTMciT4EkS3TtpX5GftBl6cEitVBfu+Ppa4O1S3v2gjlkzT2WtvXv0/Liamr++ggHXlrJ6GNPYvScuaSXjvUTS3DO5YZDYh6sjRdjDD0HamnZ/D7NW9+jt/oAkpFO9sknknPmyZSe6G+hxTuq9pJd005ujb9YG6AzJ5WCT1op/qiZvgwX0mfU5x4ExphVwKqAY99zaLs4nD6TbtLRM9O6AiAvdUzcIvBQJVQjsUvsaof4fm1u2fgeGEP2ghNiNfyEECxrIFjebsEnrSG/dkcr2hA82gZLuHNOOZHsk0+gc+sOWlevo3HdGhrfXE1q2Rhyp80ma/qR9DScRf1rS+ltyiU1r4Wis9Yy+viP/PoMJdR9ba107P2E9o+307b7Q3prDoIIGdMnMfrsRWSddKzfVnC+zyHw+QYSqw/zwo9bkID/MAFGNff2i3hqlxunf0L1uYeeWAh2KHPdt02FiKQCeViTjwkjWAnVWNslTVvfI33yBEqOs/qL1wTSUGLnadZNH+0YgYezCCMS0QZniwQGRtvgL9yjZs1g1KwZ9LW00r5hEx3vbabhrTU0rH0NuBs4Fjie3qajqXlqIt11htHHHcSVkYGkpGDcbkxvL+7ODvpaW+ltbqSnvo6u2mq6DlTSe8DKz5f0NDJmTCF3ySmMmjOL1II8zzj8xTrcwl+xfG+4eu2lOPA7oNN3Ql31OPTEQrCDmuseVgI3Am8BVwCvJtK/huAlVGNll7SXQfeBGnr2VZF/1QWhHzBMcPI066aPpmJ+yYC2BZ+0Iu5De0AGm7SKRZlWCE+4wZqczD1jIblnLMTd2UXVd47C3fov4G3g78C90AcNr1k/QREhtaSItInl5CyaT/rUI8iYPAFJS/W5ZovfQ8LdtzPRH+KBtdANlv+tDC1RC3aY5vo9wEMishOoxxL1hBKshOqmtpttHxNq/z9bO2TT+yBC+XlTB7RPxn/McAg3dzdQ2OFQZB3M+4xFmVYvwYQbAsS7Ohd36xXAlT4tGoBPgCqKbn4Id1c39PaBS5DUVFxZmbhyc0jJH01qUT7iM4lp9d0RdFyBz8WOeL0n3CmQEnx1fj92UXdWQzex33JYCUZMPOxQ5roxphP//4KkwKmEaiQ7zoQqSt/y4ftkTJtEWlEuENkGBclKuLm7kQi7nZUSC4vESyjhBktg949ppOdAge9RoICUwilkL9zi+DgLe3EOHEMgiSqlWz9tNEUfNfstsPduuBt4zM4WUQ976Em6ScdkIBY7zvTbIVU1FFxz0YDzwzW6hvBzd8MR9lApY5FaJBCdcJfd8DIVd1yM6Tr0dV8yuin/zEt+0Xi4BPuATnTNc6edYgKPSZ8hxcbvVg976FHBtmEwO87Y2SGtmz8AYOw5k4DDI7qG8Cu5BVshOX59bX8GSagoPBKLBCITbi9eAS88w/qbVT+4lJ6D+aQVN1J2w8v9x8Mh1N85Uft02uGU+ROutaUMLSrYDoSz40woO6R5xwekT5lIWvHokNcbLtE1hL+Hn52wA/0V/4p2NCMO36rtovNIom0IT7i9+Insp6rgUy8EtU8GPCaC8TiRrO8B3bMxeVDBjgPtZdDTUE/P3kryL1824Hyyr2wMh3AquTmtkPTi8skcCcTp63akog2RCbcvsfhGFM7fOtGlc8PdaFcFOvGoYMeJtg+tAvZlZ1v7DQb750/WyCoWZDV0O+bxenG7iOjrdqQWiRdf8YxUvAd7nVAkWqx1yfnwQgU7Btj5180fbyKtvISMccFTAQ93wskkaC3JJKuhO+Kv24MVbhgoqtEI+GC+MQ21UNtF0lpadfihgj1IgvnXfR3tdG3fzehzTx9w7nCwQyIh2MQjHMrn9V1w490BJVwBj0a4vQzV3yUREbVTJB3J/IGSHKhgx5j2MmjbuA3cbkrOPAIYuXYIOE88+hJJml8wYiHc8SCSv3E8ypg6RdKRzh8oiUcFO0b42iFtH27FlZtD1ozgWSQjgcAMg2A1tb3tov2avjRrJ9k17WTu7iKnu4vW9AzeGXdEyA0TYk2st+saLMEi5kjnD5TEooIdJYG7y5i+Ptp2biPrhNkDSqnGo57xcMCbYRBOPm8sdkAJvE5udxen7bFe61iL9pS6Wk6q3NP/wdA5OWNQ4ppd0267SUQsPOVgC528Xram6w0PVLAHgZN/3V4GHR/vwnR0Mur4mcDhs1gmFoSTzxvuKspg2EXpaW43i6p2MmWCFbHGwjaZUlfL4j3b/T4Ysnd0AZFFxN4PGKdsmmg95WALnTRdb3ihgh1j2j7aCqmplC32L5g8UqPrQEIJRLirKH0J9H3DidKDvf6+Yh6s3fhNtTHJsrD7gPElWk9ZF76Eh6sHcqrCrIaVIFSwoyAwnc8YQ+vOzWQeNYWUUVp6cjBEKi52vm+0k2nhfpjGagPbUB5zpJ5ysIlL73HvXpsq2sMLFewY0nPwAL21deQuPRVQO2SwhIrCs2vaKdzZjMsTDNmV/gysMBePybRgKYvZNe1hi6FTPwaomz56UPZK4MRlRlM3ObWdukhmmKOCHSHB/Ou2tVb5zbKlE/zOqR0SO7Jr2geUBHWi12OPxMsCCFYrpfijZgo/bqGtOCPkoiAnGyhSsQbnLBu78gC6SGb4oYIdJX6bFezaQtq4MtJL8hM3oMOcgk9awxLrvgzXgN1vYo1X6OyyOwRI6TV+QukU1cbSY46lHaMkHyrYgyQwna+vo52unZ/0r25UOyQ+hCMwQ5lL3FaaRfFHzY7nw41qY5WtEWplqV17Zfigf60Y0F4Gbdv9Vzd6UTsktgQTGINlgwzGSojXmOyIZ1TbMCkHt81w7FIGdZHM8COqCFtECoHHgElYG999yhjTYNOuD9jkubvXGDNwC5ZhQLD6Ibq6cWhomJRj62EbgYMzhlaowfLUpc84bqNlRzyj2nBWlhqCb4SsJC/RWiLfAV4xxtwuIt/x3P+2TbsOY8zxUV4r6ehP5wtY3ehkh2h0PXh8U9XcKeB2g3jy99ypQv3U3LDEJ5a1OuxWbtrtLh7vbJVAvPbKEWuqHdvE299X4kO0gn0xsNhz+wFgNfaCfVjh619bqxs/xnR0MmbxZL92I60yX7wIFMaUPkv4Io2oY12rwy4jw5tSCFYU216QPqjSsbEgFqtGleQiWsEuNcbs99yuBkod2mWKyAagF7jdGPO0U4cicgtwC0Cma3j4a63bNiNpqeTMmRy6sRIxsarbHKt+fKN0J/YsOrTStT7sngf2H43ID2bVqJLchBRsEXkZKLM59Z++d4wxRkScFpkdYYypFJEpwKsisskY87FdQ2PMCmAFQF7qGKf+hpxA/9pvdeOOLWTOnE5KZrraIXEg3isKoyksZUc0EWwsvwXokvTDj5CCbYxZ6nRORGpEZKwxZr+IjAVqHfqo9PzeJSKrgTmArWAnO4HpfN3VVfTVNVB03cl+x9UOiR2x+mofr8JSvrhd0F6QHtEGDKH6j2aBixZ3OryI1sxaCdzouX0j8ExgAxEpEJEMz+1i4BRga5TXTQray6B16yYQYfQ85y2mNLqODrtUtcF8tY9FP07RuDelsLUkk5zaTlI9GRreCDm7pj2q/nWBiwLRC/btwFkisgNY6rmPiMwVkbs9bY4GNojIRuA1LA97WAl2sHS+lu0fkDFtEmn52bpYJk60lWZRN300vRmuqHKtY9GPUzTuXVmZ1dDtGCFH27+iRDXpaIypA5bYHN8AfM5z+5/AMdFcJ9nw+tfddQfoqaym/PNn+Z1XOyT2xOqrfbT9hJrIizZC1olCJRi6ND0CAtP5Wtd8AEDewiMdH6N2yOFFqIm8aH1ynShUgqGCHQUt2zeSPmk86SX5/XaIRteHP8Gi9FhEyDpRqDihxlgInNL5eurr6P6kgqLFOtmoHCJWfrsy/BGRc0XkIxHZ6VkJHnj+ayKyVUQ+EJFXROQIu3580Qg7TALtkJY3NgKQd+rRiRqSkqRohKyISApwJ3AWUAG8IyIrAxIu/gXMNca0i8j/A/4HuCpYvxphD5Lmbf8ifdJ4MsoKbO0Qja4VZUQzD9hpjNlljOkGHsUq5dGPMeY1Y4w333MdMD5UpxphB8HJDuk+eICevZWM/azjmiJFUYYZKd19ZO9tC7d5safchpcVnlXaXsYB+3zuVwDzg/R3M/B8qIuqYIfBADvktX+BCPmLZvYf18lGRRlRHDTGzI1FRyLyaWAucHqotirYEWKMoXnLu2RMn0R68WjbxTJqhyjKiKcS8N3cdbznmB8ishSrLtPpxpiuUJ2qhx0mXjukq6qC3uoDjDlbJxsVRXHkHWC6iEwWkXTgaqxSHv2IyBzgLuAiY4xtHaZAVLAd8PrXA+yQje9Cagp5px6tk42KothijOkFlgMvAtuAx40xW0TkRyLi3XHrF0AO8ISIvC8iKx2660ctkQgwfX00b3qPUcceTWrOKGDAbmiKoigAGGNWAasCjn3P53bEWQsaYYeB1w5p2/Eh7pZWSs+3n2zU6FpRlHiigm2Dkx3SuHU9rtxsRp84VSvzKYoy5Khgh0lvWysdG7dRtGQ2kpoy4LxG14qixBsV7BB47ZCW9zdAXx+FZx2vhZ4URUkIKtgB2NkhbaWGxvfXkT5lIplHjEnU0BRFGeGoYAfBG1137tlN7/5aSi44tv+cTjYqijLUqGCHoL0Mmt75JzIqg/xFM3WyUVGUhBGVYIvIlSKyRUTcIuK4rj5UXdhkwc4O6W1rpWXLRgqXHENKZjqg0bWiKIkh2gh7M3AZsMapgU9d2GXATOAaEZnp1D5Z8NohzRvWQ28fxeedqNG1oigJJSrBNsZsM8Z8FKJZyLqwyYDtZOOYPhrffZOcY48gc6I12ajRtaIoiWIoPGy7urDjHNomFW0fbaGvvpHiC07S6FpRlIQTspaIiLwMlNmc+k9jzDOxHpCI3ALcApDpCn/j0ljSWp5CexnUv7SGlKICRs+fAVRrdK0oSkIJKdiDKVASQFh1YX2utwJYAZCXOsZEee2wsLNDOqsq6Nq+i7GfXcLxZdVDMQxFUZSgDIUlErIubDLhja4b33wdyUin6Ow5A9podK0oSiKINq3vUhGpABYCz4nIi57j5SKyCpzrwkY37NhhF133NDXSsvlfFJ1zPCdMrgd0GbqiKIknqnrYxpingKdsjlcB5/ncH1AXNhnxpvI1vrUGDIy5eB7Q7tdGo2tFURKFbmAQQEteB00b3iL/tKM5abYl1t7oWsVaUQ5junqQ3Y7Ta0nBiF6a7muHeKPrpvVrMZ1dlFy+EFArRFGU5GFEC3YgrYXdNKxbQ+6JU1kw3+13TqNrRVESzYgVbLvJxqYNb+FubaPkU6cAGl0ripJcjFjB9qW1PIXWoh4a/vka2bMncspif2tfo2tFUZKBESnYdtF187vr6WtspvSa0wCNrhVFST5GpGD74o2u6998meyZEzhlSbrfeY2uFUVJFkasYPtF1xvW0dfQTOm1ixARja4VRUlKRpxge+0Q8ETXhd3UrX3Z8q6XanStKEryMuIEGwIyQ9avxd3UQtmnF/tF1yrWiqIkGyNKsAOj65b8DurXvkruiVM55Qxd9KkoSnIzogQb/KPrxrWrcbe1U3b9YkCXoCuKktyMGMEOjK6bs5tp+Odq8k6bycKFRicaFUVJekaMYIN/dF3/2kuYvj7GeqJrLxpdK4qSrIwIwQ6MrhtTamjasI6iZScw77hOtUIURRkWjAjBBv/ounb1SlwZqSxZPj2BI1IURYmMw16wA6Prg2076Ni4jZJPnUJmwSiNrhVFGTYc9oINh+pdt5W4qX35adJK8lhy86F9gVWsFUUZDhzWgu0bXQM0v/c2Pfv2M/amM0nJSNXMEEVRhhXRbsJ7pYhsERG3iMwN0u4TEdkkIu+LyIZorhkp3ui6Jb+Dg688R/bMCZx+Wb5aIYqixBUROVdEPhKRnSLyHZvzGSLymOf8ehGZFKrPaCPszcBlwJow2p5hjDneGOMo7LEkMLquf/Ul3K1tLP7WHERkKIagKMoIRURSgDuBZcBM4BoRmRnQ7GagwRgzDfg18PNQ/UYl2MaYbcaYj6LpI574etcddfsoPOt4io4u1uhaUZR4Mw/YaYzZZYzpBh4FLg5oczHwgOf2k8ASCRFNijEm6pGJyGrgG8YYW7tDRHYDDYAB7jLGrAjS1y3ALZ67s7Gi+ERSDBxM8BggOcaRDGOA5BhHMowBdBxejjDGjImmAxF5Aet5hEMm0Olzf4WvronIFcC5xpjPee5fD8w3xiz3abPZ06bCc/9jTxvH1zFkxSMReRkoszn1n8aYZ0I93sOpxphKESkB/iEiHxpjbG0Uz5Ne4bn2hqGyUJxIhjEkyziSYQzJMo5kGIOOI7YYY85N9BhCEVKwjTFLo72IMabS87tWRJ7C+roQju+tKIoyHKkEJvjcH+85ZtemQkRSgTygLlincU/rE5FsEcn13gbOJvE2h6IoSjx5B5guIpNFJB24GlgZ0GYlcKPn9hXAqyaERx1tWt+lIlIBLASeE5EXPcfLRWSVp1kpsFZENgJvA88ZY14I8xKOXvcQkgxjgOQYRzKMAZJjHMkwBtBxJCXGmF5gOfAisA143BizRUR+JCIXeZrdAxSJyE7ga8CA1L9AYjLpqCiKosSfw3qlo6IoyuGECraiKMowIWkEO1mWuUcwjqDLTmMwjkIR+YeI7PD8LnBo1+d5Ld4XkcBJjcFeO+ZLauM0jptE5IDP8/9cHMZwr4jUenJm7c6LiPzWM8YPROSEBIxhsYg0+bwO34vDGCaIyGsistXz//FlmzZxfy1GPMaYpPgBjgaOBFYDc4O0+wQoTuQ4gBTgY2AKkA5sBGbGeBz/A3zHc/s7wM8d2rXG+Lohnxvwb8AfPbevBh6Lw98hnHHcBNwR5/flIuAEYLPD+fOA5wEBFgDrEzCGxcDf4/w6jAVO8NzOBbbb/D3i/lqM9J+kibBNkixzD3Mc4Sw7jRbfZasPAJfEuH8n4rKkNk7jiDvGWuBVH6TJxcCDxmIdkC8iY4d4DHHHGLPfGPOe53YLVubDuIBmcX8tRjpJI9gRYICXRORdzzL2RDAO2Odzv4KBb95oKTXG7PfcrsZKj7QjU0Q2iMg6EbkkBtcN57n1tzFW+lITUBSDa0c6DoDLPV+/nxSRCTbn481QvBfCYaGIbBSR50VkVjwv5LHA5gDrA04ly2tx2BJypWMsGepl7nEeR9QEG4fvHWOMERGn/MsjPK/HFOBVEdlkjPk41mNNUp4FHjHGdInIF7Ci/jMTPKZE8B7W+6BVRM4Dngbisv+diOQAfwW+Yoxpjsc1FGeGVLBNkixzj8E4wll2GtU4RKRGRMYaY/Z7vlbWOvThfT12iVWEaw6W9ztY4rKkNh7jMMb4XvNuLN9/qInJeyEafIXTGLNKRH4vIsUmSBGhwSAiaVhi/bAx5m82TRL+WhzuDCtLRJJnmXs4y06jxXfZ6o3AgMhfRApEJMNzuxg4Bdga5XXjsqQ2HuMI8EcvwvJVh5qVwA2eDIkFQJOPlTUkiEiZdw5BROZh/V/H9APU0/89wDZjzK8cmiX8tTjsSfSsp/cHuBTL8+oCaoAXPcfLgVWe21OwsgU2AluwLIwhH4fn/nlYM+Ufx2kcRcArwA7gZaDQc3wucLfn9snAJs/rsQm4OUbXHvDcgB8BF3luZwJPADuxyg1MidN7ItQ4fuZ5H2wEXgOOisMYHgH2Az2e98XNwK3ArZ7zglWo/mPP38AxwymOY1ju8zqsA06OwxhOxZo/+gB43/Nz3lC/FiP9R5emK4qiDBOGlSWiKIoyklHBVhRFGSaoYCuKogwTVLAVRVGGCSrYiqIowwQVbEVRlGGCCraiKMow4f8DacJS4eRkx+0AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Generate Features from Data\n", "\n", "# change this to play arround with feature functions\n", "#feature_fn = affine_features\n", "#feature_fn = quad_features\n", "feature_fn = cubic_features\n", "features = feature_fn(samples)\n", "\n", "num_iterations = 25\n", "\n", "w_bce, l, l_info = minimize_with_sgd(features, labels, np.zeros(features.shape[1]),\n", " schedule=(lambda t: 0.25),\n", " num_iterations=num_iterations,\n", " batch_size=1)\n", "print(\"Final loss\", l)\n", "\n", "plt.figure()\n", "plt.title(\"Cross Entropy Loss\")\n", "plt.grid(\"on\")\n", "plt.xlabel(\"Update Steps\")\n", "plt.ylabel(\"Negative Bernoulli Log-Likelihood\")\n", "plt.semilogy(l_info[1])\n", "\n", "plt.figure()\n", "plt.title(\"Bernoulli LL Solution\")\n", "pred_grid = np.reshape(sigmoid(feature_fn(flat_plt_grid) @ w_bce), plt_grid_shape)\n", "\n", "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], pred_grid, levels=10)\n", "plt.colorbar()\n", "#This is just a very hacky way to get a black line at the decision boundary: \n", "plt.contour(plt_grid[..., 0], plt_grid[..., 1], pred_grid, levels=[0, 0.5], colors=[\"k\"])\n", "\n", "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n", "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n", "plt.legend([s0, s1], [\"c0\", \"c1\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2) Effect of different Batch Sizes and Number of Iterations (1. Point)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Play around with the batch size and number of iterations and briefly describe your observations about convergence speed and monotonicity of the loss curve." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.11" } }, "nbformat": 4, "nbformat_minor": 1 }