Exercise_1/exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EXERCISE 1 - ML - Grundverfahren WS 21/22\n",
    "\n",
    "**Exercise 1**: Ge Li ge.li@kit.edu\n",
    "\n",
    "**Exercise 2 & 3**: Philipp Becker philipp.becker@kit.edu\n",
    "## Submission Instructions\n",
    "Please follow the instruction from Exercise ZERO!\n",
    "\n",
    "\n",
    "## 1.) Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1) Matrix Vector Calculus (1 Point)\n",
    "Given the following element-wise expression of a matrix-vector product,\n",
    "rewrite it in matrix form:\n",
    "\n",
    "\\begin{align*}\n",
    "        g &= \\alpha \\sum_i \\sum_j \\sum_k z_k x_{ij} q_i y_{jk}\\\\\n",
    "        &= \\alpha \\sum_i^I \\sum_j^J \\sum_k^K z_k x_{ij} q_i y_{jk}\\qquad (\\text{mit }I,J,K\\in \\mathbb{N})\\\\\n",
    "        &= Y\\cdot X \\cdot Q \\cdot Z \\cdot \\alpha\\\\\n",
    "        &= \\begin{bmatrix} y_{11} & \\cdots & y_{1J}\\\\ \\vdots & \\ddots & \\vdots\\\\ y_{K1} & \\cdots & y_{KJ}\\end{bmatrix} \\cdot\n",
    "        \\begin{bmatrix} x_{11} & \\cdots & x_{1I}\\\\ \\vdots & \\ddots & \\vdots\\\\ x_{J1} & \\cdots & x_{JI}\\end{bmatrix} \\cdot\n",
    "        \\begin{bmatrix} q_1 \\\\ \\vdots \\\\ q_I\\end{bmatrix} \\cdot\n",
    "        \\begin{bmatrix} z_1 & \\cdots & z_K\\end{bmatrix} \\cdot\n",
    "        \\alpha\n",
    "\\end{align*}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2) Derive Ridge Regression Weights (4 Points)\n",
    "Derive the optimal solution of weights in Ridge Regression using matrix form, i\n",
    ".e. $\\boldsymbol{w}= ?$\n",
    "\n",
    "Hint: You will need derivatives for vectors/matrices. Start\n",
    "from the matrix objective for ridge regression as stated here\n",
    "\n",
    "\\begin{align*}\n",
    "L &= (\\boldsymbol{y}-\\boldsymbol{\\Phi} \\boldsymbol{w})^T(\\boldsymbol{y}-\\boldsymbol{\\Phi} \\boldsymbol{w}) + \\lambda \\boldsymbol{w}^T \\boldsymbol{I} \\boldsymbol{w}. \\\\\n",
    "\\end{align*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ L = (y-\\Phi w)^T (y - \\Phi w) + \\lambda w^T I w $$\n",
    "$$ = w ^T \\Phi^T\\Phi w - y^T \\Phi w - w^T\\Phi^Ty + y^Ty + \\lambda w^T I w $$\n",
    "$$ = w^T\\Phi^T\\Phi w - 2 y^T\\Phi w + y^Ty + \\lambda w^T I w $$\n",
    "$$ = y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w $$\n",
    "$$ \\frac{\\delta x^TAx}{\\delta x} = (A + A^T)x:  \\frac{\\delta w^T(\\Phi^T\\Phi + \\lambda I)w}{\\delta w} = ((\\Phi^T\\Phi + \\lambda I) + (\\Phi\\Phi^T + \\lambda I^T))w = 2(\\Phi^T\\Phi + \\lambda I)w $$\n",
    "$$ \\frac{\\delta y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w }{\\delta w} = -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w $$\n",
    "$$ -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w = 0, (\\text{to find optimum}) $$\n",
    "$$ (\\Phi^T\\Phi + \\lambda I)w = \\Phi^Ty $$\n",
    "$$ w^* = (\\Phi^T\\Phi + \\lambda I )^{-1} \\Phi^Ty $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ridge Regression - Code\n",
    "Let's first get the data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db55d74c0>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEGCAYAAACZ0MnKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhBElEQVR4nO3df3icZZ3v8fc34UdJ7TlC2hUOJZOuF+xFf2hrYxF7uopWqIAgHBFLqLBCoy519ZKDFsqxyLVhzzno4sURwdZFOW0sAtoVXV3a7rGsP4A1LeU3SEuTkBYhTQVKI4U23/PH80x4MplJJsnMPM/MfF7XNVdm7ueZmW+Sdr657+9934+5OyIiIvmoiTsAEREpH0oaIiKSNyUNERHJm5KGiIjkTUlDRETydljcARTb5MmTvbGxMe4wRETKypYtW/a4+5TM9opPGo2NjbS3t8cdhohIWTGzzmztGp4SEZG8KWmIiEjelDRERCRvFV/TyObNN9+ku7ub119/Pe5Qqt6ECROYOnUqhx9+eNyhiEgeqjJpdHd3M2nSJBobGzGzuMOpWu5Ob28v3d3dTJs2Le5wRCQPiRyeMrO/MrNtkdurZvYlM7vOzHZF2s8cy+u//vrr1NfXK2HEzMyor69Xj0+kkNraoLERamqCr21tBX35RPY03P0ZYDaAmdUCu4D1wN8AN7n7N8b7HkoYyaDfg0gBtbVBSwv09QWPOzuDxwDNzQV5i0T2NDJ8GNjh7lnnDIuISGjFircSRlpfX9BeIOWQND4FrIs8XmZmj5rZ7WZ2dLYnmFmLmbWbWXtPT09pohQRiVtX1+jaxyDRScPMjgDOAe4Om24F3kkwdPUC8M1sz3P3Ve7e5O5NU6YMWQU/egUeI3z55Zf5zne+M6bnfutb36Iv8y+JYfzgBz9g2bJlw56zefNmfve7340pHhFJkIaG0bWPQaKTBvBRYKu7vwjg7i+6+yF37wdWA/OKHkF6jLCzE9zfGiMcR+IoZdLIh5KGSIVobYW6usFtdXVBe4EkPWksJjI0ZWbHRY6dBzxe9AiKMEa4fPlyduzYwezZs7nqqqu48cYbee9738u73vUuVq5cCcD+/fs566yzePe7383MmTP50Y9+xM0338zu3bs57bTTOO2003K+/ve//31OOukk5s2bx29/+9uB9p/97GeccsopzJkzh4ULF/Liiy/S0dHBbbfdxk033cTs2bP59a9/nfU8EUmw9GjIkiVw1FFQXw9mkErBqlUFK4IDwVz5JN6AiUAv8J8jbWuAx4BHgXuB40Z6nblz53qmJ598ckhbTmbuQR9j8M0s/9fIsHPnTp8xY4a7u993332+dOlS7+/v90OHDvlZZ53l999/v99zzz1++eWXDzzn5Zdfdnf3VCrlPT09OV979+7dfsIJJ/hLL73kBw4c8Pe///1+xRVXuLv73r17vb+/393dV69e7V/+8pfd3X3lypV+4403DrxGrvOKZVS/DxEZbO1a97q6wZ9PdXVB+zgA7Z7lMzWRU24B3H0/UJ/RtqTkgTQ0BENS2doLYMOGDWzYsIE5c+YA8Nprr/Hss8+yYMECrrzySr761a9y9tlns2DBgrxe76GHHuKDH/wg6VrOhRdeyB/+8AcgWNR44YUX8sILL/DGG2/kXFCX73kikgDDjYYUsocRSvrwVPyKPEbo7lx99dVs27aNbdu2sX37di677DJOOukktm7dyqxZs7j22mu5/vrrx/1eX/jCF1i2bBmPPfYY3/3ud3Muqsv3PBFJgBLMmIpS0hhJc3MwJphKFWyMcNKkSezbtw+AM844g9tvv53XXnsNgF27dvHSSy+xe/du6urquPjii7nqqqvYunXrkOdmc8opp3D//ffT29vLm2++yd133z1w7JVXXuH4448H4I477sgaz3DniUgClWDGVJSSRj6am6GjA/r7g6/j7PLV19czf/58Zs6cycaNG7nooos49dRTmTVrFp/4xCfYt28fjz32GPPmzWP27Nl8/etf59prrwWgpaWFRYsW5SyEH3fccVx33XWceuqpzJ8/n5NPPnng2HXXXccFF1zA3LlzmTx58kD7xz72MdavXz9QCM91nogkUAlmTEVZUO+oXE1NTZ555b6nnnpq0IepxEu/D5ExaGsL6hZdXXDMMUHb3r1BD6O1ddx/3JrZFndvymxPbCFcRERyyNxjqrc36F2sWVOU4neUkkYZO+WUUzhw4MCgtjVr1jBr1qyYIhKRkijxjKkoJY0y9tBDD8UdgojEocQzpqJUCBcRKTclnjEVpaQhIlIu0tuFdHYGSwCiijhjKkpJQ0SkHEQ3T4Vgw5B04ijGHlM5KGnEYDy73J555pm8/PLLw57zta99jU2bNo3p9YejbdZFYpSt+O0eJIwCrB/Ll5JGHgp9yd3hksbBgweHfe4vfvEL3v72tw97zvXXX8/ChQvHGt64KGmIFEmMxe8oJY0RFOFyGkO2Rt+8eTMLFizgnHPOYfr06QB8/OMfZ+7cucyYMYNVq1YNPLexsZE9e/bQ0dHBySefzNKlS5kxYwann346f/7znwG49NJLueeeewbOX7lyJe95z3uYNWsWTz/9NAA9PT185CMfYcaMGVx++eWkUin27NkzJFZtsy6SEDEWvwfJtvVtJd3GuzV6KpV9Z/RUKu+XGCK6Nbq7+69+9Suvq6vz5557bqCtt7fX3d37+vp8xowZvmfPnjCeYGv0nTt3em1trT/88MPu7n7BBRf4mjVr3N39kksu8bvvvnvg/Jtvvtnd3W+55Ra/7LLL3N39iiuu8BtuuMHd3X/5y186MGTL9VJts66t0UWGsXbtWx9EmZdqKMAW6LlQblujJ0WpeoTz5s0btAX5zTffzPr16wF4/vnnefbZZ6mvH7RTPNOmTWP27NkAzJ07l46Ojqyvff755w+c85Of/ASA3/zmNwOvv2jRIo4+eujl1rXNukjMMld+p4vf6VpGAbYLGS0NT42gVD3CiRMnDtzfvHkzmzZt4oEHHuCRRx5hzpw5WbcnP/LIIwfu19bW5qyHpM8b7pzR0jbrIiWQkOJ3VGKThpl1mNljZrbNzNrDtmPMbKOZPRt+HfrncYEVYwPJkbY3f+WVVzj66KOpq6vj6aef5sEHHxz7m+Uwf/587rrrLiC4ENSf/vSnIedom3WRmCWk+B2V2KQROs3dZ/tbOy0uB/7N3U8E/i18XFRFuJzGoK3Rr7rqqiHHFy1axMGDBzn55JNZvnw573vf+8bxHWS3cuVKNmzYwMyZM7n77rs59thjmTRp0qBztM26SMySUvyOSOzW6GbWATS5+55I2zPAB939BTM7Dtjs7n813Otoa/TsDhw4QG1tLYcddhgPPPAAn//859m2bVsssej3IZIhve15euV39HO6rq4kC/nKcWt0BzaYmQPfdfdVwDvc/YXw+B+Bd2R7opm1AC0ADTFm5CTr6urik5/8JP39/RxxxBGsXr067pBEBBJZ/I5KctL4r+6+y8z+AthoZk9HD7q7hwlliDDBrIKgp1H8UMvPiSeeyMMPPxx3GCKSaaTid8wSW9Nw913h15eA9cA84MVwWIrw60vjeP1ChCnjpN+DSIYEFr+jEpk0zGyimU1K3wdOBx4H7gUuCU+7BPjpWF5/woQJ9Pb26gMrZu5Ob28vEyZMiDsUkfil9yvK9bmUkKH2pA5PvQNYb8EOjocBP3T3fzWz3wN3mdllQCfwybG8+NSpU+nu7qanp6dgAcvYTJgwgalTp8Ydhki8MusYmUq07Xk+Ejt7qlCyzZ4SEUmU9DUysomp+F2Os6dERKpDrnqFWSKK31GJrGmIiFSVBC7iy0VJQ0QkLgm4fOtoKWmIiMQhIZdvHS3VNERE4pDwRXy5qKchIhKHhC/iy0VJQ0SkwNKlipqa4Ougy0OXySK+XDQ8JSJSQJnr9Do7g8cAzZTPIr5ctLhPRKSAcq3TS6WggxwH0yfEvINtlBb3iYiUQM5SRWc/we5HWSRwEV8uqmmIiBRQznV6DFPgTngdI0pJQ0SkgM48M8s6PfbTyjXZn1AGdYwoJQ0RkQJpa4M77hg8Mcro5xK+TzPrhj4hwYv4clFNQ0SkQLKu16OGX3A28IXBBxK+iC8X9TRERAokZxGcjJpFmQ1JRSlpiIgUSF5F8DIckopKZNIwsxPM7Fdm9qSZPWFmXwzbrzOzXWa2LbydGXesIiIAtLXR+trfUcf+Qc2DiuDpqbVlmjAguTWNg8CV7r41vFb4FjPbGB67yd2/EWNsIiKDhcvAm/v6gD2s4Aa6aKCBLlq55q0ieBlNrc0lkUnD3V8AXgjv7zOzp4Dj441KRCSHSAW8mXXZZ0qVcR0jKpHDU1Fm1gjMAR4Km5aZ2aNmdruZHZ3jOS1m1m5m7T09PaUKVUSq1XA705qVfR0jKtFJw8zeBvwY+JK7vwrcCrwTmE3QE/lmtue5+yp3b3L3pilTppQqXBGpNiPtWJtKQX9/2dcxohI5PAVgZocTJIw2d/8JgLu/GDm+Gvh5TOGJSLXL3M42U4UMR2VKZE/DzAz4J+Apd//HSPtxkdPOAx4vdWwiUuXSvYuLL86dMCpoOCpTUnsa84ElwGNmti1suwZYbGazAQc6gM/GEZyIVKmRehdQVjvWjkUik4a7/wawLId+UepYREQGZNsnJFMFTKsdTiKHp0REEiU9JJXrAkppFVrHiFLSEBEZTnpIaqSEUcF1jKhEDk+JiCTGSENSdXVVkSzS1NMQEckmnyGpKuldRKmnISKSKZ9ZUmV6PYzxUk9DRCQtnzUYUBUF71yUNEREQAXvPGl4SkSqW1tbUOweKVlA1Q5JRSlpiEj1yqd2kVbFQ1JRGp4SkeqTb+0ircqHpKLU0xCR6jLa3oWSxSDqaYhIdVDvoiCUNKT6pD88ampg8uTglnm/sTE4r4ShlOgtq1O+M6Mg6F2sXVtRF04qJCUNKX/5JIH0fTNYsiT48HCH3t7glnm/szM4z2zk1xzjJ370cyz9li0tShwFpd5FwZnnukxhhWhqavL29va4w5DxSk+L7OqCY44J2vbuDe7v2wdvvBFvfBAkGHeorw8e790bbJPd2pr1QyjXDhWa1TlO0Sm06d/JSFS7GMLMtrh7U2Z72fU0zGyRmT1jZtvNbHnc8UgRpf9KHKl3kISEAW99OOXqsTQ2wt/+7UCvqKuzP+vLdHWVLOLKkzkMlU/CUO9iVMoqaZhZLXAL8FFgOsGV/KbHG5UUVLZEAfn950+qdOydnXDrrQPJr4Hs2aGhpltjVKM12mEoUO1ijMoqaQDzgO3u/py7vwHcCZwbc0wyXpWYKPLQyjXUsX9QWx37aT30lez1FFXKB8v17yYf6l2MWbkljeOB5yOPu8O2Qcysxczazay9p6enZMHJKMSdKCy8mnB9fXAze+t+9HgRNbOOVSwlRQdGPyk6WMVSmlk37FBXm11EY+3z1Fg/jdZB2+S/q45kkjnh4TOfGf2/G/Uuxs/dy+YGfAL4XuTxEuDbwz1n7ty5Lgmxdq17KuUO7mbB10LcDj/cvb4+eM36+pHvp1JBLPnEms9rFvr7Gea2lsVex2uDmut4zddyUfBgtN9nEmX72Y/nZ5x+Xrn+PGICtHu2z+FsjUm9AacC90UeXw1cPdxzlDQSYu1a97q68X1opv/zJ/GDsdAfdDluKXZmPZRiZ/n9zKKK9QdFEr/XMlEpSeMw4DlgGnAE8AgwY7jnKGnELPphUOZ/JUbzQt7hZD7p858f14ejcSj7j4lDY/u5ZiaTdHxj6Z3l8zMoVU+tri72fy/lriKSRvB9cCbwB2AHsGKk85U0YjDevxoTlCjSsnWUxv25NIbeyah7GoW8ZUs0CRq6S+K/m3JWMUljtDcljRIb6zBUwv/D5+ospVJFeLNhkknumsbi0nwwj/O2lsWeYqcbhzzFzvHFHa1lJfTfTTlT0pDiGsswVMITRVSuP5bNShhE+DNey0WequmKfPBeFHsyyDdhjDvhldG/mXKnpCHFM8rexVoWe6r2eTf6y+b/fkl7GqNVoiL8eG+jGlorlwJ+BVPSkMIbQ+9i7eGXet0Rbw7+a7MMapZFqWkUW8KSyYhFfPUiEkVJQwprNL2LyIdBqn5fcv9iH8GYZk8lUa7ZTLlmTxUi0ZgN39Mo6x9oZcqVNLTLrYxOdAfRfKRSg3Z5rakJPioymUF/9v37JAmG22V4pPvhTr9tNA+5YJ42l02uXLvc6nKvkr8CXCazoSF7vmloKFCMUhzNzeP+ZE8/O517htk1XhKs3PaekjgU8EI2ra1BPomqqwvapfI1NwfbPvX3a/uncqWehgyvAL2LqPQh/bUpUp5U05Dh5bq8XKaM2oWIlLeKuXKflEB0C+qREoa2mhapKhqeksFGMxyl3oVI1VHSkMFWrBg5YWiepEjV0vCUBMIhqbbO99PITmo4RCM7aWPxW+eY6TKZIlVOPQ0ZGJJq6zuXFlbTx0QAOmmkhdUANKd+F9QtRKSqqadRzTLWX6zghoGEkdbHRFbY/9RCChEB1NOoXlkK3l1kX5bd5SdoOEpEgAT2NMzsRjN72sweNbP1Zvb2sL3RzP5sZtvC220xh1reshS8G+jKempDykoRkYiUgcQlDWAjMNPd30VwWderI8d2uPvs8Pa5eMIrc+khqSzrL1q5hjr2D2rTFh8iEjVi0jCzL5jZ0aUIBsDdN7j7wfDhg8DUUr13xUsPSeVYsNfMOlaxlFRtN4ZropSIDJFPT+MdwO/N7C4zW2RmpRyr+Azwy8jjaWb2sJndb2YLcj3JzFrMrN3M2nt6eoofZbkYaQ1GXR3Na8+i4+BU+t20yFtEhshr76kwUZwO/A3QBNwF/JO77xjTm5ptAo7NcmiFu/80PGdF+F7nu7ub2ZHA29y918zmAv8MzHD3V4d7L+09FZHrYhag1d0iMsi4rqcRfmj/EfgjcBA4GrjHzDa6+1dGG4y7Lxwh2EuBs4EPh1eQwt0PAAfC+1vMbAdwEqCMMJL0BXSGSxhagyEieRgxaZjZF4FPA3uA7wFXufubZlYDPAuMOmmM8H6Lwtf8gLv3RdqnAHvd/ZCZ/SVwIvBcId+7Io20l5Qq3SIyCvn0NI4hGCIaVD11934zO7sIMX0bOBLYGJZPHgxnSv01cL2ZvQn0A59z971FeP/KMlwdQ0NSIjJKup5GpRrpWt66KLeIDEPXCK8m+Wxvrotyi8gYJHFxn4xXHlNrVccQkbFQ0qhEXdm3AwG0tbmIjIuSRiVJbxEy0tRaJQwRGSPVNCqFptaKSAmop1EpRppaqyEpESkA9TQqRa46hplWe4tIwainUe5GqmNoaq2IFJB6GuVMdQwRKTH1NMqZ6hgiUmLqaZQz1TFEpMTU0yhHqmOISEzU0yg3qmOISIzU0ygjbW3QeMkHqOnbRyM7aWPx4BNUxxCRIlNPo0wMdDAOTQWgk0ZaWA1AM+tUxxCRklBPo0xkmyjVx0RWcEPwQHUMESmBxCUNM7vOzHaZ2bbwdmbk2NVmtt3MnjGzM+KMs9S6OrMXvbtoUB1DREomqcNTN7n7N6INZjYd+BQwA/gvwCYzO8ndD8URYEm1tdFgC+j0ob2JhtrdqmOISMkkrqcxjHOBO939gLvvBLYD82KOqTRWrKDVl1PH/kHNddZH6x1TlTBEpGSSmjSWmdmjZna7mR0dth0PPB85pztsG8LMWsys3czae3p6ih1r8XV10cw6VrGUFB0Y/aToYJUvVb4QkZKKJWmY2SYzezzL7VzgVuCdwGzgBeCbo319d1/l7k3u3jRlypTCBl9KGYv4mllHB9Pop5YOptGc+m288YlI1YmlpuHuC/M5z8xWAz8PH+4CTogcnhq2VSYt4hORBErc8JSZHRd5eB7weHj/XuBTZnakmU0DTgT+o9TxlYw2IxSRBEri7Kn/bWazAQc6gM8CuPsTZnYX8CRwELiiomdOaTNCEUmgxCUNd18yzLFWoDrGZBoaoLMze7uISEwSNzxV9dLF787OoFcRpTqGiMRMSSNJ0sXvdA/D/a3EoTqGiCRA4oanqlq24rd7kDBUxxCRBFBPI0lyFb9ztYuIlJiSRpLkKnKr+C0iCaGkkQQqfotImVDSiJuK3yJSRlQIj5uK3yJSRtTTiJuK3yJSRpQ04qbit4iUESWNuLW2BsXuKBW/RSShlDTikp4xtWQJHHUU1NcHBXAVv0UkwVQIj0PmtTJ6e4PexZo1ShYikmjqacQh24ypvr6gXUQkwZQ04qAZUyJSphKXNMzsR2a2Lbx1mNm2sL3RzP4cOXZbzKGOnWZMiUiZSlxNw90vTN83s28Cr0QO73D32SUPqtBaW4de/1szpkSkDCSup5FmZgZ8ElgXdywFkZ4tVVMT1C4uuSSYKaUZUyJSRhLX04hYALzo7s9G2qaZ2cPAq8C17v7rbE80sxagBaAhCUM+mbOlOjvhjjuUKESk7Ji7l/5NzTYBx2Y5tMLdfxqecyuw3d2/GT4+Enibu/ea2Vzgn4EZ7v7qcO/V1NTk7e3tBY1/1NI72GbS/lIiklBmtsXdmzLbY+lpuPvC4Y6b2WHA+cDcyHMOAAfC+1vMbAdwEhBzRsiDZkuJSIVIak1jIfC0u3enG8xsipnVhvf/EjgReC6m+EZHs6VEpEIkNWl8iqEF8L8GHg2n4N4DfM7d95Y6sDHR/lIiUiESWQh390uztP0Y+HHpoxmHtrZgplRXFxxzTLDH1N69QQ+jtVVFcBEpO4lMGhVB+0uJSAVK6vBU+dP+UiJSgZQ0ikUzpkSkAilpFItmTIlIBVLSKBbNmBKRCqSkUWi6Ip+IVDDNniokzZgSkQqnnkYhhTOm2lhMIzup4RCNfU/Q9sWH4o5MRKQglDQKqauLNhbTwmo6acSpoZNGWnr/gba2uIMTERk/JY1CamhgBTfQx8RBzX1M1PIMEakIShqF1NpKF9mn1Gp5hohUAiWNQojMmGqo2ZX1FC3PEJFKoKQxXukZU52d4E5r/1epY/+gU7Q8Q0QqhZLGeGXsMdXMOlaxlFRtt5ZniEjF0TqN8cpSrGhmHc39d0J/fwwBiYgUj3oa46U9pkSkisSWNMzsAjN7wsz6zawp49jVZrbdzJ4xszMi7YvCtu1mtrz0UWehPaZEpIrE2dN4HDgf+Pdoo5lNJ7jc6wxgEfAdM6sNrw9+C/BRYDqwODw3Xs3NQdEildIeUyJS8WKrabj7UwBmlnnoXOBOdz8A7DSz7cC88Nh2d38ufN6d4blPlibiDNFLueryrSJSJZJY0zgeeD7yuDtsy9U+hJm1mFm7mbX39PQUPsKMabZ0dgaPtVeIiFS4oiYNM9tkZo9nuZ1bzPd191Xu3uTuTVOmTCn8G+hSriJSpYo6POXuC8fwtF3ACZHHU8M2hmkvLV3KVUSqVBKHp+4FPmVmR5rZNOBE4D+A3wMnmtk0MzuCoFh+bywRapqtiFSpOKfcnmdm3cCpwL+Y2X0A7v4EcBdBgftfgSvc/ZC7HwSWAfcBTwF3heeWnqbZikiVMnePO4aiampq8vb29sK/sGZPiUgFM7Mt7t6U2a5tRMaquVlJQkSqThJrGsmV3gK9pib4qim2IlJl1NPIV3ptRnqqbXptBqjHISJVQz2NfGlthoiIkkbetDZDRERJI29amyEioqSRN63NEBFR0sibtkAXEdHsqRFpEZ+IyAAljeFomq2IyCAanhqOptmKiAyipDEcTbMVERlESWM4mmYrIjKIksZwNM1WRGQQJY3haJqtiMggmj01Em2BLiIyIJaehpldYGZPmFm/mTVF2j9iZlvM7LHw64cixzab2TNmti28/UXRAtQW6CIiWcXV03gcOB/4bkb7HuBj7r7bzGYSXNr1+MjxZncvwmX4IrQ2Q0Qkp1h6Gu7+lLs/k6X9YXffHT58AjjKzI4saXBamyEiklOSC+H/Ddjq7gcibd8Ph6b+h5lZrieaWYuZtZtZe09Pz+jeVWszRERyKlrSMLNNZvZ4ltu5eTx3BvC/gM9GmpvdfRawILwtyfV8d1/l7k3u3jRlypTRBa61GSIiORWtpuHuC8fyPDObCqwHPu3uOyKvtyv8us/MfgjMA/5vIWIdpLV1cE0DtDZDRCSUqOEpM3s78C/Acnf/baT9MDObHN4/HDiboJheeFqbISKSk7l76d/U7Dzg/wBTgJeBbe5+hpldC1wNPBs5/XRgP/DvwOFALbAJ+LK7HxrpvZqamry9vbgTrkREKo2ZbXH3piHtcSSNUlLSEBEZvVxJI1HDUyIikmxKGiIikjclDRERyZuShoiI5K3iC+Fm1gN0xh3HKE0m2Iermuh7rg76nstHyt2HrI6u+KRRjsysPdushUqm77k66HsufxqeEhGRvClpiIhI3pQ0kmlV3AHEQN9zddD3XOZU0xARkbyppyEiInlT0hARkbwpaSScmV1pZp7eGr6SmdmNZva0mT1qZuvDrfIrkpktMrNnzGy7mS2PO55iM7MTzOxXZvakmT1hZl+MO6ZSMLNaM3vYzH4edyyFoqSRYGZ2AsHW8NVyrdmNwEx3fxfwB4Jt8iuOmdUCtwAfBaYDi81serxRFd1B4Ep3nw68D7iiCr5ngC8CT8UdRCEpaSTbTcBXgKqYreDuG9z9YPjwQWBqnPEU0Txgu7s/5+5vAHcCI14GuZy5+wvuvjW8v4/gg/T4eKMqrvAqpGcB34s7lkJS0kio8Frqu9z9kbhjiclngF/GHUSRHA88H3ncTYV/gEaZWSMwB3go5lCK7VsEf/T1xxxHQRXtGuEyMjPbBByb5dAK4BqCoamKMtz37O4/Dc9ZQTCc0VbK2KT4zOxtwI+BL7n7q3HHUyxmdjbwkrtvMbMPxhxOQSlpxMjdF2ZrN7NZwDTgETODYJhmq5nNc/c/ljDEgsv1PaeZ2aUE14D/sFfuIqJdwAmRx1PDtopmZocTJIw2d/9J3PEU2XzgHDM7E5gA/CczW+vuF8cc17hpcV8ZMLMOoMndy3GnzLyZ2SLgH4EPuHtP3PEUi5kdRlDo/zBBsvg9cJG7PxFrYEVkwV8/dwB73f1LMYdTUmFP47+7+9kxh1IQqmlIknwbmARsNLNtZnZb3AEVQ1jsXwbcR1AQvquSE0ZoPrAE+FD4u90W/hUuZUY9DRERyZt6GiIikjclDRERyZuShoiI5E1JQ0RE8qakISIieVPSEBGRvClpiIhI3pQ0RErIzN4bXi9kgplNDK8tMTPuuETypcV9IiVmZn9PsB/RUUC3u/9DzCGJ5E1JQ6TEzOwIgv2mXgfe7+6HYg5JJG8anhIpvXrgbQT7bE2IORaRUVFPQ6TEzOxegqv1TQOOc/dlMYckkjddT0OkhMzs08Cb7v7D8FrhvzOzD7n7/4s7NpF8qKchIiJ5U01DRETypqQhIiJ5U9IQEZG8KWmIiEjelDRERCRvShoiIpI3JQ0REcnb/wdwUQG2woCiYgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from typing import Tuple\n",
    "\n",
    "# Load data\n",
    "\n",
    "training_data = np.load('training_data.npy')\n",
    "test_data = np.load('test_data.npy')\n",
    "\n",
    "test_data_x = test_data[:, 0]\n",
    "test_data_y = test_data[:, 1]\n",
    "\n",
    "training_data_x = training_data[:, 0]\n",
    "training_data_y = training_data[:, 1]\n",
    "\n",
    "# Visualize data\n",
    "plt.plot(test_data_x, test_data_y, 'or')\n",
    "plt.plot(training_data_x, training_data_y, 'ob')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.legend([\"test_data\", \"training data\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the lecture notebook, we will use polynomial-features here again.\n",
    "The following functions will be used for:\n",
    "- calculating polynomial features\n",
    "- computing the mean and std of the features (training data) as normalizer\n",
    "- normalize other data (test) features using the normalizer (mean and std)\n",
    "- evaluating the model\n",
    "- calculating the Mean Squarred Error for assigning a performance to each\n",
    "model. <br><br>\n",
    "\n",
    "Note we will use the mean and the standard deviation to normalize our features\n",
    "according to:\n",
    "\\begin{align*}\n",
    "    \\boldsymbol{\\tilde{\\Phi}} = \\frac{\\boldsymbol{\\Phi}(\\boldsymbol{x}) - \\boldsymbol{\\mu}_{\\Phi}}{\\boldsymbol{\\sigma}_{\\Phi}}, \n",
    "\\end{align*}\n",
    "where $\\boldsymbol{\\tilde{\\Phi}}$ are the (approximately) normalized features to any input\n",
    "$\\boldsymbol{x}$ (not necessarily the training data), $\\boldsymbol{\\mu}_{\\Phi}$ is the mean of the features applied to the training data and $\\boldsymbol{\\sigma}_{\\Phi}$ is the standard deviation of the features applied to the training data for each dimension.<br>\n",
    "\n",
    "Normalization is a standard technique used in Regression to avoid numerical problems and to obtain better fits for the weight vectors $\\boldsymbol{w}$. Especially when the features transform the inputs to a very high value range, normalization is very useful. In this homework we will use features of degree 10. Since the input range of the data is roughly from -4 to 4 this will lead to very high values for higher order degrees. By normalizing each dimension of the feature matrix, we will map each dimension of the feature matrix applied to the training data to a zero mean unit variance distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_polynomial_features(data: np.ndarray,\n",
    "                            degree: int) ->np.ndarray:\n",
    "    \"\"\"\n",
    "    Function to create Feature Matrix. Extends the feature matrix according to\n",
    "    the matrix form discussed in the lectures.\n",
    "\n",
    "    :param data: data points you want to evaluate the polynomials,\n",
    "                 shape: [n_samples] (we have 1-dim data)\n",
    "    :param degree: degree of your polynomial, shape: scalar\n",
    "    :return polynomial_features: shape [n_samples x (degree+1)]\n",
    "    \"\"\"\n",
    "    polynomial_features = np.ones(data.shape)\n",
    "    for i in range(degree):\n",
    "        polynomial_features = np.column_stack((polynomial_features, data ** (i + 1)))\n",
    "    return polynomial_features\n",
    "\n",
    "\n",
    "def get_mean_std_features(polynomial_features: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"\n",
    "    Function for calculating the mean and standard deviation of the features\n",
    "    :param polynomial_features: shape: [n_samples x (degree+1)]\n",
    "    :return mean_feat: mean vector of the features,\n",
    "            shape:[1 x (degrees+1)]\n",
    "    :return std_feat: standard deviation (for each dimension in feature matrix),\n",
    "                      shape: [1 x (degrees+1)] \n",
    "    \"\"\"\n",
    "    mean_feat = np.mean(polynomial_features, axis=0, keepdims=True)\n",
    "    mean_feat[:, 0] = 0.0 # we don't want to normalize the bias\n",
    "    std_feat = np.std(polynomial_features, axis=0, keepdims=True)\n",
    "    std_feat[:, 0] = 1.0 # we don't want to normalize the bias\n",
    "    return mean_feat, std_feat\n",
    "\n",
    "\n",
    "def normalize_features(polynomial_features: np.ndarray,\n",
    "                       mean_train_features: np.ndarray,\n",
    "                       std_train_features: np.ndarray) ->np.ndarray:\n",
    "    \"\"\"\n",
    "    Normalize features\n",
    "    :param polynomial_features:  features to be normalized,\n",
    "                 shape: [n_samples x (degree+1)]\n",
    "    :param mean_train_features: mean of the feature matrix of the training set,\n",
    "                 shape: [1 x (degrees+1)]\n",
    "    :param std_train_features: std of the feature matrix of the training set,\n",
    "                 shape: [1 x (degrees+1)]\n",
    "    :return norm_feat: normalized features, shape: [n_samples x (degree+1)]\n",
    "    \"\"\"\n",
    "\n",
    "    # note: features: (n_samples x n_dims),\n",
    "    #       mean_train_features: (1 x n_dims),\n",
    "    #       std_train_features:  (1 x n_dims)\n",
    "    #       due to these dimensionalities we can do element-wise operations.\n",
    "    #       By this we normalize each dimension independently\n",
    "    norm_feat = (polynomial_features - mean_train_features) / std_train_features\n",
    "    return norm_feat\n",
    "\n",
    "\n",
    "def eval(Phi:np.ndarray, w:np.ndarray)->np.ndarray:\n",
    "    \"\"\"\n",
    "    Evaluate the models\n",
    "\n",
    "    :param Phi: Feature matrix, shape: [n_samples x (degree+1)]\n",
    "    :param w: weight vector, shape: [degree + 1]\n",
    "    :return : predictions, shape [n_samples] (we have 1-dim data)\n",
    "    Evaluates your model\n",
    "    \"\"\"\n",
    "    return np.dot(Phi, w)\n",
    "\n",
    "\n",
    "def mse(y_target:np.ndarray, y_pred:np.ndarray)->np.ndarray:\n",
    "    \"\"\"\n",
    "    :param y_target: the target outputs,\n",
    "            shape: [n_samples] (here 1-dim data)\n",
    "    :param y_pred: the predicted outputs,\n",
    "            shape: [n_samples](we have 1-dim data)\n",
    "    :return : The Mean Squared Error, shape: scalar\n",
    "    \"\"\"\n",
    "    diff = y_target - y_pred\n",
    "    return np.sum(diff ** 2, axis=0) / y_pred.shape[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3) Implement Ridge Regression Weights (2 Point)\n",
    "The following function will calculate the weights for ridge regression. Fill in the missing code according to the formula for calculating the weight updates for ridge regression. <br>\n",
    "Recall that the formula is given by \n",
    "\\begin{align*}\n",
    "    \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y}\\\\\n",
    "    (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I}) \\boldsymbol{w} &= \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n",
    "\\end{align*}\n",
    "where $\\boldsymbol{\\Phi}$ is the feature matrix (the matrix storing the data points applied to the polynomial features).\n",
    "Hint: use np.linalg.solve for solving for the linear equation.\n",
    "If you got confused because of the normalization described before, don't worry, you do not need to consider it here :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_weights_ridge(Phi:np.ndarray,\n",
    "                       y:np.ndarray,\n",
    "                       ridge_factor:float)->np.ndarray:\n",
    "    \"\"\"\n",
    "    :param Phi: Feature Matrix, shape: [n_samples x (degree+1)]\n",
    "    :param y: Output Values, [n_samples] (we have 1-dim data)\n",
    "    :param ridge_factor: lambda value, shape: scalar\n",
    "    :return : The weight vector, calculated according to the equation shown before,\n",
    "            shape: [degrees +1]\n",
    "    \"\"\"\n",
    "    ##################\n",
    "    ##TODO\n",
    "    #################\n",
    "    identity_matrix = np.identity(Phi.shape[1])\n",
    "    #for Ax=b\n",
    "    A = (Phi.transpose().dot(Phi) + ridge_factor * identity_matrix)\n",
    "    b = (Phi.transpose().dot(y))\n",
    "    weights = np.linalg.solve(A,b)\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For demonstrating ridge regression we will pick the polynomial degree of 10. In the lecture notebook we have seen that this model is highly overfitting to the data.\n",
    "We will investigate the role of the ridge factor $\\lambda$. For that purpose we first need to calculate the weights for different $\\lambda$ values. <br>\n",
    "We will pick $\\lambda = [1e-{6}, 1e-{3}, 1, 3, 5,10,20,30,40,50, 1e2, 1e3, 1e5] $ to see the differences of the values. <br><br>\n",
    "\n",
    "Practical note. We use here very high values for $\\lambda$ for demonstration\n",
    "purposes here. In practice we would not choose a model where we know from\n",
    "beginning that it is highly overfitting. When choosing an appropriate model, the value needed for $\\lambda$ automatically will be small (often in the range of $1e^{-6}$ or smaller)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's do it on polynomial degree 10 and see the results\n",
    "\n",
    "# first we get the mean and the standard deviation of the training feature matrix, which we will use for normalization\n",
    "train_features = get_polynomial_features(training_data_x, 10)\n",
    "test_features = get_polynomial_features(test_data_x, 10)\n",
    "mean_train_feat, std_train_feat = get_mean_std_features(train_features)\n",
    "norm_train_features = normalize_features(train_features, mean_train_feat, std_train_feat)\n",
    "norm_test_features = normalize_features(test_features, mean_train_feat, std_train_feat)\n",
    "\n",
    "\n",
    "# now we can calculate the normalized features for degree 10\n",
    "ridge_factors = [1e-6, 1e-3, 1, 3, 5, 10,20,30,40, 50, 1e2, 1e3, 1e5]\n",
    "weights_ridge = []\n",
    "\n",
    "for lambda_val in ridge_factors:\n",
    "    weights_ridge.append(calc_weights_ridge(norm_train_features, training_data_y, lambda_val))\n",
    "\n",
    "# We further have to perform the predictions based on the models we have calculated\n",
    "y_training_ridge = []\n",
    "y_test_ridge = []\n",
    "\n",
    "for w in weights_ridge:\n",
    "    y_training_ridge.append(eval(norm_train_features, w))\n",
    "    y_test_ridge.append(eval(norm_test_features, w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are interested in the mean squarred error on the test and the training data. For that purpose we calculate them here and plot the errors for different $\\lambda$ values in log space. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db551f340>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEaCAYAAAD+E0veAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA38ElEQVR4nO3dd3hUZfbA8e8hVKkCQSlKEaRICRpQUZSAhQ4qIgj2VbHAigLqKqs/K6JYUBcXFFlXpCwsKwiuooIINoKigIC0sDQhgAQQkJL398eZCZOYnpncucn5PM88zNyZue+ZDHPPfct9X3HOYYwxxgCU8DoAY4wx0cOSgjHGmDSWFIwxxqSxpGCMMSaNJQVjjDFpLCkYY4xJY0nB+JKITBKRpwqprDNF5KCIxGTx/OMi8m5hxOInIvKGiIz0Og6TN5YUTKZEJElEDgcOhsHba4Ucw0IRORIoe7eI/FtEauZjP05EGmbz/M0iciJQzn4R+UFEugefd879zzlXwTl3Ir+fJT9EpIOIpAbiOiAia0XklsKMoSCcc4Occ096HYfJG0sKJjs9AgfD4O3ezF4kIiUz2ZbpWXVWsnn9vc65CsDZQBXgpbzsNw++CpRTBfgbMFVEqkSorLzYHoirEjAUmCAijcNdSGbfoSmeLCmYPAucWS8RkZdEZA/weKA5Z5yIzBOR34AEEWkaONvfJyKrRKRnyD7+8PrsynTO7QVmAs2ziOl2EVkvIntFZLaI1ApsXxR4yQ+BM+7rcignFfgnUB5oFNhHvUBto2TgcX0R+Txw9j4fqJ4hlhtFZLOI7BGRkYFa12WB50qIyEMisiHw/HQRqZpdTIG4nHNuHrAXaJmbfeUQx+MiMkNE3hWR/cDNIlJZRN4SkR0isk1EngomaxFpGPjMKYFa27TAdgn8P9gVqGWtEJHmgefSNfFl9R0FnnMiMkhE1gX+v7wuIpLT38WEnyUFk1/nAxuB04CnA9uuD9yvCHwDzAE+BmoAg4HJGc5yQ1+/OLvCRKQ6cA3wfSbPdQSeBfoCNYHNwFQA59wlgZe1CtR2puVQTgxwC3AssJ/MvAcsQ5PBk8BNIe9vhtY0BgRiqQzUDnnvYKA3cClQC/gVeD27mAL7LRFIqtWB9TntKxdxAPQCZqC1o8nAJOA40BBoDVwB/Cnw2ifR7/JUoA7wamD7FcAlaE2uMvod7Mkk/iy/oxDdgTZo0usLXJnDn8VEgnPObnb7ww1IAg4C+0Jutweeuxn4X4bXTwLeCXncHvgFKBGybQrweGavzyKGhcChQNnb0ANXbMj7nwrcfwsYHfK+CuhBvV7gsQMaZlPOzejBcF/gfYeBviHP1wvsoyRwZuC15UOefw94N3D/r8CUkOdOAY4ClwUerwY6hTxfM1BmyUzi6gCkBuL6HTgB3BfyfJb7ykUcjwOLQp4/LVBGuZBt/YEFgfvvAOOBOhli7Aj8DFwQ+l3n8zu6OOT56cBDXv8OiuPNagomO72dc1VCbhNCntuSyetDt9UCtjhtjgnaTPqz1cz2kdGQQNm1nXMDnHPJmbymFiFn9c65g+jZasYz4+x87Zyrgp4Jz0aTWmZqAb86534L2bY5w/Npn8s5d4j0Z851gVmBJpJ96IH9BHpQzsz2QFyVgLHoQTg3+8opDkj/968LlAJ2hOzv72gtD2AEIMC3gabAWwP7/Qx4Da2h7BKR8SJSKZPPkZvv6JeQ+4fQxGEKmSUFk1+ZTa8bum07cIaIhP4fOxM9489uH/mxHT2oASAi5YFqGcrKlcDB6i7gBhFpnclLdgCnBsoIOjPD83VCYikXiCVoC9AlQ7It65zLNlbn3O/Ag0ALEemdi33lFAek//tvQWsK1UP2Vck5d06g/F+cc7c752oBdwJ/k8CILufcWOfceUAztBlpeCYfIWzfkYksSwomUr5Bz/ZGiEgpEekA9OCP7cjhMAW4RUTiRKQM8AzwjXMuKfD8TqBBbnfmtFP7TbQJJuNzm4FE4P9EpLSIXIx+rqAZQA8RaScipdFmmtAO0zeAp0WkLoCIxIpIr1zGdRQYExJXdvvKKY6M+96B9hmMEZFKgT6Ms0Tk0sC+rxWRYJL5FU0oqSLSRkTOF5FSwG/AEbTJK6OcviMTJSwpmOzMkfTXKczK7RsDB7AeQBdgN9rpeaNzbk24g3TOfQKMREcn7QDOAvqFvORx4B+BZpG+udzty0BXEWmZyXPXox3te4HH0Pb2YCyr0A7gqYFYDgK70LNwgFfQ5qmPReQA8HVgX7k1EThTRHpkt69cxJGZG4HSwE/ogX8G2k8B2gH8jYgcDJT5Z+fcRrRZa0Lg9ZvRJqHnM+44F9+RiRLinC2yY0ykiEgFtKO4kXNuU3GPw0Q/qykYE2Yi0kNETgm0m78ArEBHcxXLOIy/WFIwJvx6oR2r29EL4Po5b6rk0RKH8RFrPjLGGJPGagrGGGPS+HoSrOrVq7t69ep5HYYxxvjKsmXLdjvnYjN7zpdJITAcr0fDhg1JTEz0OhxjjPEVEclqXi9/Nh855+Y45+6oXLmy16EYY0yR4sukEBhqNz4lJcXrUIwxpkjxZVKwmoIxxkSG7/sUjDHeOnbsGFu3buXIkSNeh2IyKFu2LHXq1KFUqVK5fo+vr1OIj4931tFsjLc2bdpExYoVqVatGrZYWvRwzrFnzx4OHDhA/fr10z0nIsucc/GZvc+XzUfGmOhx5MgRSwhRSESoVq1anmtwvkwK1tFsTHSxhBCd8vO9+DIpFLij+fvv4dFHwxuUMcYUAb5MCgX2ySfw9NPw3/96HYkxpoD27NlDXFwccXFxnH766dSuXTvt8dGjR7N9b2JiIkOGDMmxjHbt2oUl1oULF1K5cuW0+OLi4vjkk0/Csu9w8WVHc8joo9vXrVuX9x38/js0bw4lS8KPP0IeeuaNMemtXr2apk2beh0GAI8//jgVKlRg2LBhaduOHz9OyZLRMdBy4cKFvPDCC3zwwQdZvsY5h3OOEiVKZPo4K1l9zsy+nyLX0Vzg5qMyZeCll2DNGnjttfAGZ4zx3M0338ygQYM4//zzGTFiBN9++y0XXnghrVu3pl27dqxduxbQg3T37t0BTSi33norHTp0oEGDBowdOzZtfxUqVEh7fYcOHejTpw9NmjRhwIABBE+s582bR5MmTTjvvPMYMmRI2n5zIykpicaNG3PjjTfSvHlzvvjii3SPt2zZwvDhw2nevDktWrRg2rRpafG0b9+enj170qxZs7D87aIjfXqhWzfo3BkefxwGDIAaNbyOyBjfu+8+WL48vPuMi4OXX877+7Zu3cqXX35JTEwM+/fv54svvqBkyZJ88skn/OUvf2HmzJl/eM+aNWtYsGABBw4coHHjxtx1111/GOP//fffs2rVKmrVqsVFF13EkiVLiI+P584772TRokXUr1+f/v37ZxnXF198QVxcXNrjmTNnEhMTw7p16/jHP/7BBRdcQFJSUrrHM2fOZPny5fzwww/s3r2bNm3acMkllwDw3XffsXLlyj8MO82v4psURLS20KKFdjqPH+91RMaYMLr22muJiYkBICUlhZtuuol169YhIhw7dizT93Tr1o0yZcpQpkwZatSowc6dO6lTp06617Rt2zZtW1xcHElJSVSoUIEGDRqkHZj79+/P+CyOKe3bt/9D81FSUhJ169blggsuSNsW+njx4sX079+fmJgYTjvtNC699FKWLl1KpUqVaNu2bdgSAvg0KYTtiuYmTWDwYD0NGTQIzj03HOEZU2zl54w+UsqXL592f+TIkSQkJDBr1iySkpLo0KFDpu8pU6ZM2v2YmBiOHz+er9cUNN7MHuf2fQVVPPsUQv31r1C9OgwZAj7sdDfG5CwlJYXatWsDMGnSpLDvv3HjxmzcuJGkpCSAtDb/cGnfvj3Tpk3jxIkTJCcns2jRItq2bRvWMoJ8mRTCqkoVeOYZWLIEwvxFGmOiw4gRI3j44Ydp3bp12M7sQ5UrV46//e1vdO7cmfPOO4+KFSuS1UlrsE8heJsxY0aO+7/qqqto2bIlrVq1omPHjowePZrTTz893B8D8OmQ1KCwzX104gS0aQPJyToiKczVMWOKsmgakuqlgwcPUqFCBZxz3HPPPTRq1IihQ4d6HVbxGJIadjExMHYsbN0Kzz3ndTTGGB+aMGECcXFxnHPOOaSkpHDnnXd6HVK++LKmUOCL17Jy/fUwaxasXg229rMxuWI1hehWLGoKEVtk57nndKjq8OHh3a8xxviEL5NCxJxxBjz8MMyYAQsWeB2NMcYUOksKGQ0bBnXrwp//DBEYpWCMMdHMkkJG5crBCy/AihUwYYLX0RhjTKGypJCZa66BDh10+ou9e72OxhiTjYJMnQ06qdyXX36Z6XOTJk0iNjY23XUFP/30U7g/QlTx5TQXEScCr7wCrVvDY4/Bq696HZExJgvVqlVjeWAWvsymzs7JwoULqVChQpZrJlx33XW8ls1syhmnrM7tVN3RNKV3KKspZKVlS50Padw4WLnS62iMMXmwbNkyLr30Us477zyuvPJKduzYAcDYsWNp1qwZLVu2pF+/fiQlJfHGG2/w0ksvERcXxxdffJGr/Wecsjrj4yNHjnDLLbfQokULWrduzYLAwJVJkybRs2dPOnbsSKdOnSL2+Qsi+tJULoRtQrycPPEETJminc6ffKI1CGNM1qJg7mznHIMHD+b9998nNjaWadOm8cgjjzBx4kRGjRrFpk2bKFOmDPv27aNKlSoMGjQo29rFtGnTWLx4cdrjr776Ckg/ZfXChQvTPR4zZgwiwooVK1izZg1XXHEFP//8c9r7fvzxR6pWrZrvP0kk+bKmELHrFDKqVk0Tw2efwX/+E9myjDFh8fvvv7Ny5Uouv/xy4uLieOqpp9i6dSsALVu2ZMCAAbz77ru5brq57rrrWL58edqtXLlyAH+Ysjr08eLFixk4cCAATZo0oW7dumlJ4fLLL4/ahAA+rSkUqkGD4I034IEHoEsXKFvW64iMiV5RMHe2c45zzjkn7Yw+1Ny5c1m0aBFz5szh6aefZsWKFfkuJ1qmug43X9YUClXJktrpvGkTvPii19EYY3JQpkwZkpOT05LCsWPHWLVqFampqWzZsoWEhASee+45UlJSOHjwIBUrVuTAgQNhjaF9+/ZMnjwZgJ9//pn//e9/NG7cOKxlRIolhdzo1Amuukqn2N62zetojDHZKFGiBDNmzODBBx+kVatWxMXF8eWXX3LixAkGDhyY1vk7ZMgQqlSpQo8ePZg1a1aWHc3Tpk1LNyQ1q+Groe6++25SU1Np0aIF1113HZMmTUq3OE808+WEeEFhmzo7NzZtgqZNoU8fePfdwinTGB+wCfGiW7GYEM8T9evrFBiTJ0MuzhSMMcaPLCnkxUMPQa1aunRnaqrX0RhjTNhZUsiLChVg9GhYtgwisM6rMcZ4LaqSgoiUF5FEEenudSxZuv56aNdOp9hOSfE6GmOMCauIJgURmSgiu0RkZYbtnUVkrYisF5GHQp56EJgeyZgKLDgvUnIyPPWU19EYY0xYRbqmMAnoHLpBRGKA14EuQDOgv4g0E5HLgZ+AXRGOqeDi4+GWWzQ5BK5SNMaYoiCiScE5twjIOPd0W2C9c26jc+4oMBXoBXQALgCuB24XkUxjE5E7Ak1MicnJyZELPifPPKNrLwwd6l0MxhgARCRtWgnQGUhjY2Pp3l1bomfPns2oUaNyta+CTsUN/p6O24tpLmoDW0IebwXOd87dCyAiNwO7nXOZDu9xzo0HxoNepxDZULNx2mnw17/qMNV586BrV89CMcYvRi8ZTZtabUion5C2bcGmBSzdvpQRF43I937Lly/PypUrOXz4MOXKlWP+/PnUrl077fmePXvSs2fPXO2roFNxg7+n446qjmYA59wk59wH2b1GRHqIyPgUrzt6Bw+Gs8/W2kIuzyCMKc7a1GpD3xl9WbBJp5JesGkBfWf0pU2tNgXed9euXZk7dy4AU6ZMoX///mnPTZo0iXvvvReAm2++mSFDhtCuXTsaNGjAjBkzcrX/4jIdtxc1hW3AGSGP6wS25Zpzbg4wJz4+/vZwBpZnpUvDSy9Bt266EM8DD3gajjHRLqF+AtP7TKfvjL7cFX8X4xLHMb3P9HQ1h/zq168fTzzxBN27d+fHH3/k1ltvzfKAvGPHDhYvXsyaNWvo2bMnffr0yXbfx44dKzbTcXuRFJYCjUSkPpoM+qH9CLlWaOsp5EbXrnp74gkYOFCblYwxWUqon8Bd8Xfx5KInGXnJyLAkBNBpsZOSkpgyZQpdc2jO7d27NyVKlKBZs2bs3Lkzx32vXbs2bTpugBMnTlCzZs20cgcMGEDv3r3p3bt3rmLNqvkop+m4Bw8eDER2Ou5ID0mdAnwFNBaRrSJym3PuOHAv8BGwGpjunFuVl/0W2noKufXii3DoEDzyiNeRGBP1FmxawLjEcYy8ZCTjEselNSWFQ8+ePRk2bFi6pqPMhE5Ol5v534LTcQfXVFixYgUff/wxoNNx33PPPXz33Xe0adOG48eP5zv+aJiOO9Kjj/o752o650o55+o4594KbJ/nnDvbOXeWc+7pvO43avoUgho31tXZJk7Uq52NMZkK9iFM7zOdJxKeSGtKCldiuPXWW3nsscdo0aJFWPYX1Lhx42IzHXfUdTTnRtTVFABGjoTYWJ0XycczzxoTSUu3L03XhxDsY1i6fWlY9l+nTh2GDBkSln2FKl26dLGZjtumzg6nt96CP/1Jp9YeMMDraIwpFDZ1dnQrFlNnR13zUdAtt8B558GIEXDwoNfRGGNMnvkyKURl8xFAiRIwdixs3w65vHrSGGOiiS+TQlRr106bjl54ATZu9DoaYwqFn5uhi7L8fC++TApR23wUNGoUxMTA8OFeR2JMxJUtW5Y9e/ZYYogyzjn27NlD2bJl8/Q+62iOlKefhkcfhQULoEMHr6MxJmKOHTvG1q1bOXLkiNehmAzKli1LnTp1KFWqVLrt2XU0W1KIlCNHdF3nli3ho4+8jsYYY9IUudFHvlC2rF7Q9vHHEJhx0Rhjop0vk0LU9ykEDRqk6zo//7zXkRhjTK74MilE7ZDUjKpUgTvvhGnTICnJ62iMMSZHvkwKvnLffbqu80sveR2JMcbkyJJCpNWpo9ctvPkm7NnjdTTGGJMtSwqFYdgwnVp73DivIzHGmGz5Min4pqM5qHlzXYhn7Fg4fNjraIwxJku+TAq+6WgONWIEJCfDP/7hdSTGGJMlXyYFX7rkEmjbVudEOnHC62iMMSZTlhQKi4jWFjZsgFmzvI7GGGMyZUmhMPXuDQ0bwujRtjqbMSYqWVIoTDExOhJp6VL4/HOvozHGmD+wpFDYbrwRatSwqS+MMVHJkkJhK1cOBg+GefNgxQqvozHGmHR8mRR8d51CRnffDaecoiORjDEmivgyKfjyOoVQVavC7bfDe+/Bli1eR2OMMWl8mRSKhKFDdQTSyy97HYkxxqSxpOCVunWhXz8YPx5+/dXraIwxBrCk4K3hw+HgQXjjDa8jMcYYAEp6HUCx1qoVXHmlTpQ3dKgu4WmMKb4OH9Y50nbt0n+Dt8weP/88XHNN2EOwpOC14cPhssvg3XfhT3/yOhpjTDgdOpTrg7xLTkZ++y3T3RwvUYoD5WqQUjqWvSViSeYsKq2P5cIIhGxJwWsdO8K552rWv/VWKGEtesb40tGj8OqrnJg6HffLLmT3LmKOHMr0pcdKlObXUjXYUyKWXS6WHccbsf14DZKJTbvt4uTjA6mVqFJaOPVUHbxYtSo80DoyH8OSgteCE+X16wezZ+v8SMYY/3CO1P/M5vA9wyi/Yz3fcgHraJ/uAB88yKeUiuX4qbGUrlaRqtXSH+SrVoU6p0LLkMfB5ytX1llyCkPUJAURaQr8GagOfOqcKz7LlF1zDdSvD889B716aaIwxkS9Y8t+ZPcNQ6m5+jM205RR1efR4J4unHEGNKgK52U46JcrF/0/74gmBRGZCHQHdjnnmods7wy8AsQAbzrnRjnnVgODRKQE8A5QfJJCyZLwwANw772wZAlcfLHXERljsnFgwy42DhxJ86/fpDRVeKbmq9R95k7eGlCKUqW8jq5gIt2APQnoHLpBRGKA14EuQDOgv4g0CzzXE5gLzItwXNHnllugWjWdVtsYE5V2JP3OvI7Pk9qwEc2+nsh/6gzmh3+t4+Ft9zLgZv8nBIhwUnDOLQL2ZtjcFljvnNvonDsKTAV6BV4/2znXBRiQ1T5F5A4RSRSRxOTk5EiFXvhOOUUnypszB1av9joaY0yINasdr182i8P1m9F1wQjW12zP2n+t4JotL9OxT9WobxLKCy+GutQGQif82QrUFpEOIjJWRP5ONjUF59x451y8cy4+NjY20rEWrnvu0UZHmyjPmKjw5ZcwNGE5O5p15J5Pr6bcqWXZPvG/nLf9A5r3aeJ1eBERNeMfnXMLnXNDnHN3Oudez+61vp8lNSvVq+uw1H/+E7Zv9zoaY4ql1FQdCNij7U5WXXQ7YxaeS9tyKzgw6nVq7vqBWrdc6XWIEeVFUtgGnBHyuE5gW675fpbU7Nx/P5w4Aa+84nUkxhQrv/8OEydC66ZHWNLrOd5LbMStJSZxYvBQym9bR8UH79ZBIUWcF0lhKdBIROqLSGmgHzA7LzsosjUFgAYN4NprdT6kovj5jIkyKSk6vqN+Pcfc22YyN6kZz/EQ5bsnELN6FaXGjtELBoqJiCYFEZkCfAU0FpGtInKbc+44cC/wEbAamO6cW5WX/RbpmgLo1Bf79+sMqsaYiNi2TX9qZ5wBUx/8jv8e6cBM+lD77PIwfz4lZr8PZ5/tdZiFTpxzXseQZyLSA+jRsGHD29etW+d1OJFx2WU6CmnTJihd2utojCkyVq3SsRyTJ0Ps8R28W+9ROiS9jVSrBk89BbfdVuSbiURkmXMuPrPnoqajOS+KfE0BdOqL7dt1dTZjTIEcPw6ffAI9ekDz5vCfqUeYcd6zbDnlbBK2/hN54AFYtw7uvLPIJ4Sc+LKmEBQfH+8SExO9DiMynIPWreHYMVixwibKMyaPfv8dPv0U/v1veP992L0bqlV1jOs0g6u/Hk7Mls0619jzz0PDhl6HW6iKXE2hSHc0B4log+dPP8G84neBtzH5ceiQJoGBA6FGDejWDaZPhyuugM+e+ZqdTS7h2n/1JaZqFfjsM5g1q9glhJxYTSGaHTum/2Hr1oVFi7yOxpiotH8/zJ0LM2fChx9qYqhWTeeW7NP9CJftmUap8a/D0qWaKZ5+WqeVKaxpR6NQdjWF4t14Fu1KldLrFu67D776Ci6MxJIaxvjPnj16gdnMmTB/vi5lcPrpcNNNOunwpWduouSEcXD7RH1x06bw6qv6gooVvQ4/qvmyplAsRh8FHTwIZ54JHTpovdiYYmrHDvjPfzQRLFyo13jWrQtXX62J4MLzUykx/yN4/XVtci1RQvsM7rlHfz9FaYKiAsqupuDLpBBU5JuPgkaO1CrvmjXFcty0Kb42b9ZzoZkzdR4i56BxY00CV1+tixbKr3v1UuRx42DjRjjtNLjjDr3VqeP1R4hKlhT8btcurS3cdBP8/e9eR2NMRP38syaBmTNh2TLd1qrVyRpBs2aBk/7ERPjb32DKFDhyBNq311rBVVfZtT05sD4Fv6tRA26+GSZNgv/7P208NaaIOHxYr9N8/31NBKsC8xucf74uRnj11SEDhI4cgX9O1yaib7+F8uX1t3H33dCihVcfoUjxZU2hWPUpBK1bp/Xmhx/WpiRjfOLgQW0GSkr6479JSVoRBu0CaN9ek8BVV+n0E2k2bdL5wN56SzuOmzTRRHDjjbqAsckTaz4qKvr00atx/vc/G0FhokZKSvYH/T170r++dGntIK5bF+rV01v9+jqzS40aIS9MTYWPPtImorlzNWv06qVNRAkJ1nFcANZ8VFQMH6716zffhKFDvY7GFAPOwb59Jw/woQf74P19+9K/p2zZkwf7+Hj9NzQBnHZaDhfo790Lb7+tHccbNugbHn3UOo4LidUU/KZDBx1hsWEDRWJBWBM1fv1V2/NXrtRb8P7u3elfV778yQN86ME+eD82Np8n8cuWaV9BsOP44ou1VnD11dZxHGb5rimIyEDn3LuB+xc555aEPHevc+618IZqcjRihF67P22aXstvTB799pvOnhI8+AdvoYv9VawI55yjw/ybNNHmneBBv2rVArbcOKcXHaxercOsV6+Gr7/WpFC+vI6yu/tuaNmygJ/U5Ee2NQUR+c45d27G+5k9LkzFsqM5yDn9sYjADz9Yu6rJ0u+/6zE39Kx/5Urtsw0qW1aHeDZvrkmgeXO9nXFGGP5rHT+uha1effIWTAL79598XaVKWni/fpoQrOM44grSpyBZ3M/scaFxzs0B5sTHx9/uVQyeEYFhw3QY3kcfQefOXkdkPHb8OKxfn/6sf9UqHbB24oS+pmRJHbzWtq0uAx5MAg0ahGEKoEOHYO3a9Af91as1gKNHT76uZk2dbuKGG/Tfpk21GlKzpp3cRJGckoLL4n5mj01h6d8fHnlE1xC0pFDsbNmiE3wuWADff6/H4eCxV0TH9J9zjg5WC575N2oUhmb53bvTH/SDSWDzZq3BgvYgn3WWHuy7dUt/8LcagC/klBSaiMiPaK3grMB9Ao8bRDQyk7XSpXX00bBhOvNjmzZeR2QiKDlZ5/r59FNNBsEW0+rV9cy/c+eTTT9NmsAppxSwQOd0IMM33+ht+XJNAKE9zuXKadXjwgt1xtHgwb9RIyhTpoABGC/l1KdQN7s3O+c2hz2iPCiWo4+C9u/XqS+uuEInjDdFxv79OlP6Z5/p7YcfdHvFinDppdCpE3TsqEkgLGsv7d2rVwcHk8C33568uOCUUyAuTjsegmf8TZtqr7Mt/ORb+e5TyHjQF5FqwCXA/5xzy8IXosmzSpXgrru0CWnDBq2yG186ckQne/vsM60NLF2qfQFlysBFF+kF7B076pj/Aq8UefSonvkHE8A332iHBGjbU7NmeoHY+efr7Zxziv3ylMVNTjWFD4CHnHMrRaQm8B2QCJwFjHfOvVwoUWahWNcUQIf11aunV3e++qpW3U3UO35c53ILJoElS3SkUEyMtgQGawLt2unooHxzTq9pCU0A339/sgOiZs2TB//zz9esY1fKFwsFGX1U3zm3MnD/FmC+c+5GEakILAFeDl+YJs9q1tTTyEce0fbdq6/Wq57PP9/ryEyI1FQdERRMAp9/DgcO6HOtWumQ/I4d4ZJLtAKYb6HNQN9+q7dgP8App+hBf8iQk0mgTh0b9WP+IKeawnLnXFzg/qfABOfc1IzPFbZifZ1CZn75BcaO1WkB9u3To8uIEdCli7X7emTDBk0An36qo4SSk3V7o0aaADp10ovTY2PzWUBqqnY2LFlyshYQ/C0Em4FCawHWDGRC5HtCPBGZA3wMbAUmojWHfSJSDkh0zp0TiYBzq9g3H2V04IDOi/Tii7B1qx4Ihg/XIaw2TUDEOae1gWef1WQAUKuWJoBOnbSV78wzC1BASoquPTlvni5G/Msvuj1jM9B55xWwymGKuoIkhRrAE0BN4HXn3MeB7QnAec65FyIQb65ZUsjCsWMwdSo8/zysWAG1a+s6z3fcYQeLCEhN1bUAnn1WO4lPP13/3L1760J5+W6hcU6vQps3T29LlmiHxKmnwpVXak0wIcGagUye2dTZxZVzetXz6NHahlGpEgwaBH/+s57CmgI5dgzee08Xglm9Wq8OHjFCZ2rIdwfxwYNa3Qgmgi1bdHtcHHTtqrfzz7emIFMgBakpzM5ux865ngWMrUAsKeRBYqLWHGbM0GEuN9ygF781bep1ZL5z6JAuCfz887q0RcuW8NBDcO21+TxW//zzySTw+ec6OqhiRbj8ck0CnTtrbc+YMClIUkgGtgBTgG/IMN+Rc+7zMMaZZ5YU8mHDBu1zePttXQexRw89vb3oImuCyMG+fbrey8sva8fxRRfpQnhdu+bxT3fkiB78g4kgeJ1A06YnawMXX2z9QCZiCpIUYoDLgf5AS2AuMMU5tyoSgeaVJYUCSE7Wuetfe02vXr3wQk0OPXvaiKUMdu6El17SwV379+uJ+1/+oktH5lpSknYOz5unvdCHD+tUER07ahLo0kXnpzamEISlT0FEyqDJ4Xng/6JhLQVLCmHw229aa3jxRZ3m+OyztVnphhsKeOWU/23apE1EEydqi86112ozUevWuXjz0aPaMRysDfz0k25v0EAniuvaVeesKFcuop/BmMwUKCkEkkE3NCHUA2YDE51z28IcZ55ZUgij48d1qc/Ro+G773QJxCFDdCqNU0/1OrpCtWoVjBqlC4CVKKEdxyNG5OKC8W3bTtYGPvlEhwiXLq0H/2CzUKNG1kxnPFeQ5qN3gObAPGBqyNXNESEivdEEVAl4KzgENiuWFCLAOR2pNHq0jlwqX16Hsv75zzoJWhH29dc6rHT2bP3Yd94J99+fTR/vsWPw1VcnE8GPgUmEzzhDm4O6dtULFCpUKLTPYExuFCQppAK/BR6GvlAA55zLcdC7iEwEugO7nHPNQ7Z3Bl4BYoA3nXOjQp47FXjBOXdbdvu2pBBhP/yg7SdTp+oMbdWr6yyZTZrotBrB+/Xq+XaIpHN6Pdizz+r01FWragXp3nuhWrVM3rBjB/z3v5oE5s/XC8pKltSO4WDfwDnnWG3ARDVPr1MQkUuAg8A7waQQ6MD+Ge3E3gosBfo7534KPD8GmOyc+y67fVtSKCSbN+tQ1jVrdIWtNWtOztsA2kTSsOEfE0bjxlG7sEpqKsyapclg2TK9bGPYMLj99gwn9seP6xQSwauIv/9et9eqdTIJXHaZXRRofKUgE+IVmHNukYjUy7C5LbDeObcRQESmAr1EZDUwCvgwq4QgIncAdwCcWaA5A0yu1a0LDzyQftuePZoggklizRptjH///ZNrQIJOwRBaqwgmizPP9GSU09GjMHmyXnC2dq3msgkTtF89bW2YnTu16WzePPj4Y/j1V722o107zSJdupxcJ9uYIsarOn9t9PqHoK3A+cBg4DKgsog0dM69kfGNzrnxwHjQmkIhxGoyU62aHiTbtUu//ehRna45tFaxZo02Qe3bd/J15crpSKfQhNG4sXZwV66sjfphPOg++dlodiS2Yc6rCWzdqhcIj5y4gFMaLuVP7R7Q+SmCtYFg7fP003Wuii5d9EKyKlXCFo8x0SqqGoKdc2OBsTm9LmSW1MgHZfKmdOmTB/lQzmmTUzBJBBNGYqI2TaWmpn99TIwmh8qV9WAcvJ/xcXb3y5Rh506dQPaV2W34rUtfWsVPZ/z4BMqcPovrPriJ6UvawlWjteZTooRer/HUU9o01KqVXbNhih2vksI24IyQx3UC23LFOTcHmBMfH397uAMzESICNWro7ZJL0j935Ihe1bt2rc7/n5Jy8rZv38n7GzeevL9//8nF4rNwLKYMnKjMjVTh5lMrs2bFmdwa35mv3otlXM1tTP8XJBxacfK6gcsv155mY4oxr5LCUqCRiNRHk0E/4PrcvtlqCkVM2bK64HDz5jm/Nig1Va8DyJA8NnyfwoJ/72PT8hROJYU2TVM4t/4+Kqam0CglhbvWVuLJc7YxkktJ+NcLcO65VhswJkTEk4KITAE6ANVFZCvwmHPuLRG5F/gIHZI6MS9TZ1hNwVCiRFpTkXM6SnT0GB1WWqUK3P0XGDhYuwWCFmxawLgZfRkZP5JxieNIqHaABEsIxqRTGKOP+mexfR56UVyeWU3BgF47Nm2aXme3YoUuKzBmjA4rzbjU8IJNC+g7oy/T+0wnoX4CCfUS0j02xihfniY55+Y45+6oHKVj4E1k/fYbvPKKDie94QYdATtpkk4Ae//9ma89v3T70nQJIKF+AtP7TGfp9qWFG7wxUc4W2TG+sWuXTur6+uu6Rn379vDgg7YUtTF55enFa5FgzUfFy8aN2iw0cSL8/jv06qUT1F14odeRGVP0+PL8ypqPiodly6BfP51Y9M03YeBAnYF61ixLCMZEii9rCqbock5nnX7uOV2LplIlGD5cJ6mzZaWNiTxfJgVrPip69u2DOXN0hbPvv9cpk0aP1umrba45YwqPL5OCXafgf87pevUffKC3xYt1QtImTeCtt2DAgJAJ6owxhcaXScH409GjsGgRzJ2riSC4Xn2LFtpE1K2b9hXYSCJjvGNJwUTUzp068egHH+gs1AcOaA2gUye9pqBbN51F2xgTHXyZFKxPIXo5B8uXn2wWWrpUt9WuDddfr0mgY0edGdsYE33s4jVTYL/9piOF5s7V27ZtOilq27bQvbveWrWyNWmMiRZF7uI1473Nm0/2DXz2mV5UVrEiXHmlJoEuXXSWbGOMv1hSMLly4gR8/bUmgblzdQI60PmH7rpLE0H79rrGjjHGvywpmGzt3g0vvgjjx+viZCVL6sF/zBhNBGef7XWExphw8mVSsI7myEtOhhde0MnnDh2Cq6+Gvn3hiitsqWJjijJfjgi3uY8iZ+dOvWagXj1NCr16wcqVuoxy376WEIwp6nxZUzDh98svOq3EG29op/H118Ojj0Ljxl5HZowpTJYUirnt2zUZ/P3vupLZwIHwl79YX4ExxZUlhWJq61adiXTCBJ1z6MYbNRlYN40xxZslhWJmyxYYNUrXJ0hNhZtvhocfhgYNvI7MGBMNLCkUE5s3w7PP6uplALfeCg89pB3KxhgTZEmhiNu0SZPBpEk6zcSf/qTJwCahM8ZkxpdJwa5TyNmGDfDMM/DOOxATo4vVPPgg1KnjdWTGmGhm1ykUMevWaT9B48bw3ntw992aIF591RKCMSZnvqwpmD9auxaefhomT9b1CoYM0YvQatb0OjJjjJ9YUvC59evhscdg6lQoWxaGDtVkcNppXkdmjPEjSwo+9sMPkJCgy1wOGwYPPGDTVRtjCsaSgk/99BNcdhlUqACffw7163sdkTGmKPBlR3Nx9/PPusZxqVK64pklBGNMuFhNwWc2bNA1jlNTYeFCaNTI64iMMUWJJQUf2bxZE8Lhw5oQmjb1OiJjTFETNc1HItJARN4SkRlexxKNtm3ThLB/P8yfDy1aeB2RMaYoimhSEJGJIrJLRFZm2N5ZRNaKyHoReQjAObfROXdbJOPxq19+0YSQnAwffQTnnut1RMaYoirSNYVJQOfQDSISA7wOdAGaAf1FpFmE4/Ct5GQdZbRtG3z4IbRt63VExpiiLKJJwTm3CNibYXNbYH2gZnAUmAr0yu0+ReQOEUkUkcTk5OQwRht99u6Fyy/XzuU5c+Cii7yOyBhT1HnRp1Ab2BLyeCtQW0SqicgbQGsReTirNzvnxjvn4p1z8bGxsZGO1TMpKXDllbB6Nbz/vl6kZowxkRY1o4+cc3uAQbl5bVGfJfXAAejSRa9Y/ve/4YorvI7IGFNceFFT2AacEfK4TmBbrhXlWVJ/+w26d4dvv4Vp0/S+McYUFi+SwlKgkYjUF5HSQD9gdl52ICI9RGR8SkpKRAL0yuHD0KsXLF6ss51edZXXERljiptID0mdAnwFNBaRrSJym3PuOHAv8BGwGpjunFuVl/0WxZrC77/DNdfAZ5/B22/Dddd5HZExpjiKaJ+Cc65/FtvnAfPyu9+i1qdw7JgmgQ8/hAkT4MYbvY7IGFNcRc0VzXlRlGoKx4/DgAE6wui113QNZWOM8Yovk0JRceKELp35r3/BmDFwzz1eR2SMKe58mRSKQkdzaircfrt2KD/zDNx/v9cRGWOMT5OC35uPnNNawdtv61KaD2d5qZ4xxhQuXyYFP3NO11F+4w148EFNCsYYEy18mRT82nzkHDz0ELzyCtx3Hzz7LIh4HZUxxpzky6Tg1+ajxx+H0aPhrrvgxRctIRhjoo8vk4IfPfMMPPEE3HabDj21hGCMiUaWFArBmDHwyCMwcCD8/e9Qwv7qxpgo5cvDk5/6FF57DYYNg2uv1dFGMTFeR2SMMVnzZVLwS5/C+PEweLBOcjd5MpSMmonKjTEmc75MCn7wzjswaBB07apTYJcq5XVExhiTM0sKETB9OtxyC3TsCDNmQJkyXkdkjDG5Y0khzGbP1gnu2rXTSe7KlfM6ImOMyT1fJoVo7Wj++GPtUG7dGubOhfLlvY7IGGPyxpdJIRo7mj//HHr3hqZN4b//hUqVvI7IGGPyzpdJIdp89RV06wb16sH8+VC1qtcRGWNM/lhSKKDvvoMuXaBmTfj0U4iN9ToiY4zJP0sKBbByJVx+OVSpogmhZk2vIzLGmIKxpJBPa9fCZZdB2bKaEM480+uIjDGm4Cwp5MPGjdCpk06F/emncNZZXkdkjDHhYRMv5NGWLZoQDh+GBQugSROvIzLGmPDxZU3Bq+sUfvlFE8LevfDRR9CyZaEWb4wxEefLpODFdQq7d2sfwvbt8OGHEB9faEUbY0yhseajXPj1Vx1ltGEDzJunU1gYY0xRZEkhBwcO6HUIq1bpvEYJCV5HZIwxkWNJIRuHDkH37pCYqLOddu7sdUTGGBNZlhSycOSIzmW0eDG8957eN8aYos6SQiaOHtXZTufPh0mT4LrrvI7IGGMKhy9HH0XS8eO6HsIHH8C4cXDTTV5HZIwxhceSQojUVF0xbcYMePFFXU7TGGOKk6hpPhKR8sDfgKPAQufc5MIs3zlNAu++C089BUOHFmbpxhgTHSJaUxCRiSKyS0RWZtjeWUTWish6EXkosPlqYIZz7nagZyTjysg5uO8+mDABHnlEb8YYUxxFuvloEpBuIKeIxACvA12AZkB/EWkG1AG2BF52IsJxpXEOHn4Yxo7V2sGTTxZWycYYE30imhScc4uAvRk2twXWO+c2OueOAlOBXsBWNDFkG5eI3CEiiSKSmJycXOAYn3wSnntOm47GjAGRAu/SGGN8y4uO5tqcrBGAJoPawL+Ba0RkHDAnqzc758Y75+Kdc/GxBVzm7Pnn4bHHdITR669bQjDGmKjpaHbO/QbckpvXikgPoEfDhg3zXd5rr8GIEXoNwltvQQkbh2WMMZ7UFLYBZ4Q8rhPYlmsFnSV14kQYPBh69YJ//hNiYvK1G2OMKXK8SApLgUYiUl9ESgP9gNl52UFB11OoWxeuuQamTYNSpfK1C2OMKZIiPSR1CvAV0FhEtorIbc6548C9wEfAamC6c25VXvZb0JpCp056gVqZMvl6uzHGFFkR7VNwzvXPYvs8YF5+9xuOPgVjjDF/5MvuVS9WXjPGmOLAl0nBGGNMZPgyKRS0o9kYY0zmfJkUrPnIGGMiw5dJwRhjTGT4MilY85ExxkSGL5OCNR8ZY0xkiHPO6xjyTUSSgc35fHt1YHcYw/GynKJSRmGVU1TKKKxy7LNEXxkFLaeucy7TGUV9nRQKQkQSnXPxRaGcolJGYZVTVMoorHLss0RfGZEsx5fNR8YYYyLDkoIxxpg0xTkpjC9C5RSVMgqrnKJSRmGVY58l+sqIWDnFtk/BGGPMHxXnmoIxxpgMLCkYY4xJY0nBGGNMGksKxhhj0vg+KYjIRBHZJSIrI7T/KiKySUSOi8gREbkwzPsvKyLfisgPIrJKRP4vD++N6GcvaHx5LCdJRFaIyHIRSQzjfjP9G4lIZxFZKyLrReShAuz/DBFZICI/Bf4+fw53GYF9Zfk9hLOcwP5iROR7EfkggmVk+n0XtJy8/Cay++7CWEah/b7D9htyzvn6BlwCnAusjND+/wE8FyhjFVAlzPsXoELgfingG+CCaPjsBY0vj+UkAdUL4/8HEANsABoApYEfgGb53H9N4NzA/YrAz0CzcJaR3fcQ7nIC+78feA/4INx/r+y+73CUk5ffRFbfXZjLKLTfd7h+Q76vKTjnFgF7M24Xkfoi8r6IJAYydeO87ltEKqNfzEOBMpxzbl84y3DqYOBhqcAtV+OEs/rs4VSQ+KJBFn+jtsB659xG59xRYCrQK5/73+Gc+y5w/wCwGqgdzjIC+87qewhrOSJSB+gGvBmyOaxlZKPA5eTleJDNdxfOMsL++w7HcSc7vk8KmRGRUuh/6vudzg3yOHpgz6v6QDLwNjAXqCUi5cNcRrC6vhzYBcx3zn2Tn/1ESiHF54CPRWSZiNwRgf2Hqg1sCXm8lVwcDHIiIvWA1ujZYNjLyOJ7CHc5LwMjgNSQbZH4e2X2fUfqe8nxt5rhuwtrGeH8/eTwWcLyGyqZ3zdGud7AOcBMEQH9nF+EvkBEPgFOz+S9jzjn3g/cL4lW3wajf/xE9AsYGcYycM6dAOJEpAowS0SaO+ci1k+QV4UU38XOuW0iUgOYLyJrAmdKviAiFYCZwH3Ouf2B/xNhldn3EM79i0h3YJdzbpmIdAjnvjPxh+87gmX1JpvfasbvLtxlhPn3k2U5hOk3VFSTQiv0wPtWVi9wzl2Wi/1sBbY6574JnEnsR5NEOMsIff0+EVkAdAaiJikERTI+59y2wL+7RGQW2pQQqaSwDTgj5HGdwLZ8CZy9zQQmO+f+HYkyQmX4HpaEsZyLgJ4i0hUoC1QSkXeB18NYBpDl9x3OzxIqy99qFt9dWMsICtPvJ8tywvUbKpLNR8AO4EoRKQEgIi0kH6duzrlfgC0hbXYVgJ/CWYaIxAbOIBCRcsDlQCTPmvKkMOITkfIiUjF4H7iCyCbFpUCjQNtsaaAfMDs/Owp8528Bq51zL0aijEA5WX0PYSvHOfewc66Oc65eYD+fOecGRuCzZPV9h7WcEJn+VrP57sJZRrh/P1mVE77fUEF7qr2+AVMCf6hj6Jn9bUA5YAawFlgOvFuA/ccBewL7T0XPXMJWBtAS+B74MfAl/rUgnz0Cf998x5eHMhqgI01+QEd4PRLJ/x+B7V3R0SYbClIecDHalvtj4P/BcqBrOMvI6XsIZzkh++xAYPRRBD5Llt93QcvJy/Egu+8ujGWE9fedTTlh+w3ZhHjGGGPSFNXmI2OMMflgScEYY0waSwrGGGPSWFIwxhiTxpKCMcaYNJYUjDHGpLGkYIwxJo0lBWPCLHCV6WYRucvrWIzJK0sKxoSZc24FOkXDjV7HYkxeWVIwJjJ2obNZGuMrlhSMiYxRQBkRqet1IMbkhSUFY8JMRLoA5dGFmay2YHzFkoIxYSQiZdE1ve8GVgBhXQjHmEizpGBMeD0KvOOcS8KSgvEhSwrGhElgMabL0XWOwZKC8SFbT8EYY0waqykYY4xJY0nBGGNMGksKxhhj0lhSMMYYk8aSgjHGmDSWFIwxxqSxpGCMMSbN/wPQBe2UKBFgJgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "training_error_ridge = []\n",
    "test_error_ridge = []\n",
    "\n",
    "for i in range(len(y_training_ridge)):\n",
    "    training_error_ridge.append(mse(training_data_y, y_training_ridge[i]))\n",
    "    test_error_ridge.append(mse(test_data_y, y_test_ridge[i]))\n",
    "\n",
    "error_fig_ridge = plt.figure()\n",
    "plt.figure(error_fig_ridge.number)\n",
    "plt.title(\"Error Plot Ridge Regression\")\n",
    "plt.xlabel(\"$\\lambda$\")\n",
    "plt.ylabel(\"MSE\")\n",
    "x_axis = [\"$1e-{6}$\", \"$1e-{3}$\", \"$1$\", \"$3$\", \"$5$\",\"$10$\",\"$20$\",\"$30$\",\"$40$\",\"$50$\",\n",
    "          \"$1e2$\", \"$1e3$\", \"$1e5$\"]\n",
    "plt.yscale('log')\n",
    "plt.plot(x_axis, training_error_ridge, 'b')\n",
    "plt.plot(x_axis, test_error_ridge, 'r')\n",
    "# let's find the index with the minimum training error\n",
    "min_error_idx = np.argmin(test_error_ridge)\n",
    "plt.plot(x_axis[min_error_idx], test_error_ridge[min_error_idx], 'xg')\n",
    "plt.legend(['Training Error', 'Test Error', 'Min Test Error'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db58462b0>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuu0lEQVR4nO3deXiU1d3/8ffJnslCQkggEJKgArJJgLCJqFVUFIpiVdRotY8a24qXferPR2lUrIpdtK11qQrWluKIBbS4VCtFBUUQDJvsEMhCEpYsZF8myZzfHzMJk2QmC8lktu/runIlc99nZk5YPjn5nnOfW2mtEUII4Vv8XN0BIYQQfU/CXwghfJCEvxBC+CAJfyGE8EES/kII4YMCXN2BrhowYIBOTk52dTeEEMJjbN++vVhrHWvvnMeEf3JyMpmZma7uhhBCeAylVK6jc1L2EUIIHyThL4QQPkjCXwghfJDH1PztaWhoID8/n7q6Old3RViFhISQkJBAYGCgq7sihOiAR4d/fn4+ERERJCcno5RydXd8ntaakpIS8vPzGTZsmKu7I4TogEeXferq6oiJiZHgdxNKKWJiYuQ3MSE8gEeHPyDB72bk70MIz+Dx4S+EEN7qy4On+ds32Zgazb3+2hL+Qgjhpl7bcJTlm3MI8Ov936h9K/yNRkhOBj8/y2ejsUcvV1ZWxl/+8pdzeu6LL75ITU1Nl9v//e9/Z+HChR222bBhA5s3bz6n/ggh3MvhU5VsyynltimJ+En494DRCOnpkJsLWls+p6f36AdAX4Z/V0j4C+E93tmaR5C/HzdNSnDK63v0Us9uyciAtmFbU2M5npZ2Ti/52GOPcfToUVJSUrjqqquIi4tj1apV1NfXM3/+fH79619TXV3NLbfcQn5+Pk1NTTzxxBOcOnWKwsJCfvCDHzBgwAC+/PJLu6//t7/9jd/85jdERUUxfvx4goODAfjoo4949tlnMZlMxMTEYDQaqa2t5fXXX8ff35+3336bl19+mbKysnbtBg4ceE7fqxCi79Q1NPH+jnyuGTuImPBg57yJ1tojPiZNmqTb2r9/f7tjDimltWXM3/pDqa6/RhvZ2dl6zJgxWmutP/vsM33fffdps9msm5qa9Jw5c/TGjRv1mjVr9L333tvynLKyMq211klJSbqoqMjhaxcWFuqhQ4fq06dP6/r6en3xxRfrBx54QGutdWlpqTabzVprrZctW6Z/+ctfaq21Xrx4sX7++edbXsNRO2fr1t+LEKKd1ZnHddKjH+stR4t79DpApnaQqb4z8k9MtJR67B3vBevWrWPdunVMmDABgKqqKo4cOcLMmTN5+OGHefTRR5k7dy4zZ87s0utt3bqVyy+/nNhYy26sCxYs4PDhw4Dl4rYFCxZw4sQJTCaTwwuqutpOCOFejFtzOT82jKnD+jvtPXyn5r9kCRgMrY8ZDJbjvUBrzaJFi9i1axe7du0iKyuLe+65hxEjRrBjxw7GjRvH448/ztNPP93j93rwwQdZuHAhe/bs4Y033nB4UVVX2wkh3MeBExXszCvjtimJTr1uxnfCPy0Nli6FpCRQyvJ56dJzrvcDREREUFlZCcA111zDW2+9RVVVFQAFBQWcPn2awsJCDAYDd9xxB4888gg7duxo91x7pk6dysaNGykpKaGhoYHVq1e3nCsvL2fIkCEALF++3G5/OmonhHBfxq25BAX4cdOxb3t1dWJbvlP2AUvQ9yDs24qJiWHGjBmMHTuWa6+9lttvv53p06cDEB4ezttvv01WVhaPPPIIfn5+BAYG8tprrwGQnp7O7NmzGTx4sN0J3/j4eJ566immT59OVFQUKSkpLeeeeuopbr75ZqKjo7niiivIzs4G4Ic//CE33XQTH3zwAS+//LLDdkII93Sm2sSa7flcH1FL1M/vO7tIpXl1IvRahinLnID7S01N1W3v5HXgwAFGjRrloh4JR+TvRYhz88oXR3hh3WHWffIsI/Z8275BUhLk5HT59ZRS27XWqfbO+U7ZRwgh3FhdQxN/35zL5SNjGbF3q/1GeXm99n6+VfZxU1OnTqW+vr7VsRUrVjBu3DgX9UgI0dc+2FVAcVU96TPPc/rqRJDwdwtbtzr4KS+E8Alms2bZ19mMDmlk+qxUS/ArZbkaqVkvrk4EKfsIIYTLbTxcRNbpKtI/+AuqecSvteUHAPTK6sS2ZOQvhBAupLXmtY1Hia85w5zvP297stuTvF0lI38hhHChLUdL2JZdyk+/+SeB5qb2DXpxkteWhL8QQriI1po/rT/MoMgQFpw5YL9RL07y2vKp8O/l7fzPeUvn6667jrKysg7bPPnkk6xfv/4ce9a5p556ihdeeKHDNmvXrmX//v1O64MQvm5TVjHf5ZzhgS/+Tkj20bM1/ma9PMlry2fC3wnb+TsM/8bGxg6f98knnxAVFdVhm6effppZs2ade+d6gYS/EM6jteaP72xmcGUxt2x4t/mgUyd5bflM+He0nf+5st3Pf/LkycycOZN58+YxevRoAG644QYmTZrEmDFjWLp0acvzkpOTKS4uJicnh1GjRnHfffcxZswYrr76amprawG4++67WbNmTUv7xYsXM3HiRMaNG8fBgwcBKCoq4qqrrmLMmDHce++9JCUlUVxc7LC/S5YsYcSIEVxyySUcOnSo5fiyZcuYPHky48eP50c/+hE1NTVs3ryZDz/8kEceeYSUlBSOHj1qt50Q4txsPFzEztoAFn6zkuAmmwGj7SSvk4Lf+j6u36u/Kx893c/fCdv5t9rP/8svv9QGg0EfO3as5XxJSYnWWuuamho9ZswYXVxs2Zu7eS//7Oxs7e/vr3fu3Km11vrmm2/WK1as0Fprfdddd+nVq1e3tH/ppZe01lq/+uqr+p577tFaa/3AAw/o5557Tmut9aeffqoBh/cIyMzM1GPHjtXV1dW6vLxcn3/++S17/zf3S2utMzIyWt7Ltg8dtWtL9vMXomNNTWY996Wv9cU//auu9wvo3WCyQQf7+fvMyN/RnElvzqVMmTKl1Z75L730EuPHj2fatGkcP36cI0eOtHvOsGHDWjZtmzRpEjkOlnTdeOON7dps2rSJW2+9FYDZs2cTHR3tsG9ff/018+fPx2AwEBkZybx581rO7d27l5kzZzJu3DiMRiP79u2z+xpdbSeE6Njav6xmT0E5D3/9NkFmO2ViJ03y2vKZ8Hfydv4AhIWFtXy9YcMG1q9fz5YtW9i9ezcTJkywu59+860ZAfz9/R3OFzS366jNubr77rt55ZVX2LNnD4sXL3a4739X2wkhHKtZYeR3B+sZX3iYG/ZtaN/AiZO8tnwm/J2wnX+He/KXl5cTHR2NwWDg4MGDfPutnR36emjGjBmsWrUKsNxJ7MyZMw7bXnrppaxdu5ba2loqKyv56KOPWs5VVlYSHx9PQ0MDRpsZ8Lbfn6N2Qoiue+Of33AqvD9PfrEUP9rsquzkSV5bPnWFby9v599qP//Q0NBWN0efPXs2r7/+OqNGjWLkyJFMmzat997YavHixdx2222sWLGC6dOnM2jQICIiIuy2nThxIgsWLGD8+PHExcUxefLklnPPPPMMU6dOJTY2lqlTp7YE/q233sp9993HSy+9xJo1axy2E0J0TWFZLW+MnMXcA18xqeBg65NKOeVKXkdkP38PVl9fj7+/PwEBAWzZsoWf/exn7Nq1y9Xd8vm/FyHsMhr5xUeH+SQhhS+W3U9CRVHr807YxqGj/fx9auTvbfLy8rjlllswm80EBQWxbNkyV3dJCGGP0cimZ15m7Q1P8ODmd9sHfx/V+W1J+Huw4cOHs3PnzlbHSkpKuPLKK9u1/fzzz4mJiemrrgkhbNQ++RS/mvUow0oLeGDLqtYnk5Iswd8HdX5bTg9/pVQOUAk0AY1a61SlVH/gn0AykAPcorV2PFspuiwmJsYtSj9CiLNeTLyEvOh4Vr6ziJBG09kTfVznt9VXq31+oLVOsak9PQZ8rrUeDnxufSyEEN7FaGRvyiW8OXk+C3Z/xvTje1qf74P1/I64aqnn9cBy69fLgRtc1A8hhHAOo5GGn/6MReNvIrqmgl99+Vbr8y6o89vqi/DXwDql1HalVLr12ECt9Qnr1yeBgfaeqJRKV0plKqUyi4qK7DURQgj3lJHBnybOZ0/8cJ7971/oV1999lwfrud3pC/C/xKt9UTgWuABpdSltiet+0/YXW+qtV6qtU7VWqfGxsb2QVe77+KLL+60zddff82YMWNISUlp2bhNCOHdtuhIXpt2Ewt2f8bsw1vOnmiu87sw+KEPwl9rXWD9fBr4FzAFOKWUigewfj7t7H44y+bNmzttYzQaWbRoEbt27SI0NLTT9lprzGZzb3RPCNHXjEbKRozmf+c+zLDSQp78vM0SbBfW+W05NfyVUmFKqYjmr4Grgb3Ah8Bd1mZ3AR84sx/OFB4eDlj28rn88su56aabuPDCC0lLS0NrzZtvvsmqVat44oknSLP+pH/++eeZPHkyF110EYsXLwYgJyeHkSNH8uMf/5ixY8dy/Phxh+0cbQOdlZXFrFmzGD9+PBMnTuTo0aMO308I4QRGIzo9nUVjbqDE0I8/f/Q8YQ02e2C5uM5vy9lLPQcC/1KWmxMEAO9orf+jlPoOWKWUugfIBW7p6Rv9+qN97C+s6OnLtDJ6cCSLfzimy+137tzJvn37GDx4MDNmzOCbb77h3nvvZdOmTcydO5ebbrqJdevWceTIEbZt24bWmnnz5vHVV1+RmJjIkSNHWL58OdOmTeu03cqVK1m2bBm33HIL7733HnfccQdpaWk89thjzJ8/n7q6Osxms8PXufTSSzv/hoQQ3ZORwbKx1/DpyBks+vItxp06evaci9bzO+LU8NdaHwPG2zleArS/EsnDTZkyhYSEBABSUlLIycnhkksuadVm3bp1rFu3jgkTJgBQVVXFkSNHSExMJCkpqWUPoI7a2dsGurKykoKCAubPnw9ASEhIh68j4S9E79voF8NvL7ub6w5uIn3b+2dPuHA9vyNec4Vvd0boztKV7Zm11ixatIj777+/1fGcnJxWW0J31K7t+3Q0iezodYQQvchoJOc3L/LgvP9jRHEez3/yIq3uxusmdX5bPrOls7u45ppreOutt6iqqgKgoKCA06fbz3d3tV2ziIgIEhISWLt2LWDZ9K2mpqbbryOE6CajkYqFD5E+9Sf4aTNL33/Wbev8trxm5O8prr76ag4cOMD06dMBy4Tx22+/jb+//zm1s7VixQruv/9+nnzySQIDA1m9erXD14mLi3PSdyiEb6l7YjHpsx/mWP8h/H31YhLLT5096WZ1fluypbPodfL3IryF0QgZGZCXZ6nctMpxo5Gmxx9nYcptfDpyBi9+9AI37N9w9slKgYuXbMuWzkII0U1GI6SnQ02N5XFuruUxQBqWJZ2LZ9zFpyNn8Pjny1oHP7hlnd+WhL8QQtiRkXE2+JvV1FiO304GT09P4+2Jc7j/2zXcm9nmUiU3rfPb8vgJX08pW/kK+fsQ3iIvz8Hx3CaeHHEtf0u9np9kfsBjG//euoEb7NvTFR498g8JCaGkpISYmBisF5IJF9JaU1JS0nKNgRCeLDHRUuppTZN4zVZWpMwlfet7LNrwt9ZLOp1wK0Zn8ejwT0hIID8/H9nx032EhIS0XOgmhCe77jp4/XVo/mVWBTQxcG4mjDzDA5v/yf/7ekXr4PeAUo8tjw7/wMBAhg0b5upuCCG8jNEIy5efDX6/0HrifpRJ8OAzPP7FX7nnu7Vng18pO0uB3J9Hh78QQjiD7WRvQEwlcT/KxD+8DtYmcu/htWcbelCZpy0JfyGEaKN5stcwqpCY2d+jG/w59e40Ggr7nW3kYWWetiT8hRDCltFIYsBMKi6tJDI1h7r8aIo/mEhTVQhJ5FjauPGVu10l4S+EEM2MRg4ueob4tAwY2J+K74ZxZsOFYPbDQDVL+JVb7tB5Ljx+nb8QQvSGJrPm9RVfMu+W39IQrrjjvQ1EfmFAmSGJHJZyH2msdPsrd7tKRv5CCJ+3e9m7PP5dKXtS5nPtoW949rNXiamt4FleaN3Qw+v8tiT8hRA+60y1iedf/ZiVJWHEBpp4+YPfMffg19i9ZNQL6vy2JPyFED6nrqGJt175F68d19QEBPOT7R/yv5uMRJjs3BjJYPCI7Rq6S8JfCOEz6hqaWLM9n1c+/p6TjaFcmbeN/9u4nJHF7fZxsPCy0b4tCX8hhNerrm9k5bY8ln51jNOV9UwszuHPn73B1Px9jp/kwRdwdYWEvxDCa+WWVPOPLbmsyjxOZV0j08Ma+NOXr3HxtnX26/rNvGhi1xEJfyGEV6lvbGL9/tOs3n6cjYeL8FeK2WMH8T/Vh5n4i3vab9LflheXemxJ+AshPJ7ZrMnMPcPH3xfy4e5CymoaiO8XwoNXDCetIJOBv/q5vf2ZW/PSiV1HJPyFEB7J1GhmW3Yp6w+c4tO9JzhVUU9IoB9XjR7EzZMSmHHBAPxXvgMPpMto3w4JfyGExzheWsOmrGK+PlLE14eLqaxvJDjAj0tHxDL3onhmjRpIWHCAZU/mqzM6H+2D10/sOiLhL4RwS1prjpfWsi2nlG3ZJWzNLiW3xDKCHxgZzJyL4rly1EAuuWAAoUH+Z5/Y9s7rHfGBiV1HJPyFEG6huKqevQXl7C0oZ9fxMnbmlVFSbQIgyhBIalJ/7pqezMzhA7ggLrz9rVuNRstG/F0Z7YNPlnpsSfgLIfpUjamRo6erySqq5ODJSg6eqOTgyQpOVdS3tDk/NozLR8aRkhjFlOT+DI8Lx8/PzuJM28BX6uyttzriYxO7jkj4CyF6XY2pkeOlteSV1pBbUk1OSTU5xTUcK6qisLyupV2Qvx/nx4Uz/bwYxg7px9gh/Rg9OJLIkMDO36Rteacrwe/jo31bEv5C9ILmAWhenkfezrVb6hqaOF1Rz6nKOk6WWz4Ky2spLKulsKyOgrJaSq3lmmaRIQEMGxDGlGH9uSAunPNjw7kgLpzkAWEE+ndzZ/nulndARvt2SPgL92ebrP37W46Vlvbs615M6LYD0Nxcy2PwjKxpMmsqahsorTFRWm2ipMrEmRoTJVX1FFeZKKk2UVRZR1FlPUWV9VTUNbZ7DUOQP/H9QhgSbWDskH4kRIeS2N9AYn8DQ/sbiDYEtq/Rd8e5lHeayWjfLqW784foQqmpqTozM9PV3RC9qauhXlkJJpPj1zlXzSESE2P/vbv4AyI52f4gtC9XEDY2mamqb6SyrpHy2gYq6xqpqGugoraB8toGKuoaKa8xUV7bQFltA2U1luNnrMccxUBESAAxYUHERgQTGxHMgPBg4iKCiYsMYWBkCIMiQ4iPCiEiOKBn4d6R7qzesSWjfZRS27XWqXbPuSr8lVKzgT8D/sCbWuvfdtRewt+D2Qv5kpLuj+BcoQs/IPxys9F2dopRCsxmxy+ttaa2oYmq+kaq65uoqmu0ft1ItcnydVWd5XGl9euq+saWkK+sa2j5usbU1Om3EREcQJQhiH6hgUQZAokyBBEVGki0IZDosCCiDUFEhwURExZEf+tHSKB/h6/rVOdS3mn++5LRPtBx+Luk7KOU8gdeBa4C8oHvlFIfaq33u6I/wgkc/ZpeUnK2jbsHP7TqtwbqA4KorG2iOiiUqthhVDeFknjePoqC++MX1IgKasQvsAkV1Ei/4HIe+s0JqgcnUFVvCejmcK+pb6La1Ii5C38ESkF4UABhwQFEhAQQHmL5PCQqlPDgs48jQgKJCAkgMiSQyFDL536hgUSGBhIeHIC/vdUy7kbKO33GVTX/KUCW1voYgFLqXeB6QMLfkzn6j+sGIV/vH0BlcBgVwWGWzyGWz1VBBiqCDVQFG6gKMlBp/boyyPK5Oii05Xh1UChNfvZGwrnE0np0qhv8CDNVszOnjLDD+YTrBqIb6hlaXUFYkB9hkycRPn4MYcGWUA8P9icsKKAlzMOCA4iwnjME+TuvpOJqbX8rtC3xdfXfjZR3zomrwn8IcNzmcT4w1UV9ET3Rh4GvgcogA2cMkZSGRlIWGkFZSETL5/KQcMpDwikLtXxdYQ378pBw6gODO319g6mWiPoawk01hNfXEmGqJq6qlDDr8TBTLWGmWsKtny1fW45vMP2AP5se5rgpkaGmQp7TGZabfTtgXH07GUwkj0QS/QpYYn6U+TGfWU728oS0W+hK6c/2t8LOSHmnx1xS81dK3QTM1lrfa318JzBVa72wTbt0IB0gMTFxUm53an/CeXryq3kbjcqP4rBoisKiOB3en6KwaIrDoig2RFEcFkVpaD9Kw/pREhbFmeBwGv0dj1ci6quJqq2kX2Md/eqr6Vd5hn40EmGqoV9ZMRH+EGmqIaLkFBGmGiLqqomor7YGfi3+uoMCfS8ychvpLKOGsJZjBqpZyn2tf2DYm2/wpB8KvfjvpBUJ/C5zuwlfpdR04Cmt9TXWx4sAtNa/cfQcmfB1sXP4j9yo/DgRMYDCyFgK+sVxImIAJyIGcDJiACcjYjgV3p/isCi0ar/OO8JUQ0x1GTGNtfQ/P5H+I86jf7hlMjLKEET/MNsJyyAiQwO7XNM2GiHjoSrySgwkkseSsOdIC3m/9WSuEyekk8kml+R2x5PIIYdhnb9AZ5PQTlrS2kpnK7Wc8ecn5Z1uc8fwDwAOA1cCBcB3wO1aa4f3VJPwd4EuBH5NYDDHooeQGx1PTvRg8qIGkRsVT36/OE5ExrarkUfVVTGoupRBZacZRD1x0ycRd3EqcTZLCWMjgp22ysTeqkGHmeKkpah+NKFp/wNPYcaME75v2xLJddfBJ5/07JqJvlypJeWdHnG78AdQSl0HvIhlqedbWusOt9aT8O9jbVKyOjCEg7HJHBmQyOEBiRwZkMjRmAQKI+NaPW1A1RkSy08ytOwUQ8tPkVB+iiEhisH338XgO25pvfuiCzhlTX43l7L2eOTvBozcRgbPWeYsyGMJv+pwjqPLAgMhMtLzSlxuyi3Dv7sk/PuI0UjFU8+w1xTM94OGsyf+AvbHnUdOdHxLeSbUVMcFJce5oOQ455UWcF5pPslnCkkqO0l4Q51bj9T8/OwPWDtbk39OHPxQMHI76SztvObvpro8Z9FVMrp3Ggl/0aHjpTV8e6yEHZ9/x/asUxzpn9AS9AllJxl76iijTmdbPopyGFJ+Gj90+9FfzB9J+/NUt/7P6w5X42I0YnxoKxklv7RZ7fMYac2rfdz8Arhz/s3F0yewPZCEv2jlTLWJr7OK+epwEVuOllBQVgtAZF0VEwsOMrHwICmFhxh3Movoukq7r2F39OoB83Hdqvm7khtfFd2tOQsZ1buUhL+P01pztKiKz/ad4r/7T7E7vwytIaq+imk5u5met4epeXsYUZyHHx38e7D5j5xctZfckvB2TTzhjngevQNndza5c9IPiw5H/jGpZ/vhcX+43kfC30cdLarig50FfLznBMeKqgEYPzSKH9Sf4LJXnuGinL1dX9veZuTWp7Vzce7a/qTrhdU+xpJrSFfLqNGGlrdxy9+ehPvt7SOcp6KugbU7C1idmc+egnL8FEw7L4afBBZx1WvPMujQHktyN3W8EVgLB/+rExPt184TE3vhmxC9Jy2t1xM5DcCTf3sSgIS/19hbUM7yzTl89H0hdQ1mxgyO5PE5o5g3fjBxH70Hv7ApdHc1+Duo0y5ZYr927qP3wvY5TviZIvqYhL8HM5s1Gw8XsfSrY2w5VkJYkD/zJyRw+5RExiX0s/zKf2c3t8SFLv0O33xKRn9CeCYJfw+ktWb9gdP88b+HOXCigvh+IWRcN4oFU4aevfdpd2+AcQ6rMmT0J4TnkvD3MN9kFfP7/xxkd345yTEG/nDzeOalDG5/H9SMjM6D39/fMjsrw3YhfE4375wsXCWvpIb0f2SS9uZWiqtM/P6mi1j/y8v40aSE1sFvNEJyMsbci0kmGz+aSCYbI7e1fkGDAZYvt4R/To4EvxA+Rkb+bs7UaObVL7N4beNRAvwUj1wzknsuGWZ/4zNrqcdYc32ry+9zSSadZQCWy+/lghshfJ6Evxvbk1/OI2t2c/BkJdenDGbRtaMY1C+kfcM29zrN4LlWV94C1BBGhvotaSvmSOgLIST83VFjk5mXPj/CqxuOEhMWxF/vSuXKUQPtN7YzsZuH/cX2eXqoBL8QApDwdzunK+p4cOVOtmaXcuOEISz+4Rj6GQIdP8HOxG4ieXYvv09M8tL7wAohuk0mfN3I5qPFXPfSJnbnl/HCzeP544IUx8Fvndi1t4Z/Cb/CQHWrY3IBlhDCloz83cQ7W/N44oO9JMcYMN47lZGDIhw37mQNf/Oe6hn+vyevaQiJSUrmd4UQrUj4u5jZrHl+3SFe23CUy0bE8srtE4gI6WC0n9GFK3YNBtKWziEtLaH3OyyE8AoS/i5kajTz8OrdfLS7kNunJvL0vDEEtL1Yq1lXr9iVZZxCiC6Q8HcRU6OZhe/sYN3+Uzw6+0J+etl5KNXBhGxXrtj1hM30hRBuQSZ8XcDUaObnRkvwP/XD0fzs8vMdB38HE7utyIyuEKIbZOTfxxqazPzcuJ31B07z9PVj+PH0ZMeNpdQjhHASCf8+pLVm0ft7WH/gNM9cP4Y7Owp+6LzUI7dPEkKcIyn79KE/rT/Cmu35PHTl8I6DvyulnqQkCX4hxDmTkX8feXdbHi99foSbJyXwi1nDHTfsSqlHJnaFED0kI/8+8O2xEjLW7uXSEbE8d+O4nq3qkYldIUQvkPB3stMVdSx8ZydJ/Q28evuE9jddaSsvz/E5KfUIIXqJlH2cqLHJzMKVO6mub8R471THV+7C2at3tbZ/Xko9QoheJOHvRM+vO8S27FJeXJDSo716pNQjhOhtUvZxko2Hi3hj4zHumJbIDROGdNy4ozq/lHqEEE4g4e8E5bUNPLrme4bHhfP4nNGOG3a2pFMpub+uEMIppOzjBM9+vJ+iqnreuHOS/XvtQteWdCbavyOXEEL0lIz8e9kXB0+xens+P7vsfMYPjbLbxmiE5Lsuw6+mkmSyMXJb+0ZS5xdCOJGEfy8qr2lg0ft7uHBQBA9eeYHdNs0D/tymBDR+5JJMOsta/wCQOr8QwsmcFv5KqaeUUgVKqV3Wj+tszi1SSmUppQ4ppa5xVh/62p/WH6aosp4Xbh5PcID9ck/GQ1XtKj01hJHBc5YHzUs6JfiFEE7k7Jr/n7TWL9geUEqNBm4FxgCDgfVKqRFa6yYn98WpDp6sYMW3udwxLYmxQ/rZb2Q0kldip8QD5JEopR4hRJ9xRdnneuBdrXW91jobyAKmuKAfvUZrza8/3E9ESAC/vGqE44YZGSRi/wreRP9CKfUIIfqMs8N/oVLqe6XUW0qpaOuxIcBxmzb51mPtKKXSlVKZSqnMoqIiJ3f13H269yRbjpXw8NUjiTIEOW6Yl8cSfoWB6laHDVSzZHmCBL8Qos/0KPyVUuuVUnvtfFwPvAacD6QAJ4A/dPf1tdZLtdapWuvU2NjYnnTVaWpNTSz59wFGxUdy+xQHSzOb1/NrTRorWcp9JJGDwkwSOSyNWSS5L4ToUz2q+WutZ3WlnVJqGfCx9WEBMNTmdIL1mEd665tsCspq+cMt4/H3s7Nbp531/GmsJI2VlgcGA/x5aR/1VgghLJy52ife5uF8YK/16w+BW5VSwUqpYcBwYJuz+uFMlXUNLP3qGD8YGcu082LsN5KtG4QQbsiZq31+r5RKATSQA9wPoLXep5RaBewHGoEHPHWlz1ubciivbeCXV4103MjRFs3NWzcIIYQLOC38tdZ3dnBuCeDRaxrLaxp4c9Mxrh49kHEJdpZ2drZFs2zdIIRwIdnb5xy9uekYlXWN/K+9pZ2yRbMQws3J9g7noLTaxFubspkzLp5R8ZHtG0idXwjh5mTkfw7+9k02NQ1NPOToRuxS5xdCuDkZ+XdTramJt7/NZdaogYwYaHN3rua1/H5+lg97pM4vhHATMvLvpvd35nOmpoF7Lxl29mDbGn+TncVLUucXQrgRGfl3g9ms+eumbMYN6ceUYf3PnnBU4/f3t5R6pM4vhHAzMvLvho2HizhWVM2fb01BKZureR3V+M1my4cQQrgZGfl3w5ubjjEoMoTrxsW3PuGoli81fiGEm5Lw76L9hRV8k1XCXRcnE+hv/WOzvQG7arOvj9T4hRBuTMK/i/6xJYfQQP+zO3e23I8x1/JY67M/AKTGL4Rwc1Lz74Lq+kY+2l3InIvi6WcItBy0N8mr9dnbMAohhBuTkX8X/Pv7E1Sbmrh1ss1O1I4meR0dF0IINyLh3wX/zDzOebFhTEqKPntQJnmFEB5Mwr8TWacr2Z57hgWpQy3LO2WSVwjhBST8O/HP744T4Ke4cWKCTPIKIbyGTPh2wNRo5v0dBVw5Ko7YiGCZ5BVCeA0Z+Xfgi4OnKKk2cetkax1fJnmFEF5Cwr8D7+0oYGBkMJeOiLUckEleIYSXkPB3oKKugY2HipgzbjD+K9+RSV4hhFeR8Hfgv/tOYWoyM+fE9zLJK4TwOjLh68C/95xgSFQoE597QCZ5hRBeR0b+dpTXNPD1kSLmXBSPkkleIYQXkvC347N9J2lo0sy9KF4meYUQXknC346P95wgMaiJcZekyCSvEMIrSfi3UVpt4pvDp5mz+UOUTPIKIbyUhH8b/9l7kiYUfnuCSSYbP5pIJhujvvXsJK8EvxDCw8lqnzY+3XuCmNIKnjr9O2oIByCXZNJZBrnpSOwLIbyBjPxtVNc3svVYKSVHz28J/mY1hJHh/zsX9UwIIXqXhL+NTVnFmJrMnDpynt3zeU1D+rhHQgjhHBL+Nr74eDMRphoGFlTaPZ+YpOweF0IITyPhb2V+28gX+TVcejSTJeYMDFS3Oi+rO4UQ3kTC32rfH16nKCyaK45mksZKlnIfSeSgMMvqTiGE15HVPlafhw1FaTOXH8sEII2VpLHSsr4/x+zi3gkhRO/q0chfKXWzUmqfUsqslEptc26RUipLKXVIKXWNzfHZ1mNZSqnHevL+venLUTOYUHiImNqK1idkGwchhBfqadlnL3Aj8JXtQaXUaOBWYAwwG/iLUspfKeUPvApcC4wGbrO2danTlXXsjknmirxdrU9IoV8I4aV6FP5a6wNa60N2Tl0PvKu1rtdaZwNZwBTrR5bW+pjW2gS8a23rOkYjG+b9BIArTuyDmBhLqUcK/UIIL+asCd8hwHGbx/nWY46O26WUSldKZSqlMouKinq/l0YjpKfzRdQw4iuKGHVkF9TWwooVso2DEMKrdRr+Sqn1Sqm9dj6cPmLXWi/VWqdqrVNjY2N7/w0yMmiqrWNL4kXMzNmJAsuNWzIyev+9hBDCjXS62kdrPescXrcAGGrzOMF6jA6O9728PPbHnUd5aAQzcna3Oi6EEN7MWWWfD4FblVLBSqlhwHBgG/AdMFwpNUwpFYRlUvhDJ/Whc4mJbE4aD8D0vO9bHRdCCG/W06We85VS+cB04N9Kqc8AtNb7gFXAfuA/wANa6yatdSOwEPgMOACssrZ1jSVL2HzeBC4oziOu+ozlmKzwEUL4gJ6u9vmX1jpBax2stR6otb7G5twSrfX5WuuRWutPbY5/orUeYT3nmpQ1GiE5GdNdP+G7waO4+NRhWeEjhPApvneFr3WFDzU1fD9kFDWBIVycu8uywkdCXwjhI3xvb5+MDMuKHmBz0niUNjP1yHZZ4SOE8Cm+N/K3WcmzOfEiRp86RnRdJeRVubBTQgjRt3xv5G9dyVMXEMSOIaO4uHmVj6zwEUL4EN8L/yVLwGBg+5BRmAICuTj3e1nhI4TwOb4T/tYVPtx5J4SG8s2F0/A3NzFZVcgKHyGEz/GNmr/NCh8ASkrYMnAkF4VpwrPs7UsnhBDezTdG/jYrfMBS798bN4wpO75wYaeEEMJ1fCP82+zVs2fgBTT4BzLpwDYXdUgIIVzLN8K/zUqeHUMuBGCiqnRFb4QQwuV8I/ytK3yabR8yiuSyEwx4cpELOyWEEK7jG+GflmZZ0ZOUhFaKHYljmJgcIyt8hBA+y7vDv3l5p5+fZdJ3yRLyiiopDolk4uUTXd07IYRwGe9d6tl2eWduLqSns+N3gUAYk5KiXdo9IYRwJe8d+bdZ3glATQ3bv9xOeHAAIwZGuKZfQgjhBrw3/B3cinF7ZAITEqPw91N93CEhhHAf3hv+djZqqwoK5VBsMhMSpeQjhPBt3hv+bZZ3AuxOHofZz0/q/UIIn+e94W+zvLP5Fo3b738EpSBlaJSreyeEEC7lveEPlh8AOTlgNkNODtujEhkRF0G/0EBX90wIIVzKu8PfhtaaXcfLmJAY5equCCGEy/lM+OefqaW8toFxCf1c3RUhhHA5nwn/PQXlAIwbIuEvhBA+E/57C8oJ8FNycZcQQuBL4V9YwYiBEYQE+ru6K0II4XI+Ef5aa/YWlDN2SKSruyKEEG7BJ8L/RHkdpdUmqfcLIYSVT4R/82TvGAl/IYQAfCT89xWU4++nGB0vZR8hhAAfCf89BeVcEBsuk71CCGHlE+G/t7CCsVLyEUKIFl4f/qcq6iiqrJeVPkIIYaNH4a+UulkptU8pZVZKpdocT1ZK1Sqldlk/Xrc5N0kptUcplaWUekkp5dS7quyVK3uFEKKdno789wI3Al/ZOXdUa51i/fipzfHXgPuA4daP2T3sQ4f2FJSjFIweLCN/IYRo1qPw11of0Fof6mp7pVQ8EKm1/lZrrYF/ADf0pA+d2VtQwfmx4RiCvPde9UII0V3OrPkPU0rtVEptVErNtB4bAuTbtMm3HnOavQXlUvIRQog2Oh0OK6XWA4PsnMrQWn/g4GkngEStdYlSahKwVik1prudU0qlA+kAiXbuydsZU6OZS4YP4JILBnT7uUII4c06DX+t9azuvqjWuh6ot369XSl1FBgBFAAJNk0TrMccvc5SYClAamqq7m4/ggL8eOHm8d19mhBCeD2nlH2UUrFKKX/r1+dhmdg9prU+AVQopaZZV/n8GHD024MQQggn6elSz/lKqXxgOvBvpdRn1lOXAt8rpXYBa4Cfaq1Lred+DrwJZAFHgU970gchhBDdpyyLbtxfamqqzszMdHU3hBDCYyiltmutU+2d8/orfIUQQrQn4S+EED5Iwl8IIXyQhL8QQvggCX8hhPBBHrPaRylVBOS6uh/dNAAodnUn+ph8z75BvmfPkKS1jrV3wmPC3xMppTIdLbPyVvI9+wb5nj2flH2EEMIHSfgLIYQPkvB3rqWu7oALyPfsG+R79nBS8xdCCB8kI38hhPBBEv5CCOGDJPz7iFLqYaWUVkp5/W3FlFLPK6UOKqW+V0r9SykV5eo+OYNSarZS6pBSKksp9Zir++NsSqmhSqkvlVL7lVL7lFIPubpPfUUp5W+9Le3Hru5Lb5Hw7wNKqaHA1UCeq/vSR/4LjNVaXwQcBha5uD+9znqzoleBa4HRwG1KqdGu7ZXTNQIPa61HA9OAB3zge272EHDA1Z3oTRL+feNPwP8BPjG7rrVep7VutD78lta37vQWU4AsrfUxrbUJeBe43sV9ciqt9Qmt9Q7r15VYwnCIa3vlfEqpBGAOlptQeQ0JfydTSl0PFGitd7u6Ly7yP3jn3dqGAMdtHufjA0HYTCmVDEwAtrq4K33hRSyDN7OL+9GrOr2Bu+icUmo9MMjOqQzgV1hKPl6lo+9Za/2BtU0GllKBsS/7JpxLKRUOvAf8Qmtd4er+OJNSai5wWmu9XSl1uYu706sk/HuB1nqWveNKqXHAMGC35X71JAA7lFJTtNYn+7CLvc7R99xMKXU3MBe4UnvnxSQFwFCbxwnWY15NKRWIJfiNWuv3Xd2fPjADmKeUug4IASKVUm9rre9wcb96TC7y6kNKqRwgVWvtaTsDdotSajbwR+AyrXWRq/vjDEqpACyT2VdiCf3vgNu11vtc2jEnUpYRzHKgVGv9Cxd3p89ZR/7/T2s918Vd6RVS8xfO8AoQAfxXKbVLKfW6qzvU26wT2guBz7BMfK7y5uC3mgHcCVxh/XvdZR0RCw8kI38hhPBBMvIXQggfJOEvhBA+SMJfCCF8kIS/EEL4IAl/IYTwQRL+QgjhgyT8hRDCB/1/VprG48RWS0oAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let us visualize the newly fitted model with the optimal lambda value here\n",
    "x = np.linspace(-5, 5, 100)\n",
    "new_features = get_polynomial_features(x, 10)\n",
    "new_norm_feat = normalize_features(new_features, mean_train_feat, std_train_feat)\n",
    "y_pred = eval(new_norm_feat, weights_ridge[min_error_idx])\n",
    "\n",
    "plt.plot()\n",
    "plt.plot(test_data_x, test_data_y, 'or')\n",
    "plt.plot(training_data_x, training_data_y, 'ob')\n",
    "plt.plot(x, y_pred)\n",
    "plt.legend([\"test_data\", \"training_data\", \"inference\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 1.4) Error Plot (1 Point)\n",
    "In the lecture we have seen and analyzed the plot of polynomial degrees \n",
    "against the error (slide  47).\n",
    "Similarly, now please analyze the relationship between the error and the \n",
    "different values of $\\lambda$, as well as the reason behind it.\n",
    "\n",
    "Hint: Do not forget that we are in log space. Small changes in the y-axis mean high differences in the error values. <br><br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\lambda$ is regularization term such that the model cannot fit the training data perfectly anymore. Increasing $\\lambda$ strongly encourages the weight values to decay towards zero, unless supported by the data. In our case, a polynomial feature with degree 10 is highly overfitting to the training data and, therefore, with higher $\\lambda$ values (10-15), we can punish overfitting, which leads to lower MSE values. Increasing the $\\lambda$ value leads to an decrease in the error due to the higher punishment of high weight values and tackles though this the high polynomial degree. Of course, with low $\\lambda$ values, the MSE on the training data is low because there is no punishment of overfitting.<br>\n",
    "On the other hand, if $\\lambda$ is increased too far, the introduced penalty will force the regression to become to simple and underfit the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probability Basics and Linear Classification\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First Example (Two Moons)\n",
    "\n",
    "Let us start by loading a very simple toy dataset, the \"two moons\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db53bbfd0>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAi+0lEQVR4nO3df5Ac5X3n8fdXqxVafD6tfh2GXQnJgeBEWCCyBz5zhR3JNpiKJYXwy6lgkcMl+4LPZ6hTIYorhVPsszAuQ1z2hShALC4pQJKJELEdApIpV9mBaIlA/PBhBP7BrmVDJKQ7W0II8b0/pkeane2e6dnp6R/Tn1fV1s5098w+M2r108/3+T7PY+6OiIiU16SsCyAiItlSRSAiUnKqCERESk4VgYhIyakiEBEpuclZF2AiZs2a5fPmzcu6GCIihfLkk0/+q7vPrt9eyIpg3rx5DA8PZ10MEZFCMbOfhm1XaEhEpORUEYiIlJwqAhGRkitkH4GISKcdOXKEkZER3njjjayL0rKpU6cyODhIb29vrONVEYiIhBgZGeGd73wn8+bNw8yyLk5s7s7evXsZGRlh/vz5sV6jikBya8vOUW59+AV+vv8Qp/T3serCM1i+aCDrYklJvPHGG4WrBADMjJkzZ/Laa6/Ffo0qAsmlLTtHufGBZzh05CgAo/sPceMDzwCoMpDUFK0SqGq13Ooslly69eEXjlUCVYeOHOXWh1/IqEQi3UsVgeTSz/cfamm7SBkcPnyYK664gtNOO43zzjuPn/zkJ4m8ryoCSd2WnaOcv24781d/i/PXbWfLztFxx5zS3xf62qjtImVw1113MX36dHbv3s11113HDTfckMj7qiKQVFVj/6P7D+Ecj/3XVwarLjyDvt6eMdv6entYdeEZKZZWJL44Nzituueee1i4cCFnnXUWV111FQ8++CArVqwA4NJLL2Xbtm0kscqkOoslVY1i/7WdwNXHyhqSIuhEcsNzzz3H5z//eX7wgx8wa9Ys9u3bxwUXXMCcOXMAmDx5MtOmTWPv3r3MmjWrrfKrIpBUtRL7X75oQBd+KYS4Nzit2L59O5dddtmxi/yMGTPaLmcUVQSSqlP6+xgNuejXxv7zMn4gL+WQ/EsruWFgYIBXXnmFwcFB3nrrLQ4cOMDMmTPbfl/1EciETDQe2iz2H9aH8Ln7n2LR2n9MJOYaV9y+DBHoTHLD4sWL2bRpE3v37gVg3759LF26lA0bNgCwefNmFi9enMhYB7UIpGXtxEObxf7DmtgArx88kuqAsk409aV7rbrwjDH/J6D95IYFCxZw00038YEPfICenh4WLVrEHXfcwVVXXcVpp53GjBkzuO+++5IovioCaV27F8lGsf9GTek0L8QaxyCt6FRyw4oVK45lCVVt2rSprfcMo4pAWtbJi2RUH0KSf6NWVD9AnL6MZu8h5VLk5Ab1EUjLOjnYK6wPIem/UdWoHyDuOAb1JUg3SKQiMLO7zexVM3s2Yr+Z2VfNbLeZ7TKzc2r2rTCzF4OfFWGvl+zVdg7/+vBb9PaM7aBKarDX8kUDfPGS99LfN34e9aQHlDULcX3xkvcy0N+HAQP9fXzxkveOu+PTnEjSDZIKDX0D+BpwT8T+jwKnBz/nAX8BnGdmM4A/BYYAB540s63u/npC5ZKYGoU36juH9x86Qu8kY/qJvew/eCTxcEi1iZ1EyCXqPbbsHI0MQVXDT3Ga+upLkG6QSEXg7t8zs3kNDlkG3OOVsdCPm1m/mZ0MfBB4xN33AZjZI8BFwL1JlEviaZYFFHbXe+Rt58Qpk9m55iMdK1e7MdeozzX8031888no0E0r4adW+hJE8iqtPoIB4JWa5yPBtqjt45jZSjMbNrPhVhZckOaahTeKetcb9bnufeKV0BRVaD38pDmRpBsUprPY3de7+5C7D82ePTvr4nSVZhf6os4EGvW5jjaYpCusH6CRuH0JIkn43ve+xznnnMPkyZPZvHlzYu+bVvroKDCn5vlgsG2USniodvtjKZVJAs3CG50YLJOkVlNAe8xCK4OB/r4JXcCLnDYoxTJ37ly+8Y1v8OUvfznR902rRbAV+ESQPfQ+4IC77wEeBj5iZtPNbDrwkWCbpKhZeCPPd70TSQH9+HlzFM6R5O3aCLedCTf3V37v2tj2W9ZPQz1v3jwWLlzIpEnJXroTaRGY2b1U7uxnmdkIlUygXgB3vwP4NnAxsBs4CPxxsG+fmf0ZsCN4q7XVjmNJT5xRkXm9623Uv/H91YuPHVP/uYZOnaFBYJKcXRvhoc/CkaAFeuCVynOAhZdP6C3DpqHulKSyhj7eZL8D10bsuxu4O4lyyMTl9ULfTLP+jajPVdTPKzm1be3xSqDqyKHK9glWBJqGWgotzSkXlL4puXBgpLXtOVOYrCEphrSnXOiG9M1OLHEoKZs22Nr2GMKmoe4UVQSSqLSnXMhzR3YcmquoSyxZA711rdDevsr2Caqdhvqss87i+uuvZ8eOHQwODrJp0yY+9alPsWDBgjYLXqHQUAl1MnSTxeCzIsf7te5Bl6j2A2xbWwkHTRusVAIT7B+oCpuGemQk+XCTKoKS6cQi27UUs29NUUdtS4iFl7d94c+KQkMl0+nQTTfE7NNU1FHb0l1UEZRMp+9Aix6zT5sqznzzBtOR5Fmr5VZoqGTSCN0UOWaftk4tcSjtmzp1Knv37mXmzJmJLBCfFndn7969TJ06NfZrVBGUTN7nDSojVZz5NDg4yMjICEWc7Xjq1KkMDsZPXVVFUDK6AxWJp7e3l/nz52ddjFSoIigh3YGKSC1VBClKc+qFTumGzyAiY6kiSEmn8/fT0A2fQUTGU/poStKeeqETuuEziMh4ahGkpBtGkHbDZ0ibQmlSBGoRpKQbRpB2w2dIkyaUk6JQRZCSbhhB2g2fIU0KpUlRKDSUkm7I3++Gz5AmhdKkKFQRpChu/n6e48oagxCfZmKVolBFkKGwCz6gFM0uoek8pCgSqQjM7CLgz4Ee4E53X1e3/zbgd4OnJwL/zt37g31HgWeCfT9z96VJlCnvonLyT5g8SQuVdIlOhNLy3FqU4mq7IjCzHuDrwIeBEWCHmW119+erx7j7dTXH/xdgUc1bHHL3s9stR9FEdSTWb6tSXLmYkgylaUCfdEoSWUPnArvd/WV3fxO4D1jW4PiPA/cm8HcLrdULu+LKoiwk6ZQkKoIB4JWa5yPBtnHM7FRgPrC9ZvNUMxs2s8fNbHnUHzGzlcFxw0WcFrZe1IV9+om9StGUUMpCkk5JexzBlcBmd6+9rTnV3YeAPwRuN7PfCHuhu6939yF3H5o9e3YaZe2oqJz8P/3YAq3wJaE0oE86JYnO4lFgTs3zwWBbmCuBa2s3uPto8PtlM3uMSv/BSwmUK9eadSTqwl8OrXT+KgtJOiWJimAHcLqZzadSAVxJ5e5+DDN7DzAd+KeabdOBg+5+2MxmAecDX0qgTIWgnPxya7XzVwP6pFMsicWZzexi4HYq6aN3u/sXzGwtMOzuW4Njbgamuvvqmte9H/hL4G0qYarb3f2uZn9vaGjIh4eH2y63SFrC7vxvffiF0AFnUAkJ6iIvSTOzJ4NQ/NjtSVQEaVNFIEVSf+cPlZBOVKpw7THqH5IkRVUEmnROpMOi0j57zBq+TqmhkhZVBCIdFpXeedR9XOZY3NeKJEkVgUiHRaV3VlODBxqkfyo1VNKgikCkwxqt47B80QDfX72Y2684WwMJJTOafTTnNMlY8cVJ+1RqqGRJWUM5FpVtokwSEZkIZQ0VkCYZq7FrI9x2JtzcX/m9a2PWJRLpGgoNZSROyEeTjAV2bYSHPgtHgs994JXKc4CFl2dXLpEuoRZBBqohn9H9h3COTy2wZefYKZo0yVhg29rjlUDVkUOV7SLSNlUEGYgb8mmUbVIqB0Za2y4iLVFoKANxQz7KJAlMG6yEg8K2i0jbVBFk4JT+vtDJxsJCPpqhFFiyZmwfAUBvX2W7iLRNoaEMKOTTooWXw8e+CtPmAFb5/bGvqqNYJCFqEWSgWchHg8hCLLxcF36RDlFF0ET1ojy6/xA9Zhx1D50rvtWLd1TIp9XFSkSkBHZtrGTJHRip9I0tWZPojZFCQw3UpnlCZbZIGJ/uGTcdNA4NIhORMarjaA68AvjxcTQJDqpURdBA2EW5qvbinOTFW4PIRGSMFMbRqCJooNnFt7o/yYu3BpGJyBgpjKNRRdBAs4tvdX+SF29lFInIGFHjZRIcR6OKoIGwi3JV7cU5yYv38kUDxxYrMY4vXqKOYpGSWrKmMm6mVsLjaBLJGjKzi4A/B3qAO919Xd3+q4FbgWrv6dfc/c5g3wrgvwfbP+/uG5IoUxJq0zwbZQ3FGQHcSlaRBpGJyDHV7KAOZg21vR6BmfUAPwI+DIwAO4CPu/vzNcdcDQy5+2fqXjsDGAaGAAeeBH7H3V9v9DeLth6B1hUQkTzo5HoE5wK73f1ld38TuA9YFvO1FwKPuPu+4OL/CHBRAmXKFaWEikieJREaGgBqZwQbAc4LOe4PzOwCKq2H69z9lYjXht4im9lKYCXA3LlzEyh2epQSKlnYsnOU//HQc7x+8AgA/X293Lx0gVqhaerwQLCkpNVZ/BAwz90XUrnrb7kfwN3Xu/uQuw/Nnj078QJ2klJCJW1bdo6yavPTxyoBgP2HjrBq09MTGugoE5DCQLCkJFERjAJzap4PcrxTGAB33+vuh4OndwK/E/e13UApoZK2Wx9+gSNHx/f/HXnbFZJMS4EWVEoiNLQDON3M5lO5iF8J/GHtAWZ2srvvCZ4uBX4YPH4Y+J9mNj14/hHgxgTKlCtaV0DS1ijs2GifJjxMUIEWVGq7InD3t8zsM1Qu6j3A3e7+nJmtBYbdfSvwWTNbCrwF7AOuDl67z8z+jEplArDW3fe1W6Y8UkqopClqzYvqvjCa8DBhBVpQKZE+Anf/trv/prv/hrt/Idi2JqgEcPcb3X2Bu5/l7r/r7v+n5rV3u/tpwc9fJ1EekbJbdeEZ9PbYuO29kywyJKnstoSlMBAsKRpZLNKFli8a4NZLz2L6ib3HtvX39XLrZWdF3t0ruy1hBVpQSesRiHSpVsORrSyhKjEVZEEltQhEBFB2W5mpRSAigLLbykwVQUxKq5MyUHZbOakiiEFpdSLSzdRHEENUWt3n7n+K89dt15D9pO3aCLedCTf3V37ncEi+SCJycq6rRRBDo/Q5tQ4SVp2fpTo0vzo/CxQi+0Ikthyd62oRxNAsfU6DbhJUoPlZRNqSo3NdFUEMjZasrNKgm4Q0mp8lJ81okUTkaC4iVQQx1K4jHEWDbhISNQ9L3/TCTOkrJTPRG5QUFqWPSxVBTMsXDfD91Yu5/Yqzx7UOenuMXx9+i/mrv6XO43ZFzc8CuWlGd6MtO0c5f912ncOtamfNgRzNRaTO4hbVD7rpP7GXX73xFvsPVRYAUedxm6IW6n5gZfjxOZzSt2iUHt2C+hXH3vx19A1Ksw7fFBalj6vtxeuzkKfF689ftz10fpaB/j6+v3pxBiXqUredGTGl7xy47tn0y9NFdA7HVJ/l05DBzfs7XaKWdXLx+lLTjI0pyVEzutvoHI4pLMsnSg7XHGhEFUGbtB5xSgo0pW/R6ByOKW4YsoA3KOojaNOqC88YE18FzdjYMQWZ0jctSc1/pXM4pqgVx/pmwJR3ZB7nb4cqgjZpxkbJwpado6za/PSxBepH9x9i1eangdY7eHUOx7Rkzfg+gt4++OgtE7/w13c+q7M4vjx1FssE5OTkL7JFa/+R1w8eGbf9xN5JTH/HCbqgd0qS525Y53NvX0dDnlGdxWoRSLpyNL9KkYVVAgAHj7zNwaCTV2mgHZBkeLLRFBNFnGvIzC4ysxfMbLeZrQ7Zf72ZPW9mu8xsm5mdWrPvqJk9FfxsTaI87dDAmg5rNr+KppFIlObByrEcTTHRdovAzHqArwMfBkaAHWa21d2frzlsJzDk7gfN7D8DXwKuCPYdcvez2y1HEjSwJgXN5hJSayGW/r7eY4MYm1EaaE5FdT4XdIqJc4Hd7v6yu78J3Acsqz3A3b/r7geDp48DuUyyjVp3QHdUCWo0v0qOZmPMu5uXLqB3ksU6VmmgOZWjsTFJVAQDQG21NhJsi3IN8J2a51PNbNjMHjez5QmUZ8I0sCYFjU7+HDWV8275ogFuvewsBvr7MCqjgP/ofXO1+HyR5GhsTKqdxWb2R8AQ8IGazae6+6iZvRvYbmbPuPtLIa9dCawEmDt3bkfKd0p/X+hQe91RJajR/Crb1uamqVwEYesLD506Q2mgRZKTsTFJVASjwJya54PBtjHM7EPATcAH3P1wdbu7jwa/Xzazx4BFwLiKwN3XA+uhkj6aQLnH0cCalESd/FF52gUbpZklLT4vE5FEaGgHcLqZzTezKcCVwJjsHzNbBPwlsNTdX63ZPt3MTggezwLOB2o7mVNVu+5Atbn9xUveq/9YaclRU1mkTBIZUGZmFwO3Az3A3e7+BTNbCwy7+1YzexR4L7AneMnP3H2pmb2fSgXxNpVK6XZ3v6vZ39OAMpFkJDVNhRRD1IAyjSwWKan6dGmohELVCu5emoZaRMZQurRUqSIQKSmlS0uVKgKRktI6BAnokilRVBGIlNSqC8/QALR2tLNwfc6oIhApKaVLt6mLpkTRNNQiJaYBaG3ooilRVBEkSDnZkjc6JzsoR7OHtkuhoYRUc7JH9x/COT6FtdYzkKzonOywHM0e2i5VBAlRTrbkjc7JDuuiKVEUGkqIcrIlb1o9JxVGmoCczB7aLrUIEqKcbMmbVs5JhZHKTRVBQpSTLXnTyjmpMFK5KTSUkGoTWk3rCdq1MXyxGpmwVs5JhTbLTRVBgpSTPUFhi9Y/sBJ+9jj83leyLVvBxT0ntTpfuSk0JNkLG6GJw/DdhRyuX0QKbUbokrmEmlFFINmLHInphRyuX0SabiJEF80l1IxCQ5K9qBGaUMjh+kWl0GadRnMJdVn/lVoEkr0lawAL31fA4frSJbpoLqFmVBFI9hZeDkP/iXGVQbPh+iWJ30pGom5CuvDmRBWB5MPvfQUuWR9/uH6J4reSkS6aS6iZRCoCM7vIzF4ws91mtjpk/wlmdn+w/wkzm1ez78Zg+wtmdmES5ZGCWng5XPcs3Ly/8rtRHLaL5oKXnOqiuYSaabuz2Mx6gK8DHwZGgB1mttXdn6857BrgdXc/zcyuBG4BrjCz3wauBBYApwCPmtlvuvvYIY4i9eLGbzVQTdrRJXMJNZNEi+BcYLe7v+zubwL3AcvqjlkGbAgebwaWmJkF2+9z98Pu/mNgd/B+Io3Fid8qfCQSSxIVwQBQm/s3EmwLPcbd3wIOADNjvjYRW3aOcv667cxf/S3OX7ddk2kVXZz4rcJHkoUCJjEUZhyBma0EVgLMnTu3pddWZ1asTqpVnVkRUN50UVWb643CPiVK/5OcCJsu5aHPVh7nOMSURItgFJhT83ww2BZ6jJlNBqYBe2O+FgB3X+/uQ+4+NHv27JYKqJkVu1SzzuUSpf9JwiZ6V1/QVmgSFcEO4HQzm29mU6h0/m6tO2YrsCJ4fCmw3d092H5lkFU0Hzgd+OcEyjSGZlYsqRKl/0mCdm2EB68d27f04LXxKoOCtkLbrgiCmP9ngIeBHwIb3f05M1trZkuDw+4CZprZbuB6YHXw2ueAjcDzwD8A13YiY0iLxpRUidL/JEHfuQGOvjl229E3K9uhcWuhoK1Qq9yYF8vQ0JAPDw/HPr6+jwAqMyuWflItERnv5mnR+y75q7F9AFBpZVZvMOr7COr3Z8zMnnT3ofrtheksbkfcBTq0ZqsUmc7fFDSbiC5OEkMOlaJFEIdaDVJkOn8TdMt8OLRv/Pa+GXDodSDsmmmVpIWci2oRaK6hgDKLcqiA+dhZ0fmboI/eApN6x26b1FvZ3qwPoKDnbClCQ3EosyhnCpqPnRWdvwlqFt4J6wNYsqbQ56wqgoDWbM2ZEi0KkgSdvwmLmmOoUSVx25mFPWcVGgpozdacaTcfu6BN9InS+ZuiqIGMBR1DAGoRHBM3s0hSErV8ZZx87AI30SdK528OtHPOZkxZQ5JP7eRj33ZmxH/IOZU7OJFOyPkYAlDWkBRNO6OCJ9JEL1koSTqgwCPZFRqS/JrooiCtNtFLGEqSDinoQjZqEUj3aXWyuYLOGCmSFFUE0n1abaIXONtDJAkKDUl3qm+iV/sAwgYIFTjbQyQJahFI92u2drHWLZCSU0Ug3a9ZH0CBsz1EkqDQkHS/OH0ABc32EEmCKgLpTrs2Hp8PxiZB2MJ36gMQAVQRSDeqHxcQVgmoD0Aaqb2RKMjiMu1QRSDdJ6xPAMB6wN8uxX9saUMJBxiqs1i6T1SfgB8NUkVHKpWFppGQMCUcYKiKQLpPZOzfolNIRapKOMCwrYrAzGaY2SNm9mLwe3rIMWeb2T+Z2XNmtsvMrqjZ9w0z+7GZPRX8nN1OeUSA8HEBGOPWmu3yuzxpoNEkg82Wo+xC7bYIVgPb3P10YFvwvN5B4BPuvgC4CLjdzPpr9q9y97ODn6faLI9I+LiA0AXH6eq7PImgAYbjtFsRLAM2BI83AMvrD3D3H7n7i8HjnwOvArPb/LsijdWvIjVtTvhxze7yND1199EAw3HazRo6yd33BI9/AZzU6GAzOxeYArxUs/kLZraGoEXh7ocjXrsSWAkwd+7cNostpbNkTfSi41FKmD1SChpgOE7TFoGZPWpmz4b8LKs9zitLnUUud2ZmJwP/G/hjd3872Hwj8B7g3wMzgBuiXu/u6919yN2HZs9Wg0JaNJG7vBJmj6QuixZXCfsAmmnaInD3D0XtM7NfmtnJ7r4nuNC/GnHcvwW+Bdzk7o/XvHe1NXHYzP4a+G8tlV6kFa3e5ZUweyRVWbW4JtI67HLt9hFsBVYEj1cAD9YfYGZTgL8D7nH3zXX7Tg5+G5X+BS0oK/mhO8fOyqrFVcI+gGba7SNYB2w0s2uAnwKXA5jZEPBpd/9ksO0CYKaZXR287uogQ+hvzWw2ldy+p4BPt1kekeR0+s6xZNMYjJNli6tkfQDNtFURuPteYEnI9mHgk8HjvwH+JuL1i9v5+yIdVb1QdOJirY5oLQiUI5prSKSRTt05NgqLdHNFUNsK6psOPVPg6JvH95c8Vp8VTTEhkoU8dkR3OoOnfiDXoX3gDn0zUKw+W2oRiGQhb2GRNEJVYa2gt4/AlHfADT9O5m/IhKhFIJKFvE1jkEYGTx5bQQKoIhDprKhwS95SGNO4SCsdN7cUGhLplGbhljylMKYRqtJArtxSi0CkU4o0RUUaoao0W0GaLLAlahGIdEqRYuKdHDNR/3c63QqK2/Fd9gF9NVQRiHRK3jKDmslTqKodccZoaEDfGAoNiSSlPhxx+kfylRnUiiKHVuK0xIoUtkuBKgKRJIStejV8FzCpeAOmmq3glXdxspOKFLZLgUJDIkkIu8MEOPJr4G24ZH3+K4CqqLvlv/s0PLDyeDy9emzeYuxxspOKFrbrMLUIRJLQ6E6yaCGHqM/iRznWQnjwWtjyJ/lsNcTJTsrbgL6MqUUgkoSoO8yqIoUcmn0WGDtRXFWeJs1r1vGdVpZUQagiEElCWDiiVpFCDs0+SyN5rfCiUkVLeuGvp4pAJAnVC8p3bqjMqlmraCGH+rtlmxSEhWLIY4WnVNGm1EcgkpSFl1dm0bzkr/Izh9BELbwcrnsWbt4Pv3/H+Hh6zxSY1Dt2W14rPKWKNqUWgUjSui3kEBVPD9uWx8+tVNGmVBGIyHGNYulhF/k8XvjrKVW0KYWGRKSi6APJoihVtKm2KgIzm2Fmj5jZi8Hv6RHHHTWzp4KfrTXb55vZE2a228zuN7Mp7ZRHRNrQrbH0vK39kEPthoZWA9vcfZ2ZrQ6e3xBy3CF3Pztk+y3Abe5+n5ndAVwD/EWbZRLpDmnPjtnNsfRu67dJWLuhoWXAhuDxBmB53BeamQGLgc0Teb1I5jo5MVsWYRqtIFZa7VYEJ7n7nuDxL4CTIo6bambDZva4mS0Pts0E9rv7W8HzEWAg6g+Z2crgPYZfe+21Nost0qZOX6izCNMoll5aTUNDZvYo8K6QXTfVPnF3NzOPeJtT3X3UzN4NbDezZ4ADrRTU3dcD6wGGhoai/o5IOuLMed+OLMI0mnahtJpWBO7+oah9ZvZLMzvZ3feY2cnAqxHvMRr8ftnMHgMWAd8E+s1sctAqGARGJ/AZRNLX6Qt1VimPiqWXUruhoa3AiuDxCuDB+gPMbLqZnRA8ngWcDzzv7g58F7i00etFcqnT8fQ0wjRFXnxGEtVuRbAO+LCZvQh8KHiOmQ2Z2Z3BMb8FDJvZ01Qu/Ovc/flg3w3A9Wa2m0qfwV1tlkckHZ2+UHc65TFPYwZUIWXOKjfmxTI0NOTDw8NZF0PKrsiLn992ZkToaU5ljqG01E8IB5UKVXn+HWFmT7r7UP12TTEhMlFFjqdPpI+jExVfpzvdJRZNMSFSRq32cXQqlNTNg9gKRBWBSBm12sfRqXENGsSWC6oIRMqo1c7oTt25axBbLqiPQKSsWunj6NS4Bg1iywVVBCLSXNg6xknduRe5071LKDQkIs1pKueuphaBiMSjO/eupRaBiEjJqSIQESk5VQQiIiWnikBEpORUEYiIlFwhZx81s9eAn3b4z8wC/rXDf6MTilpuUNmzUNRyg8o+Eae6++z6jYWsCNJgZsNh07XmXVHLDSp7FopablDZk6TQkIhIyakiEBEpOVUE0dZnXYAJKmq5QWXPQlHLDSp7YtRHICJScmoRiIiUnCoCEZGSU0UQMLPLzOw5M3vbzCLTuszsIjN7wcx2m9nqNMsYUZ4ZZvaImb0Y/J4ecdxRM3sq+NmadjnrytLwOzSzE8zs/mD/E2Y2L4NijhOj3Feb2Ws13/MnsyhnGDO728xeNbNnI/abmX01+Gy7zOyctMsYJka5P2hmB2q+89wsbWZmc8zsu2b2fHBt+a8hx+Tje3d3/VT6SX4LOAN4DBiKOKYHeAl4NzAFeBr47YzL/SVgdfB4NXBLxHG/yvo7jvsdAn8C3BE8vhK4vyDlvhr4WtZljSj/BcA5wLMR+y8GvgMY8D7giazLHLPcHwT+PutyRpTtZOCc4PE7gR+FnDO5+N7VIgi4+w/d/YUmh50L7Hb3l939TeA+YFnnS9fQMmBD8HgDsDy7osQS5zus/UybgSVmZimWMUwe/+1jc/fvAfsaHLIMuMcrHgf6zezkdEoXLUa5c8vd97j7vwSP/x/wQ2Cg7rBcfO+qCFozANQu3DrC+H/YtJ3k7nuCx78AToo4bqqZDZvZ42a2PJ2ihYrzHR47xt3fAg4AM1MpXbS4//Z/EDTxN5vZnHSKlog8nttx/Qcze9rMvmNmC7IuTJggvLkIeKJuVy6+91KtUGZmjwLvCtl1k7s/mHZ54mpU7ton7u5mFpUPfKq7j5rZu4HtZvaMu7+UdFlL7iHgXnc/bGafotKqWZxxmbrdv1A5t39lZhcDW4DTsy3SWGb2b4BvAp9z9/+bdXnClKoicPcPtfkWo0DtXd5gsK2jGpXbzH5pZie7+56gSflqxHuMBr9fNrPHqNydZFERxPkOq8eMmNlkYBqwN53iRWpabnevLeOdVPpviiKTc7tdtRdWd/+2mf0vM5vl7rmYjM7MeqlUAn/r7g+EHJKL712hodbsAE43s/lmNoVKR2amGTjB318RPF4BjGvZmNl0MzsheDwLOB94PrUSjhXnO6z9TJcC2z3oWctQ03LXxXaXUokJF8VW4BNBFsv7gAM1IcfcMrN3VfuPzOxcKte0rG8agEpGEHAX8EN3/0rEYfn43rPuWc/LD/D7VOJzh4FfAg8H208Bvl1z3MVUev9fohJSyrrcM4FtwIvAo8CMYPsQcGfw+P3AM1QyXZ4Brsm4zOO+Q2AtsDR4PBXYBOwG/hl4d9bfc8xyfxF4Lvievwu8J+sy15T9XmAPcCQ4z68BPg18OthvwNeDz/YMEZlzOSz3Z2q+88eB92dd5pqy/0fAgV3AU8HPxXn83jXFhIhIySk0JCJScqoIRERKThWBiEjJqSIQESk5VQQiIiWnikBEpORUEYiIlNz/B+YEMR/KIohBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from typing import Tuple, Callable\n",
    "\n",
    "data = dict(np.load(\"two_moons.npz\", allow_pickle=True))\n",
    "samples = data[\"samples\"]\n",
    "labels = data[\"labels\"]\n",
    "\n",
    "c0_samples = samples[labels == 0]  # class 0: all samples with label 0\n",
    "c1_samples = samples[labels == 1]  # class 1: all samples with labe 1 \n",
    "\n",
    "plt.figure(\"Data\")\n",
    "plt.scatter(x=c0_samples[:, 0], y=c0_samples[:, 1], label=\"c0\")\n",
    "plt.scatter(x=c1_samples[:, 0], y=c1_samples[:, 1], label=\"c1\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us also define some plotting utility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def draw_2d_gaussian(mu: np.ndarray, sigma: np.ndarray, plt_std: float = 2, *args, **kwargs) -> None:\n",
    "    (largest_eigval, smallest_eigval), eigvec = np.linalg.eig(sigma)\n",
    "    phi = -np.arctan2(eigvec[0, 1], eigvec[0, 0])\n",
    "\n",
    "    plt.scatter(mu[0:1], mu[1:2], marker=\"x\", *args, **kwargs)\n",
    "\n",
    "    a = plt_std * np.sqrt(largest_eigval)\n",
    "    b = plt_std * np.sqrt(smallest_eigval)\n",
    "\n",
    "    ellipse_x_r = a * np.cos(np.linspace(0, 2 * np.pi, num=200))\n",
    "    ellipse_y_r = b * np.sin(np.linspace(0, 2 * np.pi, num=200))\n",
    "\n",
    "    R = np.array([[np.cos(phi), np.sin(phi)], [-np.sin(phi), np.cos(phi)]])\n",
    "    r_ellipse = np.array([ellipse_x_r, ellipse_y_r]).T @ R\n",
    "    plt.plot(mu[0] + r_ellipse[:, 0], mu[1] + r_ellipse[:, 1], *args, **kwargs)\n",
    "\n",
    "# plot grid for contour plots\n",
    "plt_range = np.arange(-1.5, 2.5, 0.01)\n",
    "plt_grid = np.stack(np.meshgrid(plt_range, plt_range), axis=-1)\n",
    "flat_plt_grid = np.reshape(plt_grid, [-1, 2])\n",
    "plt_grid_shape = plt_grid.shape[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2): Classification using Generative Models (Naive Bayes Classifier)\n",
    "\n",
    "We first try a generative approach, the Naive Bayes Classifier.\n",
    "We model the class conditional distributions $p(\\boldsymbol{x}|c)$ as Gaussians, the class prior $p(c)$ as\n",
    "Bernoulli and apply Bayes rule to compute the class posterior $p(c|\\boldsymbol{x})$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a small recap, recall that the density of the Multivariate Normal Distribution is given by\n",
    "\n",
    "$$ p(\\boldsymbol{x}) = \\mathcal{N}\\left(\\boldsymbol{x} | \\boldsymbol{\\mu}, \\boldsymbol{\\Sigma} \\right) = \\dfrac{1}{\\sqrt{\\det \\left(2 \\pi \\boldsymbol{\\Sigma}\\right)}} \\exp\\left( - \\dfrac{(\\boldsymbol{x}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x}-\\boldsymbol{\\mu})}{2}\\right) $$\n",
    "\n",
    "and we already saw how to implement it in the python introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def mvn_pdf(x: np.ndarray, mu: np.ndarray, sigma: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Density of the Multivariate Normal Distribution\n",
    "    :param x: samples, shape: [N x dimension]\n",
    "    :param mu: mean, shape: [dimension]\n",
    "    :param sigma: covariance, shape: [dimension x dimension]\n",
    "    :return p(x) with p(x) = N(mu, sigma) , shape: [N] \n",
    "    \"\"\"\n",
    "    norm_term = 1 / np.sqrt(np.linalg.det(2 * np.pi * sigma))\n",
    "    diff = x - np.atleast_2d(mu)\n",
    "    exp_term = np.sum(np.linalg.solve(sigma, diff.T).T * diff, axis=-1)\n",
    "    return norm_term * np.exp(-0.5 * exp_term)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Practical Aspect:** In practice you would never implement it like that, but stay\n",
    "in the log-domain. Also for numerically stable implementations of the multivariate normal density the symmetry and\n",
    "positive definitness of the covariance should be exploited by working with it's Cholesky decomposition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum likelihood estimator for a Multivariate Normal Distribution is given by\n",
    "$$ \\boldsymbol{\\mu} = \\dfrac{1}{N} \\sum_{i}^N \\boldsymbol{x}_i \\quad \\quad \\boldsymbol{\\Sigma} = \\dfrac{1}{N} \\sum_{i}^N (\\boldsymbol{x}_i - \\boldsymbol{\\mu}) (\\boldsymbol{x}_i - \\boldsymbol{\\mu})^T. $$\n",
    "\n",
    "This time, before we use it, we are going to derive it:\n",
    "\n",
    "### Exercise 2.1): Derivation of Maximum Likelihood Estimator (5 Points):\n",
    "\n",
    "Derive the maximum likelihood estimator for Multivariate Normal distributions, given above.\n",
    "This derivations involves some matrix calculus.\n",
    "Matrix calculus is a bit like programming, you google the stuff you need and then plug it together in the right order.\n",
    "Good resources for such rules are the \"matrix cookbook\" (https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf) and the Wikipdia article about matrix calculus\n",
    "(https://en.wikipedia.org/wiki/Matrix_calculus ). State all rules you use explicitly\n",
    "(except the ones given in the hints below). \n",
    "\n",
    "**Remark** There are different conventions of how to define a gradient (as column-vector or row-vector). This results in different ways to write the Jacobian and thus different, usually transposed, matrix calculus rules:\n",
    "- In the lecture we define the gradient as column-vector \n",
    "- In the Wikipedia article this convention is referred to as \"Denominator Layout\". It also contains a nice explanation of the different conventions for the gourmets among you ;) \n",
    "- The Matrix Cookbook uses the same convention (gradient as column vector)\n",
    "- Please also use it here\n",
    "\n",
    "**Hint** Here are two of those rules that might come in handy\n",
    "\n",
    "$\\dfrac{\\partial\\log\\det(\\boldsymbol{X})}{\\partial \\boldsymbol{X}} = \\boldsymbol{X}^{-1}$\n",
    "\n",
    "$\\dfrac{\\partial \\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x}}{\\partial \\boldsymbol{x}} = 2 \\boldsymbol{A}\\boldsymbol{x}$ for symmetric matrices $\\boldsymbol{A}$ (hint hint: covariance matrices are always\n",
    "symmetric)\n",
    "\n",
    "There is one missing to solve the exercise. You need to find it yourself. (Hint hint: Look in the matrix cookbook, chapter 2.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Solution\n",
    "\\begin{align}\n",
    "    p(\\boldsymbol{x_i}) \n",
    "    &= \\mathcal{N}\\left(\\boldsymbol{x_i} | \\boldsymbol{\\mu}, \\boldsymbol{\\Sigma} \\right)\\\\\n",
    "    &= \\dfrac{1}{\\sqrt{\\det \\left(2 \\pi \\boldsymbol{\\Sigma}\\right)}} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\n",
    "    &= \\dfrac{1}{\\sqrt{(2 \\pi)^{K} |\\boldsymbol{\\Sigma}}|} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n",
    "\\end{align}\n",
    "\n",
    "With K as the dimensions of the quadratic matrix $\\Sigma$\n",
    "\n",
    "##### calculate the likelihood\n",
    "\\begin{align}\n",
    "    \\text{lik}(\\boldsymbol{\\theta};D) \n",
    "    &= \\prod_i p(x_i)\\\\\n",
    "    &= \\prod_i^{N} \\dfrac{1}{\\sqrt{(2 \\pi)^{K} |\\boldsymbol{\\Sigma}}|} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n",
    "    &= \\prod_i^{N} 2\\pi^{-\\frac{K}{2}} |\\boldsymbol{\\Sigma}|^{-\\frac{1}{2}} \\exp\\left( - \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\n",
    "     &= (2\\pi)^{-\\frac{NK}{2}} |\\Sigma|^{-\\frac{N}{2}} \\exp\\left( - \\dfrac{\\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\right)\\\\\\\\\n",
    "    \\log\\text{lik}(\\boldsymbol{\\theta};D) \n",
    "    &= \\sum_i \\log p(x_i)\\\\\n",
    "    &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu}) \\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### derive $\\mu$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "    \\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial\\mu}\n",
    "    &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
    "    &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}^T-\\boldsymbol{\\mu}^T) \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
    "    &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} - \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu} - \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} + \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
    "    &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}+ \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
    "    &= -\\dfrac{1}{2}\\sum_i^N -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + 2\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
    "    &= -\\sum_i^N -\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + \\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
    "    &= -\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for Optimum: $-\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu} = 0$\n",
    "\\begin{align} \n",
    "   0 &= -\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n",
    "   0 &= \\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n",
    "   0 &= -N\\boldsymbol{\\mu}\\sum_i^N \\boldsymbol{x_i}^T \\\\\n",
    "   \\boldsymbol{\\mu} &= \\frac{1}{N}\\sum_i^N \\boldsymbol{x_i}^T \\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### derive $\\Sigma$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\frac{\\partial\\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial \\boldsymbol{\\Sigma}}  &= \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\left(\\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right) \\\\\n",
    "&= -\\frac{N}{2}\\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\log(|\\Sigma|) - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
    "\\end{align}\n",
    "Because $(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})$ is scalar, we can take its trace and obtain the following form \n",
    "\\begin{align}\n",
    "    &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\text{Tr}\\left((\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right)\\\\\n",
    "\\end{align}\n",
    "The rule on page 10 (formula 63) of the matrix cookbook allows us to derive the trace\n",
    "\\begin{align}\n",
    "    &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T\\\\\n",
    "    &= -\\frac{N}{2}(\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T = 0 \\\\\n",
    "    &= -N \\Sigma^{-1} + \\Sigma^{-1}(\\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T) \\Sigma^{-1} = 0\\\\\n",
    "\\end{align}\n",
    "Multiply from both sides with $\\Sigma$\n",
    "\\begin{align}\n",
    "-\\Sigma N + \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T &= 0 \\\\\n",
    "\\Sigma = \\frac{1}{N} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Implementation**\n",
    "\n",
    "Lets reuse one of the implementations from the zeroth-exercise for that "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def mvn_mle(x: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"\n",
    "    Maximum Likelihood Estimation of parameters for Multivariate Normal Distribution\n",
    "    :param x: samples shape: [N x dimension]\n",
    "    :return mean (shape: [dimension]) und covariance (shape: [dimension x dimension]) that maximize likelihood of data.\n",
    "    \"\"\"\n",
    "    mean = 1 / x.shape[0] * np.sum(x, axis=0)\n",
    "    diff = x - mean\n",
    "    cov = 1 / x.shape[0] * diff.T @ diff\n",
    "    return mean, cov\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use this maximum likelihood estimator to fit generative models to the samples of both classes. Using those models and some basic rules of probability we can obtain the class posterior distribution $p(c|\\boldsymbol{x})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.2) Generative Classifier (2 Points)\n",
    "\n",
    "Given a way to fit the class conditional using our Maximum Likelihood estimator, we can implement the generative classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.83\n"
     ]
    }
   ],
   "source": [
    "# Fit Gaussian Distributions using the maximum likelihood estimator to samples from both classes\n",
    "mu_c0, sigma_c0 = mvn_mle(c0_samples)\n",
    "mu_c1, sigma_c1 = mvn_mle(c1_samples)\n",
    "\n",
    "# Prior obtained by \"counting\" samples in each class\n",
    "p_c0 = c0_samples.shape[0] / samples.shape[0]\n",
    "# LEAVE AS EXERCISE\n",
    "p_c1 = c1_samples.shape[0] / samples.shape[0]\n",
    "\n",
    "def compute_posterior(\n",
    "        samples: np.ndarray,\n",
    "        p_c0: float, mu_c0: np.ndarray, sigma_c0: np.ndarray,\n",
    "        p_c1: float, mu_c1: np.ndarray, sigma_c1: np.ndarray) \\\n",
    "        -> Tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"\n",
    "    computes the posteroir distribution p(c|x) given samples x, the prior p(c) and the\n",
    "    class conditional likelihood p(x|c)\n",
    "    :param samples: samples x to classify, shape: [N x dimension]\n",
    "    :param p_c0: prior probability of class 0, p(c=0) \n",
    "    :param mu_c0: mean of class conditional likelihood of class 0, p(x|c=0) shape: [dimension]\n",
    "    :param sigma_c0: covariance of class conditional likelihood of class 0, p(x|c=0) shape: [dimension x dimension]\n",
    "    :param p_c1: prior probability of class 1 p(c=1) \n",
    "    :param mu_c1: mean of class conditional likelihood of class 1 p(x|c=1) shape: [dimension]\n",
    "    :param sigma_c1: covariance of class conditional likelihood of class 1, p(x|c=1) shape: [dimension x dimension]\n",
    "    :return two arrays, p(c=0|x) and p(c=1|x), both shape [N]\n",
    "    \"\"\"\n",
    "    # TODO: compute class likelihoods \n",
    "    # TODO: compute normalization using marginalization\n",
    "    # TODO: compute class posterior using Bayes rule\n",
    "    p_x_c0 = mvn_pdf(samples, mu_c0, sigma_c0)\n",
    "    p_x_c1 = mvn_pdf(samples, mu_c1, sigma_c1)\n",
    "    p_x = 1 / samples.shape[0]\n",
    "    p_c0_given_x = (p_x_c0*p_c0)/p_x\n",
    "    p_c1_given_x = (p_x_c1*p_c1)/p_x\n",
    "    return p_c0_given_x, p_c1_given_x\n",
    "\n",
    "\n",
    "p_c0_given_x, p_c1_given_x = compute_posterior(samples, p_c0, mu_c0, sigma_c0, p_c1, mu_c1, sigma_c1)\n",
    "# Prediction\n",
    "predicted_labels = np.zeros(labels.shape)\n",
    "# break at 0.5 arbitrary\n",
    "predicted_labels[p_c0_given_x >= 0.5] = 0.0  # is not strictly necessary since whole array already zero.\n",
    "predicted_labels[p_c1_given_x > 0.5] = 1.0\n",
    "\n",
    "# Evaluate\n",
    "acc = (np.count_nonzero(predicted_labels == labels)) / labels.shape[0]\n",
    "print(\"Accuracy:\", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the class likelihoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db5734730>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABHU0lEQVR4nO2dd3hUVfrHPyeNEHpCbwlVmoCAKKgI2BVx7QpiW3/YUMR1Leuuy6rs6uquYMHeQbErNmygKODSpPdOqCEQekiZ8/vjnTGTZGYySe7MvXfmfJ7nPHdm7p1739zc+d73vuc971FaawwGg8EQ+yTYbYDBYDAYooMRfIPBYIgTjOAbDAZDnGAE32AwGOIEI/gGg8EQJxjBNxgMhjjBCL4h5lFK/UUp9YrddhgMdqNMHr7B6SilNgFpQBut9WHvZzcB12itB9po14/AyUARUAwsBm7XWi+1yyaDIRTGwze4hURgtN1GBGCU1ro2kA78CLxtrzkGQ3CM4BvcwhPAPUqp+oFWKqUmKKW2KqUOKKUWKKVO81s3Vik1yfv6a6XUqDLfXayUusT7upNS6jul1F6l1Gql1BXhGKe1LgamAF389ttXKTVHKZWnlNqhlHpWKZXiXfecUuo/ZeyYqpQa433dXCn1kVIqRym1USl1Z5n9zvf+rbuUUv8Nx0aDwQi+wS3MRzzoe4Ksnwf0RDztd4APlFKpAbZ7F7ja90Yp1QXIBL5UStUCvvN+vzFwFTDRu01IvEI+HPjV7+NiYAzQEOgHnAHc5l33JnC1UirB+/2GwJnAO97PPkdCRC2837tLKXWO97sTgAla67pAO+D9iuwzGMAIvsFdPATcoZRqVHaF1nqS1jpXa12ktf4PUAM4LsA+PgF6KqUyve+HAx9rrY8BQ4BNWuvXvfv5DfgIuDyETU8rpfKAg8Ao4B9+Ni3QWv/q3dcm4EXgdO+6ucB+RMxBbi4/aq13AScCjbTWD2utC7TWG4CXvdsAFALtlVINtdaHtNb+NxmDIShG8A2uQWu9DPgCuL/sOqXUPUqplUqp/V4Brod41mX3cRD4khLxvBqY7H2dCZzkDcHkefczHGgawqw7tdb1gZrIDeNDpVR3r00dlVJfKKV2KqUOAP8sY9ObwDXe19dQEv/PBJqXseMvQBPv+j8CHYFVSql5SqkhIewzGH4nyW4DDIZK8ndgIfB7/Nsbr78X8ZaXa609Sql9gAqyj3eBvyulZgKpwAzv51uBn7TWZ1XWKK21B/hZKbUOOBtYAjwP/AZcrbU+qJS6C7jM72uTgGVKqR5AZ+BTPzs2aq07BDnWWkrCQZcgN5kMXwaTwRAM4+EbXIXWeh3wHnCn38d1kNTIHCBJKfUQUDfEbr5CvOiHgfe8Yg3y9NBRKTVCKZXsbScqpTqHY5tSqh/Sabvcz64DwCGlVCfg1jJ/SzbS9/A28JHW+qh31VzgoFLqPqVUTaVUolKqm1LqRO9xrlFKNfLanef9jgeDoQKM4BvcyMNALb/33wDTgDXAZiAf8ZID4o3Xf4y3k9Tv84OId34VsB3YCTyO9AcE41ml1CGl1CFEuP+qtf7au+4eYBgS338ZuVGV5U3gePzSOb0ZP0OQTuiNwB7gFSRMBXAusNx7zAnAVX43C4MhKGbglcFgI0qpAUhoJ1ObH6MhwhgP32CwCaVUMjKY7BUj9oZoYATfYLABb79AHtAMGG+rMYa4wYR0DAaDIU4wHr7BYDDECY7Nw2/YsKHOysqy2wyDwWBwFQsWLNijtS43Gh0cLPhZWVnMnz/fbjMMBoPBVSilNgdbZ0I6BoPBECcYwTcYDIY4wQi+wWAwxAmOjeEbDAZDpCksLCQ7O5v8/Hy7Tak0qamptGzZkuTk5LC/YwTfYDDELdnZ2dSpU4esrCyUClZc1XlorcnNzSU7O5s2bdqE/T0T0jEYDHFLfn4+GRkZrhJ7AKUUGRkZlX4yMYJvMBjiGreJvY+q2G1COoaIU1wMeXnS9u2T5YEDkJ8Px46Vb4WFkJAgLTFRmv/rtDSoXbt0q1VLlg0bymuDwVAeI/iGKrN/P2zaBNu3B267d4vAHzwYXbtq1oRGjaBxY1n6XrdsCZmZ0rKyoH59cKlzZ4hxjh07xrXXXsuCBQvIyMjgvffew4rKA0bwDSHZvx9WroR160q39ethz57y2zdsCM2bS+vaFRo0kFa/fklr0ADq1IHUVKhRo3xL8l6VHo88Hfgvi4rgyBE4dEja4cMlrw8ehNxcudHk5EjbvRuWL5dl2XBnnTolN4COHaFz55KWkRHhE2swhODVV1+lQYMGrFu3jilTpnDffffx3nuB5s+pHEbwDYAI6Zo1sGQJLF0qyyVLYMuWkm2UgtatoX17uPRSaNcO2rSBFi1E4Js2FcG2Cl8IpywNGlR+X1rLDWrz5vJt40aYPh2O+s0Z1aiRCH+XLnDCCdCnD3TrBikpVf97DIZgvPXWWzz55JMopejevTu7d+9m7NixAFx22WWMGjUKrXW1+xuM4MchWsPWrfDrryVt4UKJn4N42J06wamnwvHHi9B16CBhECsFPZooVRLe6dOn/HqPR8R/5crS7d134YUXZJuUFDkfffpA795w4onyPtBNyeA+7roLFi2ydp89e8L48aG3Wb58OY8++iizZ8+mYcOG7N27lwEDBtCqVSsAkpKSqFevHrm5uTRs2LBa9hjBjwM8Hli8WLzY2bNF4Ldvl3WpqSJet98uF2f37iL2bhX2qpKQIE8rbdrA+eeXfK41bNgACxZImz8fpkyBF1+U9fXqyY1xwABpvXtDJcbBGAxMnz6dyy+//HcxT09Pj9ixjODHIFpLjP2HH6RNny6xbYC2bWHgQOjXD04+WQTehCmCo5SErtq1gyuukM985/fXX2HmTGlffinr0tLk3J55ptw4jj/edAy7hYo88WjSokULtm7dSsuWLSkqKmL//v1kWNCxZAQ/Rjh2DGbMgM8+g6+/lvAESHx9yBA44wwYPFjeG6qHUtKP0b49XHONfLZrF/z8s4j/jz/CAw9Ia9ECzjtPxP+MM6BuXVtNNziQwYMHc/HFF3P33XeTkZHB3r17GTp0KG+++Sb9+vXjww8/ZPDgwZaMFzCC72L27oWvvoKpU2HaNMlSqVULzjoL7r1XBKZjR+NhRoMmTeCyy6SBhMymTZP/z/vvwyuvSN/I6afD5ZfDJZdIf4LB0LVrVx588EFOP/10EhMTOeGEE3jhhRcYMWIE7du3Jz09nSlTplhyLMfOadunTx9tJkApz/798PHH8M474tEXF0t2zNChcNFF4sWnptptpcGfwkLpO/nqK/j0U8mGSkiAQYMkTHTxxUb87WLlypV07tzZbjOqTCD7lVILtNYBUhNMaQVXUFAgoZorrhBxv/FG6Ui8916JI2/bJp2I559vxN6JJCeLZ//447BqlWSCPPCApLzefDM0awZnny0ZQS4s2mhwESak42CWLJGUwClTZMRqo0Zw000wfDicdJIJ1bgRpaBHD2mPPCLZUx98AJMnw7BhMjBt+HD44x8l/99gsBLj4TuM/Hx4+23o319E4fXXpdPvyy/Fk3/mGcmuMWLvfpSSVNhx4+SJ7bvv5H/9yivQq5e0iRNlFLHBYAVG8B3Chg1wzz2S1XHttZJG+d//ishPnizhGpPfHbskJEgq5zvvSIfvM89I+uftt0OrVhK+27rVbisNbscIvs0sXAhXXikjWSdMkE7X77+XWO+YMRDBMRgGh5KeDqNGwW+/SWfvWWfBf/4jg8KuugrmzrXbQoNbMYJvA1rLgKizzpKRmdOmiXe/ebPEc884w4RsDEK/fpLWuWGDDP3/+mvpvzn9dMnSMhgqgxH8KKK1xOL79pXH92XL4LHHJFvj8celAJmhNJMnSw2fhARZTp5st0X2kJkJTz4J2dkyInTdOnkaHDgQfvrJbusMVjNz5kx69epFUlISH374oWX7NYIfJX75RWqtDBkiA6ZeekmqNN53n9RjMZRn8mQYOVKefLSW5ciR8Sv6ICWdR4+W0g4TJkhO/8CBIv4//2y3dQaraN26NW+88QbDhg2zdL9G8CPM4sVwwQVw2mnyI33+eYnP/9//mZz5injwQal978+RI/J5vJOaCnfeKdfU+PFS2XPAABnBu3693dYZKstbb71F9+7d6dGjByNGjCArK4vu3buTkGCtRJs8/Aixa5d472++KbnVjz0Gd9whxbUM4eFfiz+cz+ORmjXF4/+//xPh/+c/JWx4111yYzS1eyrBgrtg3yJr99mgJ/QeH3KTQOWRI4Xx8C2mqAieflpq2LzzjqTTbdgg4m/EvjyhYvStWwf+TrDP45m0NPjLXyTEM2wY/Pvfkvn18stSHtvgXEx5ZJfyyy+SN71kiQyVf/ppOO44u61yLr4YvS9s44vRg4w2HTeu9HoQYRs3Lvq2uoXmzWWw3qhR4uWPHCkD+V5+2VyLFVKBJx4LGA/fAg4ckEfq006TEggffSSpluYHFpqKYvTDh0vndmampKlmZsr74cOjb6vb6N1bSjW//rpMWdmjh4QVCwvttsxQlsGDB/PBBx+Q6520IpIhHbTWjmy9e/fWbmD6dK0zM7VOSND6z3/W+tAhuy2yj0mT5FwoJctJk0Jvr5TWkn9TuilVsr+MjJLPMzIq3mc07XcLO3Zofemlcg5POEHr336z2yLnsGLFCrtN0Fpr/cYbb+iuXbvq7t276+uuu07PnTtXt2jRQqelpen09HTdpUuXgN8LZD8wXwfRVduFPVhzuuAfPqz16NFyBjt00Hr2bLstspdJk7ROSyst3GlpoUUzMzOw4PvENjm5/LqUlMgIcVXsdxsffaR106ZyXp96SmuPx26L7Mcpgl9VjOBHgUWLtD7uODl7d9wR3169j1DiHYxQIhtsfxXtM5r2u5E9e7QeOlT+tgsvlPfxTLwJvonhV5I335RqlQcPSs2bp5+WWabinaqkUIaK0Yf6XnXTMgNlBoVjfyyM+s3IkElYJkyAb76Rap1mwFYcEexOYHdzmod/9KjWI0eKZzRokNa7dtltkbOw2kOOlIcf7KnCv68g0LFiMeQzf77W7dtL/9PTT8dniGfFihXa49I/3OPx2BPSAV4DdgPLgqxXwNPAOmAJ0KuifTpJ8Ddv1rp3bzlb99+vdWGh3RY5B//wS9lO2OoIYiRi+JMmaZ2YGFjYMzJCC3qshnz27y8J8dx2W/xd2xs2bNA5OTmuE32Px6NzcnL0hg0byq0LJfhW5eG/ATwLvBVk/XlAB287CXjeu3Q8ixbJpBRHjsij8EUX2W2RPUyeLOmSW7bIwCdfLrx/nrzWEprRWsIz48ZVPYXS973Ro2VuAJBwxIQJ4e/T3+b0dAnDFRcH3nbvXslXL/s3+o4Vq6N+69aVOZIfeACeeEKKsr33nowOjwdatmxJdnY2OTk5dptSaVJTU2nZsmXlvhTsTlDZBmQR3MN/Ebja7/1qoFmo/TnBw//2W61r19a6VSutly2z2xr7qGoYxE4C2RyqVWRzrHr4/rz6qtZJSVp36iRPtQZ3ggM6bVsA/vP1ZHs/K4VSaqRSar5Sar7dd9y33pJZptq2hTlzoGtXW82xlWADpHyed1mi6fUG60gNZHMwwhm9O25c+dIYsTbq98YbZZrFHTukENuGDXZbZLCcYHeCyjZCe/hfAKf6vf8B6BNqf3Z6+OPHi/d2xhla5+XZZoZjCDZAKpppk4EI1ZEars2JieH3CcTqwKyyzJ+vdXq61i1aaL1qld3WGCoLDvDwtwGt/N639H7mOJ57TmqQXHopfPWVqVUPwYuVZWTY6/WGKs0QToG1tDRJsw23T2D4cNi0SYqRbdoUuyUeeveW2bQKCmRmreXL7bbIYBXREvypwLVKOBnYr7XeEaVjh82LL0rRqYsugnffhZQUuy1yBsHCGRMm2FvrJlRHaiCbk5PlJmXq8lRM9+4yk1ZCAgwaZGrsxwzBXP/KNOBdYAdQiMTn/wjcAtziXa+A54D1wFIqCOdoG0I6r74qj/gXXKB1fn5UD+0KqhPOiFQopKKO1HgJwUSSlSslvNO+vda7d9ttjSEcMKUVQvPVVzL45JxzZICVwToiOWDJDYOhYuGmM2uW1qmpWvfta8qIuAEj+CFYulTrOnWkiqC5mAUrRSrS6YxOFlQ33JDC5ZNPxCkaMkTroiK7rTGEIpTgK1nvPPr06aPnz58f0WPs3g19+0rn1Ny5UNkxDLFI2UlJQGLhVY13JySI1JVFqdifiSkrSyZ1KUtmpnT6uo3nn4fbboO//Q0efthuawzBUEot0Fr3CbQubounFRTAxReL6E+dasTeh9UTh8fzNIWxNjr31lvhhhvg0Ufh22/ttsZQFeJW8B96CGbPlrS8PgHvhfGJ1SIVDwOWghGLN7tnn5VBiMOHwzZHJlYbQhGXgj9jhkzyPHIkXH653dY4C6tFKp6nKYzFm11aGnzwARw9CldfHbw2kcGZxJ3g790LI0ZAx47w3//abY3ziIRIxcuApbLE6s2uUyeYOFHq6L/wgt3WGCpD3HXaXnWVVAf89Vfo1cvy3ccEgSpjul2kDNaiNZx7rtSZWrkSWpSrjGWwC9Np6+WHH6T069/+5i6xj/ZMS1XxyGNhNihD+CglXn5hIdx5p93WGMIlbgTfd2G2bQt//rPd1oSPL01y82bxqjZvlvdOElQ32Giwnnbt4O9/lyfmL76w2xpDOMRNSGf8eBgzBj77DIYOtWy3EccNudxusNEQGQoLoVs3SE2F336TJzyDvcR9SCc3VzyRc8+FCy+025rK4YZcbjfYGEniOZyVnCy/rSVLxNM3OJu4EPxnn4UDB2QKN6XstqZyuCGX2w02RgoTzoIrr4TOnWHs2NgfPe12Yl7wDx2Cp5+WME63bnZbU3nckMvtBhsjhdUjk91IYqJ4+cuXw0cf2W2NIRQxL/ivvCK59/ffb7clVSOcXG67Qwqxmm8eDvEezvJx+eWSEDFxot2WGEIR0522xcUigG3bymQOsUDZHPnzz5fyEFYVOzNUDtNhXcI//ynX5po10KGD3dbEL3HbaTt9OmRnwx132G2JNQSKF7/wggkp2IlV4Sy7n9Ks4Prrxf7XXrPbEkMwYlrwJ0+GunVhyBC7LbGGQPHiYA9o8RZSsAsrwlmx0vHbvDlccAG88YapseNUYjakc/QoNGkCl10WOx5HsNrygYjHkIJbiaWw0JQpUlRtzhw4+WS7rYlP4jKk8+WXcPBgbMWxg6U5lk01jZcMmVghljp+zzpLrsdvvrHbEkMgYlbwv/tOwjkDB9ptiXUEixffckt8Zsi4gXBi87E0jiEjQ+aXMBOkOJOYFfyZM+HUUyVHOFYIFi+eODE+yw87jbLifttt5WPzI0bI/85f/GNtHMM550g12rw8uy0xlCUmBX/3bli1Ck4/3W5LrCdea8s7nXAzqHx9MP4ds7E2jmHgQLk+582z2xJDWWJS8H/+WZYDBthrhyF+qEwGlQ//9NlYupH7So8vWGCvHYbyxKTgL1wooZwTTrDbEkO8UNUOVjd2zFZEgwbQsqVMjGJwFjEp+CtXQvv2UKOG3ZYY4oVwM6jC/Z7bad8e1q2z2wpDWWJS8DdskAsuHomFEZtupKIMKoiv9NnMzNh8enE7MSn427bJI2W8EQsjNsvGvR06LrAcFWVQaQ1vvx07HbMV0bgx5OS45/8XL8Sc4Hs8MuFJ48Z2W2I9FXnvbi/VO3aszErmEwmt5f3Ysd4NNk6GT7PgnQRZbnTWnayijtdY6pitiPR0OHYM8vPttsTgT8wJ/uHDIhR169ptibWE4727ecSm1pK3PWFCieiPGSPv8/JAb5wMc0fCkc2AluXckY4TfYOwbJksc3PttcNQmpgTfJ9HkZpqrx1WE4737uYRm0rBU0/B6NEi8gkJshw9Wj5Xix+E4jInoPgILHbJ40uc8dtvsty3z147DKWJOcGPVcLx3t0+YtMn+v489ZS3s/NIkBMQ7HODrVx3nSx9HdYGZxBzgp+cLMvCQnvtsJpwvHe3j9j0hXH8+T2mnxbkBAT73GArBQWyjLUnbbcTc4Jfp44sDxyw1w6rCdd7d2vHoH/MfvRosd8X3hkzBnSPcZBY5gQkpkEPlzy+xBm5ufJbTEmx2xKDPzEn+ImJ0mEba51Fobz3WMi9Vwrq1/eL2fvF9OvXB9VmOPR9CdIyASXLvi9BG5fc0eKMnTuhUSO7rTCUQ2vtyNa7d29dVTp31vqSS6r8dUuZNEnrzEytQevERFlmZsrnZbdRqvy6cPaflib79bW0tMrtw0l4PKHfG9zBSSdpfcYZdlsRATZM0vqTTK0nK1lucN4PDZivg+hqkt03nEiQmQkbN9ptRUkqpS+7xjftmy+l0of/Nv7rwgnHhMrecUs4x5+yo1ErKk1gcB5aw9q1MttcTOFLDfZli/lSg8E1T5oxF9IB6NxZyiN7PPbaEUiMffhEubqDpdyce2+ITbZtg717oXt3uy2xmBhIDY5JD79bN5nTds0a6NTJPjsqEt1Q68MV7NatA8+H6obce4NFFBdA0SHwFICnsGSpC73vi+RRSSVBQhKoxNKvE9MguQ4k1LDkkcpXB79nz2rvylnEQGpwTAr+qafKcuZMewU/mBj7r4fQgj15snj7W7bIZ+PGlQ7VjBtXOiQE7sq9N/ihNRzLhfxdgduxPVB4oHQrOgjFFtUvUImQVEfEP6m2LFPSoUYjSG0szf91zeZQsxmo0oGC77+HWrXgxBOtMcsxpLX2jvQO8LlLsETwlVLnAhOAROAVrfVjZdZfDzwBbPN+9KzW+hUrjh2IDh2gWTP48cfSsfJoE0iMffiLcjDBLtsHECi+71uGuikYHET+Hji4Fg5v8mubS5aBxFsllYhtSn1IawnJdb3iXNf7ujYk1oCEZFDJsvQ1lQRo0MXi7esiv9eFUHRUbhxFh6DwoLwuPCTLY7lwYCXk74bio+VtS0iGtFZQK1OEr1YmdXMyuf3yDqR4OgMNI3gyo0yPcaVj+OC61GClq1nOTimVCKwBzgKygXnA1VrrFX7bXA/00VqPCne/ffr00fPnz6+yXcOGieBv22Zvx5/PQ9+8WVJGi4ulU9lflIN58VlZgb3/zEzJsTc4FK3hyFbYv1LE8sBK7+sVIqD+1GgItbJEMGtliXjWbAqpTUpaSgNn9F4XHRbh97Wj2d6bVUnTR3eg8NOUGhlQtzPU7STLep2hQU95MnAjGydLzP7IFrnB9RjnuA5bpdQCrXWfgOssEPx+wFit9Tne9w8AaK3/5bfN9URZ8F9+WbzhJUvg+OOrvBtbSUgIXF5WKfs7pA1etAcOroe9C2DfQlnuXQiFeSXbpKSL0NX1tY5Qq42IfHLtsA81ebKMS/CNMcnIkIFpTnqae/65Ap58eCszPltD6/qrvDe6VXLTO7anZMOazaBBL0jv7W29oGaLyN/YXCDY1SWU4FsR0mkBbPV7nw2cFGC7S5VSA5CngTFa661lN1BKjQRGArSuZq/jRRfBrbfKj+Sxxyre3omYDlkHUpAHObMh5xfYMxv2/SaxdICEFKjfHTKvFC+2XhfxbGs0qraQTZ4MN95YUrIARPhvuEFeO0X0X3sjhdpN29HqpHagziu9Mn+PPOXs/a3kBrnja7lpAqQ2hcanQaPTZFnveEhItM64GEirrC5WePiXAedqrW/yvh8BnOTvzSulMoBDWutjSqmbgSu11oND7be6Hj7ABReIh795s3jLbqNsDB8kvu+m+jiu50g27P5ZBD7nZ8hbBmiJi6f3gvQ+JR5qva4S044AwcJ74JwQ35Il0KMHjB8vTyJhUXQY9i2RG0Dur3KufVkvyfWgYX9oPACanQUNTijXQVwpPs0K0umaCX/YVPX9OoxIe/jbgFZ+71tS0jkLgNbaP3D5CvBvC45bISNGwNVXw08/waBB0TiitZgOWRsoOgK7f4Id38KObyQUAdIp2rA/HH+5eJ8ZfSEpLfS+LKSyKbwVZXdFgtdfl+KFlTpOUi1o1E8aXh/x8BbvTdbbFj8grUYjaHYOND8Pmp4NqZXsEI6BtMrqYoWHn4SEac5AhH4eMExrvdxvm2Za6x3e1xcD92mtTw61Xys8/CNHoGlTGDIE3nmnWrsyxDL7V8G2z2HntyI0nmOQmAqNBkCzs6HJIAnVJNiXxVwZD9+OJ8PDh8WOQYPggw8s3nn+bu8NeJrchI/tAZQ8XbUcCq0vh7rHVbwf4+FXX/C9BzgfGI+kZb6mtR6nlHoYqekwVSn1L2AoUATsBW7VWq8KtU8rBB/gnnukCNfatdC2bbV3Z4gFtJb48daPpR3wXor1uooH2ewciSMn1bTXTj8CxfBBPOrXXy8t5HZkdz3xBNx7L8yeDf36ReYYgMT79y6A7dNg+1cSBgKo102Ev/Vl0ncSiLIxfJC0yhgrwhdxwY8EVgn+9u3Qpo10br3wggWGGdyJ1tLJuuVDEfkjW2SgUePTodUl0GIo1GpV8X5sJNwsnWhndx0+LDeZ3r1h2jTr9x+SI9tg60fyf835BdCSCZU1HNpeJ2MW/InzLJ2YF3yQbJ3XXpOCas2bW7JLg1s4uA42vg2bJsGhDVI+oNnZXpG/UPLEY4xoe/hR8+4r4sh2yP4EtrwPu2cCSv7XbW+ElhfJwLQ4IJTguzB3pfLce68MeHrkEftsiIWa9a7h2F5Y+zx82x8+7wDLHoHabaHfW3BpDpw+FdpeH5NiD9Gd6nLvXnj8cTj7bJvFHiCtOXS8Hc78CYauh25/hf0rYNaV8ElzmH+n9NfEM8HqJtvdqlMPPxB33CH15hcssHS3YRFrNesdicejdc7/tJ59ndbv1tB6Mlp/0VXr5Y9rfXir3dZFnerMsVAZbrpJ5nlYvDgy+682xUVab/9W61+u0vrdFLkupp+r9bZpMTvZAiHq4cdFSAcgLw+OO046bmfNim5efkUZFibVMgDhxlqLjsDmKbB2onTmJdWGNiOg3U3evG0HlCSIUWbNkkKF99wjYR3Hk78b1r4o10r+Ton1d7oLsq6RFNsYie/HfQzfx5tvwvXXSzzfN0IxGgTrRPNhBlOVIZxsiiPbYfV4WPeylDGo1wU63CZin1zXDqvjisJC6NVL5o5esUKqY7qG4gKJ8696SrK1UhtDkzNh6yfg8SsQ59IMHiP4XjweOO00mRxl8WJo2bLi71hBKA/fh1NGSzqCYPnSyRnS8Za/3fuBglaXQsdRMhrTePNR46GHpE/s00+ljIkr0VoGdi17FHZ+F3gbF+box32nrY+EBMlZPnZMRuH6phyMNIE60cpiZqjyI9jIx8JcP7FHMm5a/gGanG7EPop8/z08+qg8JTtW7DdOFsfhnQRZbgyQJaGUOAqDvw2+nxgbhRtXgg/QsSM884yUTn788egcc/hwCdlkZgbfRmuTvfM74U4o4cl31fRy0SDS2WA7d8I118g0os88Y+2+LcMXEjyyGdAlRdICib6PtCA/zpSGoeOxLiPuBB8kjn/llfJYOmdOdI45fLiEbCZNCu7t+yY4iXvR7zEOElLD2zbGPLDq4CupsHmzaJTV11NxsYj9gQPw/vsOi9v7e/S/Xlf5uWd7jJOYfSkUFOTAjLMhb6nVFttCXAq+UjLqtnVruPRS2FquUHPkqMjbr8wE5jFJfg7k/k9mYsIbpqnZElKC5My7aHq5SPPgg+VnV7PqetIaxoyBH34Qz75r1+rv0zLKevQ6SKw2lHPQZrh00KZlAkqWJ70OvZ+W7K+ve0oef+GhCPwB0SOuOm3Lsnw59O8vj76//AJ16kT0cOUwE5z44SmCtS/Akr/J1Hptb4TjHyoZGh8ndVCqQySvp//+F/70J7j7bvjPf6q3L8sJ1slflqp2wB7LhSUPyWC+Wq2h78tSrtmhmE7bIHTtKpX9li+XEE9RUXSPH2wik7ib4CRnFkzrAwvukNry5y+Fk14qXQclkAcWg2JfnRh8pK6nDz4Qsb/sMofm24cT1qvO3LM1MuDE5+Csn6WK6oyz4dcboWBf1fZnJ8FGZNndrB5pG4oXXpDRryNHRnfwXdyPwD26S+tZI2T04ycttd78fsyOfgyHQNeDUlrfemvVv1/d62nGDK1r1ND6lFO0Pnq06vuJKJ9kyjVUtr2TqPVkJes3hHESNkzy7ivEd4qOav3b/bLvj5tpvXO6tX+LBRBipK3twh6sRVPwtdb6/vvlbNx+e/RFPxpD4B3H5g+0/rCh/HDeq+sV/czwfpgxSmZmabH2bxkZ4V0jVl5P33+vdc2aWnfpovWePVXfT8TZMEnrKWmlxX5KWuWupcruI3e+1p930vqdBK2X/ENKODiEUIIf1zF8f7SGP/9Z4pOjRsHTT5vU7ohwLBfm3Q5b3pOJvI/ukPRKH764PMTEMPfKUNGIbB/RGJn97beSY9++vXTUNm4cuWNZQnXLIlRlcpTCQzDvVqnE2mQw9J8MNZtWxXpLMTH8MFBK4pN33w3PPit1x6t6LzSVMYOQPRW+7ArZH0P3R8BTXFrsQTplF4yufB51DBBurD3SmVxffw1Dh0rtqRkzXCD2IOL+h00wzCPLyjoHVZn+MLm2VGA96VXYM0cyeXKilOddRYzg+6EUPPmkpJ8980zgGYYqItK50K6kuAAW3AUzL4LUpnDOPCldezRIPmxBbuXzqGOAcePCf6qM1Mjst94Sz75LF/HsG1Zy2ljXEiy9t6K0X6Wg3Y1wzlxIqgM/DIItVs/xaB1G8MuglIR1xo6FN96ACy6A/fvD/34kc6FdyeEt8P3psHoCHDdafhgNesi6yubQx/ggq+HD4ZZbwhN9qzO5tJaBiNddJxUwf/hBZtSKGwINvKpMZk/9bnD2HMky++UKWPFvR47QNYIfAKXg738Xwf/xR/kBhDs4K5jnFZe1crZPg2m9YP9yOPV96D0eElNK1gf7kSXH7yCriRPh7bdlYJ5SIropKaW3sXoyk/x8udk88og81U6bBg0aWLd/V2BF2m9qQzjjB2h9JSy6T+L7nigV7AqXYL25drdoZ+kE47vvtK5bV+tmzbSeM6fi7YNlWmRmRtpSB+HxaL3iCUlv+/J4rfevDr5toFQ4K7IuYohIZnJt26Z1//5yjf7rX3GdFWsdnmJJ3ZyM1nNukPdRBJOWWT2WLNG6TRutk5K0Hj8+9I8i7nPriwu1nnurXOw/X6514eGq7SecnGhDKSp7Y5g2TetGjeT6fP/9aFgYZyz+uy2ibwTfAvbu1XroUDljl16qdV5e8G3jNre+8JDWM4bIRb7wz1H3bOKZyjgahYVa/+Uvsk23blqvWBF9e+OGxQ95Rf/GqP0ejOBbhMej9b//LXN4tm9vz/y4juVojtZf95aBKGsm2m1N3BFuKHHLFq1PO03W3XST1oer+ABmCBOPR+vFfytxgqJAKME3nbaVQCkZnDVjhmTenHSSdO5WNnUz5sjfA9PPkM7ZAZ9Bh1vttijuqChZQGt4+WWpH7VwoXQMv/xyxRPzGKqJUnD8P+Q3sfIJWP+areYYwa8Cp50GS5fCVVfBww9D376waJHdVtmET+wProEBU6HFELstiktCFU7btAnOOkvGg5x4IixZInXtY5pwZryKFkpJmeWmZ8Pcm2HXj7aZYgS/iqSni5f02Wewa5f8kMaOlekT44ayYu/gkrGxTqBpNGvWhIEDoVs3mDtX5oD4/nto29YWE6NHVWa8ijQJSXDqe1CnA/x8CRzaaI8Zthw1hhg6tKS88j/+IT+uL7+026ooUHQEfjzPiL1D8J9YRylo0kRKIrz5JpxyCixbBjffHCf1oRY/6MyR2in1YeAXMkHLnBEyB0SUMYJvAenpMnXhtGmQmAhDhkhbty78fbiq/o6nGGZfIzMBnfKeEfsoEuo6GT4cfv5ZnI9du2RKwnfflesyruZYqEpdnGhRuy30mShzQKyI0qTafhjBt5BzzpH46BNPwE8/SQfZX/5ScWkG19XfWXQfZH8iI2dbDrXbmrgh1HVy+LCMlO3UCT79VMokrFol/Uxx4dX7U9W6ONEiaxhkXgVLx0LuvKge2pRHjhA7dsB990mcPz0d7r1Xyi4Hmvg5K0t+vGXJzJQON0ex7iXpeOp4B/R52m5r4opg10mDBpCcDLt3y6xUp5wC48dLhk7r1hLfj2QpZcfhhukwC/bBV90huR6ct0hi/BZhyiPbQLNmUnlwwQJJ37z/fmjXTqpwlu3YdU39nX2LYP4d0Ow86PWU3dbEHcGuh3374PjjYdYs+MMfpFCfa54WI4EbpsNMaQC9J0gq8/qXo3ZY4+FHiVmz4K9/lWJsrVqJ93/DDZJZ4QoPv+gwTOsNhQfhvMVSKMoQVYJdJ02awM6dobdx1LVkELSWcsr7l8GF66RT1wKMh+8ATjkFpk+H776Dli0lvNO6tQzcuvfe8il1VldErDYLRsOBNdB/UmTF3kn50w5i1y7o2bP85zVrSjlvH655WjRI50qvp+DYXlj2SFQOaQQ/iigFZ54p3v7PP8tN4OGH4U9/gpNPhubNZZvMzMhPYVcptn4C61+Frg9Ak0GRO06g/Ok5I2DubZE7psNZsQJuukmcg6lToXdv8ehBrpOXXy59nYQagBVzxIJzkH4CtL0O1k6UcS0Rxgi+DSglNfY/+0wyKUaMkJvA9u0weLBk+Vx+ud1Weik6DAvuhPo94PixkT1WoPxpNKx7wZ0/5iqSny8x94EDJdPrnXfgj3+E1ath/nwJ32gtIZqyTkGgAViOe1q0AicOrqoqnf8Mxfmw7sWIH8oIvs0cd5x485s3S1rd2rVwxRUS9rnvvsrl8keEZY/CkWw48TlISI7ssYLmSWv7B81EgWXLZC7l5s2l9MHWrfDPf0o4ZuJE6NCh4n2UHYDluKdFq3Dq4KqqUK8LNDsH1j4n04FGENNp6zCKi+Hbb+VH+vnn8n7gQBg2DC69VFI8o8aB1fDV8ZA5DPq9EfnjfZrl9dgCoWSCan82TpYf+JEtkmPdY5yzMjHCYOdO+OAD8eJ//VXSKy+5BP7v/2DQIBlgZQjAOwlAIO0KcJ24ge3fwI/nQr+3oU31Ch2ZTlsXkZgI550Hn3wint0jj8C2bZJa17QpXHihiMOhQ1Ew5rd7IbEm9IzSiMAe44Ago4TKDppx8SN9bq7E3s84A1q0gDvvlP/nk0/K/3rKFFlnxD4ETh9cVVmanQ212sDmKRE9jLmkHEzz5pLK6Yvd3nmnVOUcPlzqpFxxhZR0yM2NwMHzlsK2qdDpT1CzSQQOEIA2w6H9LZQT/UCTSbvskX7bNnlqu+ACuXGPHCkhmwcflFDO0qXSed+okd2WuoTqTjruNJSCVhfDzu+g8EDEDmOJ4CulzlVKrVZKrVNK3R9gfQ2l1Hve9f9TSmVZcdx4QSnJznjySYn1z5wJ118vyxEjRPxPOw0ef1yyOiyJ0i1/DJJqQ8dRFuysEvSdKI+1FQ2aqaheis0ZHB6PVKh86CHo1Uv6ZG6+Wf4/Y8bIgLzVqyVLq2vXqJoWG7hhcFVlaXkxeApg+9cRO0S1Y/hKqURgDXAWkA3MA67WWq/w2+Y2oLvW+hal1FXAxVrrK0Pt18kx/MmTxTOze+i6xyOe/xdfSLzfV5O/TRtJ/xw8WOL/TZtWcscH18MXHaHT3XDCExZbbRHB4v1pmeLlRXlovdawcaNMjvPjj9IPs3u3hGX69y8pqNelSxzWtjGEh6cYPm0hqc+nvFvl3YSK4Vsh+P2AsVrrc7zvHwDQWv/Lb5tvvNvMUUolATuBRjrEwZ0q+L4CVkf8tCQtzRmZENnZIv5ffine/wHvk2HnztIBOGgQDBggTwQhWfgnWPMMXLQZajaLuN1VIlS9lMUPBr8Z/GGTJYf3CfxPP5WI/Natsq5xY4nBX3ABnHsuZGRYckhDVXFT5/6sYbD7J/hDdpU9g0gL/mXAuVrrm7zvRwAnaa1H+W2zzLtNtvf9eu82e8rsayQwEqB169a9NwcaI24zbhm6XlQEv/0mYjRjhgz0OnxY1mVlySxdvtarl19RN08xfNYKMvrCgE9tsj5Mgv2QI5DBsWcPzJsnYZq5c+V1To6sa9RInqQGDpSbaqdOxot3DG4opObPmudg/ihxtmpVrQM6lOBbV6LNArTWLwEvgXj4NpsTELcMXU9Kklm4TjxRSjcUFkr4Z/bsEtF6/33ZNiFB4sg9e8IFvWdwZaMd5NQaTkPtcOFqMzzwjzatdRAPv+IfUGGhjH1Ytkwmtlm+XOLtG70TFCklYZkhQ+TcDhhgwjRRZ+5tsP4lmUhEJUK7kdL3E4hQnftOFPwGPWWZt7TKgh8KKwR/G9DK731L72eBtsn2hnTqAZHILYk4rVsH9vCdPnQ9ORn69ZPmY/fu0l7r9OkwqOZkDvStQ+t+Q0iqITeCrl2hfXuZGq9NG1lmZDhY5ILF8L0ZHB6PZM1s3FjSVq0ScV+9WkQf5O9r1046zG+9teRpqE4dG/4mgzD3Nlj3fMl7XVzyvtEp5Z/4nDwZSiDqdZHl/hXQ4gLLd29FSCcJ6bQ9AxH2ecAwrfVyv21uB47367S9RGt9Raj9mhi+DWgPng8bsyflPD7Z+TbLl5d4urt3l960Th0R/zZtJH20SRPpHPa1Jk2k1awZRfO19Fvs2gU7f/uOXfM/YefuJHYd6cDOhPPYmteejRvlhl3gN6DRNyK1W7eSG1y3bhKaiab9hjB4N0lEvhwJkJha/iafWBMKAviWFvbnWM4H9aDNdVWebyKiIR2tdZFSahTwDZAIvKa1Xq6UehiYr7WeCrwKvK2UWgfsBa6q7nHtwifqobJ0nJLFU2n2ryShMJfGvQZx80WlVx06VOINb9ggbeNGCX/88kvwsQA1akDdutLq1St5XbeurEtOlpaUVPI6OVm88MJC6YsoLCz9+sgREfZArej3aULP8jYZzNa4sQxy6tlTRrL6blZt2sj/qEaNCJ3TauLaaylSBBR7AE/g0E1CTRH+IE97jqRmCzhaNkhiDaa0gsW4+glg7Qsw71a4cC3UaV+prxYUSCfmzp3Sdu2SlpdXIsb795cW52PHyot6YaGIPXhvAklFJKmjJCcWkJzkISm1Fml10krdOPxvKA0alH7KaNpUwk9uHLXq6mspUgT18IOhZDBfuDF/J/BeLXlauaxqUe+IZulECrcKvluyeAIyazjsmg4Xb7c1QO/xyOHVJpdlWFhMpa8lXaaXvez7WKBsDN9HQi3wHC7/eXIGeI666xp6x/s/G1Y1bTa1dKKIW7J4ArJvITQ8yXaRSEjwmuCy8glWU6lraclYWDimZJi11vJ+ydiI2GYbfSdC+1vFUwdZtr8VTnoxcKkFhfuuoRqNxMOPAEbwLca1E1B4iuDQeqjbyW5LSqhKhkUsTIrhJexrSWsozIPVE0pEf+EYeV+YZ1GtDQfRdyJcXSQe8NVF8j5YqYWCvYH34dQsHYD63SA9oINebYzgW4xrJ6A4ugM8hVC7rd2WlFDZiogurqAZiLCvJd9UeceNFpF/N0GWx42Wz2MtrBOMNsMl82aYR5ZthruzqmZBHiTXj8iujeBbjGsnoMj3zoKd6qBSCpWtiFjZEJDDnwYqdS35RN+feBL7YLixquaxPVAjMhNfOGqkbawwfLgLBL4sBftkGaELrUr4OtXCrYNSmRBQ2SH3vqcB/+M6gLCvJV8Yx5+FY4zoV/YashtPoaRk1sqMyO6N4BuE4qOyTHTYSKNg5RMCUZmSCm4bch8K/5i9L4zjew9G9CtzDdnNwfWgPRELrZqQjkH4vWPPxcJQmcd3tw25D4VSEvP1j9n7YvrJ9eNb7N3G3nmyjFCnrfHwDUJCiiw9hfbaUR0CPb43P1/ezxlR+nG+GgXWHEn3saXz7n2ib8TeXeTOg6RaULdzRHZvBN8gJHsrghVFbnq1qOD/+B4qTl9BgTVXUlbcjdi7j9y54t0nJEZk9yakYxBqNJRl/u7Q27mJiuL0sTZFnsHdHNsLexdAw/4RO4Tx8A1CWktZHnZhDLssv0+MEmQCHV+c3k2deYbYZ9tU0EUymXmEMB6+QUiuA6mN4eBauy2pHqUGXwXBrXF6QwkOH0NRJbZ8JNdmhDpswQi+wZ963SBvsd1WVI9AYZxSKOnINbiXGBtRDUDBftj5LbS+LKJ9L0bwDSWk95ap1Yrz7bak6lSYVqlh45vuFod4JxaL6m14AzwFkDUsoocxgm8oodGpctHt+Z/dllSdcMI1bheHeCFY2CaWxlCAFC5cPR4anSZOVwQxgm8oofEAUAnyaOlWAg2+CkSwcguxFhd2K6HCNm4siBaK7E/h8CbodHfED2UE31BCSn1oMhg2v+fekrpl0y1VkHzmsuIQi3FhNxMqbOPGgmjB0BpWPgG120GLCyN+OCP4htJkXi118XPn2W1J1fEvk3vym+GJQyzGhSNFNJ6EQoVtYmkMxdYPZbBVl/sjNtjKH5OHbyhNq0tkXtvN70LDvnZbU33CrZYYa3HhSBGtKqMVlb6IhTEUxfnw271Qvwe0vSEqhzQevqE0KfWh+QWwaRIUhUpvdBGBJsYoSyTiwrHYJxCtJ6FYCtsEY9V4id33fioq3j0YwTcEotMYmYRh/Wt2WxI9rBaYWO0TiNaTUCyFbQJxeDMsHwctL4Img6J2WCP4hvI0Pg0anSKdSW6unlkZrBaYWOsT8D2tEKQzPxIZMuE8mbkRTzHMuRZQ0Gt8VA9tYviGwHR5AH4aApvegbbX2W1NdLAyLmxnn8DvtYQsmuGpbNy+LLEWaok0q5+C3TPh5NehdlZUD208fENgmp8PDXrBkr9C4SG7rXEfduWKRyKUFKpcRayFWiLNviVyPlteDG2i70gZwTcERino8wwcyYZlj9htjfuwq9MxEqGkoE8lKrZCLZGmYB/8chmkNIC+L9oyX4ERfENwGvWXdLFV/4X9K+22xrkEysaxq9MxEqGkWBvZageeIvjlCsnKOe0jSG1kixlG8A2h6fkYJNWGuTdLZ5OhNKFCKHZ0OkZCnCP5tBKLqauBWHg37PweTnxBEiJswgi+ITSpjaH3eMj5GZY/arc1zsNp2TiREOdIPa1U1N8QKzeDtc/DmmfguDHQ7kZbTTFZOoaKaXOteCfLHobGA6HJ6XZb5BycNkI33JHFVdmv1U8oFd0sozGiN9JseAvm3S6DGU/4t93WoLRDi2T16dNHz58/324zDD4KD8K03lB0GM5bZFsM0hFsnAzzR0NhbvBt0jIljGM3VqdoWsk7CQTO61chSis45LyGw+b3YfbV0HgQDPwCElOjclil1AKtdcBps0xIxxAeyXXg1PfhWC78fDEUHbXbInvYOBl+vSG02DslL93pIZNQ/Q1Oe3KqLNlTYfZwmZD89M+iJvYVYQTfED4NekL/SZAzG+aMAO2x26Los/hB0CFGHzspLz1YyGTONfBhQ7lx2Vn6IVR/g5szgza9K+mX6b1g4JeQVMtui37HCL6hcrS+DHr9B7Z+BAv/ZLc10Sekh+mwvPRQthbklr9xRbuzOVRnsFuLp616CmYPE89+0DeQXNdui0phOm0NlafTGDi8RaZlq5EB3f5qt0XRI1hs2bfOSYSyNRjRDpmU7Qz2hZmObIHkdEisCQV7ndf/UBbtgUX3wconodVl0P9tx4Rx/DEevqFq9PoPZI2AJX+DxX9z7wxZlaXHOFDJ5T9PSHGe9xnudI/+2HnTKtvnUJgLxUeh39vOenIqS9ERmH2NiH2H2+GUKY4UezAevqGqqATo9wYk1pD8/OKjcMITtgwXjyo+0fHP0knJgN4TnCdIpVI0w/D07Q6ZhErTdNq59XFoA8y8GPKWQo9/QZf7HP0bMIJvqDoqQWqCJKbCqv9IymafZyAhxi8rN8y2VDYds/2tsPHN0oKakAKJdaDQISETt2XmbP8aZg0TgR/4FTQ/126LKiTGf5mGiKMSoPfTkomw4nE4vFEeaVPq221Z/BJoGsKNb0p1xu1fOTMnHyqe1tApeIpk8pKl/4D63WHAx1C7rd1WhUW1BF8plQ68B2QBm4ArtNb7AmxXDCz1vt2itR5aneMaHIZSUnOndnuZD/fbfnD651Cnvd2WxSfBQiPbv3L2oKUe48rX3bc7zFSWg+slJXnPHMi6Rp5wkyrZT2Ij1e20vR/4QWvdAfjB+z4QR7XWPb3NiH2s0v4mGPw9HMuBb06CnT/YbZF1WD1IKZKDntwWGvHh5GkNtYZ1L8PXPWD/Cug/Gfq95Sqxh2qWVlBKrQYGaq13KKWaAT9qrY8LsN0hrXXtyuzblFZwMYc2wE8XSknlrg/A8WMhIUBmi1sINONTYlrVxcjq/ZXl0yz3lyVwEkd3SLXYbZ9DkzNkpqparey2KiiRLK3QRGu9w/t6J9AkyHapSqn5SqlflVJ/CLYzpdRI73bzc3JyqmmawTZqt4Vz5kplwOX/hO9Ok5uAW7G6ImakK2y6ddCS0/AUw+pn4YtOsONbmX928LeOFvuKqDCGr5T6HmgaYFWpq1NrrZVSwR4XMrXW25RSbYHpSqmlWuv1ZTfSWr8EvATi4VdovcG5JNWCk16BpmeLN/tVTzhxImQNd3TaWkCsDpFEIuRSNivH6R20Tid3Psy7BfYukGu4z7NQt4PdVlWbCgVfa31msHVKqV1KqWZ+IZ3dQfaxzbvcoJT6ETgBKCf4hhgk8wrI6CuFpOaMkEnRT5wY9cmbq4XV2SNW7y9YVk4k499OrsJZHY7thaV/hzXPQc2mknHW+gr3OSlBqG5IZyrgm4n3OuCzshsopRoopWp4XzcETgFWVPO4BjdROwvOnCmPxDkz4cuusPK/kt7mBqwOkVi9v2hPwhKJidLtpjgfVjwBU9vB2onQcRRcsBIyr4wZsYfqC/5jwFlKqbXAmd73KKX6KKVe8W7TGZivlFoMzAAe01obwY83EhKh02i4YAU0GQS//Ukyefb8ardlFWN19ojV+6tMiMiK7CCnzfJVHTzFMknJ5x1h0b0y/eB5i6HP05BSz27rLMdMgGKIPlrD1g9h/p2Qv1MemXv+yzWDVxxHuFk5VmUHhZq4ZJhLSmZrD2z7ApY8BHmLIb2PlAZpMtBuy6qNmQDF4CyUgtaXw4VrodtD8sP7ohMsuFtiqIbKEW6IyCrP3M216j3FsPk9+LonzLwIig5JnP6c/8WE2FeEEXyDfSTXhu7/EOFvcy2smSAx1KWPQEGe3da5h3BDRFZlB7kx7dNTCBvegC+7wKyrpP+o39swZJU3Th8fUmhCOgbnkLdUvM1tn8vEER1HwXF3xff8uVZi5YAst2TpHMuF9a9I1s2RrTJrW9e/QquLY1bkQ4V0jOAbnMe+RTJga8uHMgFGh1ug092Q1sJuy9xNpEf4Ool9i2HNM7BpsmTgNBkk11DzC2Iq6yYQoQTfVMs0OI8GPWXC9P0rYfm/YPUEaa0uEa+/0Wkx/6ONCKXq4zvcM68Kxfmw9VNY9zzsninOQpvr5Jqp381u6xyB8fANzufQRlj7vDyaF+yD+sfLjzhruKMmiDbYgNaQO1fi85unQGEe1MqS66PdjZDSwGYDo48J6Rhig6IjsPldWP2MpNIl15OUzrbXyaTRxuuPH45sk3DNhjfgwErx5ltdCm2vl/BNjMbnw8EIviG20Br2zIa1L0L2xzLTVu120GaENJPPH5sc3gpbP5IxHDmz5LOG/aHtDZLmG4MDpaqCEXxD7FJ4CLZ+DBvfgl3TAS0i0OoSycQw4u9uDm+GLR/Blg8g1zsqu353EfjWV0Ddjvba50CM4Bvig8NbYdMk2PK+ZPoA1O8hwt/qEqjXzYR9nE5xAeT8IpU+d3wtk42AdOS3vhxaXWZEvgKM4Bvij0MbYesnkP2J9/FfQ6020OxsaU0Gm3l3nYDWcGi9zI6242tZFh2SCdYbD4Bm50HLoWa6zEpgBN8Q3xzdBds+g21fwq4ZUHRQOvXS+0Kzs6DpWZBxIiSm2m1p7KM1HFgFu3/ytplwdLusq5UpAt/8PLkhJ1dqkjyDFyP4BoMPTyHs+R/s/E5mMdo7VwppJaRAem+pltiwv7SawSZwM4RNwX6ZRGTvPEmfzPkF8r3TZtRsBo1P97aBUPc4E3KzACP4BkMwCvLE08yZJZk/ufPAUyDrareTyVsa9CxpqY3ts9XpFOyDvOWwb6Gcx73z4MDqkvW120LDU6DJ6dBogIRpjMBbjhlpazAEI6U+tLxIGkDxMdi7EPbMkptAzizJ/fdRsxnU7wkNekDdztKBWKcj1Ei3w3p7KNgH+1fB/uXetkyWR3eUbFOzuYTJsq6B9BMhow/UyLDPZgNgBN9gKE1iDWjUT1rne+SzY3tloNe+RSVt53eg/WbsqtFQhL9uR6jdHmq1hrRWkNZSmlv6B7SWPo6jO+DwFpl8vmwrzCvZPrEm1Osi877W6yqtQQ9T98ihGME3GCqiRrqM3mwyqOSz4gI4vBEOrIGD3nZgjfQLHH0jwD4aifDXbAGpDeUGkZIhXm+NDO/7dCkVkZgGSWmyTEisut2eIvAcg8IDEroqyBPvvND7ujBPOrTzd4jA+1rZmvkJyZLhVLstNDxZlnU6SH2aWllxParVbRjBNxiqQmKKdDLWPa78uqIjcCRbyvGWW26FvEVwbI8U+6qIhBpe8a8JKhFQXoH1Ln2vPQWyP88xCUt58qUzuiKS6kiYqmYzCcGkNit5n9ZS+jFqNq/ejcfgGIzgGwxWk5QmoZ2KBggVHZF67QW5cgMo2CdlIoqOiJdddETeFx+B4qMi4NoD6NJL7ZFQVEINCR35v06oIXMLpNSH5Pqy/P11PfeEmgyWYATfYLCLJG/oplYruy0xxAkm+GYwGAxxghF8g8FgiBOM4BsMBkOcYATfYDAY4gQj+AaDwRAnGME3GAyGOMEIvsFgMMQJRvANBoMhTnBseWSlVA6wOYxNGwJ7ImxOVTG2VQ1jW9UwtlWNWLMtU2vdKNAKxwp+uCil5ger/Ww3xraqYWyrGsa2qhFPtpmQjsFgMMQJRvANBoMhTogFwX/JbgNCYGyrGsa2qmFsqxpxY5vrY/gGg8FgCI9Y8PANBoPBEAZG8A0GgyFOcJ3gK6UuV0otV0p5lFJB05WUUpuUUkuVUouUUvMdZtu5SqnVSql1Sqn7o2RbulLqO6XUWu+yQZDtir3nbJFSamqEbQp5HpRSNZRS73nX/08plRVJeypp2/VKqRy/c3VTlOx6TSm1Wym1LMh6pZR62mv3EqVUr2jYFaZtA5VS+/3O2UNRtK2VUmqGUmqF9zc6OsA2tpy7MG2z5txprV3VgM7AccCPQJ8Q220CGjrNNiARWA+0BVKAxUCXKNj2b+B+7+v7gceDbHcoSueqwvMA3Aa84H19FfCeg2y7Hng2mteX97gDgF7AsiDrzwe+BhRwMvA/B9k2EPgi2ufMe+xmQC/v6zrAmgD/U1vOXZi2WXLuXOfha61Xaq1X221HIMK0rS+wTmu9QWtdAEwBLoq8dVwEvOl9/SbwhygcMxThnAd/mz8EzlBKKYfYZgta65nA3hCbXAS8pYVfgfpKqWYOsc02tNY7tNYLva8PAiuBFmU2s+XchWmbJbhO8CuBBr5VSi1QSo202xg/WgBb/d5nE6F/bhmaaK13eF/vBJoE2S5VKTVfKfWrUuoPEbQnnPPw+zZa6yJgP5ARQZsqYxvApd5H/w+VUk6ZmNau6ytc+imlFiulvlZKdbXDAG9o8ATgf2VW2X7uQtgGFpw7R05irpT6HmgaYNWDWuvPwtzNqVrrbUqpxsB3SqlVXg/ECbZFhFC2+b/RWmulVLB83EzveWsLTFdKLdVar7fa1hjgc+BdrfUxpdTNyJPIYJttcjoLkevrkFLqfOBToEM0DVBK1QY+Au7SWh+I5rErogLbLDl3jhR8rfWZFuxjm3e5Wyn1CfKYXm3Bt8C2bYC/N9jS+1m1CWWbUmqXUqqZ1nqH9zF1d5B9+M7bBqXUj4i3EQnBD+c8+LbJVkolAfWA3AjYUmnbtNb+dryC9JE4gYhdX9XFX8S01l8ppSYqpRpqraNSuEwplYwI6mSt9ccBNrHt3FVkm1XnLiZDOkqpWkqpOr7XwNlAwMwBG5gHdFBKtVFKpSCdkRHNhvEyFbjO+/o6oNzTiFKqgVKqhvd1Q+AUYEWE7AnnPPjbfBkwXXt7sCJMhbaVie0OReKuTmAqcK034+RkYL9fKM9WlFJNfX0wSqm+iP5E4waO97ivAiu11v8Nspkt5y4c2yw7d9HohbayARcjsbVjwC7gG+/nzYGvvK/bIpkVi4HlSLjFEbbpkmyANYjnHC3bMoAfgLXA90C69/M+wCve1/2Bpd7zthT4Y4RtKncegIeBod7XqcAHwDpgLtA2itdZRbb9y3ttLQZmAJ2iZNe7wA6g0Hut/RG4BbjFu14Bz3ntXkqITDYbbBvld85+BfpH0bZTkX69JcAibzvfCecuTNssOXemtILBYDDECTEZ0jEYDAZDeYzgGwwGQ5xgBN9gMBjiBCP4BoPBECcYwTcYDIY4wQi+wWAwxAlG8A0GgyFO+H8oXENEGzt+CgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Naive Bayes\")\n",
    "plt.scatter(x=samples[labels == 0, 0], y=samples[labels == 0, 1], c=\"blue\")\n",
    "draw_2d_gaussian(mu_c0, sigma_c0, c=\"blue\")\n",
    "plt.scatter(x=samples[labels == 1, 0], y=samples[labels == 1, 1], c=\"orange\")\n",
    "draw_2d_gaussian(mu_c1, sigma_c1, c=\"orange\")\n",
    "plt.legend([\"c0\", \"c1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the final posterior distribution for the case $p(c=1|\\boldsymbol{x})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.5, 2.5)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAEKCAYAAAC48pk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABusklEQVR4nO29eZxkZXX//z639t57emZ6NjZhGBZBlBE0LI4L2yjbT1SMUSQQY4I/QxQVl0RjjEvka4hLTAii4s+vyFe/6CiDgOjIEEUZkH1xhm2Y6Vl736q7qu7z++Peqq7l3tq7q7r7vOdV01W3nrr3udVdpz73nPOcI8YYFEVRFEVRlMZjNXoCiqIoiqIoioMKM0VRFEVRlCZBhZmiKIqiKEqToMJMURRFURSlSVBhpiiKoiiK0iSoMFMURVEURWkSVJgpiqIoiqI0CSrMFEVRFEVRmgQVZosUEfmCiFxdx/19RkQ+U+Vr/yAix9drLoqiLGzUfikLGRVmixARWQa8B/ivRs/F5Trgs42ehKIozY/aL2Who8JscfJeYLMxZrLRE3HZBLxeRFY0eiKKojQ970Xtl7KAUWG2QBGRF0Tk4yLypIgMisi3RSTqPn0e8BuP17xTRJ4QkQkReVZENtRpLkeKyICIvMp9vEpEDqT3b4yJAw8C59TjeIqizG/UfimLGRVmC5t34RiLI4GjgU+5208AnskeKCIfdp9/N9AGXAS8UI9JGGOeBT4G/H8i0gJ8G/iuMWZL1rCngFfU43iKoiwI1H4pixIVZgubrxtjXjLGDAD/ArzT3d4FjKYHuTkbnwb+3BjzkDHGNsY8Zox5QUS+JCJbReR7IhKqdiLGmP8GdgC/B1YCn8wbMurOS1EUBWq0X8Cgm5g/JiIvr2Uiar+UuUSF2cLmpaz7LwKr3PuDQHvWc28CHjPGPJL9YhF5BbDaGHMG8DRwSY3z+W/g5cDXjDFTec+1A0M17l9RlIVDTfYLmADeDPyoTvNR+6XMCSrMFjaHZN0/FOhz7z+KExpIswRvo/JnwF3u/V8Ap1U7ERFpA64HvgV8RkSW5A05Fsg3rIqiLF5qsl/GmIQx5kA9JqL2S5lLVJgtbK4SkTWuEfkk8EN3+2bgdVnj/gicLiKvEIe1InIs0A2MuGOGcQxgtfw7sM0YcyVwO/Cf6SfcpN6Tgbtr2L+iKAuLWu1XPVH7pcwZKswWNv8bx+P1HPAs8Dl3+83ARhGJARhjfus+93OcXInbgBjOVWiH+5pOYKCaSYjIhcC5wN+4mz4EvEpE3uU+Ph/YYozp83q9oiiLklrtV11Q+6XMNWKMafQclFlARF4ArjTG/NLn+c8D+40x1xfZx0nAh4wx7xGRTwDPG2N+4DP2MwDGmM9UMdffA1cYYx6v9LWKoiw86mG/ssZ+B7iumH1R+6U0E8FGT0BpDMaYT5Qx5mER2SciW4GdOBWuZ2Mup87GfhVFWZiUY78ARGQzcBKwTkT+yxjznVmYi9ovpa6oMFOKYoz5SJlDt8zmPBRFUSrFGLOxzKFbZnMeilIJGspUFEVRFEVpEjT5X1EURVEUpUlo6lBm2IqamNVeeqCiLGJGUgcPGmOWVfKaDRsiZmDQLmvsY48m7zTGnFvV5BYxar8UpTxm04bNR/vV1MIsZrXz2o6LGj0NRWlq7hy88cVKXzMwaHP75qVljT10zd7yBio5qP1SlPKYTRs2H+2XhjIVRVEURVGaBBVmiqIoiqIoTYIKM0VRFEVRlCahqXPMFGU2aO2K8raPvJHeI3oQSxo9nbIwtmHf8/38ny/fw/hQvNHTURSlQcxH+wVqwypBhZmy6HjbR97IceuPJhKIITI/DJsxhp6eHt72EfjOJ29v9HQURWkQ89F+gdqwStBQprLo6D2iZ94ZNREhEojRe0RPo6eiKEoDmY/2C9SGVYIKM2XRIZbMO6MGjmGbT6ELRVHqz3y1X6A2rFxUmCmKoiiKojQJKswUpQmYnp7i6o99gLMu2MDb3nMRu/p2NXpKiqIoZaH2q75o8n8TY4zBJoVtUlk/bQw2tnF+GvcfGMjuRy/Ofzn/xMIigIWFJQHnvvtzvrrGFwr/5ye30tHRyd2btnD7nT/jun//Itd/6euNnpai1IRtbGyS2CZFyrVhM/bLYLAhbcNM3ovFsWAgWFi5NizLfgUIIKI+hkai9qu+qDCbRWyTYtrEmbbjJMyUe4uTsJ37STPt3hLOT5KkTMK9JUmRnLO5BggSkCABCRGQIEFCBCREUMIEJUxIwoQkQsiKOD8lSkgihK0oYYliSWDO5jrX/GxzK1/5Rjd79gZYuSLFh64a5PyN4zXt8yc//zHfuvm/ERHWrT2GgYF+PvDXVwNwzhvP47Nf+jTGGBXMSsMwxmRs1nSW3UqYOAkzTcJMkzRTGfuVMo79SpoEKdeWmQK1NTs4Ii1IkLQNCxHM3PJsmEQIWa79kihhK0qA0IL9rKn9mn+oMKsC26SI2+NM2RNMmjGm7ImZm5lk2v2ZNNO++7AIZIxFQMIEJUSLxBxxRMgVSUECBLEkiCUWAYLOtaNYSNYVpHNVSeYnkDGIBoMxxvWuzXjabFKkTArbJEmRyhKDzs+0sY3bYyRNgoSZwiblez5BCRORGGErRkRaCFsxolYLEWklajm3iNVCQObXn9zPNrfyqc/1EI87V+R9e4J86nPOqqJqjdv2Z//EN2/8Oj/49o9Z0r2EoeEh/uLKd7ByxUoAgsEg7W3tDA4NsqR7SX1ORFFcjDFMm0ni9gRxe8yxZcaxX9P2pGvDJpk2k0WElRByBU/QtV9hiRG02jPCKG2/AhLAwrFhluvdcjxglvvFPePVz5lnVjTAiR6kbZgTOUhHEhxBmMyIwWyBOGEmHVtmT5Ek4fueWARc2zVjwyJWjIg1Y7+i0kpQwvNKbKj9mp/Mr2/JOcI2NnF7jAl7hAl7hMnUGHF7lEl7jEl7jGkzWfCaAEEiVgthidEW6GaJtdr9kEdnvEtZV2rzTaAApExyxvNnx5lOX03bcabNJFP2JFNmgpHUQaYSk6Q8DGFYYsSsNmJWG1GrnVigjRargxarg6jVhtVkIYmvfKM7Y9TSxOMWX/lGd9WG7f4Hfsu5b9qYMVpdnV21TlNRMjjCK+7arhEm7FHiru2atEeJ2+NuCHEGQQhLjIjVQtRqoSPQQ8SKEZaYY78sx7uU9jYF56GHyRjb9fSl7daMNzAtRKfsSSbtMYbt/UybwiKoAUJErVZaAu1ErTZiVjsxq92xYYF2ghJuwJn5o/ZrfjL/1EGdSF81jqUGGU8NM24PM+H+jNtjOVeKguV+CNtYFjqEqNU2cxXl3prtAzkbpL14UVqhjMil43Ebz3gX4/Y4k7YjckdS/exLvJjzBSEIUaudVquD1kAXLe7PtkAXYWlM3Z49e71P1G97tfQu62XP3j2s6F1JMplkdGyU7q7uuh5DWVgkTYLx1JBzs0cy9msiNVJwUeRcELXTGVhGb+iILNvVRtS9oJxvQqtSRCwndEkU6Cw53jY2U/Y4cZO2YeNM2uOuyB1lMLmvICoSligtVietgU5aAp20Wp20BbqIWR0NuehU+zU/WRTCbMqeZCw1wFhqkNHUoCPG7KGcD1WAIC2BTjoDS1kZfhkx14vTEuggIi0L3mjNBkEJ0xYI0xbw/oAaY5gyE0ykXM+kPcJEaoRxe4TBqb05OXZBCdNmddEWWEJboJu2QDftgSWEreisnsPKFSn69hR+TFau8A/rluI1r/4zPvDhv+a9f3El3V3dDA0P8YbXvYnbfv5jXvmKV3HnPXfwmle/Vv/mFMDxVI+lhnJs2HhqkLjJ9ngIMauNVquT7kgvLVaHY8MCHcSstnnpoW80lljEAu3EaPcdkzDTrlfS8UxOpIaZsEc4kHiJ6ek/ZcYJFi1WR8ZutQUcW9Zidczq51zt1/xkQX1ajTFM2COMpvoZSfUzkuxnLDXAlJnIjAlJhLZANyvDR9JqOd6Y1kCXiq8GICJE3Ry0JazMeS4t2sZTQ65Xc4gxe4h9iefZNf10ZlxEWmgP9NAR7KEj0EN7YCkxq61uv8sPXTWYk6MBEI3afOiqwar3ufbIo3n/FR/g3X91KZZlcdwxx/NPn/gXPvIPf89ZF2ygs7OTf/vC1+ox/TlDRF4ARoEUkDTGrBeRJcAPgcOBF4C3G2MGxfnl/DuwEZgA3muMecjdz2XAp9zdfs4Y8113+8nAd4AYsBn4O2PM3GSWzyHTdjzHfo2m+pmwhzMefIsArYEuukMrcuxXi9WxoBfgNCshCRMKLqWDpQXPJc0046nhHPs1kjrIvsTzmTEBQrQHumnPsl/tge66/S7VfpVHs9mveSvMjDHEzTgjyQMMJQ8wkjrISOpgxgsmCG2BbnpCq50//EAPbYHuReGyXwhki7ae0OrM9nQIejQ14N76GU0O0B/flfnyCkmEjsBSOoNL6QwspzO4jIjVUtU80nkY9V7VdPH5b+Xi89+as+2r//ofNe2zCXi9MeZg1uNrgXuMMV8UkWvdxx8DzgPWurdTgW8Cp7qG8NPAepziCQ+KyCZjzKA75q+A3+MYtnOBO+bmtGaHpEkwkjzIcOoAw64Nm7RHM89HrTbaA0voDR9Oe2AJ7YElDQuJKZUTlDCdwWV0BpflbE+HoDP2KzXAnqkdvMRTgONdaw8soSOwlK6gY79arc6qSoKo/aqIprFf80aY2SbFSKqfoeRehpL7GUruz3jC0n/IK0IvozO4lI7AUloDXeq+X4CIiLtiqoWloTWZ7SmTzBi69Jfc8/FHM2ItZrXRGeilK7gcYypz45+/cbxmQ7ZIuRDY4N7/LrAFx7BdCNzsXjHeLyJdIrLSHXu3MWYAQETuBs4VkS1AhzHmfnf7zcBFzCNhZoxh0h5lKLnPuaX2M5oahKy/z47AMtZEjqEzsJT2QM+sh+mVxhCUUIFgy472OParn72J5zLRgQAhV6QtxzZ/gcEUrGL1Q+1X1TTMfjW1ckmaabZPbmMwuZfh5IFMuYaY1U53cGXmaqIj0KNu/EVOQIJ0BZfTFVzOIZFjAUesjaQOMuwK+cHkHvYmnmXKTDKS6icoIQIEnZ8SavAZzC1jdpitk4eWOXrvUhHZlrXhBmPMDXmDDHCXiBjgv9zne40xe9I7AXrd+6uBl7Jeu8vdVmz7Lo/tTU3KJHkx/gSDyb0MJvdmVnMHcb6YXxY9ia7gMjoDywhbsQbPVmkkIkJrwFk0sCL8MsARa+P2EMPJA44NS+3nufjDbsSgHytju5yfssga+ZRvw+af/WpqYTZhj/B8/BHaAz0cEjmWrmAv3cHeqsNSyuIiIEG6gyvoDq7IbJtMjRKSKGGJOPWNmGbKjfQ7Qs0pSKne1hwOGmPWlxhzujFmt4gsB+4WkaeznzTGGNfoLRrG7SGenvwdUWmlJ7SK7uAKugLLaQss0XQKpSQiklnotDpyNOCEQcPi1FlLkXDEvvupStfGTF9olutRWwTMO/vV1N8+LVYHr+24aFGUolDmhlig3Sn5YbUBYLBzCuommWDKTCBIVueDEFY59UEWMcaY3e7P/SJyG3AKsE9EVhpj9riu/v3u8N3AIVkvX+Nu281M6CC9fYu7fY3H+KYmZrXz6raNxAL+q/oUpRKCEsKSQI5zYqbbgiPU0p5ZJxoQXpQRgUppNvvV1L7PdFVpRZktBIughIlarZml7DGrnaBESJkkcXuMMbfEStweJ2n8q4cvVkSkVUTa0/eBs4HHgU3AZe6wy4Cfuvc3Ae8Rh9cAw27I4E7gbBHpFpFudz93us+NiMhr3BVR78naV9MSkoiKMmXWCUiIiNVCi9Xp5FdbnUTEEW5TZoJxe5jRVD+T9igJM1VQXHix04z2q2aPmYgcAtyME381OPHbf88b47u8VFGaCcHK9NMDsElleprOXI1mt6NZfLkdHvQCt7nhuSDwv40xvxCRB4BbReQK4EXg7e74zTi2YAeOPbgcwBgzICL/DDzgjvtsOpEW+FtmlpvfQR0T/9WGKQuJdEusCC0YbDca4EQEEmbKGeN600JWRKMBTWi/6hHKTAIfNsY85KrOB0XkbmPMk1ljPJeX1uHYijKrWAQIi9OaxmDcsMF0prUL4DZKjhCScNUi7YEHf8/n/9c/88z2p/nKF77KuW/aWM/TmFWMMc8Br/DY3g+80WO7Aa7y2ddNwE0e27cBL695st6oDVMWJPkXmk7ahnOhOWUmmEpNZPVtrr5VoNqvzHN1sV81X+obY/akrxyNMaPAUxSuOMgsL3WXjKaXlyrKvEEQN+zp1JdqtbqISCzTW3U0NcB4atht/lxZuGDlytV84TNf5i3nXjBLs1f8UBumLBYC4vR0bg100R5YkulPPG0mGbed2mpxe5yUSZbeWRZqv+pLXZP/ReRw4JU4RdSy8VtGuidvHCLyPuB9QCZBW1EaSfvwT1i6/zqCyT6SwVUcXH4No50XZXqHRmjFJkXCdhq8x+1x4owTkJB7tVroSfvJz3/Mt27+b0SEdWuP4cuf+zcALGvRh0UbSq02TO2X0mz42S/B7R0qUQwmE+pMp2xYaW+bFcHKkwpqv2aXugkzEWkDfgxcbYwZqXY/bv2QGwA6g8sW1fJ6pfloH/4JvXs+geWudAold9O75xMAjHZelBln4ayUitCSK9LMGHHIhApCEmH7s3/imzd+nR98+8cs6V7C0PDQ3J+YUkA9bJjaL6WZKNd+CZKxT45ImyJhppkyk0ylJp2UDitKSCLseHaH2q9Zpi7yVkRCOAbt+8aY/+sxxG95qaI0NUv3X5cxamksM8nS/df5viYt0toC3bRaXYQlRsokmbRHGU31s/X3v+bsN53Hku4lAHR1ds3mKShloDZMWYhUY78ckRalxepww52tCELcHmc0NcC9v/8VZ73pHLVfs0jNwsxdrfQt4CljzFd8hvktL1WUpiaY7Ktoez5OzbRW2gNLaLE6CEqYFEkSJs5YaohpE8+0jVIag9owZaFSq/1ywp0xWgNdtAW6nZxabJJmmtFUP3F7PNORR6kf9fCYnQa8G3iDiDzs3jaKyPtF5P3umM3AczjLS/8bZ+moojQ9yeCqirYXIyhhYlY7rzv1Tfzqnl8xNDRE3B5j18DzTNpjFSfcKnVDbZiyIKmn/XIiAa1sOPVN/OqeXzM2PMG0mWT3wAuM28OZVepK7dScY2aMuQ+K934otrxUUZqZg8uvycnRALAlxsHl11S9z6OPXMffXPH/8v73/zWWZXH0uqN56yVv5aMf/QgjIyP86t5f8rX/vJ7bf3RXPU5BKYHaMGWhMhv2a+2RR/M3V3yAv/rrK7Esi3XrjuGSS97KNR/9sNqvOtHULZkUpdGkE2S9VjXVwsXnv5WLz39r5rHBcM8dW5i249ikECym7AnCVlQL2CqKUhVzZb8Afn3HVqZNnKSZBmDSHiUsMe07XAX6jilKCUY7L6rZkJVCEKeQbSDmdhmIZwpAhiVK2IpphW5FUSpmLuwXzLRQtEkxbcdJmDgJM+V0IZCYtlesABVmitJk5Bq4SaZNnOlUnKCEMUYTbRVFaV4sAkStViK0kDBxpuxJJsyIm6MWa/T05gUaI1EWHcY2OClDzY1j4JwuAxFpIWUSTKbG2TZ6BwMJXRCoKIuR+WK/0lGA9sASYlY7IEzaY0wmR9kZf1IXOxVBhZmy6Nj3fD9Tqcl5YdxgZsl60ETZ+/xBRlMDPDB2O38YvV0FmqIsMuab/QIISYRWq5OAHeaFZ3fy1ORv2Tp8Ky/GH1eB5oGGMpVFx//58j287SPQe0QPYhVdjNc0GNuw7/l+bv9fD3Bm5zvYNfU0z8Uf4YGx2xs9NUVR5pD5aL9gxobd/ZXHWd+2kWfjD/H05P0qzDxQYaYsOsaH4nznk/NX0AQkyGHRl7MmcgwDyT08NHZno6ekKMocMd/tF0BPaBU9oVX0J/poDyxhe3xbo6fUVKgwU5R5SkCCLAsdUnqgB2OpKPeNrC1z9B+qOoaiKEoxekKVF7pNU74Nm3/2S3PMFEVRFEVRmgQVZoqiKIqiKE2CCjNFURRFUZQmQYWZoiiKoihKk6DCTFEURVEUpUlQYaYoiqIoitIkqDBTFEVRFEVpElSYKYqiKIqiNAkqzBRFURRFUZoEFWaKoiiKoihNQl2EmYjcJCL7ReRxn+c3iMiwiDzs3v6xHsdVFEWpFbVfiqI0E/Xqlfkd4OvAzUXGbDXGvKVOx1MURakX30Htl6IoTUJdPGbGmHuBgXrsS1EUZS5R+6UoSjMxlzlmrxWRR0TkDhE53m+QiLxPRLaJyLZpE5/D6SmKovii9ktRlDmhXqHMUjwEHGaMGRORjcBPgLVeA40xNwA3AHQGl5k5mp+iKIofar8URZkz5sRjZowZMcaMufc3AyERWToXx1YURakFtV+KoswlcyLMRGSFiIh7/xT3uP1zcWxFUWYfEQmIyB9F5Ofu4yNE5PciskNEfigiYXd7xH28w33+8Kx9fNzd/oyInJO1/Vx32w4RubYB56b2S1EWOM1kw+pVLuMHwO+AdSKyS0SuEJH3i8j73SGXAI+LyCPAV4FLjTHq5leUhcPfAU9lPf4S8G/GmKOAQeAKd/sVwKC7/d/ccYjIccClwPHAucB/uIYyAHwDOA84DninO7ZuqP1SFIUmsmF1yTEzxryzxPNfx1mOrijKAkNE1gBvBv4F+JDrXXoD8OfukO8CnwG+CVzo3gf4EfB1d/yFwC3GmCngeRHZAZzijtthjHnOPdYt7tgn6zV/tV+KsrhpNhs2V8n/iqI0ERPJEI8OrCp3+FIR2Zb1+AY3yT3N9cBHgXb3cQ8wZIxJuo93Aavd+6uBlwCMMUkRGXbHrwbuz9pn9mteytt+arkTVxRlYVKBDStlv6DJbJgKs/nEoSvrs5+de+qzH2WxcNAYs97rCRF5C7DfGPOgiGyY01kpiqKUxtd+QXPaMBVmjaZeYmu2jqkiTinOacAFbhmJKNAB/DvQJSJB94pzDbDbHb8bOATYJSJBoBMnkT69PU32a/y2K4qi1ErT2TBtYj4XHLrS/9bszNd5K3OCMebjxpg1xpjDcRJff2WMeRfwa5ykeYDLgJ+69ze5j3Gf/5WbSL8JuNRd8XQETp2wPwAPAGvdFVJh9xib5uDUFEVZBDSjDVOPWb1ZLKLF7zzVw6Y4fAy4RUQ+B/wR+Ja7/VvA99zE2AEcI4Ux5gkRuRUnITYJXGWMSQGIyAeAO4EAcJMx5ok5PRNFURYjDbNhKsxqZbEIsXLJfz9UqC0ajDFbgC3u/eeYWZGUPSYOvM3n9f+Csyoqf/tmYHMdp6ooilJAs9gwFWaVokKsMrLfLxVpiqIoilIUFWbloGKsPqg3TVEURVGKosLMjwaLsclD2ksPqiOxl0bn9HiAetMURVEUJQ8VZtnMoRiba+FVimLzmRPRln7vVaApiqIoixgVZjDrgqzZRFil+M1/VgSbetEURVGURcziFmazJMjmuxArl/zzrLtQUy+aoiiKsshYnMKszoJsroTY2IrSv67+Hevo23Y6ibF2Qm2jrFp/Hz1HPZN5vm1vssirayP7fairSFOBpiiLgr6pU9gev5i43UPU6mdt9DZWRf7Q6GkpypyyuIRZHQXZbIixcoRXMfp3rGPn1rMwqRAAibEOdm49CyAjzso5Rj3E26yINBVoirJg6Zs6hScm3oNNBIC4vZQnJt4DoOJMWVQsDmHWZIKsVgGWJt87lkoEM6IsjUmF6Nt2eo7XrNr5VSvY6i7SVKApyrwn3zuWtMMZUZbGJsL2+MUqzJRFxcIWZnUSZLWKsXoJsWy8vGNgPMcmxurj3cs/j2qEWl1Fmgo0RZmXeHnH/OxX3O6Zw5kpSuNZmMKswYKsHkIs2xsW7Bhl+Yb76Hz5jNdr962nF3jHQDz3FewYZXK593Fi+4sf2ytPLU32edYi0uoi0FScKUpTUSxfbHv84gLvmJ/9ilr9dT22ojQ7C0+Y1UGUVSPIahVj2cJp+PF17LnvLEzSEV7JkQ72bHZyxdLiLDlS7hwNbUc9W/S4w4+vY/+W00mOtBOITZKKh8E455Odpwb4CrZaRFpdBJp6zxSlaSiVL1a+F8ywNPBoWcdLC7GQjJEwUSCUc+zBxJEcTJ2oYk1peqx67EREbhKR/SLyuM/zIiJfFZEdIvKoiLyqHsfN4dCVNYuyyUPaKxZlYyuCFYuyyeWFt2z2bzk9I8rSmGSI/VtOzzwOdpQrYoSxHUf6Pjv8+Dr2bD6L5EgHIKQmWzKiLHPsVIid929g59az3JCpZARb/451BftMvycVvy9VvP8FaPsspUKawn4tMLw8Yul8MajECyYcTJ1YdERaBDrhUCFh2kmLsuxj70q8PjMmLdb6pgp6VCtKw6mXx+w7wNeBm32ePw9Y695OBb7p/qwPdRBklVCNEPMj21sV7Bil7ahnfb1hyZF24r02AF0XbqX/lrMxifxwZvHXAUzdc2zmmE5eR2l9buIx8kMN5SwsSL9XlXjRavagqfesJIlkgL4DXY2eRrPwHRppv+YxXiFD8M8LS29fG70tx6NWDK99ZR+3XBuWb790YcH8ZiHbsLoIM2PMvSJyeJEhFwI3G2MMcL+IdInISmNM7d+cNYiy2RRkxcRYmrS3KjtkOfTQSfjlWgS6RzL329c/DcDgz08nNdgBlgHb2zhlv2502zH033FWlqDzPla5lLuwoFqBVnN4U8WZUoKG2q95jFe48rGJ97oWpXi+WHaumSOubCBQ9DV+x63FhunCAqUZmascs9XAS1mPd7nbajNscyTK6i3I0uzbelpByNLPyEgoQfdb7svZ1r7+6YxAG912DAe/fw7YeXMNJIkd9xw7P3NlSQHnhwQTWKGEE+bMI39hgddigmwqFWh18Z6pOFNqY3bs1zzHO4E/5LO2EiymMh41cMRZWqD1TZ3CYxPvJT8EKSRYGniU3wx/IeOVS5lIWZ62XAxetrWahQWKMts0XfK/iLwPeB9A1GrzH1ilKJsNQVaJGMsOKaYGO8p8laHn0rtoP+8R3xHt5z0C7Qn6bz4bMxYDwGqbpOU1TzB+7ysw02FnoF3O1aVBWiYxEzEC3SMZQZgfOpVgguUbcsVi+r2YDYGm4kxpdsq2XwuAyrxNhuNbbvYNG6a3PzXxDpI4NjokY/QGH6AvcVpZZTXyjxdklCTtRK1+lgYezdkPFApFRWkW5kqY7QYOyXq8xt1WgDHmBuAGgM7gssJPYBN5ycoVZNliLJtA9wipwc6Srw8sHYH2BDs/eBWpgx0Elo7Q/fZf0376kznj2k9/smDbzg9eNSPKysRqneSwz3/T87l06NSrhEd+vtzyDfexYnnxwraVCDQVZ0qDqJ/9WkBErX5XKJU3FsjxfOWvisz2oKX5zfAXyi6rkU1IxnhD14dztnVPPVuyhIaW2VCagbkSZpuAD4jILThJs8NznZ9RriirlyDzE2PghB0zuWEFLvbcxxKeJnbSn+i/cWNGYKUOdtJ/40aAAiGWT+pguV65meMtuewuWDYFB3INYnboNE0ciO6zPPPl9mw+CzY6JT7K8aCVK86gytCmijOlOhpuv5qJwsT7bPuVQACTFZK0mGJp4NGq2i35e+W8Q5Pp4x0Tu6Vgu5fwyz8vbQmlNAN1EWYi8gNgA7BURHYBn8ZNFjDG/CewGdgI7AAmgMurOtAshy/rIcqKCTJwk+8LVlM6F9aBpSPETvoTkw8fTaq/g0CP4xkbvPX1BV4vMx1m9EevY93Fvy16vL7lQ0zt7y58wrLBCNI6iQD2eCxzvIzYWzY1M/6Af05HvNdm3zcL8+XSJT46X/5MWSHOcsUZ1OA9U3Gm5DFn9msBUJh4D2n7lb0qM9/rVKx8RjHR4+eVCzJK0JombvcQZBQRIWHaavJyVTtHRak39VqV+c4SzxvgqpoO0mBRVqsgA2DZFIN3nO5R4kIILB3m0K9+w318N6uWDWWe/Z9vXui5u6kylgof+pd38Oy/XYI9NSPsrMg0R/79j1j+xocByltynBZpPgLNL18uv/TH5PLS4gxmObSp4kzJYk7s1wLBr2J/1DrI6zo/ntmSL2Qem7jSc3+l8tS8ympYTHFsyw/rLpZKlfhQlLmi6ZL/PZnvoizL8+QXWkz1d+SIsWwiy7y9XhGf8dmkxdfOm85j6kAXkWVDHPqXd2S2AwXHLSrUfLxofvlyXoVwS4kzqCy0WZfG6IqilKRa8eLn+Sq1KjK/rMZs5n1VO0dFqTfNL8yaSJTlJ7d3Xbi1IOcqh2wR4xJYOkLqYKGAyRZZ++85KUdIdZ36JAfuenWO1ysYneL0D/5vjl7SV3T+ALytj0ezhFgpsoVaSZHmirPut9xXuGozVLhqM029Q5sVo14zZZFRj8T2csVL/rGWBh6lz65uVWSp3LB64eed05WbylzT/MKsCmZLlOUnt/ffcjZAoTjzEGRput/+awa+tbEgtHjoX94BOKIsO/Q4tb+b/rvXc8z5W9h53ysZ29dDW28/p37gVo4+73dlnSfAiR4C7tGBVSVflxZpXgJt9L7jGLz19c5K0e4RWk95nMknX0ZqsCNTZiOy/mnY5183rZzQZilxpiFNRSlOvRLbyxEvXsfqs09jVeh/mqpXpZdQPb7lZl2VqTSc5hZm4dLthvKZrfClZ//KRIjBn5+eK8yKiLJVy4bg4t+yv2PCN7S45ztn5Yg2gGQ8ws77Xsm7b//7kudVCflirZhQyxdoo/cdl7tSdLCT8QeOp+cddxeu3HRDvVEfgVZOaLMUGtJUFH/qldheTmjR71gHUyfm5KE1Ej+henzLzU0zR2Xx0tzCbJaoJqfMr39larAjU1Xfr74Y5IYHl7/x4ZwcL5gRSf+zz7su0Ni+2U9ATc+hHIF2v89K0cE7TvcN78Z77arEmYY0FaU2iuWGFast5kWp0OJ8SKLXFZhKM7OghFk53rJqE/2DHaMkR3wS992kd6/6Yn4J/WnyvVZtvQcZ27usYFxbr5PDcXrH9qL7K8V9I2tLjilHoKX6/Rcx5NdAy67b5lWYthxmNaSpKAucYsVg09vrVbdrPiTRVyoetfCsMpdU1jRxgVNs9WXXhVuRUMLjmdwih2Y6zOCtrweKi7ITl/R55n2d+oFbCUZzw6Hh6CTv+chXaxZl4Ai79K0UfnME/xWhgR63Ybob0k3XbXPEq2QKzw4/vq7gtZW0tvKj0sb0QE3dJBRlPrA2ehsWXmkWufYr7TWq97GaLYneTyR6bU+HPR2xKRkB2zd1yizPUlmsLBhhVg9vmR/xXpv29U/Tc+ldTmHWEhQrfQHeifhpjj7vd7zuUzeydNUeEJulq/Zw5b98ntMuuKuaqRelEoGWz6F/eQdWZDp3YzCJHQ/x/J9/nJ0fvIrRZ45k8OeFddvShWe9KCbOqv39KcpiZ1XkDxzfcjOQKjm21pBj+lhR6yBgiFoHi/bJbARe4lFIkLTD3Dl4A78Z/kJGeBULeyrKbKDfdC7leGva1z/Nwe9tLDmuWH2xYqIMXLH0ju3wjptLT6hOpMVZsTDniUv6ckKb6Ry55//jQpIjLc7GZAAz5txPHezk4H94F8YF/5w9RVFmh1WRP/gWes2mHiHHuSpxUS3puT09eSkJ4zSbNwRJ4qRoxO2lPDZxZU5T9XyaKWdOWVgsCI/ZbHvLsgl0jxQdn136Ip9yRFkjKeVB85q/PRXCCYekb9l4bXMo9j7W6jXTcObcIiJREfmDiDwiIk+IyD+5248Qkd+LyA4R+aGIhN3tEffxDvf5w7P29XF3+zMick7W9nPdbTtE5No5P8kFQinR1Wwhx9kmZcIUs19poeZFM+XMKdXTjPZLPWZUltvkVUg1u9fly668vWDFJdRflJ0R21n22K2Th1a079M7tvt6z7I9ZztvOq+gtEc5SChB91u8C88q85Ip4A3GmDERCQH3icgdwIeAfzPG3CIi/wlcAXzT/TlojDlKRC4FvgS8Q0SOAy4FjgdWAb8UkaPdY3wDOAvYBTwgIpuMMYXLn5WieNUhy+912cyernri3V7KC6cte7ZwW2wCdoHTdPZrQXjMZguvVkvt65+m569uJ7B0GMQQWDrM0r/9KUv/9qcELZvtX3on2971cfbfc1LZx6lElJ0R21mRKKv2NcVIi8xyenXOYABDoHuYnkvvon390+X1F1WaHuMw5j4MuTcDvAH4kbv9u8BF7v0L3ce4z79RRMTdfosxZsoY8zxO0/BT3NsOY8xzxphp4BZ3rFIhXvlfJ7TcyAktNwJOT8vs/KqFTKWhyGbOmVOqpxntl3rMKmR02zEM3nG6U+3erVsG5BRbndrfzbP/dgng5GIV85ZVKsrKpWXfBJ3PjxGYsklFLIaPaOOM3p1le8+Kec3S+PXw9CLQPcKhn7mxrLHKHJC0fBvSe7BURLZlPb7BGHND9gARCQAPAkfhXB0+CwwZY9I1TnYBq937q4GXAIwxSREZBnrc7fdn7Tb7NS/lbT+13MkrM3iVfQDq0hWgnmxYv4PLz9/GsiVjHBho49s/W8+WbUfV9RjFSoh4jdXCs01G+TZs3tmvRSHM6rWaL136IR3GzNQtCycLiq3aU2F23nSeZ1izGioVZd1/GsksIA1O2XT/ycnpqkScleLQv7wjp32Ug3F/zrj9JTxd19DlrBacVbw4aIxZX2yAMSYFnCQiXcBtwDFzMTGlfPyq3VtMNVWx1Q3rd3D1O7cSjTgrSHt7xrj6nVsB6irOioV1NWy5oJh39mtRCLN64Vn6YToM096toyoL9dWPzufHCqp6WLazfaK3pW7HSYvO7PZS7Zf8BsDpodnfQaDH7Yaw7tm6HVdpXowxQyLya+C1QJeIBN2rzjXAbnfYbuAQYJeIBIFOoD9re5rs1/htV8rEr+yDjXeeaKNWHV5+/raMKEsTjaS4/PxtdRVmfu2lvLZp2HJx0Cz2S4VZBaQG/VfoeFGsbMZsEpjyzt3y214L+e2l0r00C9pSlR82U+YZIrIMSLhGLYaT5Pol4NfAJTg5FZcBP3Vfssl9/Dv3+V8ZY4yIbAL+t4h8BSd5di3wBxz3xVoROQLHoF0K/Plcnd9CofKcqsasOly2ZKyi7bXgV9ZDhdjioRnt16IXZv071rHr+xtITcYAkOgkK8/eAsC+b57m9MDsHqH7LfcR6B7JtF/KpbAkRLGyGbNNKmIR9BBhqYiu9WgW5iKHZg5ZCXzXzdOwgFuNMT8XkSeBW0Tkc8AfgW+5478FfE9EdgADOIYKY8wTInIr8CSQBK5yQwyIyAeAO4EAcJMx5om5O73mJTtnDGzAImr1szTwKAdTJ+Z4ffxzqjzsVwPDdwcG2ujtKRRhBwbaGjCbEsxGmZ150LtX7dfs2q9FIcza9iY988z6d6xj571nY+yZ50y8hb5N5yABg0k521ODnfTfcjatpzzO+B9eXhDOLMBKceTf/6hu+WXglLwoN89s+Ii2nBwzANtyttcrv6yRzPf8srnKoZkrjDGPAq/02P4czoqk/O1x4G0++/oX4F88tm8GNtc82QVEfs6YY/Od3LFd9utJC650Ltmq0P/QZ59WRomIVENXHX77Z+tzPh8A8akA3/5Z0TSh+tHomoaVHL8BIk7t1+zbr0XtQunbdnqOKJshkBFlaUwixOSTL6Pn0rsypTJmEkXzMFaOKCvWDLycpuJpyhVVE70tDB7dQTJiYYBkxGLw6A7u7Cg/n7HYvIqdj1KaYjk0ilIuxetwFfbAPJg6MadUhq/9wmpoKG/LtqO4/gdnsK+/DdvAvv42rv/BGfX90j90pf9tPtGAc1D7NfvUxWMmIucC/45zyXajMeaLec+/F/gyMwlvXzfGNLx2QmKssgrxqcEODn5vI4HuEZb+zU+dBPeDhaHNSnPL7htZW3bZjLQ4K+U9m+htyST6V+olq0QsZtPnt9jBJ78sum9xXhfMZQ6NUh7z0YZVmjMWt3t4bOJKolY/J7Tc6IZAC0ObzVDRfsu2o+onxOab2KoHXudcJ++a2q/Zp2Zh5sZly6lq+0NjzAdqPV49CbWNkhirJKHfuQpNDTplMlrPfITxe1+RUyrDL7fs0YFVReuZVSLOoHKxVS6lRFk9vWXDj69j392F+X0rlj9Tt2OUTZ2MVjm5F/Mqh2YRMF9tWCV1uBxKhzYXRGmIORRiT7y0kV0Dr8YJPtmsWfIAxx/SpBH3MsSa2q/moB4ui4ZX5Y69NFpyjFde0qr19yGWV75SCgkUz2My02EmHz6anis353QBWHLFZt/cslKiplpPVb2oRZRV6i2buudY9tx+NqnJFtJ96ky8hb6fnU3/jnWAkwP42C1X8NCNV/PYLVfQv2NdWfll5fw9zAbp3IvenjEsmcm92LB+R864b/9sPfGpQM62Oc2hUfJpuA2rhrXR27CY8nnWL0zp4BXanLcV7RsUinRE2ak4TlYBAuwaOJUnXtoIQN/ACfzmyQ9x5yP/xG+e/BB9AyfM2dzKJut9U/vVPNQjlJmpguviV9X2rSJyJvAn4O+NMS95jEFE3ge8DyAa8loBWT96jnI8M7t+t4HUVOGqzP1bTic5kg53Fq5cSvV30H76kwWlIfoOdLHKJ5xZjucM5raheTmCsJ6iDJz3Nj+PDwATpG/b6QDs3HoWJuUstEiMdbBz61lET0jRu+bpkvNtBOXWX0rfX0CrmuY7dbNhOfbLml0PQn4dLr9Vme7MCl4ft3t8y0U0NU0SmnQ8ZYWNz3cNnkLroQfY3ncutmu/4okunui7iOmeqK/9atQFZZrLL/6j2q8mYa5WZf4M+IExZkpE/hqnz9QbvAa6rRJuAOhsWVX8si+L2EujTB5SPGfMa3Vmz1HPZARafjPzzpc727d//QqSI4Uhz0DXiO+xahFnMDcCrVwPXVWirAjRfVaW4C0kMdbuLMxI5RXzTYV44ZkziwqzqoxbA3Iv6ppDo8wFZdmwHPsVXFa2/aqW4sLqBwD8ZvgLTZtLVjYNEGOlvk94xCfgZIQXnjkzI8rS2CXsV8njucyWgFvWMey9Xe3XnFMPYVas2i0AxphsC3Aj8K91OG7die0vFGcAyzfcx57NZ2GSMx80CSWcNkMHIrDMO5xQSpwBZQu0NLUItUpDpaVCr0VFWYmE/2DHqKfYhXTun7eRmpqsrMjvXKK5F/OWBWPDvPBqPTQvcsnmQIyVK4Y8EQOm0BOJGF87VQ/75TfnWgXbgZFOejsLxdmBkc6Z38U8qLG2EKiHMHuAElVtRWSlMSb9G70AeKoOxy2gWq9Z/4519G07ncRYO6G2UZa94b6MtwxmPGfp0Ga64Gz7evfKp0pxBuULtDRzkYc2G4IMcldhLt9wH3tuP7swnClJVq2/z/19FBqxSMzfS9noUIBX/aVE0iIaTrD5azeqy795aRobVi1ezcnTnjS/1kNNG8KcJUFWkwjzYMUhf2TvzleRG840rDjkjwweOIqpSY8V+0XsV614nV8lNvHb976eq8+9nWgokdkWT4S4f8dRfPevv8qyjmEOjHTy7Xtfz5anTlCRNovULMzc7uoFVW1F5LPANmPMJuCDInIBTjXcAeC9tR63FrLFWf+OdQW5TH2bzmPf3RvoPWtLRpR1vvyZzP14r0dro7Qg8RBoaSFTjkCD8kVaPuuHn+fCAw+zJDnBQLCFny47iW2dR5T12nJXW9ZDlMGM2M1elWlFJjnktVsyoeXs3wuAFUhw+Lp7y5pn2dTRuOTnXoyOR4hFpulsd/4m5nshxoXKfLRh2Xg1J39s4koGE0dyfJsTzmz6XLJDV7Lh2Me4/MxfFwqAKqm3EMvn6BN/BcDel17peM7E0HPMI6w67V4iO/YV2C8JJFhx6v8UOAZms2B2/ntQTKil3+vs38H9O47inBMfzYi13s5hrj73dmc87u9GBVrdEWNmPQ2iajpbVpnXHn1lxa8r5wOZ/nA8dssVviUzJJhg5ca7c7xnaTzFWRof71maYgItn3JF2vrh53nX3t8TMTPemvRv1k+kVVL6omQuWQWiLJvYfv9d9u9Yx97fn8bUZAeR2AiHr7u3vomzdTAoxZaXf/efbvEMbe7rb+OyT19a87HT3Dl444PGmIqWREUOPcSsvubqssY+/3fXVLx/xckxe23HRbN+HL8cMjCc0HJj0wsygA3HPlbgrbGN44vaX4FIq7cY8+oYUy75kZhV6+/LXHRWQ70FXOyl0ZJi+Lt//VXP8Oa+4U4u+68P5m6swZ7Opg2bj/aruVsyTSdKj/GgkpBmsSKzJhmi72fnAhSIs+g+y1+cFfGeQXketDR+4ilfsF144OEcUQYzDvae5ATv3PMHdo5188vYcSWPmT/PopRoTl6NKMs2aJHYCOtO+nn9E/7rQKnWJFqIUZkL/AvNCo9N/CXQhE2588KVl5/56xxRBmC5BizHS+MhzuohxmoRYF7UW5SB9xxrEWt/dvp2rj7ldqJBD2+Y+z77Lgjw2q55aHWjuYUZOL/kKnIOyhVnJYvMGos9m88CvMUZFPGelSnQoDIvGhQKtu7kRNHxMZK8f3xrSWFW9irLGgQZFBdl2SGAqclOtj/miGMvcVa1KKvReGxYv4OPvPs3BAK5Hufs5eW6GECZC4oXmg3wxMR7gCYRZz623E8ApImGElx+5q8zgqFWMVZvIZaNV3rMzq3Od0it4iyfasXaGw97iE+85ocErTz7lfc+F10Q4IcKtJppfmEGsyrOXnbUb9j+2LkFS5uzMckQ+7ec7hnShBLeMygp0KBQEFUq1PZbHaywiyeWLvd5vqKSFzUIsuHH12UWUPhdRXqVyfBbZt5IUXb1O7cWiLI0aY9Yw5sxK4uCtdHbeGziSrxqlYFTTHZ7/OLGCrMS9ttPAGSzrGO4akFWDyG2d/+M/Qp2jLJ8w32e3wm7b/Uu9bP7odNp+bPyhVmxNI9iZJ+rl0h742EP8dFTflwgytKk3+fYS6O+CwK+fe/rS0/k0JUqzqpkfgizGiglztJf9jueeCOpRAw/45YcafctpwFleM8gV9SUyEPzE0vZgu1Nk0/y/vGtLLdHGJEY0wjhIhW/++iqqu5YKTGWppQo2/PzszNN4xNjHby45Txe3HJejkgrt0xGo0QZeBeSzSbtEduy7SiOPWIfbzn9aSzLYNvCnfcfrYn/Sl1ZFfkDg4kj2ZV4PX72q9K+mnWjiCDLzm8anYwxnbQIB/3t5/6JrooOXa0Y87Lxw4+vyymZlBxxFon1bTqvQKT51WgsVrux3HlAZYLNa6HB+17xi0z40ov0+zx5SDt3jP0Zxz76Em955UNYYrCNcOejJ5a/KEO9Z1Uxf4RZlV4z8BZn+3YdwwvPnJlJLD/q+HsAeOaRN4MpFBihNkcIFBNnUKZAg4pEWjZpYXWBeZiPchctOB+wLjOJIbcRS7aJniDEdZxV9nHqIcbSxPbDn+7ckBFl+TPMVPUfShGJjZRcZt5IUQbFc8SyPWIb1u/gnNf8KeNZCwQM57zmTzz1fG/DxZmVWLxN5BcCXuUxukPPujllgYLxDSkmW0KUZXtiOlsmSaYE2wZxDZdkGbB4MsQNj5xb8pCVirFitjyb/VtOz6lj6eBMMDnSkZPu4lejMdhRn1xYrzmXK9bGVgRZ3jLk+3z++/zGwx7inFc8SsD1rgXEcM6Jj/JU3yGVrZidBe/ZQrZh8+usavjFZn+Z79t1DNsfO9cVAJKTx7TuFbdjBXKvJiSQYNX6+2b2tb/0ByG6z8rcSnIgknsrg2u4OyPKMvPMuk1jMUALNrCLLj7BRWySk+o2h3LPLf0+pVte+ZGu6n/4unsL3v/sMhmNFmXgnyOWSgnX/+CMnLIZfi1OFKVa0uUxnLwyyTQlBzih5aaC/plzXky2jJ6VXsn+wYDBshxBJuKsyjQG9o538a9/eCv3vPgqz32NrQhmbuUwuXzmVi6lvF0mGWLf1tOI99p0XbgVyTs3CSXounAr8V67rFulVHJOe+PdntuTthS8z17etXQeWsU0SSut+cD88ZilqYPnrFi7jFPfeANAxptWbEVNWnSU+jBkC5iyPnTFhJHrXVvFUNFdRLA5QJj18sny9lsG5V6dVJsbMTXZkQktZ3szD193L4ebB3K7GVZCna/U/HLHskUZVNaiSVHKZXv84pwq/jCTR/a6zo9nxjSkmGyZtrlUsj84qzL3jnfxjk2f8Hy+EiFWKfl2OtA9QmqweO/m1KDjJUsXHh/8+emkBjsKC5JXcfw05djg7PP1ssXfeG4jnzrmVmJZF8CTqRBf/n2h+PXzrpXz+/NEQ5tlMf+EGdQszkq1y+hd83ROonkpA5D9x193kZaPK676urtYExgqOnSVGYKD1YuxSt3ExQSZFZnEnmop+vp0uDL//a+pHMYsGIBym/jqqkxlNvDLF0tvb0gx2QrtcTnJ/uAtDOotyMqxw91vuY+D39uIXw4fQGDpSObCuf28R2g/75HiO63iQjl/rqVstJdI+8X+kwG46mWbWREdZG+8m288t5FfTJ0MK3IXDOyf6GJF61DBfvdPdGUWCFSFLgwoyvwUZlCTOIuGhoknugq2+7XLSP+hlmMQyvWigfeHqlyxdt34eXy+/Ue0iH8SZ5/dVda+is2nFOV6xw557RZe/M3ZYLzfw+xwZXb+XzQ0zNoVd7NqyWMVz202P/jlNPHVVZnKbOBXHqNhTcmrsMNeq/28yE74L8f+1lOMZdN+3iPE9yxn7Jfr8RJnEp6m++1OeG/0vuMYvPX1pA52EFg6Qvfbf0376U8W7rRUbnEZwi37PMoVabH9jjhLC7R8xlYEM995NzxyLh895cc54czsPDQVZ7PD/BVmMPNLrdAwrF1xN0/suhDbhDPbymn3U41Ag8pc6cU+XNkfwk3TJ8MoXNN6B6utIQwzBRkBJkyI68bPK3vflVBNqPKwtieIviKVEVyB0CQCJBOxnKr+6fy/dKg5nujiiV0XAlQmzuboA1+s8n+5njVFqYSmakpe5cXxHWN/xvQDUd73il+wvGWIkakWWkJxwoEZG5cWAPUSZGWLMR/BtOwv7yZ69O6M6MIyYEuO+Bq97zj6b9yImXa+W1IHO+m/cSOAtzirZB4lhFq5Ii1boJ27/MFCz9n+kzPveTq0mf497Z/o4oZHzs0Jeao4qz/zW5ilqdB7lv6C3773LOKJTqKhYQ47fmvR6vLZVCLQoFDIVJPzAIUftrt4NXfxagDe3PEgV/duZmVokD2Jbq7ft5G7Rk4mWt2hCqhGjOXX0MkPUXrx4hNnFOb/mTDb955VvjCbQ1FWrPJ/+qcKMaWeNE1T8ipFWXqF/D0vvirnC/6Nhz2UIwC+9sKbnfBasX3VS5CVuTK+/fQnMwLLq9bkth+9LiPK0pjpMKM/eh3rLv6t5z7LLmFUgVBLn3MxgfaGox7kU6tvJWY53rBVsUE+dcytwEy4c2xFsOD35EX6d1qVQDt0JQxW/rKFzMIQZlCVOMv5ojfASzN/YPnlNLz6NGYLj0qWaXuJnGrFWprbR07m9pHiRqxcqk3eh8pbhOSHLeMJ7wRbv+05zPGVV7FVlyrGlNmknDwyr5IadRNvNYoyL7IFQCl7Wspe1lOMFSv2vf+ek9h503lMHegismyIqf1dnuOmiogvv/2XFGzZ8/cRacUE2tW9mzOiLE0skOCql23OCXOmfxfl2PaavGdKhoUjzKDq0GY2sZdG6Rs4ge19M+G0Um2BoHqRljmujxiqVbBVerxqqLZf275dx7D9kXMyIWUn78+7QG4oULzlVCPc4brqUmlW0iU10uHO7JIaNYuzWRBlaWoNW5YUZGWIsXK7ruy/5ySe/bdLsKcc+zW1vxs/+xVsL2G/yphHUaGWPq8KBNrKkLebakXUe3t27lkxVJzVzsISZmlqFGjb957lW06jnHBn/h9vLe1A6img6kktzXPBEcAvPnlGTp6fg+AYt9wE20QqQt/ACYXhzAbmJ+iqS6VZKVZSoyZh1kBRVpOXrIQgq7QF3olL+vjedz6SEWUzeNuv1ESY6O8P4+jzfue5v/zex6Xm6CvSyhBoaXG2J9HN6nChCPOrcwblizOlNhamMEtT5cpNv7CZX5mNUtRTqHnhl8BZT+rxYcy/iiorPJkhmJtn1gQJo7rqUmlWSpXUqIpZEmVvPOwhrnzlnUXtV9VesjoIshOX9HluH9vn1zi+EJMM8fuvv91XmHkdo5hYS8+7GoGWfr+u37eRz2blmAFM2iH+7eBG3+NCeeJMvWa1sbCFGVTlPfMrpxENDWf+2Kptpgv+IqcawXbu8gdzigWuig3yz8d9n1d0PM+XdlxSt7lVS7EPp9/77Ec80dlQQea1AvP6H5yhqy6VpqPuJTVmUZR95NQfF7VfVYmyGgWZnxjLpq33IGN7l5Ucl2ZsX2WiOHsOfiKtLIGWJc4uCD/INa13sMoaos/u4rbBV/O69qdyFo3dPnIyLC8erVFxNrssfGGWpgKB5llOQ6ZZu+LuzOPsP7haRFo25YiifPF21cs251RwBqdsxiVrfssjI0dkrjzn0v1c7ofR7322JEHSbi0Y37A6TfivwLz+B2dw2acvbdi8FMWLZiipUY5dvPKVdxa1X7fh7fmv1ktWTJCVI8bSnN6xHfnIV7nxk59gOj7Tbi4cnSQcjTM2VBgOXLpyH6d3bC/Yft/I2pLHS8+tKoHmirMLwg/m1L5cExjioiUP8Oldb/dcODZZQpyVg4qz6lg8wixNGQLNq5xGsSKn+X949RJqXuQLLL9ETUvg/z38diLDqaI1aOpFNR8+v/cZKBRsjarT5KIrMJX5RF1LalThLSs3p6yY/frbtZu5bfvJBaWAvhw/16nj6IWPKKtVkHkJqtMuuAuAW//X33JwTy9LV+7j7R/+DwBPwZZ+rtS+iwm1cgSanzi7xtxR0F+5RRL83arbfVf0FxNnmm82e9RFmInIucC/AwHgRmPMF/OejwA3AycD/cA7jDEv1OPYVVNCoBWU06iAuRRqfi0zwGlnkl21eUWr8/j4pS9w2uqnqxZr9bwC8n2f+4caX6cpC12BubCZlzasBHVpzTSLogycRPNVMW9xtjI0yJs7HszJg1odHuRfQ7fyD+YndMskfXYX142f5wi1CkVZtYIsm9MuuCsj0PLJF2x+4/yOWa1A8xNnfv2VV1lDOYsCKkFDmrNDzcJMRALAN4CzgF3AAyKyyRiTXeb4CmDQGHOUiFwKfAl4R63HLpei9Xyyc5ZqKLNRDK8/ynqJtRseOZdPvfaWnKr/aWwjOa00AKLBBBevvT8zPi3WAE9xNqcfqKzfxarInoYKsXxKrcAs1gFAaW6a3YbNaj2yBvON5zbyz8d939N+7Ul0e9baCkuKHpkEnHDc59t/BJJkEycV7KMaUVZKjHlxRmxn7uN33Mi177jRc+zWyUPL2me5Aq1ccdZHF2s8xFm6dZ9XkfLbR06uS0hTqYx6eMxOAXYYY54DEJFbgAuBbKN2IfAZ9/6PgK+LiBhjvIu+1JGK6vnMgUhLU0rwlCvc7nnxVRy/9IUcsQVOO5NIwLsPXb4RjAYT/PXxd/Db+0rnOswKTbDCshT3P34I55/5VO577K7ALKcDgNLUNK0Nm9V6ZKWYZW8ZOBXmX9HxPJes+W3OZ2vSDnH9vo18ac33S+6vRRJcw90Fwmw2RVm+EKuE/NeWEmqlBFo54uwC8zAtTBcU8ki37rsg/CD/1DOTf7Y6PMhnVztdAOpVuFwpn3o0T1wNvJT1eJe7zXOMMSYJDAOeS1RE5H0isk1Etk2beM2TK1bPpyg798zcGkDspdGyb//9k7P4159fxL7hTmwD+4Y7uf6ON7N/pPxyFMs6hmfxbDxo8PtbCRvW72Djac/kfHEYA7aBj162hY+8+ze++WeLARE5RER+LSJPisgTIvJ37vYlInK3iGx3f3a720VEvioiO0TkURF5Vda+LnPHbxeRy7K2nywij7mv+aqIePhYqqZuNqxp7Nc84ks7LuEfnnwXfZPd2AZ2T3fzj7udhPQ9Cf+aWtnkh+kqFWWnd2wvKcrOiO3M3OpJufssNj+/81q1bIgLzMN8np+whImMKDPApG0RNyG+0v4Drmu/JSPK0sSsBFf3bgaKr4wtqyjwLKbz1Eoz2q+mS/43xtwA3ADQGVxW89VosXo+vxn+QnnhgTn0pFXLlqdOYMtTJxRsv/rc24mGZj5wtin0mAEcqEDEVUXWe+iEZj44b0Izf3PJ7wgFc1eBiUBL1BVjAe8/00WUf5YEPmyMeUhE2oEHReRu4L3APcaYL4rItcC1wMeA84C17u1U4JvAqSKyBPg0sB7nu+NBN6Q46I75K+D3wGbgXOCOOTzHsmhK+zVH1FJE9hf7T/ZcgXn9vo3805pbC0RDPn10Ze5XI8pKkS+cbrvtYv71ix+nr281q1bt5qPXfoGLL65tcVL6GMU8aKd3bC9rFWc213B3QdK/AFGxiYnTkcDy6Vjg1x1ggdF09qsewmw3cEjW4zXuNq8xu0QkCHTiJNDOOn71fIDM9orCA/keniYTahuOfYzLz/w1yzqGOTDSyZ2PnshrjtqReXz/jqM458RHc8RaPBHi2/e+vv6T8fCGNTQ0UyUdreX11MtnsXQAMMbsAfa490dF5CkcD9OFwAZ32HeBLTiG7ULgZjcMeL+IdInISnfs3caYAQDXOJ4rIluADmPM/e72m4GLqJ8wa1obVnf7VS4NsmteeU6fGL0kU3tryLTQKnEiMnOhNEGI6zgLqC3R3wsvT9Ztt13MtR+9jsnJFgB27z6Eaz96HUDN4ix9zHLz0LLxDWn6JP2X43Mu12M5n2lG+1WPUOYDwFoROUJEwsClwKa8MZuAtFvvEuBXc5FfBk49H4v8L9bClhlVhweyQ3INDsttOPYxrj73dno7h7EEejuHOefER/n2va9n45f/gcv+64M81XcIU4kgxjjhuOGJGNf/4s2e3raKyH8ffN6LxRCaAee9vf/xQ0oPbBBW0knoLecGLE2H59zb+/z2KyKHA6/EuTLsdY0ewF6g173vFzostn2Xx/Z60bQ2bNbtVxORXoG5OjyIJbl5TmcOfoqj+q/js2MXMmYiGJx3YYAWPsFFbJKTqjpmNYn+//rFj2dEWZrJyRb+9Ysfr2oOlVLpnPdb1XWsmbYtrt9XvAtAIynXhjEP7VfNHjNjTFJEPgDcibPU/CZjzBMi8llgmzFmE/At4HsisgMYwDF8c4JXPZ9ZaVeSxkuQzNHV5+Vn/jrHEwYQDSW4/Mxfs+WpEzLCLXtMJFRFHZoaBOisvvezxMhYhM72yrxmIvCal7/EN//PLE1qbjlojCnZZ0pE2oAfA1cbY0ay0yiMMUZE5uRirFKa2YbNuf2aRYrlIk0ux3MFZsxKcE3rHWyaPrmgQCpAlOIhTqivtwygr8/7O9Vve7XHrsZr5sV/tp7BP47eXrEXZtyOLpTE/3lnv+qSY2aM2YwTN83e9o9Z9+PA2+pxrGrIr+fj5GbUsV1JKYoJmTqKNr8E/vT2coRbJgw6SyUf6t4qZg745o9fy4fe9RvCoZnPpTGlQwGLKMcMEQnhGLXvG2P+r7t5n4isNMbscV396UX3fqHD3cyEDtLbt7jb13iMrxvNbMMabr/mCL98plXWEADXtN5RkGvWwsyKzAvMw1x78E6W2yPstzr4z9Yz+GXsON/jVeMtA1i1aje7dxd6w1etquufZN34Zew4XjZ6gL/gDznirJQN6wxMzPrcmoVms1/1CGXOO7zCAw2rLO8VAix2K4JfAv+BEafHZDHhtmH5Vq4++2czYVC35MOG9TtqPsVsmuq9L5Mt247iK99/Hfv625xVr/1tbLr32MzjVMrbui2WHDN3hdG3gKeMMV/Jeio7/HcZ8NOs7e9xVze9Bhh2QwZ3AmeLSLe7Aups4E73uREReY17rPdk7WvR0YyfoXqsuvPLZ0rX2UoLtHxW4aw8/AK3scIewQJW2CN8bPRO3jT5pOdrauGj136BWCxXtMRiE3z02i/U/Vj14jNyIR/ibeyiCxvYRRffm3wtu1Jd+AXkF0N+GTSn/Wq6VZlzQV3blcw1RcTZt297ZU49LXBrbd32SsC/SKqxhb+55Hdz0nJovr73W7YdVfA+pMOU+XXMYKbG2SLhNODdwGMi8rC77RPAF4FbReQK4EXg7e5zm4GNwA5gArgcwBgzICL/jJPzBfDZdCIt8LfAd4AYTtJs063InCvm62eoFNfv25hT5R9m6myBI9DWBIYKXmcj/CO3EyM3LSNGkvePb+WznF/XeaYT/Ou9KjObeoUxs9kkJ+XWepuI8JkJPEPE6Tpyi4Sms1+LUphBndqVNBlp4eBXgf7bP1vPh//i3oLSD4GA8V15OBvhuIX23pd63xc6xpj7yM9Gn+GNHuMNcJXPvm4CbvLYvg14eQ3TXFAstM8QzBQy9euLed34eQUCAiCIoRvvsNtye8T3ePeNrK06nHnxxbfVVYhVQqXlMjx7Z2axafpkGCWz8jW76v9ioBnt16IVZgsVL89ONn4LyfxyDRZLOK4cirVdKvW+K4pSmttHTs4RBPHemYvITdMn86rJF3h37HcF9srvW3W/1cGjA6uqWgCwdfLQuheTLfe41eDX2DzNBeZhruFuVjFEH11cFz4vI3o3TZ/MpumTq+qXWQ7aK7MyFmWO2WLl8vO35SSw55Ov2RZZOK4o6XBlb8/YrObgKcp8ZLa+ePOFwhsiT/leROZbtkmC/GfrGUX3X8r7NBshxVqOV623LF39fw1DWMAanB6jF4QfLGs/xXpllmpirlSOeswWEaXCkganNER769SiC8flk+8di0YSc5KDpygLlba9Sd+SGbH9xdv+pPFbAACO/RqkhS4mClZlVus1g7nxnJUjAIuJsnxv2Zsmn+T941tZbkboo4sWpguq/7fITCkSKBTBSuNQYbaI8Ev+T2MJxKdCvOPad2eEyUcv27LoRJpXU3K/lUuLqSSGojQavwUA4IR/JghzwfKrMsLkH0dvnxFpA8d5irO04CmWb5YtnOol0irxxlUqyj42emdmMcQahnwaLhUXumlq9ZZpGLNyVJgtIr79s/UFqwfzWbZkzFOYXP3OrQDzRpwVywcrxeXnbyt4jzQHT1lU7Nwz522Z/Lxm0X1WJtfMbwFAmlUMsX7/C3yMGWGSLp0B+IozKH8xQL6gKleoVSLEDjt4kJN27aJlepqBYAvxZUG2dR6RM8Yvp+z941sLVqj65eClS5H4ecuKiTJl9lBhtojIXj24fMmYp9g4MNDmKUzKCdvVIobqSa3C0s8Lll+QUXPwFGWG2EujJeuZFQtnFiMtztIrCK9rv4WgRyH2Prq4hrt9S2f8Mnacb1hz/fDznLPjcZYkJ5gIh3l4zRpeXOrdpzSbeuehHXbwIOuff5GIcexXT3KCd+39PUBGnBVL9PdbiZpvv9KlSKoNYaq3bPZQYbbISK8eLFZ766OXbfF8bbGwXb29bNWIvPRrvERnJflgfiHfkbEI8elQw4WnoixUysk1S4uzfM9Zupn5V/Dug5YtWPLF2frh53nX3t9nxFDr9DTrn38RoCxxlk+2x6tckXffyFrWDz/Pa/Y8TyAv+BgxKS488DA3pU7zfX060b+PLtZ4NC4fMDEm7QirrCH67C6uGz+Pu156te/+NITZOFSYLVKK1d66/PxtnsKkWNiuWi+bF9WIPC+hmU+5+WBeId/4VIBv/vi1KsSUxUMV4cx6eM3KCWnm197qs7u4LnAWm+QkrjF3ewqT/Gbeaa/TiUv6uPDAwxlRliZiUhyzcy/fD782s62cUOdhBw9y6gsvELSdubZOT3PqCy8AuSIvP28sLQ7zRVma7qR/i6TsWmXXcRaf5yc5yf4TJsQ/j12USfSH4sn+KsoaiwqzRYxf7S0/YZIftsv2avnlMFSTHF+NyPN6TT7l5oMt9oKxijLb1BrShJnaWzksm/IWJoQKSmdkVi7uH/G1X0vyxFA55SrO2fl4RpSlCdp2gcjLx0scZpMvLMG7eOwmOQkMXJO6O8c7NleiTKkdFWZKAeUIk3I8VFBdcryfmCsm8koJwFL5YPmh0/sfL2xSrCiLjlnymkH15TPSoiK7+GyGAxE2cSosJbeYKmexaewkGINVy4YKVi76MRBsKXke+eSLuVLby3k+vyZbviDLKR6bcoTYdZyX8She0+p0AKo2dAnlizL1ltWOCjPFEz9v2ob1O/ibt/6OjrYp35WKaapNji/W03PD+h2e8/J9jYH9JTxeXqHTC86cKWQ5H1elKkojqZc4g+ICzVOcAZsOnsqmZScVbL/APMw/7P85S5j09ZKlmSTI12Kvzwl5lsNAsIUeD5FlI6wffj5ndWV2Ev8+q4MVHon7SYQvtZ/DzWN/Bh7Xn+nisWkP4ZrAEF9q/yGCEJZUZtsX2n5EqMPybLXUUFFWpP/zYkWFmVI2G9bv4EPv+k3J7gEGagr/+ZX1CAQMH7tsCx98x1ai0VTOMfzCr9f/4IyScyinPIYWk1UWLVWWzqiHOIPavGcALHP6AF9gHuZL/JgI3mIOHNtloKBALZRueZTma7HXe3rjAhgu3/Nb3rHnARIE6CCec5z/bD2j4HUThPgEFznePh+uSd1NSyC3fEhECs8xZiW4undzgTBTUdZ8qDBTyqZUSydwvFOXffrSmo6TFj8fefdvCARyjycCLTH/RQHV5IWVmwenxWSVRUsTizMo7j1LC7Rruu8mEvAXZQC76eJM+Qirlg6VmrIvaTH3qdHNBPMS+QVoJQGudyunxlrsOIZGWwpDsHKS94Hc81rVU/5cV4YGM/fLqVGm4cvGoMJMKZta87gqYcu2o3zLdmST7cmqtpF4qY4I2eMWClbCaCKvUhlNIM6gSu8Zpavcp8ttgHdSPTj5aeXwy9hx/OPo7WWNjZHkytH/4eaxP2OTnMQmTvIfnPYCZlGsG0I+exLdzSHI6uApW8g2TJtjKWVTTJikUlJW2LBex8umVk/Wt3+2nvhUoOgYLSarKFT9hVruF3hZpRj2F/f2RPdZmVs26Sr3XiQRJ2To551K7+NAV/k3/I+XzyqP8h6AI8Sybx5cN34ek3ao5DEmUyH+Y/vGomPa9ibnhShb6NQkzERkiYjcLSLb3Z/dPuNSIvKwe9tUyzGVxvHtn61nOlGYMptIWnz5e6+rqyjbsH4H0UjCt0dlNrV6srZsO4rrf3AGqZR3OvBsiE6lOVAbVgVzIM7qIdCAHIF23fh5TJnCr7xpE+CakUvZdPDUkiKoEn7F0UWy2XLpo6tQhJWYQ/rc7nrp1fzj7reT9Dg3cPJ++ya7+dzTb+cX+wsT/9NUIshUlM0utXrMrgXuMcasBe5xH3sxaYw5yb1dUOMxlQaQLicRDBps203yNzA8FuF//X9n+q7g/O4/3cLmr93Id//pFjas31H2sa5+51Y6Z3HlZz5bth2FWN4qUCyjomzhojasGmoQZ40QaG+d/CMfif6CEDYpM2O/BuwWPjr69sJ6aMAFw49zb/LL7LA/yb3JL3PB8ONli6YLzMNcwh/L+oKdMCGuGzmvjJH+3sDbR07G8pGBBjj//n/wFWVz4iUDFWUVUGuO2YXABvf+d4EtwMdq3KfSZBTULJPSKx5radHkVyzWGBgZj4CB9rapuhd+9cs1W0i5ZUoBasOqpYZG5+XmncGMJ6dUQVq/HLQ3dzzIZ1ffSsyaWbk4aYf4x91vn1mhmJeXdkH4wZyWT2sCQ3y+/UcwyoyIKyLOrukuXCkJjg0bNC0YDN0y6Vn8NZtK+ljuSXSzOjxYsH1v3NMJXFF+Vs3J/SrKKqJWYdZrjEm/43uBXp9xURHZBiSBLxpjfuK3QxF5H/A+gKilX4jNQL0q8ZfbCH25XxNx4B3XvrusOVfTa7PcjgfNxETKu2GxUjZ1tWGLzn7VKM6AWRNo4Ii0q3s354gyKCwdkS+APrL2Fzl9OAFaJME1rXf4iihwBN01rXew2mehgQHWD3y2YLuXAHtzx4NcvXYzK0OD7El0c/2+jUVrkP0HG/nUMbcSyxKEk6kQ33huJq+s0mT52RZkcXuckNQeNl5olBRmIvJLYIXHU5/MfmCMMSLilxF0mDFmt4i8DPiViDxmjHnWa6Ax5gbgBoDO4LIyMoyU2aaelfjLaYTuF74s13NVrbduPrViGkjsYUf8IYaTZSyxWuTMpQ1blParBnEGlXnPoHyBBo5oWXlcoRcJcktHlPvcKmvI14v15o4H+eySHxWIwGz2JLrL8oLle/lWhwf57OpbAfjVDm9hmA5VXvWyzayIDrI33s03ntvIL/afPPeCDIqKsil7gufij7Br6mmOjp1S+7EWGCX/so0xb/J7TkT2ichKY8weEVkJeH5LGGN2uz+fE5EtwCsBT2GmNB/VhPiqeU2xfpfTCSEaSbD5azeWFEy1NFSvtuTGXGCMYSC5h2fjf2QwuYewxDg69mqenry/0VNratSGzQHpL+E58p5BrvenmEjbG+9mVaz8EB/4hwX3JPxf4+WZy8Y2sCo0yN1r/9nX+1VsXzErwd8v3ewrzMARZ2mBln5/2kq0nco5xiwLsrg9zvPxR9k19TQGm1XhtSwPHcrTk7+r/bgLiFqT/zcBl7n3LwN+mj9ARLpFHF+liCwFTgOerPG4yhziVU6iVIivmtf4edOMARGhs20KS2Y8YH6LCarx1jUzxhj2T+/kD6M/Y9vYZiZSw6yLvYYzO9/BYdGXN3p68x21YfWkxlyi9OKASgVCOoHdyzP0jec2MpnKLSeRDvGlFw7k367ft7GgBMWkHeL6ff7lJvy8bMY4oswSp0B22vv15o4HM2Pyj++3rxVRfy8fFH8filHTSss0O/f4/v7HU8M8Mb6Ve4d/yEtTT7IyfCSnd7yNl7eeSSxQvhhfLNSaY/ZF4FYRuQJ4EXg7gIisB95vjLkSOBb4LxGxcYTgF40xatTmEdWE+Kp5jZ+XzbaFUDA3ObeYB2yhJPHbJsWe6Wd5If4YY/YgUauNY2N/xurI0QREa0PXCbVh9aZG71maarxoUJhH9Qv8Q3x+/GrHyYRHCl/zq/0nE/N5zd6XeXvmUsYiaOXar1Ler3K9fLUUWK1btf4iYnw4eYAX4o+yN/ECFhZrIus4InKiirES1GTdjTH9wBs9tm8DrnTv/xY4oZbjKI2nmhBfpa/xS76PhL3Dm34esPmYxJ/NtB1n19TT7Jx6kikzQVugmxNaXseK8JFYojWh64nasFmkzgINKhdp4IiX+/a+gvsefQVQXm4a5IYFy+Ebz3kn30d9wpvFvF9e+4onQ9z4x3NqrnY/24LMGJv9iZ28OPU4g8m9BCXMEZETOCz6ciJWS32OvcDRy26lafDzsl1+/raKPGDzKYk/m9HUADvjT9I3vR2bFD3BVbw8eiY9wdVIqYJuitKs1EmgQfVetGz8hE25gs0Pv+T7q162uaIct7SQ/PJwive94hcsbxli/0QXNzxyLve8+KqK51X3PpY+gixhT7F7+k/snHqSSXuUqNXGutiprImsIyjh+s5hgaPCTGkq/LxslXrAmjmJPxvbpNifeJGdU08ymNyLRYCV4aM4LHo87YEljZ6eotSPWRBoaWoRamnK9UQVE3B+XrZqvF/3vPiqqoQYzFJTcR9BNpLs56Wpp9gzvYMUSboCvRzdegrLQ4eph79KVJgpTc989YAVYyI1wq7pZ9g99QzTJk7MauPo2KtZHV5H2Io2enqKMnvUUaClmQ2h5kelocR6er/8mBUhlsZDkCVNgr3Tz7Fr6mmGUwfcC8ojOTRyHB3BpbM3l0WCCjNlXjBfPGDFSJkk+6ZfYPf0Mwwk9wDCstAhHBI5lqXB1YheXSqLiVkQaGnmUqiVQy3er3xmVYSl8RBjxhiGUwfYPfUMe6afI0WCVquLY2KvYVV4LSFLC8XWCxVmijKLGGMYSu1j99R29k0/R5IEMaudo6InszpyNFGrtdFTVJTGki0CZkGkgb+YabRgK8acCLBsfEKVk/YYe6Z20De9nXF7mABBesNHsCayjq5Ar+a/zgIqzBRlFhhLDbJn+ln2TO9g0h7LGLNV4bUsCa5UY6YoXsyBSMumXPFTTwE354KrGEUS+fclnmfP9LOudx+6gr0cHz2RFeEjNJl/llFhpih1YiI1wt7p59iTeJax1CAg9ARXcVT0ZJaHDycooZL7mI+IyE3AW4D9xpiXu9uWAD8EDgdeAN5ujBkUR5H+O7ARmADea4x5yH3NZcCn3N1+zhjzXXf7ycB3gBiwGfg7Y8ziaHe0mJljkVaMphJTteIjxpJmmv3TO9mbeI6DiV0YbFqsDo6MvopV4aNoCXTM8UTnjmazYSrMFKUGxlKD7J9+kb2J5xlN9QPQFVjOMbHXsiJ8xGKp2/Md4OvAzVnbrgXuMcZ8UUSudR9/DDgPWOveTgW+CZzqGsFPA+txej0/KCKbjDGD7pi/An6PY9TOBe6Yg/NSmoV8MdFgoTavKFIANmFPcSCxk32JFziY2IVNioi0cGjkOFaGj6QjsHSxePe/QxPZMBVmilIB6QTY/YkX2T/9AuP2MACdgeWsi51Cb/hlxKz51WGgVowx94rI4XmbLwQ2uPe/C2zBMWoXAje7V4v3i0iX26NyA3C3MWYAQETuBs51+1J2GGPud7ffDFyECrPFjQo1b8poiTVpj3EgsZP90y8ykNyDwSYiLayJHMOK8BGLMm+s2WyYCjNFKUHSTNOf2M2BxEscSLzEtJlEELqDKzkkchy94cPnXRK/NW1XEp5ZKiLbsh7fYIy5ocRreo0x6W+JvUCve3818FLWuF3utmLbd3lsV5QZvATJQhdrZfYlNcZmOHXQtV87M579FquTwyLH0xs+gs7AsnknxiqwYdXYL2igDVNhpih5GGMYTQ1wMLmLg4ldDCX3YjAEJczS4BqWhQ9lWfCQxbQ8/KAxpup+VsYYIyKaE6bMLcWEy3wRbVU2hZ+yJ+hP7HZt2G4SJg4IXcHlrI29muWhw2i1OuedGKuSmuwXzL0NU2GmKDju/YFEH/3J3fQndjNt4gC0BZZwWOQEloUOoSvYq5Wsy2efiKw0xuxx3fz73e27gUOyxq1xt+1mJmyQ3r7F3b7GY7yiVE+5gme2BFyVgsuPpEkwmNzLQKKPg8ndjKUGAAhLlKWh1SwLHUpPcLUWr66MhtkwFWbKomTSHmMwsYeB5F4Gkn1M2o5LPCxRekKr6Qmupie0et6FKJuITcBlwBfdnz/N2v4BEbkFJ3F22DV8dwKfF5F0A8GzgY8bYwZEZEREXoOTOPse4GtzeSLVYlAn4bynzgKqXiRNgqHkvowYG04dwGAQLLqCvayNvZqe4Go6Aj2LxSs2GzTMhjW1MBtJ9fPQ2J10B1fQFeylI7CUgDT1lJUmxBjDmD3oGjLHmMVtpyl6UMJ0B1dwWOR4loRW0WZ1qyGrEBH5Ac6V4lIR2YWzMumLwK0icgXwIvB2d/hmnGXmO3CWml8O4BqvfwYecMd9Np1EC/wtM0vN72CeJP6Ppvq5f+SnGfvVFewlYsUaPS1lHjJlTzCY3JcRY6OpfleICR2BpRwePZElwZV0B1fod2QVNJsNa+rfYEjCjKeGOZBw8ukEi45AD13BXjqDy+gMLCNmtesXqZLDlD3BcPIgw6kDDCf3MZw8QBKngXBYYnQHV3B45OV0B1fSHliifz81Yox5p89Tb/QYa4CrfPZzE3CTx/ZtwMtrmWMjCEsMEYsXp57ghanHAIhZHXQFl9EV6KUjuJSOQA+WBBo8U6WZSJkko6l+hpMHGEruZzh1IOPRtwjQGVzGEdET6Q6upCvYu2DrI84lzWbDmlqYxax2XttxEVP2JEPu1cJw6gAvTT3Fi1OPAxCSKB2BpXQGl9IRcAxd1GrTL9tFwpQ9wWhqgBFXiI2k+jPeMBDaA0tYGT6SzmAv3cFeFfLKnBG1Wjm1/Xxsk2I4dTBjw/oTfeyZfhZwLjbbAz10BnroCC6jI9BDW6BbxdoiwRFhA64QO8hI6iBjqYFMGDwiLXQFezk0cixdwRUq5BcJTS3M0kSsGL3hw+kNHw6AbWzGUgOuR+QAI6mDPB9/JPPHHJQI7YEltAeWZAxda6BLryzmMbZJMW4PM5YaZCTZz1hqgNHUAFNmIjOmxeqgK7CczshxdAaW0x7s0d+50nAsCdDtXhiAE1qPm3GGkwcYTu5nJHWQPYnneGn6acARa22BLteG9dAeWEJboNv1wOlFxXwk/TsfSw0ymhpw7FdygHF7KPO9FZIIHYEeDo+eSGdgGZ3BZZrjukiZF8IsH0ssJwwQXMohkWOB3CuPkeRBRlMD7J56hp0kM6+LWe2OSLO6aAt00RrootXqXExlD5qepEkwkRrOiLDxlPNzwh7OGDDBojXQxZLQKjoCPXQEemgP9OjvUZkXiAgxaSMWbmNF+AjA+eKesEcy9mskNUB/oo++6R2Z14UkUmi/Al1EpVUFW5NgG5u4Pca4PcRYaohx9zaWGsykUwBErTbaA90sDx/m2q+lxDTSo7jUJMxE5G3AZ4BjgVPcOKrXuHNxeksFgBuNMV+s5bheBCRIV3A5XcHl4H4/zxi7AcZTQ+6VymCmD1iakERptTppCXTQYnUQC7TTYnXQYrUTkqh+WOqIMYakmWbSHmXCHmHCHmUyNcK4PcJEajjHAwZCi9VOa6CL3vBhtAa6abcc76e685V60Cw2TERoDXTSGuhkRfhlme3T9mTGbo2lhhhLDbA38TzJ6anMGIuAa7s6Hbvl2q+Y1UHUatUSL3XGNikm7THHfqVGZmxZapgJezTnuyUsUVoDXayMHEVboJs2q5v2wBK9iFSKUqvH7HHg/wH+y2+AiASAbwBn4VS8fcDtH/VkjccuSbaxy8Y2NpP2qHM1Yw9nPDT9id30me05YwMEiVqtxKx2YlYbUauNiNVC1GolYrUStVoISni2T2XekDQJpuwJ4vY4U/Y4cePcj9ujTKbGiNtjOVeO4BivFquDntBqWqwO1xPQSavVqQJMmW2a2oaFrRg9llO6JY0xhmkzmWW/Rhi3hxhPDXIgsTNHGIAQlRZigbT9cu2WtLr3WzREmoVtbKbN5Iz9sieI22PE7XEmbcd+5V48usLYtVvLQ4fR4n7ntFpdWjdMqYqahJkx5img1If6FGCHMeY5d+wtOL2mZt2o+WGJ5SnYwAmJTtqjmSuhSXvM/UCOMpw4QMJMFbwmQJCw1UJEooStGBGJEbZihCRK2IoSkkjOLSjheWEI096thJkmYaZImLjz055i2sSZNpNM25NMuT+n7ckC0QVOSYqo1UYs0M6S0EpiVpsrdJ2rexW2SqOYjzZMRIhICxGrhSWsynnOGJu4mciyX6PEXRs2kNzLlD1eUF9NEMISI2LFCLu2y3kcJSSu/bJybdh8uGAyxmCTcm2XY7fSdmzaTJGw467tmmDKxB0bZiYL9iNYjv2y2lgaWpO5n46wqLBV6s1c5Jh59Y86dQ6OWxUBCTou50C35/Mpk3SvplyvkBlnyp5kyp5g2sSZSI0waPZ6CrhsgoQISpiAhAhKiICECEiQAEHnpwSxCGR+ilhYWIhYCBaS9c9BwDW4acNrY2OwMcb5aRsbmxQpk8ImScqkSJkkKRLYJknSJEmZBEkSJM00STNd/Bwk4orRFjoCPYRDzpdFRByPYtRqIWK1agK+Mt+ZNzZMxHLy16w2z+eNsZkykxn75diwCecCyxUmY4lBpsxknuctF4sAQQkTlLQdCxIglLFd+fbLEgshkGXD0rZrxoJl5uj+7/xz7JbBdv6ZFDYpbNd22Th2yzZJkiRImSRJkyDlXlAWO4cAQUJWlIjEiFltdAWWEbZaHLslLZnoSFjTWZQ5pqQwE5FfAis8nvqkMeanHttrQkTeB7wPnATJZiMgQV9vWza2sQu8TOkrt7QXyhE/CVLuLWHijlAySVIkM0ao3jiGMpARgZYECRIkKCEiVizH4OZ7+2aunKOau6LMC+bShjW7/RKxMmHMTpb5jjPGkCRBwo7PeJyy7Fgyy34lTYIUCabNJCk7QQpXNJkkKVJQ5w4IgmARJCBp4RfMXNhG3QvB9IVv0IoQ8rRjUb1gVJqWksLMGPOmGo/h11fK73g3ADcAdAaXzdueJpZYRCRGhNoqfRvjXjWSylw5GozrBTOebV8yvrTMlamFJZbjecPSqz9lUTGXNmyh2C8RIUSYUKD2NIO0p94Ye8aLj8nYtoJjp71pkrFerg2b8bgpykJmLkKZDwBrReQIHGN2KfDnc3DcBYGIuCGAAAU+f0VR5gK1YTVguWkYar8UpTxquvQQkYvdvlKvBW53m3giIqtEZDOAMSYJfAC4E3gKuNUY80Rt01YURakdtWGKojQbta7KvA24zWN7H06Tz/TjzTiNPxVFUZoGtWGKojQbGqxXFEVRFEVpElSYKYqiKIqiNAnzslemoig1Mp2AnXsaPQtFUZTqWMA2TD1miqIoiqIoTYIKM0VRFEVRlCZBhZmiKIqiKEqToMJMURRFURSlSVBhpiiKoiiK0iSoMFMURVEURWkSVJgpiqIoiqI0CSrMFEVRFEVRmgQVZoqiKIqiKE2CCjNFURRFUZQmQYWZoiiKoihKk6DCTFEURVEUpUlQYaYoiqIoitIkqDBTFEVRFEVpElSYKYqiKIqiNAkqzBRFURRFUZqEmoSZiLxNRJ4QEVtE1hcZ94KIPCYiD4vItlqOqShK8yEi54rIMyKyQ0SubfR8ykVtmKIozWa/gjW+/nHg/wH+q4yxrzfGHKzxeIqiNBkiEgC+AZwF7AIeEJFNxpgnGzuzslAbpiiLmGa0XzUJM2PMUwAiUp/ZKIoyHzkF2GGMeQ5ARG4BLgSaXpipDVOURU/T2a9aPWblYoC7RMQA/2WMucFvoIi8D3if+3DszsEbn5nluS0FFspV8EI5l4VyHjA353JYpS8YSR28887BG5eWOTyaF767Ie8zvBp4KevxLuDUSufU5JRlw9R+1YSeS/MxV+cxmzZs3tmvksJMRH4JrPB46pPGmJ+WeZzTjTG7RWQ5cLeIPG2MuddroPuG+Qq3eiMi24wxvrkl84mFci4L5Tygec/FGHNuo+cwV8ylDVP7VT16Ls1HM5/HQrZhJYWZMeZNtR7EGLPb/blfRG7DcR16CjNFUeYdu4FDsh6vcbc1BWrDFEUpQtPZr1kvlyEirSLSnr4PnI2TcKsoysLgAWCtiBwhImHgUmBTg+dUN9SGKcqCpunsV63lMi4WkV3Aa4HbReROd/sqEdnsDusF7hORR4A/ALcbY35Ry3HrzJyFHeaAhXIuC+U8YGGdiyfGmCTwAeBO4CngVmPME42dVXksABu2kP6+9Fyaj4VyHr40o/0SY0wjj68oiqIoiqK4aOV/RVEURVGUJkGFmaIoiqIoSpOgwozy27I0K83WTqJaROQmEdkvIvM+sVpEDhGRX4vIk+7f1t81ek7KwmS+2y9QG9ZsqP1qLCrMHNJtWebd8vesdhLnAccB7xSR4xo7q6r5DrBQatMkgQ8bY44DXgNcNY9/L0pzM2/tF6gNa1LUfjUQFWY4bVmMMbNdoXu2yLSTMMZMA+l2EvMOt2DnQKPnUQ+MMXuMMQ+590dxVvusbuyslIXIPLdfoDas6VD71VhUmM1/vNpJ6AeoiRCRw4FXAr9v8FQUpRlRG9bEqP2ae+aqV2bDqVNbFkWpCBFpA34MXG2MGWn0fJT5idovpRGo/WoMi0aY1aMtS5PSdO0kFAcRCeEYte8bY/5vo+ejzF8WsP0CtWFNidqvxqGhzPlP07WTUEBEBPgW8JQx5iuNno+iNDFqw5oMtV+NRYUZ/m1Z5gPN2E6iWkTkB8DvgHUisktErmj0nGrgNODdwBtE5GH3trHRk1IWHvPZfoHasCZF7VcD0ZZMiqIoiqIoTYJ6zBRFURRFUZoEFWaKoiiKoihNggozRVEURVGUJkGFmaIoiqIoSpOgwkxRFEVRFKVJUGGmKIqiKIrSJKgwUxRFURRFaRL+f/u8AZIarEb/AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 720x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "p_c0_given_x, p_c1_given_x = compute_posterior(flat_plt_grid, p_c0, mu_c0, sigma_c0, p_c1, mu_c1, sigma_c1)\n",
    "p_c0_given_x = np.reshape(p_c0_given_x, plt_grid_shape)\n",
    "p_c1_given_x = np.reshape(p_c1_given_x, plt_grid_shape)\n",
    "\n",
    "plt.figure(figsize=(10, 4))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=10)\n",
    "plt.colorbar()\n",
    "# plot decision boundary \n",
    "plt.contour(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=[0.0, 0.5], colors=[\"k\", \"k\"])\n",
    "\n",
    "plt.title(\"p($c_0$ | x)\")\n",
    "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n",
    "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n",
    "plt.legend([s0, s1], [\"c0\", \"c1\"])\n",
    "plt.xlim(-1.5, 2.5)\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], p_c1_given_x, levels=10)\n",
    "plt.colorbar()\n",
    "# plot decision boundary \n",
    "plt.contour(plt_grid[..., 0], plt_grid[..., 1], p_c0_given_x, levels=[0.0, 0.5], colors=[\"k\", \"k\"])\n",
    "plt.title(\"p($c_1$ | x)\")\n",
    "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n",
    "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n",
    "plt.legend([s0, s1], [\"c0\", \"c1\"])\n",
    "\n",
    "plt.xlim(-1.5, 2.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The color indicates the posterior likelihood for the respective call and the black line indicates the decision boundary. \n",
    "We achieve a train accuracy of 87%.\n",
    "For such a simple task that is clearly not great, but it nicely illustrates a\n",
    "problem with generative approaches:\n",
    "They usually depend on quite a lot of assumptions.\n",
    "\n",
    "### 2.3) Wrong Assumptions? (1 Point):\n",
    "Which untrue assumption did we make?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "data is independent and identically distributed\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Stochastic and Batch Gradients\n",
    "\n",
    "In the recap sessions with Prof. Neumann we already saw (or will see) an implementation of a Discriminative Classifier using Logistic Regression. Here we are going to extend this to stochastic and batch gradient descent.  \n",
    "\n",
    "We start by implementing a few helper functions for affine mappings, the sigmoid function, and the negative Bernoulli log-likelihood. - Those are the same as used for the full gradient case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "def affine_features(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    implements affine feature function\n",
    "    :param x: inputs, shape: [N x sample_dim]\n",
    "    :return inputs with additional bias dimension, shape: [N x feature_dim]\n",
    "    \"\"\"\n",
    "    return np.concatenate([x, np.ones((x.shape[0], 1))], axis=-1)\n",
    "\n",
    "def quad_features(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    implements quadratic feature function\n",
    "    :param x: inputs, shape: [N x sample_dim]\n",
    "    :return squared features of x, shape: [N x feature_dim]\n",
    "    \"\"\"\n",
    "    sq = np.stack([x[:, 0] ** 2, x[:, 1]**2, x[:, 0] * x[:, 1]], axis=-1)\n",
    "    return np.concatenate([sq, affine_features(x)], axis=-1)\n",
    "\n",
    "def cubic_features(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    implements cubic feature function\n",
    "    :param x: inputs, shape: [N x sample_dim]\n",
    "    :return cubic features of x, shape: [N x feature_dim]\n",
    "    \"\"\"\n",
    "    cubic = np.stack([x[:, 0]**3, x[:, 0]**2 * x[:, 1], x[:, 0] * x[:, 1]**2, x[:, 1]**3], axis=-1)\n",
    "    return np.concatenate([cubic, quad_features(x)], axis=-1)\n",
    "\n",
    "def sigmoid(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    the sigmoid function\n",
    "    :param x: inputs \n",
    "    :return sigma(x)\n",
    "    \"\"\"\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def bernoulli_nll(predictions: np.ndarray, labels: np.ndarray, epsilon: float = 1e-12) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    :param predictions: output of the classifier, shape: [N]\n",
    "    :param labels: true labels of the samples, shape: [N]\n",
    "    :param epsilon: small offset to avoid numerical instabilities (i.e log(0))\n",
    "    :return negative log-likelihood of the labels given the predictions\n",
    "    \"\"\"\n",
    "    return - (labels * np.log(predictions + epsilon) + (1 - labels) * np.log(1 - predictions + epsilon))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are also using the same bernoulli objective and its gradient as before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def objective_bern(weights: np.ndarray, features: np.ndarray, labels: np.ndarray) -> float:\n",
    "    \"\"\"\n",
    "    bernoulli log-likelihood objective \n",
    "    :param weights: current weights to evaluate, shape: [feature_dim]\n",
    "    :param features: train samples, shape: [N x feature_dim]\n",
    "    :param labels: class labels corresponding to train samples, shape: [N]\n",
    "    :return average negative log-likelihood \n",
    "    \"\"\"\n",
    "    predictions = sigmoid(features @ weights)\n",
    "    return np.mean(bernoulli_nll(predictions, labels))\n",
    "\n",
    "def d_objective_bern(weights: np.ndarray, features: np.ndarray, labels: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    gradient of the bernoulli log-likelihood objective\n",
    "    :param weights: current weights to evaluate, shape: [feature_dim]\n",
    "    :param features: train samples, shape: [N x feature_dim]\n",
    "    :param labels: class labels corresponding to train samples, shape [N]\n",
    "    \"\"\"\n",
    "    res = np.expand_dims(sigmoid(features @ weights) - labels, -1)\n",
    "    grad = features.T @ res / res.shape[0]\n",
    "    return np.squeeze(grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1) Implementation (3 Points)\n",
    "\n",
    "Finally, we can implement our batch gradient descent optimizer. When setting the batch_size to 1 it will become a stochastic gradient descent optimizer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def minimize_with_sgd(features: np.ndarray, labels: np.ndarray, initial_weights: np.ndarray, schedule: Callable,\n",
    "                      num_iterations: int, batch_size: int):\n",
    "    \"\"\"\n",
    "    :param features: all samples, shape: [N x feature_dim] \n",
    "    :param labels: all labels, shape: [N]\n",
    "    :param initial_weights: initial weights of the classifier, shape: [feature_dim * K]\n",
    "    :param schedule: learning rate schedule (a callable function returning the learning rate, given the iteration\n",
    "    :param num_iterations: number of times to loop over the whole dataset\n",
    "    :param batch_size: size of each batch, should be between 1 and size of data\n",
    "    return \"argmin\", \"min\", logging info\n",
    "    \"\"\"\n",
    "\n",
    "    assert 1 <= batch_size <= features.shape[0]\n",
    "    # This is a somewhat simplifying assumption but for the exercise its ok\n",
    "    assert features.shape[0] % batch_size == 0, \"Batch Size does not evenly divide number of samples\"\n",
    "    batches_per_iter = int(features.shape[0] / batch_size)\n",
    "\n",
    "    # setup\n",
    "    weights = np.zeros([batches_per_iter * num_iterations + 1, initial_weights.shape[0]])\n",
    "    loss = np.zeros(batches_per_iter * num_iterations + 1)\n",
    "    weights[0] = initial_weights\n",
    "    loss[0]= objective_bern(weights[0], features, labels)\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "     #--------------------------------------------------\n",
    "        # TODO: shuffle data\n",
    "        #--------------------------------------------------\n",
    "        for j in range(batches_per_iter):\n",
    "            global_idx = i * batches_per_iter + j\n",
    "\n",
    "             #--------------------------------------------------\n",
    "            # TODO: do stochastic gradient descent update!\n",
    "            #--------------------------------------------------\n",
    "\n",
    "\n",
    "            # log loss (on all samples, usually you should not use all samples to evaluate after each stochastic\n",
    "            # update step)\n",
    "            loss[global_idx + 1] = objective_bern(weights[global_idx + 1], features, labels)\n",
    "    return weights[-1], loss[-1], (weights, loss)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss curve is expected to look a bit jerky due to the stochastic nature of stochastic gradient descent.\n",
    "If it goes down asymptotically its fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final loss 0.6931471805579453\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-73-b9a1220137c9>:31: UserWarning: No contour levels were found within the data range.\n",
      "  plt.contour(plt_grid[..., 0], plt_grid[..., 1], pred_grid, levels=[0, 0.5], colors=[\"k\"])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f9db4e72d30>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAc7klEQVR4nO3deZQc1Xn38e9PEpslEEgyilnFFggYswnMomCJGIMMYjdYBmMwCEjAkEOMjQyxhfMmfg3GCQRIEDYvmyPZZjPCGIyDBSGsEmFTWAQCmUUWIMEICRACnvePumNaw0xNzTDVXdP9+5zTp6tuV3U/t3uOHt17q+5VRGBmZtaVAY0OwMzMqs2JwszMcjlRmJlZLicKMzPL5URhZma5nCjMzCyXE4WZmeVyorDKkfQVSbMkLZW0QNJvJI1pYDzPS3o7xdP+uKjguTMlHV92jEVIOkbS3Y2Ow/qfQY0OwKyWpNOBM4GTgNuAd4F9gQOBj/wjJ2lQRLxXh9AmRMTv+vpN6xi/Wa+5RWGVIWko8H3g5Ii4PiKWRcSKiJgREWekY6ZIulbSNZKWAMdIWk/STZIWS3pG0qSa99wltU6WSFoo6cepfPX0HoskvSHpQUkjexHzMZLulvQjSa9Lek7S+PTaPwJ/CVxU2wqRFJJOljQXmJvKJqXYF6e6rFfzGSHpVEnzJL0m6TxJAyStmo7ftubYdSW9JemTPazH7uk7aEvPu3eo4zxJb6b6HZnKN5d0ZzrnNUk/7+n3Z/1ERPjhRyUeZC2H94BBOcdMAVYAB5H9R2cN4C7gEmB1YHvgVWCvdPy9wFfT9hBg17R9IjAD+AQwENgJWKuLz3we+HwXrx2T4pmU3uevgZcBpddnAsd3OCeA24FhKf69gNeAHYHVgH8F7upw/O/T8RsBT7e/Z6r3D2uOPQ2YkRPr3Z2UDwNeB75K1sswMe0PBwYDS4At07GfArZJ29OAs9LvsDowptF/Q36U83CLwqpkOPBadN8Vc29E3BgRHwAjgD2Ab0fEOxHxMPAT4Oh07Apgc0kjImJpRNxXUz4c2Dwi3o+I2RGxJOczb0wtj/bHpJrX5kfEZRHxPnAl2T+m3bVOfhARiyPibeBI4PKIeCgilgOTgd0kjao5/ofp+D8A/0L2jznp8yZKUtr/KnB1N5/d0X7A3Ii4OiLei4hpwJPAhPT6B8CnJa0REQsiYk4qXwFsDKyXvnuPfzQpJwqrkkXACEndjZ29ULO9HrA4It6sKZsPrJ+2jwP+HHgydansn8qvJhsDmS7pZUnnSlol5zMPioi1ax6X1bz2x/aNiHgrbQ7pYR3m17zHUrLvYv0ujp+fziEi7gfeAsZK2grYHLipm8/uaKXPr/mM9SNiGXAE2ZjRAkm/Tp8D8C1AwAOS5kj6eg8/1/oJJwqrknuB5WTdSnlqpzx+GRgmac2aso2AlwAiYm5ETATWBX4IXCtpcGRjH+dExNbA7sD+fNgK6UtdTc/csQ4bt+9IGkzW2nmp5pgNa7Y3Sue0uxI4iqw1cW1EvNPDGFf6/JrPaP8Ob4uIvclaSk8Cl6XyP0bEpIhYj6wr7xJJm/fws60fcKKwyoiINuC7wMWSDpL0CUmrSBov6dwuznkBuAf4QRqg/gxZK+IaAElHSfpk6qZ6I532gaRxkraVNJCsD34FWRdLX1sIbNrNMdOAYyVtL2k14J+A+yPi+ZpjzpC0jqQNycYhageOrwEOJksWV3XzWUrf058ewC3Anyu7LHmQpCOArYGbJY2UdGBKXsuBpaTvSdKXJG2Q3vd1suRXxndoDeZEYZUSEecDpwNnkw1KvwCcAtyYc9pEYBTZ/4xvAL4XH17Kui8wR9JS4ALgy2lc4M+Aa8mSxBPAneT37c/QyvdR3FCwShcAh6Uroi7s7IAU698D1wELgM2AL3c47FfAbOBh4NfAT2vOfwF4iOwf6v/qJp7dgbc7PNrIWlR/R9bl9S1g/4h4jezfiNPJvtvFwOfIBuwBdgbuT9/tTcBpETGvm8+3fqj9ygwzqyhJAWwREc/kHHM58HJEnF2/yKxV+IY7s34uXR11CLBDg0OxJuWuJ7N+TNI/AI8D50XEc42Ox5qTu57MzCyXWxRmZparKccoRowYEaNGjerVucuWLWPw4MF9G1DFuc6todXq3Gr1hY9X59mzZ78WEZ3OEdaUiWLUqFHMmjWrV+fOnDmTsWPH9m1AFec6t4ZWq3Or1Rc+Xp0ldbw7/0/c9WRmZrmcKMzMLJcThZmZ5XKiMDOzXE4UZmaWq/JXPaVZKy8hWzt5ZkT8rMEhmZm1lIa0KCRdLukVSY93KN9X0lNp7eAzU/EhZHPsTwIOqHuwZmYtrlFdT1eQTf/8J2ldgIuB8WRz4U+UtDWwAR+u7vV+HWM0MzMaONdTmvHy5oj4dNrfDZgSEfuk/cnp0BeB1yPiZknTI6LjPP3t73cCcALAyJEjd5o+fXqv4lq6dClDhnS3imVzcZ1bQ6vVudXqCx+vzuPGjZsdEaM7e61KYxTrs/K6wC8CnwUuBC6StB8wo6uTI2IqMBVg9OjR0du7E303Z2twnZtfq9UXyqtzlRJFp9Li7sc2Og4zs1ZVpctjX2LlBeQ3YOXF5c3MrAGqlCgeBLaQtImkVcnWDL6pwTGZmbW8Rl0eOw24F9hS0ouSjouI94BTgNvIFrv/RUTMaUR8Zmb2oYaMUUTExC7KbwFuqXM4ZmaWo0pdT2ZmVkFOFGZmlqupEoWkCZKmtrW1NToUM7Om0VSJIiJmRMQJQ4cObXQoZmZNo6kShZmZ9T0nCjMzy+VEYWZmuZwozMwslxOFmZnlcqIwM7NcThRmZpary7meJB2Sd2JEXN/34ZiZWdXkTQo4IT2vC+wO3JH2xwH3AJVLFJImABM233zzRodiZtY0uux6iohjI+JYYBVg64g4NCIOBbZJZZXjO7PNzPpekTGKDSNiQc3+QmCjkuIxM7OKKbIexX9Kug2YlvaPAH5XXkhmZlYl3SaKiDhF0sHAnqloakTcUG5YZmZWFUVXuLsHeA8I4IHywjEzs6rpdoxC0uFkyeEw4HDgfkmHlR2YmZlVQ5EWxVnAzhHxCoCkT5KNUVxbZmBmZlYNRa56GtCeJJJFBc8zM7MmUKRFcWsnVz3dUl5IZmZWJUWuejojTecxJhX5qiczsxZS9Kqn/wZW0ORXPb397vssfy946933Gh1KXbnOraHV6txq9YWszhGBpD59X0VE/gHZVU/nATMBAX8JnBERlRvMrpnradLcuXN7fP5fnT+TZ19d1veBmZnVydP/ZzyrDur5MLKk2RExurPXmuqqp4iYAcwYPXr0pN6cf8KemzL78SfZbNPN+jiyant23rOucwtotTq3Wn0hq/PAAX3bmoBiiaJlrno6YueNGLlsHmM/11p/XDPjBde5BbRanVutvpDVuVGJwlc9mZm1sKJXPR0K7JGKfNWTmVkLKXTVU0RcB1xXcixmZlZBReZ6OkTSXEltkpZIelPSknoEZ2ZmjVekRXEuMCEinig7GDMzq54iVy8tdJIwM2tdXbYo0rQdALMk/Ry4EVje/npEXF9uaGZmVgV5XU8TarbfAr5Qsx+AE4WZWQvoMlFExLH1DMTMzKopr+vpWxFxrqR/JWtBrCQiTi01MjMzq4S8rqf2AexZ9QikL9RMCtjoUMzMmkZe19OM9Hxl/cL5eD7upIBmZvZReV1PM+iky6ldRBxQSkRmZlYpeV1PP6pbFGZmVll5XU93tm9LWgPYKCKeqktUZmZWGUXmepoAPAzcmva3l3RTyXGZmVlFFJnCYwqwC/AGQEQ8DGxSWkRmZlYpRRLFioho61CWv9C2mZk1jSKzx86R9BVgoKQtgFOBe8oNy8zMqqJIi+IbwDZkEwL+B7AEOK3MoMzMrDqKJIqJEXFWROycHmcB55QdmJmZVUORrqdDJb0TET8DkHQRsEa5YZmZWVUUShTATZI+APYF3oiI48oNy8zMqiJvCo9hNbvHky1c9N/AOZKGRcTikmMzM7MKyGtRzCa7DFY1z/ulRwCblh6dmZk1XN4UHr6pzszMcrue9oqIO2rWzl5JFdfM9noUZmZ9L6/r6XPAHay8dna7Sq6Z7fUozMz6Xl7X0/fS80fWzpZ0aJlBmZlZdRS54a4z/9ynUZiZWWX1NlGoT6MwM7PK6m2i8OyxZmYtIu+qp8foPCEIGFlaRGZmVil5Vz3tX7cozMyssvKueprfsUzS/hFxc7khmZlZlfR0jOL7pURhZmaV1dNE4audzMxaTE8TxYmlRGFmZpXV7XoUHed6krQB0AY8FhGvlBWYmZlVQ5GFi44DdgN+n/bHkk1Bvomk70fE1SXFZmZmFVAkUQwC/iIiFgJIGglcBXwWuAtwojAza2JFxig2bE8SySupbDGwopywzMysKoq0KGZKuhn4Zdo/LJUNBt4oKzAzM6uGIoniZOAQYEzavxK4LiICGFdWYGZmVg3dJoqICEl3A++Szf30QEoSZmbWArodo5B0OPAAWZfT4cD9kg4rO7DekDRB0tS2trZGh2Jm1jSKDGafBewcEV+LiKOBXYC/Lzes3omIGRFxwtChQxsdiplZ0yiSKAZ0uLFuUcHzzMysCRQZzL5V0m3AtLR/BPCb8kIyM7MqKTKYfUaaxqP9qqepEXFDuWGZmVlVFGlREBHXA9e370v6Q0RsVFpUZmZWGb0da/B042ZmLaK3icL3UZiZtYguu54knd7VS8CQcsIxM7OqyRujWDPntQv6OhAzM6umLhNFRJxTz0DMzKyafOOcmZnlcqIwM7NcThRmZpYr76qnoyLimq6ufoqIH5cXlpmZVUXeVU+D03Pe1U9mZtbk8q56ujQ9++onM7MWltf1dGHeiRFxat+HY2ZmVZPX9TS7blGYmVll5XU9XVnPQMzMrJryup5mkDP5X0QcUEpEZmZWKXldTz+qWxRmZlZZeV1Pd9YzEDMzq6ZuV7iT9ByddEFFxKalRGRmZpVSZCnU0TXbqwNfAoaVE46ZmVVNt3M9RcSimsdLEfEvwH7lh9ZzkiZImtrW1tboUMzMmkaRrqcda3YHkLUwirRE6i4iZgAzRo8ePanRsZiZNYsi/+CfX7P9HvA8cHgp0ZiZWeV0mygiYlw9AjEzs2rqdoxC0lBJP5Y0Kz3OlzS0HsGZmVnjFVm46HLgTbLupsOBJcD/KzMoMzOrjiJjFJtFxKE1++dIerikeMzMrGKKtCjeljSmfUfSHsDb5YVkZmZVUqRFcRJwVRqXELAYOKbMoMzMrDqKXPX0CLCdpLXS/pLSozIzs8oocsPdasChwChgkCQAIuL7pUZmZmaVUKTr6VdAG9mKd8vLDcfMzKqmSKLYICL2LT0SMzOrpCJXPd0jadvSIzEzs0oq0qIYAxyT1qVYTnblU0TEZ0qNzMzMKiE3USgbuT4JmF+fcMzMrGpyE0VEhKSLI8JdT2ZmLarIGMVDknYuPRIzM6ukImMUnwWOkvQ8sAyPUZiZtZQiiWKf0qMwM7PKKrJm9nxgQ2CvtP1WkfPMzKw5FFm46HvAt4HJqWgV4JoygzIzs+oo0jI4GDiAbHyCiHgZWLPMoMzMrDqKJIp3IyKAAJA0uNyQzMysSookil9IuhRYW9Ik4HfAZeWGZWZmVVFkPYofSdqbbK3sLYHvRsTtpUdmZmaVUOTyWFJiuF3SCGBRuSGZmVmVdNn1JGlXSTMlXS9pB0mPA48DCyV52nEzsxaR16K4CPgOMBS4AxgfEfdJ2gqYBtxah/jMzKzB8gazB0XEbyPil8AfI+I+gIh4sj6hmZlZFeQlig9qtt/u8FqUEIuZmVVQXtfTdpKWkE0CuEbaJu2vXnpkZmZWCV0miogYWM9AzMysmjy5n5mZ5WqqRCFpgqSpbW1tjQ7FzKxpNFWiiIgZEXHC0KFDGx2KmVnTKJQoJG0s6fNpew1Jnj3WzKxFFFmPYhJwLXBpKtoAuLHEmMzMrEKKtChOBvYgmxSQiJgLrFtmUGZmVh1FEsXyiHi3fUfSIHzDnZlZyyiSKO6U9B2ym+72Bn4JzCg3LDMzq4oiieJM4FXgMeBE4Bbg7DKDMjOz6iiyHsVBwFUR4VXtzMxaUJEWxQTgaUlXS9o/jVGYmVmL6DZRRMSxwOZkYxMTgWcl/aTswMzMrBqKLoW6QtJvyK52WoOsO+r4EuMyM7OKKHLD3XhJVwBzgUOBnwB/VnJcZmZWEUVaFEcDPwdOjIjlJcdjZmYV022iiIiJ9QjEzMyqqctEIenuiBgj6U1WvhNbQETEWqVHZ2ZmDZe3wt2Y9OyZYs3MWliRweyri5SZmVlzKnLD3Ta1O+mGu53KCcfMzKqmy0QhaXIan/iMpCXp8SawEPhV3SI0M7OG6jJRRMQP0vjEeRGxVnqsGRHDI2JyHWM0M7MGKnJ57GRJ6wBbAKvXlN9VZmBmZlYN3SYKSccDp5EtgfowsCtwL7BXqZGZmVklFBnMPg3YGZgfEeOAHYA3ygzKzMyqo0iieCci3gGQtFpEPAlsWW5YZmZWFUXmenpR0trAjcDtkl4H5pcZlJmZVUeRweyD0+YUSb8HhgK3lhqVmZlVRpHB7GE1u4+l5+jsWDMzaz5FxigeAl4FniZbk+JV4HlJD0nyHdpmZk2uSKK4HfhiRIyIiOHAeOBm4G+AS8oMzszMGq9Iotg1Im5r34mI3wK7RcR9wGqlRWZmZpVQ5KqnBZK+DUxP+0cACyUNBD4oLTIzM6uEIi2Kr5DdlX0jcAOwYSobCBxeWmRmZlYJRS6PfQ34hqTBEbGsw8vPlBOWmZlVRZGFi3aX9L/AE2l/O0kexDYzaxFFup7+GdgHWAQQEY8Ae5YZlJmZVUeRREFEvNCh6P0SYjEzswoqctXTC5J2B0LSKmSzyT5RblhmZlYVRVoUJwEnA+sDLwHbp30zM2sBRa96OrIOsZiZWQV1mSgkfTfnvIiIfyghHjMzq5i8FkXHeyYABgPHAcMBJwozsxbQZaKIiPPbtyWtSTaIfSzZVB7nd3WemZk1l9wxirQWxelkYxRXAjtGxOv1CMzMzKohb4ziPOAQYCqwbUQsrVtUZmZWGXmXx/4dsB5wNvCypCXp8aakJfUJz8zMGi1vjKLQXdtmZtbcnAzMzCyXE4WZmeVyojAzs1xOFGZmlsuJwszMcjlRmJlZLicKMzPL5URhZma5nCjMzCyXE4WZmeWqfKKQtKmkn0q6ttGxmJm1olIThaTLJb0i6fEO5ftKekrSM5LOzHuPiJgXEceVGaeZmXWt2zWzP6YrgIuAq9oLJA0ELgb2Bl4EHpR0EzAQ+EGH878eEa+UHKOZmeVQRJT7AdIo4OaI+HTa3w2YEhH7pP3JABHRMUl0fJ9rI+KwnNdPAE4AGDly5E7Tp0/vVbxLly5lyJAhvTq3v3KdW0Or1bnV6gsfr87jxo2bHRGjO3ut7BZFZ9YHXqjZfxH4bFcHSxoO/COwg6TJXSWUiJhKtsgSo0ePjrFjx/YquJkzZ9Lbc/sr17k1tFqdW62+UF6dG5EoeiQiFgEnNToOM7NW1Yirnl4CNqzZ3yCVmZlZBTUiUTwIbCFpE0mrAl8GbmpAHGZmVkDZl8dOA+4FtpT0oqTjIuI94BTgNuAJ4BcRMafMOMzMrPdKHaOIiIldlN8C3FLmZ5uZWd+o/J3ZZmbWWE4UZmaWq6kShaQJkqa2tbU1OhQzs6ZR+p3ZjSDpVWB+L08fAbzWh+H0B65za2i1OrdafeHj1XnjiPhkZy80ZaL4OCTN6uo29mblOreGVqtzq9UXyqtzU3U9mZlZ33OiMDOzXE4UHzW10QE0gOvcGlqtzq1WXyipzh6jMDOzXG5RmJlZLicKMzPL5USR9GQd7/5G0vOSHpP0sKRZqWyYpNslzU3P66RySbowfQ+PStqxsdEX09n67L2po6SvpePnSvpaI+pSVBd1niLppfRbPyzpizWvTU51fkrSPjXl/eZvX9KGkn4v6X8lzZF0Wipvyt86p771/Z0jouUfZOt1PwtsCqwKPAJs3ei4+rB+zwMjOpSdC5yZts8Efpi2vwj8BhCwK3B/o+MvWMc9gR2Bx3tbR2AYMC89r5O212l03XpY5ynANzs5duv0d70asEn6ex/Y3/72gU8BO6btNYGnU92a8rfOqW9df2e3KDK7AM9ExLyIeBeYDhzY4JjKdiBwZdq+EjiopvyqyNwHrC3pUw2Ir0ci4i5gcYfintZxH+D2iFgcEa8DtwP7lh58L3VR564cCEyPiOUR8RzwDNnffb/624+IBRHxUNp+k2ypgvVp0t86p75dKeV3dqLIdLaOd96P0d8E8FtJsyWdkMpGRsSCtP1HYGTabqbvoqd1bJa6n5K6WS5v74KhCessaRSwA3A/LfBbd6gv1PF3dqJoDWMiYkdgPHCypD1rX4yszdrU10m3Qh2TfwM2A7YHFgDnNzSakkgaAlwH/G1ELKl9rRl/607qW9ff2Yki09TreEfES+n5FeAGsmbowvYupfT8Sjq8mb6Lntax39c9IhZGxPsR8QFwGdlvDU1UZ0mrkP2j+bOIuD4VN+1v3Vl96/07O1FkmnYdb0mDJa3Zvg18AXicrH7tV3p8DfhV2r4JODpdLbIr0FbTpO9velrH24AvSFonNeW/kMr6jQ7jSQeT/daQ1fnLklaTtAmwBfAA/exvX5KAnwJPRMSPa15qyt+6q/rW/Xdu9Kh+VR5kV0c8TXZlwFmNjqcP67Up2RUOjwBz2usGDAf+E5gL/A4YlsoFXJy+h8eA0Y2uQ8F6TiNrgq8g6389rjd1BL5ONgD4DHBso+vVizpfner0aPqH4FM1x5+V6vwUML6mvN/87QNjyLqVHgUeTo8vNutvnVPfuv7OnsLDzMxyuevJzMxyOVGYmVkuJwozM8vlRGFmZrmcKMzMLJcThbUMSaNqZ1pNZVMkfbOH7/O8pBHdHPOdXsS3v6T/kfRImi30xFR+kKSte/p+Zn3FicKsHD1KFOnu26nAhIjYjmxOn5np5YPIZgU1awgnCrNE0kxJF6T5/R+XtEsqHy7pt2k9gJ+Q3cTVfs6NabLFOe0TLkr6v8Aa6X1+lsqOkvRAKrtU0sAOH78mMAhYBBDZ7J9PSdodOAA4L527WXrcmj73vyRtlT7jCkn/LmmWpKcl7Z/Kt6n57EclbVHqF2lNx4nCbGWfiIjtgb8BLk9l3wPujohtyObK2qjm+K9HxE7AaOBUScMj4kzg7YjYPiKOlPQXwBHAHum93weOrP3QiFhMdoftfEnTJB0paUBE3JPKz0jv9yxZy+Mb6XO/CVxS81ajyOb92Q/4d0mrAycBF6TPHk12F7dZYYMaHYBZHXU1DUFt+TTI1nqQtJaktckWCDoklf9a0us1x58q6eC0vSHZ3DqLOrz/XwE7AQ9mU/ewBh9OWvdhEBHHS9oW+DxZAtgbOKb2mDSL6O7AL9N7QbZITbtfRDZR3FxJ84CtgHuBsyRtAFwfEXO7+B7MOuVEYa1kEdlqZrWGAc/V7HdMJl3OcSNpLNk/6rtFxFuSZgKrd3YocGVETO4uwIh4DHhM0tUprmM6HDIAeCO1Djp9i4++ZfyHpPvJWhm3SDoxIu7oLhazdu56spYREUuBBZL2gmydZbJVze6uOeyI9NoYsplG24C7gK+k8vF8mGyGAq+nJLEV2VKb7VakAWrIJqs7TNK67Z8raePa2CQNSYmn3fbA/LT9JtkYBpGtRfCcpC+l8yRpu5rzviRpgKTNyCaEfErSpsC8iLiQbFbVzxT5vszauUVhreZo4GJJ7VM2n5P6/du9I+l/gFXIZhcFOAeYJmkOcA/wh1R+K3CSpCfIZuq8r+Z9pgKPSnoojVOcTbbK4ACy2V5P5sNEAFmr41uSLgXeBpbxYWtiOnCZpFOBw8jGN/4tvecq6fVH0rF/IJtWei3gpIh4R9LhwFclrSBb/e2fevaVWavz7LFmSeo6+mZEzGp0LL0h6Qrg5oi4ttGxWHNx15OZmeVyi8LMzHK5RWFmZrmcKMzMLJcThZmZ5XKiMDOzXE4UZmaW6/8DAQ+27rRHW/MAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvp0lEQVR4nO3de5xVdb3/8dd7mOF+G0ABARXTvORJEiQ1Tz8sNS8E3jK0x0n62SErTxdN05NaeUnz0sXy2CG19PwMM9NCxIwKMisM8Ig3QkhNQS4iIyAgzDCf3x9rbdyzZ619mb32nn35PB+Peczea6291nfvPbM+a32+N5kZzjnn6lNDdxfAOedc9/Eg4JxzdcyDgHPO1TEPAs45V8c8CDjnXB3zIOCcc3XMg4ArG0kmaf/w8U8lXRM+/ldJy7u3dNEk7RuWu7GLr/+EpN8mXS7nkuJBoMpJelnSdklvSWqR9LCkMd1drkKY2Z/M7MC49enBI2P5dEmP53MMScdI+oukTZI2SvqzpCOKKXfEMToFDDO7x8xOSPI4ziXJg0Bt+KiZ9QdGAuuAH3RlJ1292q10kgYCcwg+lyHAKOCbwI7uLJdzlcCDQA0xs7eB+4FDUssk9ZJ0k6RXJK2T9CNJfcJ1kyStkvRVSWuBn0j6hqT7JN0taYuk5yRNSNvfwZIWSHozXDclbd0CSZ9Oe57XlXqqHAl9DFHeDWBms8xsl5ltN7PfmtnT4fEbJF0u6Z+S1ofvfVBMWV+WdFza829I+n/h08fC32+Gd2ZHZX4Gko6WtCi8I1kk6ei0dQskXR3epWyR9FtJw5L+MJxL50GghkjqC3wcWJi2+HqCk+A4YH+Cq+Ar09aPILg63geYES6bAtwLDAZmAz8M998EPAT8FtgT+A/gHkmxqZwK8QKwS9Jdkk6S1Jyxfnr4cyywH9Cf8D0X6IPh78Fm1t/M/pq+UtIQ4GHgFmAo8B3gYUlD0zY7B/gUwefbE/hKF8rhXN48CNSGX0l6E9gEHA/cCCBJBCf2L5vZRjPbAnwLmJb22nbg62a2w8y2h8seN7O5ZrYL+B/gsHD5kQQnyOvNbKeZ/YEgzXJ2ad9eccxsM3AMYMCPgdclzZY0PNzkE8B3zOxFM3sLuAyYVoL02CnACjP7HzNrM7NZwN+Bj6Zt8xMzeyH8Lu4jCN7dTtKd4V3Sswnt7zfh3eScjOV3SFoq6WlJ90vqX8A+d0l6KvyZnUAZh0qaH97VdeWioCp4EKgNp5rZYKA3cAHwR0kjgD2AvsCS8B/uTeA34fKU18M0Urq1aY+3Ab3DE+JewKtm1p62/p8EdxcVzcyWmdl0MxsNHErwXr4Xrt6L4H2k/BNoBIaTrMzjpI6V/vllfvZ5nwRL7KfAiQnu70bg3yKWf9nMDjOz9wKvEPw9dxCmzfaNeO12MxsX/kyJWF+ot4ErqPG7MQ8CNSTMdz8A7CK48t0AbAfeY2aDw59BYSXy7pcVcIjXgDGS0v9u9gZWh4+3EgSdlBEFv4kyMLO/E5zUDg0XvUaQDkvZG2gjqGTPlO095vosM4+TOtbqiG0ripk9BmxMXybpXeEV/RJJf5J0UAH7+z2wJWL55nDfAvpQ2N9nJEnjJf0xLOejkkbmWcatZvY4QTCoWR4EaogCU4FmYFl4xf5j4LuS9gy3GSXpI108xBMEV6eXSGqSNIkglXFvuP4p4HRJfRU06Tyvy2+ms56Seqf99AiXK2N578wXSjpI0kWSRofPxxCksFJ1J7OAL0saG6YfvgX83MzaIsrxFEGqqCmsMD8zbd3rBOm1/WLew1zg3ZLOkdQo6eMElfhzYravdDOB/zCz8QRXy/+VxE4l/YTgjuggCmvp1lvSYkkLJZ0a7qsp3MeZYTnvBK5Nopy1oiabBNahhyTtIrhq+idwrpk9F677KkFF8MKwpclq4Dbg0UIPYmY7JX2U4J/9snBfnwyvrAG+CxxBcAX9NHAPcFzUvrrguYzn/05wtX40wd3ObpKaMk7gW4D3AxdKGgy8SXDivThcfydBquYxgpTaowSV3lGuIAgaLcAfgZ8RVKxjZtskXQv8OTz5dEifmNkbkiYD3yf4DlYCk81sQ+63X1nCYHk08Ivgoh2AXuG604GrIl622sxyXoCY2afCIP8DgoYOP5H0KeCL4Sb7A3Ml7QReMrPTwuX7mNlqSfsBf5D0DMHdxKHAvLCcPYA1YTn/A/hMRBEWmdmncpWzVsgnlXHO5SPMw88xs0MV9L1YbmZ5pVZi9jcJ+IqZTY5Z/0Hgksz1khYA083s5Sz7/ilBoF8OzDSzo4oo53Rggpl1qp+oBZ4Ocs4VLMzdvyTpY7A7FXlYjpdlFe4jNayICJoq/z37q3a/tllS6k5kGPAB4HmCILCHpKPCdU2S3lNMOWuNBwHnXE6SZgF/BQ5U0MHwPIKmtedJWkqQrptawP7+BPwC+HC4v48AAu4K0zjPEPSAj0orRTkYWByWZT5BM+bnzWwnQb3Nt8N1TxGksfIt58sE/Tmmh+U8JMdL0l97oqTlklZKujRi/XRJr+udZq3pHS3PlbQi/Dk3bfm1kl6V9Fa+5chZTk8HOedcssI6jRcI+u2sAhYBZ5vZ82nbTCcizaSgU+FiYAJBPd8SYLyZtUg6kqDeb0VGK78u8zsB55xL3kRgZdgBcSdBC7p875Q+AswLO3i2APMIGxmY2UIzW5NkQSu6dVCPfv2safCQ7i6Gc67C7Xht1QYz2yP3lvH+z6Re1rKxPfeGwDPPtD1Hx/4DM81sZtrzUcCrac9XEbRQy3RGWAH+AkFHuVdjXluyDpkVHQSaBg9h789d2N3FcM5VuBWXX5jZE7tgLRvbmT03v/H6xo5Z+7aZTci9ZVYPAbPMbIekzwB3AR8qcp8F83SQc84lbzWQPq/HaDJ6hpvZG2aWGs78dmB8vq9NkgcB55xL3iLggLAXek+CQRs7DGqXMXzFFGBZ+PhR4ISw2WszcAJd6NyZLw8CzjmXsLDH+gUEJ+9lwH1m9pykq/TOHBxfUDAnx1LgCwTDmWNmG4GrCQLJIuCqcBmSblAw90bfsMnqN4ota0XXCTjnSmNgzyY+e9jB7D2gP++M+lDZzOCVLW9x29JlbN7Z2t3FycnM5hKMF5W+7Mq0x5cRDL8S9do7CYYzyVx+CXBJkuX0IOBcHfrsYQczbuw+NPXrj6okCpgZQ7a+xWeBby96uruLUzM8HeRcHdp7QP+qCgAAkmjq15+9B1TKFAu1wYOAc3VIoqoCQIqkqklfVQsPAs45V8c8CDjnKsLOHTu4+DMzOOXoIznnlJNY/eor3V2kuuBBwDlXER6Y9TMGDh7Mw39ZyL/9+2f43jXXdHeR6oK3DnLO5fTwA7255br+rH2tgRF7tfOFy97ilNOLm3p39i/u464f3YYk3n3wIWzcsIHPXhTM6X785Mlc97X/xMyqsu6imngQcM5l9fADvfnmxQN4e3uQOFizugffvHgAQJcDwcrlf+fH3/8ed//6IZqHDmVTSwvTTz+V4XvtBUBjYyP9Bw7gzY0baR46NJk34iJ5Osg5l9Ut1/XfHQBS3t7ewC3Xdb2p5t8ef5zjJ3909wl+UHNzUWV0XedBwDmX1drXok8Tccu7aviIkax77TUA2traeGvzFgYP8aHkS82DgHMuqxF7RY+xH7c8HxOPOYZ5cx7izY0bAdjU0sKkE05g9i/uA2DenDlMPOYDXh9QBl4n4JzL6guXvdWhTgCgd5+gcrir9j/wID79hS/yqTNOo0ePHhx06KFccf0N/OcXLuCUo49k0ODB3HDbfydRfJeDBwHnXFapyt+kWwdNPevjTD3r4x2W3Tzz9qL26QrnQcA5l9Mpp79d9EnfVSavE3DOuTrmQcA55+qYp4Occw7Y3N6b323bP8+t15a0LOXkdwLOOVfHig4CksZImi/p+XC+zC9GbCNJt0haKelpSYcXe1znnHPFS+JOoA24yMwOAY4EPi/pkIxtTgIOCH9mALclcFznXA1ZvPCvnHXC8bxvzCh+O+eh7i5O3Sg6CJjZGjN7Mny8BVgGjMrYbCpwtwUWAoMljSz22M652jFy1Ciu+d73Oem007q7KHUl0YphSfsC7wOeyFg1Cng17fmqcNmaiH3MILhboHGQDyrlXCUYtPaXjHjxOpp2rKa11yjW7ncZm0acUdQ+M4eS/tYPfghAQ4NXVZZTYkFAUn/gl8CXzGxzV/djZjOBmQC9R42xhIrnnOuiQWt/yejlX6GhfTsAPXesYvTyYNz/rgaCqKGkXfdIJORKaiIIAPeY2QMRm6wGxqQ9Hx0uc85VuBEvXrc7AKQ0tG9nxIvXdXmfPpR05UiidZCAO4BlZvadmM1mA58MWwkdCWwys06pIOdc5WnaEX29FrfcVZck7gQ+APwb8CFJT4U/J0s6X9L54TZzgReBlcCPgc8lcFznXBm09sps55F9eT6ihpJ23aPoOgEzexzIOui3mRnw+WKP5Zwrv7X7XdahTgCgvaEPa/e7rMv7jBpKetr0T/Gl8/4vm998kz/Om8dtN93IgwseS+ItuCx82AjnXFapyt+kWwdFDSX9uyX/W9Q+XeE8CDjncto04oyiT/quMnmDXOecq2MeBJyrQ2YQVNVVFzOjCotd0TwIOFeHXtnyFq1b36qqQGBmtG59i1e2dH1uY9eZ1wk4V4duW7qMzwJ7D+iPsrbtqxxmQfC6bemy7i5KTfEg4Fwd2ryzlW8verq7i+EqgKeDnHOujnkQcM65EpB0oqTl4WRal0asP1/SM+EoC4+nz8Mi6bLwdcslfSRt+Z2S1kt6NqlyehBwzrmESeoB3EowodYhwNkRk239zMz+xczGATcA3wlfewgwDXgPcCLwX+H+AH4aLkuMBwHnnEveRGClmb1oZjuBewkm19otY8j9fkCqqdZU4F4z22FmLxGMuTYxfM1jwMYkC+oVw845B7y1qxd/evPdeW79+DBJi9MWzAznQkmJmkjr/Zl7kfR54EKgJ/ChtNcuzHht10fry8GDgHPOFW6DmU0odidmditwq6RzgMuBc4suWYE8HeScc8krdCKte4FTu/jaongQcM655C0CDpA0VlJPgore2ekbSDog7ekpwIrw8WxgmqReksYCBwB/K1VBPR3knHMJM7M2SRcAjwI9gDvN7DlJVwGLzWw2cIGk44BWoIUwFRRudx/wPNAGfN7MdgFImgVMAoZJWgV83czuKKasHgScc64EzGwuwayK6cuuTHv8xSyvvRa4NmL52UmWETwd5JxzdS2RIJCrF5ukSZI2pc1BfGXUds4558orqXTQT4EfAndn2eZPZjY5oeM555xLQCJ3AqXoxeacc670ylkncJSkpZIekfSeuI0kzZC0WNLiXVu3lrF4zjlXf8oVBJ4E9jGzw4AfAL+K29DMZprZBDOb0KNfvzIVzznn6lNZgoCZbTazt8LHc4EmScPKcWznnHPxyhIEJI2QgknsJE0Mj/tGOY7tnHMuXiKtg6J6sQFNAGb2I+BM4LOS2oDtwDSrphmunXOuRiUSBHL1YjOzHxI0IXXOOVdBvMewc87VMQ8CzjlXxzwIOOdcHfMg4JxzdcyDgHPO1TEPAs45V8d8UhnnnAO2tfVk6et7dXcxys7vBJxzro55EHDOuTrmQcA55+qYBwHnnKtjHgScc66OeesgV7DNTx3IG/OOoW3TABoHbWHo8Y8zcNzy7i6Wc64LPAi4gmx+6kDW//p4rLUJgLZNA1n/6+MBPBA4V4U8CLhYUVf8b8w7ZncASLHWJt6Yd4wHAeeqkAcBFynuit9ao/9k2jYNKGfxnHMJ8SBQZ/LN58dd8aN2MHXaXn22l72MzrnieRCoI4Xk82Ov7CMCAIDt7Mnmpw4s6GQddbIHspbRA4RzyUqkiaikOyWtl/RszHpJukXSSklPSzo8ieO6wmTL52dqHLQlZi/RQYBdjZH7iZMKSG2bBgLafbJf//Ck2DLGvWbzUwfmfVznXEdJ3Qn8lGAO4btj1p8EHBD+vB+4LfztEhR3lZy+PErU8qHHP97hijxgxAaBmP3Ela1tWy+ISjdlqXPwSmnnkpfURPOPSdo3yyZTgbvNzICFkgZLGmlma5I4votP9Wx/ZSRb/vfQTifPdFFX/amTavqJe9fORmx734L2E1e2IKDkr3HQloKCmHMuP+WqExgFvJr2fFW4zINAQuKukjcvOgwsPuunptbdufhMA8ct73CFnXkyj9pPvs1K4+4oGvpux1qbOmyf2new34GdXhOfunLO5VJxFcOSZgAzABoHNXdzaapHoRW5YAVXrGbeHSADU85K3bhmpVFl2uPkBR2OkVnGzCCULYg553IrVxBYDYxJez46XNaJmc0EZgL0HjWmsJxBHQvSJZ2vklMn6qjtx158B1BYk8zMu4N0L914XkHNSjM19N2+e99Rx4hKUWUrq7ckci63cgWB2cAFku4lqBDe5PUBxUs/yanPdujRBrve+UrV1MqA9z3bqU4g/eo5yWEgst2NqKk1ayWzmlp33wVkky0IpfPhLVx3k3Qi8H2gB3C7mV2fsf6DwPeA9wLTzOz+tHXnApeHT68xs7vC5dcCnwSazax/EuVMqonoLOCvwIGSVkk6T9L5ks4PN5kLvAisBH4MfC6J49azzOaStr0vGDT03UaQ6tnMnlPnMXzKAvacOo/GQZs7LE+/qs632Wgucbn5xkFbOpVh4MSnYsuUhCTfl3OFktQDuJWgZeQhwNmSDsnY7BVgOvCzjNcOAb5OcME8Efi6pFRu/KFwWWKSah10do71Bnw+iWO5QGRla3sjDU3beNc13+uwONvVc5ItbqKalabuOqLLsKDgY+TLWxK5bjYRWGlmLwKEWZCpwPOpDczs5XBde8ZrPwLMM7ON4fp5wInALDNbGC5LrKAVVzHs8pPUSS6uLqErLW4KzdmXUpLvy9WHXa0NtKzN+/9nmKTFac9nhvWZKVEtIvPtGxXXmrIkPAiUUCkrJgs9ycWVJdvVe1fkm7MvtaTfl3MZNpjZhO4uRBI8CJRIqSsmCznJ5VOWSrh676pswbaa35eranm3iIx57aSM1y5IpFQRPAiUSKmHOCjkJJerLJVy9d4VuQJctb4vV/UWAQdIGktwUp8GnJPnax8FvpVWGXwCcFnyRQx4ECiRXDn7JFJF+Z7karmS1McTcpXIzNokXUBwQu8B3Glmz0m6ClhsZrMlHQE8CDQDH5X0TTN7j5ltlHQ1QSABuCqtkvgGgmDSV9Iqgqan3yimrB4ESiRb5611syd1aLtf6jbstVxJWkyA885krpTMbC5B8/j0ZVemPV5EkOqJeu2dwJ0Ryy8BLkmynIn0E3CdDT3+cdTU2nmFNbD5b+PK2oY9qiy1UkmarW9CNj4stXMBvxMokdQV5bpfnhgxgFt0G99SpWdquZI0WwV5rgpjTyM550GgpAaOW866+0/Ke/tSpmdqtZI09Z5enzuJ9m19goWNrZ2G0G7bNJB195/E+ocnsecpC2q6nsS5Qng6qMTiT+wdx8arlfRMdwlO9iI1hEZUyi21bv2vj4+dE7kW6kmcK4QHgRKLy8eXeuycelLIfAUQBAyJmq0nca4Qng4qsez5+AXAO61U1t1/Uk3l68ulKymc9m19GH7mIzVZT+JcITwIdLNqH/J4cr8lXNQ8l5GNLaxpa+bmlpOZs3V8WcsQ2xw3y5zIjYO21Gw9iXOF8HRQieVqiljNQx5P7reEa4bdx6imFhoEo5pauGbYfUzut6Ss5ciWcksNrZ25ztM+zgU8CJRYrpN8NbdSuah5Ln0bOp58+za0clHz3JhXlMbAccsj50wYPmUB7/rP/2b4mY94/YtzMTwdVGLZTvIv3Xhe7OuqoZXKyMaWgpaXUrbUjqd9nIvnQSAP6Z2OoiZXz1a5GJ+vJnZ5taQr1rQ1M6qp8wl/TVtzxNaunlVC3ZGL5umgHDJz+kHv3yC3v+6BE1j34AlZhx6IHj4irsKyutIVN7eczLb2jqmube1N3NxycjeVyFWiSqk7ctE8COQQ3QY91N7YYWJ36FypG5WvzmbsxXdURQAAmLN1PJdvOIvVrc20G6xubebyDWf5FZ7roFLqjly0RNJBkk4Evk8wZOrtZnZ9xvrpwI28M6nCD83s9iSOXWpdqaBt2zQg67g1L914Xs2M6jln63g/6busKqnuyHVWdBCQ1AO4FTieYC7MRZJmm9nzGZv+3MwuKPZ45ZYtpx+noe/2rG3/ferD6rBu9iQ2LzoMTCBj4BFLGT5lQXcXq9sVmt/3uqPKlkQ6aCKw0sxeNLOdwL3A1AT2WxFih4QGaGiDHm0dFqmpFTOyNguNa9JYLWmgerBu9iQ2/23c7jqg1BDg62ZP6rTt5qcO5KUbz2PF5V/ipRvPq+nhqLuS3/e6o8qWRDpoFPBq2vNVwPsjtjtD0geBF4Avm9mrEdsgaQYwA6BxUPdfKWQO+5BP66C4kUPTU0vebLGybV50GJ0r78XmRYd1uBuo9h7fhcqW34+7G0gtr/TWQWoVvdbG1P/VsHI1EX0ImGVmOyR9BrgL+FDUhmY2E5gJ0HvUGIvaptxynbAz1wVBoTZy/nXLYgagy1heb/MSdDW/73VHlSuJdNBqYEza89G8UwEMgJm9YWY7wqe3AzX911DLM3nVDcVcf2Qsr+Ye310Rl8f3/H71SuJOYBFwgKSxBCf/aQQTIe8maaSZrQmfTgGWJXDcsipkPtpansmrXgw8YmlQJ9AhJRRUDqer5fmbo9zccjLXDLuvQ0poW3sT87cdzPzRV1d0usdFKzoImFmbpAuARwmaiN5pZs9JugpYbGazgS9ImgK0ARuB6cUet5yi8r7r7j8p69DPtZrzr5een6m8f67WQfXW0isqvz9/28GcPmDR7sCQqixO395VLplVRNo9Uu9RY2zvz13Y3cWIbdefoqbWumjdk2oZkn4VuKO9ga3Wm8EN22o6KGRTyF1itckn6M8ffXVkE9DVrc0cu+qKspRzxeUXLjGzCcXso5DzTRLHqxQ+dlAecuV3rbUpmFCe2mwRkhLVMqRXQzu92AbU3xVg5sl/+JmP1NT3nxn0475f7wxW3TwI5CGvDmPWUNNNAyG/f+pczQVrRa02DU2/8m+ngUa1d1gf9f16Z7Dq5mMH5SFrh7E01TIZTFfl+09dbVeAXensVc2TAcXJ7AiWGQBSMr9f7wxW3fxOIA+dOowBcdMW1lLTwMx8cGYFYJxqugKMq/RfP/s4bGdTbI6/FpuGRqX7omR+v3O2jufwXi8xbeBCetDOLhp4YMsRNX83WCs8CMSIquwbe/Edu9et++WJ4ZACHdVK08CofPDpAxbx5Nv7cmSff9CDdtoBy0gZVNsVYPQoscJ29gTi0zy12DQ0nzu4qO93cr8lnD5g0e6/g0bag7+VHWM9EFQBTwdFyDUv8MBxyxl+xm9qukNY3PAAR/dZQaPakaCHYJfBxl19q3Yo6Xyu3KPSPLXYITDuDq7NGrJ+vz5UdHXzO4EI+QwFUOsdwuKuChsysmC9GtrZ0NqLI1+5pgylSl6+o8RmBota/P7jOoLlCuzeOqi6eRCIkG++t1Y7hEF8i48o1fzP/s6AfzFjBYWi0jy19v13daA3bx1U3TwIRKjFfG+hoq4K263znQBU9z/7wHHL2f7KyIghIt4Rleap1Q5iXRnoLe4OoprqhuqZ1wlEqMV8b6Gipo782eaja7Ip4PApCxh+5iO753dQn2009N1G3FwPueqMasXkfkuYP/pq/r7vhcwffXXsnAE+zWh18zuBCLWY7+2KqKvCJ3eMrcmxgwpJ7dTD8NH59hZO8aGiq5cHgRiFnBRqNTUQpd7+2aO+21rsI5CpK5PHuOrk6aAi1UtqoB7FfbcNfbdHv0BWM9+7t/ipHx4EilSLwwe4QNx3a0b0MCLh+FG1EAh88pjSkTRE0jxJK8LfkR+qpHPDbVZIOjdt+cclPS3pOUnfTlu+j6Tfh+sWSBqdT3k8CBSpHlIDkH8lYS2J+w5tex/2nDoPIsbWqZULAB8PqKQuBX5vZgcAvw+fdyBpCPB1gvnaJwJfl9QsaShwI/BhM3sPMELSh8OX3QTcbWbvBa4CrsunMB4EihTXbLSWmpNmDiw2qqmFm/a4hyuH3N/dRSupbN/twHHLY+chrrYLgKgA7y1+SmoqwTzrhL9PjdjmI8A8M9toZi3APOBEYD9ghZm9Hm73O+CM8PEhwB/Cx/PD4+TkFcNFqoeZpaIqCRsE5wz8S02PD5Pru62F/iS5WgHV6ncbpaEV+q7Ne/NhkhanPZ9pZjPzfO3wtOl21wLDI7YZBbya9nxVuOw3wIGS9g2XnQr0DLdZCpwOfB84DRggaaiZvZGtMB4EilQPzUmzDSGR3lqk1qaezPXd1sIFgLcC6rIN2WYWk/Q7YETEqq+lPzEzk5T39I5m1iLps8DPgXbgL8C7wtVfAX4oaTrwGMGc77ty7TORICDpRILo0wO43cyuz1jfC7gbGA+8AXzczF5O4tiVoNaGD8iUbQiJVIDIdUVZrQEi23dbCxcA3gqoNMzsuLh1ktZJGmlmaySNBNZHbLYamJT2fDSwINz3Q8BD4b5mEJ7ozew1gjsBJPUHzjCzN3OVteggIKkHcCtwPMHtySJJs83s+bTNzgNazGx/SdOAbwMfL/bYxaqn9v3FuLnlZG7a456sQ0bkGkmykI5H1aTaLwDyHfenWoN4hZoNnAtcH/7+dcQ2jwLfSms5dAJwGYCkPc1sfbjuc8BZ4fJhwEYzaw+3vTOfwiRRMTwRWGlmL5rZTuBeOldIpFeE3A98WFL2EbtKzNv352/O1vH8bPPRtGfctKa3Fsl2RelDDVeu+dsOzvq9QnTDgGuG3VcXLcRK5HrgeEkrgOPC50iaIOl2ADPbCFwNLAp/rgqXAXxf0vPAn4HrzeyFcPkkYLmkFwjqGa7NpzBJpIOiKjDeH7eNmbVJ2gQMBTZk7iy8vZkB0DiodG2S66Hrf5Ku2nhm1iEjsl1ResqhMk3ut4SzBj7R4Q7PDJ58e18uap7LTXvcw5q2Zvo07PB6gwSFFbUfjli+GPh02vM7ibiaN7OzY/Z7P8FFdkEqrmI4rGGfCdB71Ji8K0wKlat9v6eKOsvWWiTbSJIXNc+tqqGG6+W7v3zog/RUx3pDCT7QZwWp+/RRTS1YzH+hB/HakEQ6aDUwJu356HBZ5DaSGoFBBBXE3SZbG3BPFeWW2bYciG1XXk0dj+rpu29u2Ba5PDNRG5e4rdQg7gqTRBBYBBwgaayknsA0goqPdKmKEIAzgT+YxV1flEe24aJ9KIjs4nLEAMeuuoKDXv4Ox666okOroN5qpc0asArveOTffbTM/1azoD7BVb+ig4CZtQEXENRmLwPuM7PnJF0laUq42R3AUEkrgQuJ6CZdbgPHLWfPqfN2jyGfPm58vQwF0VX5VvRmBotGtbPdmiq6ZUk9ffctu/rmvW3U3cGxfZclXCLXHRKpEzCzucDcjGVXpj1+G/hYEsdKUlzzvlroCVpK+Vb05tsRqZKaH9bTd3/NxtO4btgsejW8MwZSm4ldpg7LzKJTQl4nUBt87KAIPrNYdvmOMJlPsKi05of19N3P2Tqeyzac3aEe55LXz+m0rKU9+o7B6wRqQ8W1DqoEtdATtJTynVM2rtmogPmjr97dcqiSmh/W23cf1+Ir807N5xCuXR4EYlR7T9BSSp0gcqVwooIFBKmF1BV/H0WMy0/3phr8u+8o3+/bVScPAq5L8hlhcs7W8Rze6yXOGfiXyCEn+jYELYYa6Twuv6cayiPf+ph6G1G0nngQcCV1bN9lkQEgpYF2trU3eaqhGxQ6mbyrTR4EXEnlk9Z5YMsRHNt3macaSijqir/S6mNc9/Ag4Eoq2zDUEMxJcGzfZRy76ordy1K9kT0oJCPuir8S62Nc+XkTUVdSUUNGZKrkJqPlVoq5nOOu+HfF/Pt7fUx98TsBV1LpLUv2amyJ7HSUftJJKkVRSR3Q8lWqHH3szHBeH+PwOwFXBnO2jufYVVdw0eufyDmQXBLDTpfzbiKpK/fJ/ZZwwx6zSjLvQrbOfT6ZvPM7AVc2+bQ3z3emq2zKVeGZ1JV7aj+N6txUForP0Wfr3OdNP50HAVdWuU46+fZGTpeZ+inXJDZJBZuo/aQrNkfvnb3y09AK/V/LOS97zfEg4CpKoSesqKvxzOkSU5Ku8Ewq2GTbvis5+mz1IanlqRSTBwLnQcBVnFx3C5P7LeHyIQ/S3COYFCWzsrlB0G506KRWigrPuNRVOw1M7rck7xNs3H7arKHgHH1ciurwXi9x+oBF3jHMdeIVw66qTO63hOuGzWJI4zak+FmvBCWv8Ixr/tqodm7e4x4W7n05Vw65P2fFcdzMa5e8fnbBZY5LUU0buLAklc6u+vmdgKsqFzXP7TDWfZzX2po7dEArhdQJ+oY9ZnWq1JVgSI9tfGLgXzrM1xt19Z1kzj4utdQjYnymbNu7+uFBwFWVfE5a5WzrPmfreG7a457Y9Zl3KnEVx0m10snVQztqe1ffPB3kqkq2k1Z3zV9c6Im0lFffcSmqqLSZdwxzUOSdgKQhwM+BfYGXgbPMrNNfuKRdwDPh01fMbErmNs7l4+aWkztNiQiw03pw6evTyl7JObnfEvo07IidgjFKKa++8+mhbRaky7yZqIPi00GXAr83s+slXRo+/2rEdtvNbFyRx3J1LL3Z45u7+tJKK/3CAdBa2vtyzRun5XVCS3I4iagZtzKDQebzclx9p1JLf9/3QqLikkHJ60tc9Sg2CEwFJoWP7wIWEB0EnOuyzJPtkMZtbGtv4qLXP5FI80noWjPJqJY4UtC0s4F21rQ1M3/bwd02THYSva9d7Ss2CAw3szXh47XA8JjtektaDLQB15vZr+J2KGkGMAOgcZD/sbrS9swtZnC6vbIMzHbQy9/Z/fyqjXnvusP+iw0cXel97epPziAg6XfAiIhVX0t/YmYmKaavJvuY2WpJ+wF/kPSMmf0jakMzmwnMBOg9akzc/lwdKXXP3K4MTleqYR6SvFvx4SJcPnIGATM7Lm6dpHWSRprZGkkjgfUx+1gd/n5R0gLgfUBkEHAuU1JpjVINTpduW3sT87cd3OVJcZIe/M4HiHO5FNtEdDZwbvj4XODXmRtIapbUK3w8DPgA8HyRx3V1JK5HbaFpjST2E3fXkGqe+sCWIzh9wKIuD2NdrsHvnEspNghcDxwvaQVwXPgcSRMk3R5uczCwWNJSYD5BnYAHAZe3OVvHJzLufRL7ibtrSPVQPrbvsqKGZ8g29r9zpVBUxbCZvQF8OGL5YuDT4eO/AP9SzHGcSyqtUex+clW2Fnsl75W5rtx82AjnCpCrsrXYegevzHXl5kHAuQJlu5tI4kreK3NdOfnYQc4lKKn6C1e7JA2RNE/SivB3p9tESftIelLSU5Kek3R+2rrxkp6RtFLSLVLQJz2f/UbxOwHnEuZX8i6HfIbbWQMcZWY7JPUHnpU028xeA24D/h14ApgLnAg8kud+O/E7AeecK6+pBMPsEP4+NXMDM9tpZjvCp70Iz9Vhf6yBZrbQzAy4O+31Ofcbxe8EnHMO6LFzF/1e2Zrv5sPCoXBSZoajHeQjr+F2JI0BHgb2By42s9ckTQBWpW22ChhVyH4zeRBwzrnCbTCzCXErkxhux8xeBd4raS/gV5Luz7dwOYbx6cCDgHPOJSyJ4XbS9vWapGeBfwX+DIxOWz0aWB0+Lmi/KV4n4Jxz5ZXPcDujJfUJHzcDxwDLw3TPZklHhq2CPpn2+pz7jeJBwDnnyivf4XaeCIfb+SNwk5mlZmf8HHA7sJJgIM5Hsu03F08HOedcGeU53M484L0xr18MHJrvfnPxOwHnnKtjHgScc66OeRBwzrk65kHAOefqmAcB55yrYx4EnHOujnkQcM65OlZUEJD0sXCs6/ZwYKO47U6UtDwc//rSYo7pnHMuOcXeCTwLnA48FreBpB7ArcBJwCHA2ZIOKfK4zjnnElDsRPPLAMKJbeJMBFaa2YvhtvcSjHv9fDHHds45V7xy1AmMAl5Ne54+/rVzzrlulPNOINu42GaW1yh1hZA0A5gB0DgorykynXPOdVHOIJBtXOw8rQbGpD1PH/866ngzgZkAvUeNyWtSBOecc11TjnTQIuAASWMl9QSmEYx77ZxzrpsV20T0NEmrgKOAhyU9Gi7fS9JcADNrAy4AHgWWAfeZ2XPFFds551wSim0d9CDwYMTy14CT057PBeYWcyznnHPJ80llnHMOYEcreim2urJm+bARzjlXxzwIOOdcHfMg4JxzdcyDgHPO1TEPAs45V8c8CDjnXB3zIOCcc3XMg4BzztUxDwLOOVfHPAg451wd8yDgnHN1zIOAc87VMQ8CzjlXxzwIOOdcHfMg4JxzdcyDgHPO1TEPAs45V8c8CDjnXB0rdqL5j0l6TlK7pAlZtntZ0jOSnpK0uJhjOudcNZM0RNI8SSvC380x2+0Kz5lPSZqdtnyspCckrZT0c0k9w+W9wucrw/X75lOeYu8EngVOBx7LY9tjzWycmcUGC+ecqwOXAr83swOA34fPo2wPz5njzGxK2vJvA981s/2BFuC8cPl5QEu4/LvhdjkVFQTMbJmZLS9mH845V2emAneFj+8CTs33hZIEfAi4P+L16fu9H/hwuH1WjfkevEgG/FaSAf9tZjPjNpQ0A5gRPt2x4vILny1HAbMYBmzo5jJAZZSjEsoAlVGOSigDeDlS9il2B5t3bXj00Td+PCzPzXtnpLZnZjuvZRhuZmvCx2uB4TmO0QZcb2a/AoYCb5pZW7jNKmBU+HgU8CqAmbVJ2hRun/V7yRkEJP0OGBGx6mtm9utcrw8dY2arJe0JzJP0dzOLTCGFH+TM8NiLuzt9VAllqJRyVEIZKqUclVAGL0eyzOzEpPaV7byZcUwLL46j7BOeN/cD/iDpGWBTUmVMyRkEzOy4Yg9iZqvD3+slPQhMJL96BOecqzrZzpuS1kkaaWZrJI0E1sfsI3XefFHSAuB9wC+BwZIaw7uB0cDq8CWrgTHAKkmNwCDgjVxlLXkTUUn9JA1IPQZOIKhQds65ejQbODd8fC7QKaMiqVlSr/DxMOADwPNmZsB84MyI16fv90zgD+H2WRXbRPQ0SauAo4CHJT0aLt9L0txws+HA45KWAn8DHjaz3+R5iHxzbKVUCWWAyihHJZQBKqMclVAG8HJUo+uB4yWtAI4LnyNpgqTbw20OBhaH5835BHUCz4frvgpcKGklQc7/jnD5HcDQcPmFxLc66kB5BArnnHM1ynsMO+dcHfMg4JxzdaxigkClDEFRQDlOlLQ87KKdV+6twHIU1bW8yGNnfW9d7Z5egnJMl/R62vv/dAnKcKek9ZIiGzMocEtYxqclHd4NZZgkaVPa53BlCcowRtJ8Sc+H/x9fjNim5J+FKwEzq4gfgoqQA4EFwIQs270MDOvOcgA9gH8A+wE9gaXAIQmX4wbg0vDxpcC3Y7Z7K+Hj5nxvwOeAH4WPpwE/L8H3kE85pgM/LPHf5QeBw4FnY9afDDwCCDgSeKIbyjAJmFPiz2EkcHj4eADwQsT3UfLPwn+S/6mYOwGrkCEo8izHRGClmb1oZjuBewm6bCepy13Li5TPe+tS9/QSlKPkLOjUuDHLJlOBuy2wkKAN98gyl6HkzGyNmT0ZPt4CLOOdnqopJf8sXPIqJggUIDUExZJwiInusLt7dii963ZSCupaLmmhpFMTOG4+761D93SCXoxDEzh2oeUAOCNMPdwvaUzCZchHOf4W8nGUpKWSHpH0nlIeKEz/vQ94ImNVpXwWrgDlGjsIKP8QFCUuR9FK1bXczP6RdFkr1EPALDPbIekzBHcnH+rmMnWHJwn+Dt6SdDLwK+CAUhxIUn+CXqtfMrPNpTiGK6+yBgGrkCEoEihHqnt2SnrX7UTKoeK6lhcTBPJ5b13qnp50Ocws/Zi3E9SjlFsifwvFSD8Zm9lcSf8laZiZJTqgm6QmggBwj5k9ELFJt38WrnBVlQ5S5QxBsQg4QMHkDj0JKkcTaZmTpstdy4s8bj7vrUvd05MuR0a+eQpBnrrcZgOfDFvGHAlsSkvjlYWkEak6GUkTCf6vEw3K4f7vAJaZ2XdiNuv2z8J1QXfXTKd+gNMIcog7gHXAo+HyvYC54eP9CFqJLAWeI0jflL0c4fOTCVpI/KNE5RhKMOHECuB3wJBw+QTg9vDx0cAz4efxDHBeQsfu9N6Aq4Ap4ePewC+AlQRDgexXor+JXOW4Lvw7SHWtP6gEZZgFrAFaw7+L84DzgfPD9QJuDcv4DFlatpWwDBekfQ4LgaNLUIZjCOrjngaeCn9OLvdn4T/J//iwEc45V8eqKh3knHMuWR4EnHOujnkQcM65OuZBwDnn6pgHAeecq2MeBJxzro55EHDOuTr2/wF9mmuExtyvEAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate Features from Data\n",
    "\n",
    "# change this to play arround with feature functions\n",
    "#feature_fn = affine_features\n",
    "#feature_fn = quad_features\n",
    "feature_fn = cubic_features\n",
    "features = feature_fn(samples)\n",
    "\n",
    "num_iterations = 25\n",
    "\n",
    "w_bce, l, l_info = minimize_with_sgd(features, labels, np.zeros(features.shape[1]),\n",
    "                                 schedule=(lambda t: 0.25),\n",
    "                                 num_iterations=num_iterations,\n",
    "                                 batch_size=1)\n",
    "print(\"Final loss\", l)\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Cross Entropy Loss\")\n",
    "plt.grid(\"on\")\n",
    "plt.xlabel(\"Update Steps\")\n",
    "plt.ylabel(\"Negative Bernoulli Log-Likelihood\")\n",
    "plt.semilogy(l_info[1])\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Bernoulli LL Solution\")\n",
    "pred_grid = np.reshape(sigmoid(feature_fn(flat_plt_grid) @ w_bce), plt_grid_shape)\n",
    "\n",
    "plt.contourf(plt_grid[..., 0], plt_grid[..., 1], pred_grid, levels=10)\n",
    "plt.colorbar()\n",
    "#This is just a very hacky way to get a black line at the decision boundary: \n",
    "plt.contour(plt_grid[..., 0], plt_grid[..., 1], pred_grid, levels=[0, 0.5], colors=[\"k\"])\n",
    "\n",
    "s0 = plt.scatter(c0_samples[..., 0], c0_samples[..., 1], color=\"blue\")\n",
    "s1 = plt.scatter(c1_samples[..., 0], c1_samples[..., 1], color=\"orange\")\n",
    "plt.legend([s0, s1], [\"c0\", \"c1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2) Effect of different Batch Sizes and Number of Iterations (1. Point)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play around with the batch size and number of iterations and briefly describe your observations about convergence speed and monotonicity  of the loss curve."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}