From c0af75025b98f3578ab0de17689d5b27d0b88a2f Mon Sep 17 00:00:00 2001
From: paul-loedige <paul@gve-loedige.de>
Date: Wed, 10 Nov 2021 10:58:28 +0100
Subject: [PATCH] 1.2 and 1.3 finished

---
 exercise.ipynb | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/exercise.ipynb b/exercise.ipynb
index 1835d22..8f4f300 100644
--- a/exercise.ipynb
+++ b/exercise.ipynb
@@ -53,6 +53,21 @@
     "\\end{align*}\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$ L = (y-\\Phi w)^T (y - \\Phi w) + \\lambda w^T I w $$\n",
+    "$$ = w ^T \\Phi^T\\Phi w - y^T \\Phi w - w^T\\Phi^Ty + y^Ty + \\lambda w^T I w $$\n",
+    "$$ = w^T\\Phi^T\\Phi w - 2 y^T\\Phi w + y^Ty + \\lambda w^T I w $$\n",
+    "$$ = y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w $$\n",
+    "$$ \\frac{\\delta x^TAx}{\\delta x} = (A + A^T)x:  \\frac{\\delta w^T(\\Phi^T\\Phi + \\lambda I)w}{\\delta w} = ((\\Phi^T\\Phi + \\lambda I) + (\\Phi\\Phi^T + \\lambda I^T))w = 2(\\Phi^T\\Phi + \\lambda I)w $$\n",
+    "$$ \\frac{\\delta y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w }{\\delta w} = -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w $$\n",
+    "$$ -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w = 0, (\\text{to find optimum}) $$\n",
+    "$$ (\\Phi^T\\Phi + \\lambda I)w = \\Phi^Ty $$\n",
+    "$$ w^* = (\\Phi^T\\Phi + \\lambda I )^{-1} \\Phi^Ty $$"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -231,7 +246,8 @@
     "The following function will calculate the weights for ridge regression. Fill in the missing code according to the formula for calculating the weight updates for ridge regression. <br>\n",
     "Recall that the formula is given by \n",
     "\\begin{align*}\n",
-    "    \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n",
+    "    \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y}\\\\\n",
+    "    (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I}) \\boldsymbol{w} &= \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n",
     "\\end{align*}\n",
     "where $\\boldsymbol{\\Phi}$ is the feature matrix (the matrix storing the data points applied to the polynomial features).\n",
     "Hint: use np.linalg.solve for solving for the linear equation.\n",
@@ -256,7 +272,9 @@
     "    \"\"\"\n",
     "    ##################\n",
     "    ##TODO\n",
-    "    #################"
+    "    #################\n",
+    "    identity_matrix = np.identity(y.shape[0])\n",
+    "    weights = (Phi.transpose() * Phi + ridge_factor * identity_matrix)^-1 * np.linalg.solve(Phi.transpose(), )"
    ]
   },
   {