diff --git a/exercise.ipynb b/exercise.ipynb index 1835d22..8f4f300 100644 --- a/exercise.ipynb +++ b/exercise.ipynb @@ -53,6 +53,21 @@ "\\end{align*}\n" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$ L = (y-\\Phi w)^T (y - \\Phi w) + \\lambda w^T I w $$\n", + "$$ = w ^T \\Phi^T\\Phi w - y^T \\Phi w - w^T\\Phi^Ty + y^Ty + \\lambda w^T I w $$\n", + "$$ = w^T\\Phi^T\\Phi w - 2 y^T\\Phi w + y^Ty + \\lambda w^T I w $$\n", + "$$ = y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w $$\n", + "$$ \\frac{\\delta x^TAx}{\\delta x} = (A + A^T)x: \\frac{\\delta w^T(\\Phi^T\\Phi + \\lambda I)w}{\\delta w} = ((\\Phi^T\\Phi + \\lambda I) + (\\Phi\\Phi^T + \\lambda I^T))w = 2(\\Phi^T\\Phi + \\lambda I)w $$\n", + "$$ \\frac{\\delta y^Ty - 2y\\Phi^Tw^T + w^T(\\Phi^T\\Phi + \\lambda I)w }{\\delta w} = -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w $$\n", + "$$ -2 \\Phi^Ty + 2(\\Phi^T\\Phi + \\lambda I)w = 0, (\\text{to find optimum}) $$\n", + "$$ (\\Phi^T\\Phi + \\lambda I)w = \\Phi^Ty $$\n", + "$$ w^* = (\\Phi^T\\Phi + \\lambda I )^{-1} \\Phi^Ty $$" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -231,7 +246,8 @@ "The following function will calculate the weights for ridge regression. Fill in the missing code according to the formula for calculating the weight updates for ridge regression.
\n", "Recall that the formula is given by \n", "\\begin{align*}\n", - " \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n", + " \\boldsymbol{w} &= (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I} )^{-1} \\boldsymbol{\\Phi}^T \\boldsymbol{y}\\\\\n", + " (\\boldsymbol{\\Phi} ^T \\boldsymbol{\\Phi} + \\lambda \\boldsymbol{I}) \\boldsymbol{w} &= \\boldsymbol{\\Phi}^T \\boldsymbol{y},\n", "\\end{align*}\n", "where $\\boldsymbol{\\Phi}$ is the feature matrix (the matrix storing the data points applied to the polynomial features).\n", "Hint: use np.linalg.solve for solving for the linear equation.\n", @@ -256,7 +272,9 @@ " \"\"\"\n", " ##################\n", " ##TODO\n", - " #################" + " #################\n", + " identity_matrix = np.identity(y.shape[0])\n", + " weights = (Phi.transpose() * Phi + ridge_factor * identity_matrix)^-1 * np.linalg.solve(Phi.transpose(), )" ] }, {