ML_Maschinelles_Lernen_Grundlagen/Exercise_1
1
0
Fork 0
Code Issues Pull Requests Projects Releases 1 Wiki Activity

2.1 fertig

Browse Source
This commit is contained in:
Pascal Schindler 2021-11-13 18:30:06 +01:00
parent 9cca6dc1df
commit 81b5f5aac1
1 changed files with 1 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

57
exercise.ipynb
View File

@ -681,10 +681,6 @@
" \\log\\text{lik}(\\boldsymbol{\\theta};D) \n",
" &= \\sum_i \\log p(x_i)\\\\\n",
" &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu}) \\\\\n",
" &= \\text{DELTE HERE AFTER}\\\\\n",
" &= \\sum_i \\log(1) - \\log(\\sqrt{\\det(2\\pi\\Sigma)})- \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\\\\n",
" &= \\sum_i^N - \\frac{1}{2}\\log(\\det(2\\pi\\Sigma))- \\dfrac{(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})}{2}\\\\\n",
" &= - \\frac{N}{2}\\log(\\det(2\\pi\\Sigma))- \\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
"\\end{align}"
]
},
@ -702,23 +698,6 @@
"\\begin{align}\n",
" \\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial\\mu}\n",
" &= \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
" &= \\text{MYYY SOLLLTUIONNNNNN} \\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}^T-\\boldsymbol{\\mu}^T) \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} - \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu} - \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} + \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}+ \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
" &= -\\dfrac{1}{2}\\sum_i^N -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + 2\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
" &= -\\sum_i^N -\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1} + \\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
" &= -\\boldsymbol{\\Sigma}^{-1}\\sum_i^N -\\boldsymbol{x_i}^T + \\boldsymbol{\\mu}\\\\\n",
"\\end{align}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\begin{align}\n",
" \\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial\\mu}\n",
" &= \\frac{\\partial}{\\partial\\mu}\\left(- \\frac{N}{2}\\log(\\det(2\\pi\\Sigma))- \\dfrac{1}{2}\\sum_i^N(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right)\\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}^T-\\boldsymbol{\\mu}^T) \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} - \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu} - \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} + \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
" &= \\frac{\\partial}{\\partial\\mu}-\\dfrac{1}{2}\\sum_i^N \\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{x_i} -2\\boldsymbol{x_i}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}+ \\boldsymbol{\\mu}^T\\boldsymbol{\\Sigma}^{-1}\\boldsymbol{\\mu}\\\\\n",
@ -753,8 +732,7 @@
"metadata": {},
"source": [
"\\begin{align}\n",
"\\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial \\boldsymbol{\\Sigma}} &= \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu}) \\\\\n",
"&= \\text{MMMYYY SOLUTTION} \\\\\n",
"\\frac{\\partial\\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial \\boldsymbol{\\Sigma}} &= \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\left(\\log(-\\frac{NK}{2}(2\\pi))-\\frac{N}{2}\\log(|\\Sigma|) - \\frac{1}{2} \\sum_i^{N}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right) \\\\\n",
"&= -\\frac{N}{2}\\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\log(|\\Sigma|) - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
"\\end{align}\n",
"Because $(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})$ is scalar, we can take its trace and obtain the following form \n",
@ -774,39 +752,6 @@
"\\end{align}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\begin{align}\n",
" \\frac{\\partial \\log\\text{lik}(\\boldsymbol{\\theta};D)}{\\partial \\boldsymbol{\\Sigma}} \n",
" &= \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\left(-\\frac{N}{2}\\log(\\det(2\\pi\\Sigma))- \\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right)\\\\\n",
" &= -\\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\frac{N}{2}\\log(\\det(2\\pi\\Sigma)) - \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
" &= -\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}\\dfrac{1}{2}\\sum_i^N (\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
" &= -\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\\\\n",
"\\end{align}\n",
"Because $(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})$ is scalar, we can take its trace and obtain the following form \n",
"\\begin{align}\n",
" &= -\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N \\frac{\\partial}{\\partial \\boldsymbol{\\Sigma}} \\text{Tr}\\left((\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1} (\\boldsymbol{x_i}-\\boldsymbol{\\mu})\\right)\\\\\n",
"\\end{align}\n",
"The rule on page 10 (formula 63) of the matrix cookbook allows us to derive the trace\n",
"\\begin{align}\n",
" &= -\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T\\\\\n",
"\\end{align}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$-\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T = 0$ to optain the optimum:\n",
"\n",
"\\begin{align}\n",
" 0 &= -\\frac{N}{2}(2\\pi\\Sigma)^{-1} - \\dfrac{1}{2}\\sum_i^N -\\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T\\\\\n",
" &= N(2\\pi\\Sigma)^{-1} - \\sum_i^N \\left(\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{x_i}-\\boldsymbol{\\mu})(\\boldsymbol{x_i}-\\boldsymbol{\\mu})^T \\boldsymbol{\\Sigma}^{-1}\\right)^T\\\\\n",
"\\end{align}"
]
},
{
"cell_type": "markdown",
"metadata": {},