added all missing parts to the report

.gitignore

@@ -1,3 +1,5 @@
+experimentation.py
+
 # ---> TeX
 ## Core latex/pdflatex auxiliary files:
 *.aux
@@ -23,7 +25,7 @@
 # *.pdf
 
 ## Generated if empty string is given at "Please type another file name for output:"
-*.pdf
+LaTeX/Final_Project_Loedige.pdf
 
 ## Bibliography auxiliary files (bibtex/biblatex/biber):
 *.bbl

LaTeX/Final_Project_Loedige.tex

@@ -8,13 +8,51 @@
 
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
-\usepackage{listings}
 \usepackage{xcolor}
 \usepackage{hyperref}
 \usepackage{multicol}
 \usepackage{setspace}
 \usepackage{graphicx}
+\usepackage{float}
 \usepackage{xurl}
+\usepackage{cleveref}
+\usepackage{listings}
+\definecolor{mGreen}{rgb}{0,0.6,0}
+\definecolor{mGray}{rgb}{0.5,0.5,0.5}
+\definecolor{mPurple}{rgb}{0.58,0,0.82}
+\definecolor{backgroundColour}{rgb}{0.95,0.95,0.92}
+\lstset{
+    language=python,
+    backgroundcolor=\color{backgroundColour},   
+    commentstyle=\color{mGreen},
+    keywordstyle=\color{magenta},
+    numberstyle=\tiny\color{mGray},
+    stringstyle=\color{mPurple},
+    basicstyle=\ttfamily\scriptsize,
+    breakatwhitespace=false,         
+    breaklines=true,                 
+    captionpos=b,                    
+    keepspaces=true,                 
+    numbers=left,                    
+    firstnumber=0,
+    stepnumber=1,
+    numbersep=5pt,                  
+    showspaces=false,                
+    showstringspaces=false,
+    showtabs=false,                  
+    tabsize=2,
+    literate={~}{{$\mathtt{\sim}$}}1
+}
+\lstset{literate=%
+    {Ö}{{\"O}}1
+    {Ä}{{\"A}}1
+    {Ü}{{\"U}}1
+    {ß}{{\ss}}2
+    {ü}{{\"u}}1
+    {ä}{{\"a}}1
+    {ö}{{\"o}}1
+}
+
 
 % allow deeeep lists
 \usepackage{enumitem}
@@ -130,48 +168,85 @@ The Street View House Numbers (SVHN) dataset was created at Stanford University
 and 531\,131 additional, somewhat less difficult samples, to use as extra training data" (\url{http://ufldl.stanford.edu/housenumbers/}).
 The dataset provided by TensorFlow uses the "MNIST-like 32-by-32 images centered around a single character" version.
 
+\section{Methods}%
+\label{sec:Methods}
+\subsection{Dense Neural Network}%
+\label{sub:Dense Neural Network}
+The first method tried to find a good classification model was to use regular Dense Neural Networks.
+Through experimentation it was determined that the classification accuracy only varied slightly when using a different amount of hidden layers (see \cref{sub:Dense Neural Network Comparison}).
+
+\subsection{Convolutional Neural Network (CNN)}%
+\label{sub:Convolutional Neural Network}
+When it comes to image classification CNNs are often used as they reduce the dimensionality of the input.
+This in turn simplifies the problem and makes it easier to classify.
+Multiple configurations for CNNs where tried (see \cref{sub:CNN Comparison})
+The best CNN found during experimentation had the following configuration:
+\lstinputlisting[firstline=90, lastline=108]{./exported_code.py}
 \clearpage
 
-\section{Code}%
-\label{sec:Code}
+\section{Comparison}%
+\label{sec:Comparison}
+While the Dense Neural Networks can be trained efficiently trained (a few seconds per epoch),
+they are also less accurate than the CNNs.
+As can be seen in \cref{fig:val_acc all models} the Dense Neural Networks achieved a validation accuracy of around 70 \%.
+The CNNs on the other hand achieved around 80 -- 85 \%.
+
+It can also be seen that the type of model has a bigger impact on the accuracy than its complexity.
+Even the CNN with just one convolutional layer performs better than the Dense neural network with 5 hidden dense layers.
+
+\subsection{VGG16}%
+\label{sub:VGG16}
+As can be seen from the comparison of the two previous methods CNNs hold a clear advantage when it comes to image classification.
+In order to find an even better model for classification it therefore makes sense to look at CNNs developed by other researchers.
+VGG16 is one CNN that is commonly used for these kind of tasks and is also available as part of the texttt{tensorflow.keras} library.
+While the attempt at transfer learning was unsuccessful (see \cref{sub:Transfer Learning VGG16}),
+training the whole model pre-trained on the "Imagenet" dataset with a learning rate of $0.001$ produced a model with even higher accuracy than the CNNs mentioned before.
+While the training of this model is very time intensive it gave a validation accuracy of over 92 \% (\cref{fig:val_acc all models}).
+
+\begin{figure}[H]
+    \centering
+    \includegraphics[width=\textwidth]{images/overall_val_acc.pdf}
+    \caption{Validation Accuracy of the different models}
+    \label{fig:val_acc all models}
+\end{figure}
+
+\section{Summary}%
+\label{sec:Summary}
+In this project it has been shown that Convolutional Neural Networks outperform Dense Neural Networks when it comes to classifying the SVHN dataset.
+Additionally, if the goal is to achieve the highest accuracy possible one should use a pre-existing model designed for solving problems in a similar context.
+
+\clearpage
+\appendix
+\section{Appendix}%
+\label{sec:Appendix}
+
+\subsection{Dense Neural Network Comparison}%
+\label{sub:Dense Neural Network Comparison}
+\begin{center}
+    \includegraphics[width=.8\textwidth]{images/dense_neural_net_acc.pdf}\\
+    \includegraphics[width=.8\textwidth]{images/dense_neural_net_val_acc.pdf}
+\end{center}
+
+
+\subsection{CNN Comparison}%
+\label{sub:CNN Comparison}
+\begin{center}
+    \includegraphics[width=.8\textwidth]{images/cnn_acc.pdf}
+    \includegraphics[width=.8\textwidth]{images/cnn_val_acc.pdf}
+\end{center}
+
+\subsection{Transfer Learning VGG16}%
+\label{sub:Transfer Learning VGG16}
+\begin{center}
+    \includegraphics[width=.7\textwidth]{images/transfer_learning_acc.pdf}
+    \includegraphics[width=.7\textwidth]{images/transfer_learning_val_acc.pdf}
+\end{center}
+
+\subsection{Code}%
+\label{sub:Code}
 
 The following code is an exported and slightly formatted version of a Jupyter Notebook that is also available at \url{https://git.ploedige.com/Intelligent_World_Informatics_V/Final_Project}.
 
-\definecolor{mGreen}{rgb}{0,0.6,0}
-\definecolor{mGray}{rgb}{0.5,0.5,0.5}
-\definecolor{mPurple}{rgb}{0.58,0,0.82}
-\definecolor{backgroundColour}{rgb}{0.95,0.95,0.92}
-\lstset{
-    language=python,
-    backgroundcolor=\color{backgroundColour},   
-    commentstyle=\color{mGreen},
-    keywordstyle=\color{magenta},
-    numberstyle=\tiny\color{mGray},
-    stringstyle=\color{mPurple},
-    basicstyle=\ttfamily\scriptsize,
-    breakatwhitespace=false,         
-    breaklines=true,                 
-    captionpos=b,                    
-    keepspaces=true,                 
-    numbers=left,                    
-    firstnumber=0,
-    stepnumber=1,
-    numbersep=5pt,                  
-    showspaces=false,                
-    showstringspaces=false,
-    showtabs=false,                  
-    tabsize=2,
-    literate={~}{{$\mathtt{\sim}$}}1
-}
-\lstset{literate=%
-    {Ö}{{\"O}}1
-    {Ä}{{\"A}}1
-    {Ü}{{\"U}}1
-    {ß}{{\ss}}2
-    {ü}{{\"u}}1
-    {ä}{{\"a}}1
-    {ö}{{\"o}}1
-}
 
 \lstinputlisting{./exported_code.py}
 

LaTeX/exported_code.py

@@ -5,7 +5,7 @@ from tensorflow.keras import layers, models, applications, optimizers
 import tensorflow_datasets as tfds
 import tkinter.messagebox # for notifications
 
-#  Load Dataset
+# Load Dataset
 # https://www.tensorflow.org/datasets/keras_example
 # split into training and test data
 ds_train, ds_test = tfds.load(
@@ -42,54 +42,32 @@ configs = {}
 configs['regular Neural Network (3 hidden layers)'] = {
         'model': models.Sequential([
             layers.Flatten(input_shape=(32, 32, 3)),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
+            layers.Dense(256, activation='relu'),
+            layers.Dense(128, activation='relu'),
             layers.Dense(64, activation='relu'),
             layers.Dense(10, activation='softmax')
         ]),
         'optimizer': 'adam',
         'loss': 'sparse_categorical_crossentropy',
         'metrics': ['accuracy'],
-        'epochs': 10,
+        'epochs': 30,
         'batch_size': 64
     }
 
 configs['regular Neural Network (5 hidden layers)'] = {
         'model': models.Sequential([
             layers.Flatten(input_shape=(32, 32, 3)),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
+            layers.Dense(1024, activation='relu'),
+            layers.Dense(512, activation='relu'),
+            layers.Dense(256, activation='relu'),
+            layers.Dense(128, activation='relu'),
             layers.Dense(64, activation='relu'),
             layers.Dense(10, activation='softmax')
         ]),
         'optimizer': 'adam',
         'loss': 'sparse_categorical_crossentropy',
         'metrics': ['accuracy'],
-        'epochs': 10,
-        'batch_size': 64
-    }
-
-configs['regular Neural Network (10 hidden layers)'] = {
-        'model': models.Sequential([
-            layers.Flatten(input_shape=(32, 32, 3)),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(64, activation='relu'),
-            layers.Dense(10, activation='softmax')
-        ]),
-        'optimizer': 'adam',
-        'loss': 'sparse_categorical_crossentropy',
-        'metrics': ['accuracy'],
-        'epochs': 10,
+        'epochs': 30,
         'batch_size': 64
     }
 
@@ -132,7 +110,6 @@ configs['convolutional neural network (3 convolutions)'] = {
 vgg16_base_model = applications.VGG16(input_shape=(32,32,3), include_top= False)
 # freeze the VGG16 model
 vgg16_base_model.trainable = False
-
 configs['frozen VGG16 base model'] = {
         'model': models.Sequential([
             vgg16_base_model,
@@ -229,28 +206,4 @@ plt.title("Validation Accuracy")
 plt.show()
 
 # Notify when done
-tkinter.messagebox.showinfo("DONE", "DONE")
-
-# Save data for future use
-# store histories with pickle
-import pickle
-histories = {}
-for config_name, config in configs.items():
-    histories[config_name] = config['history']
-with open('savepoint.pkl', 'wb') as out:
-    pickle.dump(histories, out)
-
-# store models with tensorflow
-for config_name, config in configs.items():
-    config['model'].save(f'saved_models/{config_name}')
-
-# store models in HDF5 format
-for config_name, config in configs.items():
-    config['model'].save(f'hdf5_models/{config_name}.h5')
-
-# Test Save File
-new_model = tf.keras.models.load_model('hdf5_models/fully trainable VGG16 base model.h5')
-print(new_model.summary())
-# Evaluate the restored model
-loss, acc = new_model.evaluate(ds_test)
-print('Restored model, accuracy: {:5.2f}%'.format(100 * acc))
+tkinter.messagebox.showinfo("DONE", "DONE")