diff --git a/lab09/lab_solved.ipynb b/lab09/lab_solved.ipynb new file mode 100644 index 0000000..e0a51e4 --- /dev/null +++ b/lab09/lab_solved.ipynb @@ -0,0 +1,1574 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "ExecuteTime": { + "end_time": "2020-03-31T12:22:16.372625Z", + "start_time": "2020-03-31T12:22:16.345534Z" + } + }, + "source": [ + "EE-311\n", + "======\n", + "\n", + "Lab 9: Convolutional Neural Networks\n", + "------------------------------------\n", + "\n", + "created by Francois Marelli, Arnaud Pannatier and Roberto Boghetti on 05.04.2022\n", + "\n", + "In this lab, we will illustrate the basics of PyTorch, an open source framework for deep learning.\n", + "\n", + "We will also have a look at Convolutional Neural Networks (CNNs), and see why the convolution operation is very useful when working on images." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Import libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "ExecuteTime": { + "end_time": "2020-04-28T07:21:13.343695Z", + "start_time": "2020-04-28T07:21:13.318856Z" + } + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "\n", + "from tqdm.notebook import trange\n", + "\n", + "from sklearn.model_selection import train_test_split\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction to PyTorch and Tensors\n", + "\n", + "PyTorch is an open source machine learning framework that implements efficient algorithms for deep learning. It shares many similarities with NumPy, and in this section we will give a brief overview of the basic functionalities needed in this lab.\n", + "\n", + "Instead of arrays, PyTorch uses Tensors to store data. They are very similar to NumPy's ndarrays, with a few additional functionalities. Mainly, Tensors can be moved to GPU and implement automatic differentiation (autograd).\n", + "\n", + "### Creating Tensors\n", + "\n", + "Tensors can be created in similar ways to NumPy arrays:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tensor from list:\n", + " tensor([[1., 2., 3.],\n", + " [4., 5., 6.]])\n", + "\n", + "Zero tensor:\n", + " tensor([[0., 0., 0.],\n", + " [0., 0., 0.]])\n", + "\n", + "Empty tensor:\n", + " tensor([[0., 0., 0.],\n", + " [0., 0., 0.]])\n" + ] + } + ], + "source": [ + "# From a list\n", + "a_tensor = torch.Tensor([[1, 2, 3], [4, 5, 6]])\n", + "\n", + "print('Tensor from list:\\n', a_tensor)\n", + "\n", + "# Full of zeros\n", + "zeros = torch.zeros(2, 3)\n", + "\n", + "print('\\nZero tensor:\\n', zeros)\n", + "\n", + "# Empty\n", + "empty = torch.empty(2, 3)\n", + "\n", + "print('\\nEmpty tensor:\\n', empty)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "PyTorch and NumPy can be interfaced easily. It is possible to go from array to Tensor, and back." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tensor from array:\n", + " tensor([[1, 2, 3],\n", + " [4, 5, 6]])\n", + "\n", + "Array from Tensor:\n", + " [[1 2 3]\n", + " [4 5 6]]\n" + ] + } + ], + "source": [ + "# NumPy array\n", + "a_array = np.array([[1, 2, 3], [4, 5, 6]])\n", + "\n", + "# Create Tensor from array\n", + "a_tensor = torch.from_numpy(a_array)\n", + "\n", + "print('Tensor from array:\\n', a_tensor)\n", + "\n", + "# Convert Tensor to array\n", + "a_array = a_tensor.numpy()\n", + "print('\\nArray from Tensor:\\n', a_array)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Tensor properties\n", + "\n", + "Just like arrays, Tensors are characterized by their `shape` and `type`.\n", + "\n", + "Note that the number of elements in a Tensor is given by `nelement`, which is a function." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Shape: torch.Size([2, 3])\n", + "Type: torch.int64\n", + "Elements: 6\n" + ] + } + ], + "source": [ + "print('Shape:', a_tensor.shape)\n", + "\n", + "print('Type:', a_tensor.dtype)\n", + "\n", + "print('Elements:', a_tensor.nelement())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Operation over Tensors\n", + "\n", + "Mathematical operations over Tensors are very similar to NumPy. Standard operators apply element-wise.\n", + "\n", + "Some operations require the type to be float (conversion is not automatic). Casting a Tensor to a different type is quite simple." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Sum:\n", + " tensor([[3., 3., 3.],\n", + " [3., 3., 3.]])\n", + "\n", + "Product:\n", + " tensor([[2., 2., 2.],\n", + " [2., 2., 2.]])\n", + "\n", + "Dot:\n", + " tensor([[6., 6.],\n", + " [6., 6.]])\n" + ] + } + ], + "source": [ + "shape = 2, 3\n", + "\n", + "ones = torch.ones(shape)\n", + "twos = torch.full(shape, 2)\n", + "\n", + "sum_result = ones + twos\n", + "prod_result = ones * twos\n", + "\n", + "# Type conversion to float\n", + "ones = ones.float()\n", + "twos = twos.float()\n", + "\n", + "dot_result = ones @ twos.T\n", + "\n", + "print('Sum:\\n', sum_result)\n", + "print('\\nProduct:\\n', prod_result)\n", + "print('\\nDot:\\n', dot_result)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As with NumPy, it is easy to compute the sum or mean of a Tensor.\n", + "\n", + "If needed, they can be converted back to a scalar by using the `item` function." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Sum: tensor(21.)\n", + "Mean: tensor(3.5000)\n", + "\n", + "Scalar mean: 3.5\n" + ] + } + ], + "source": [ + "a_tensor = a_tensor.float()\n", + "\n", + "print('Sum:', a_tensor.sum())\n", + "print('Mean:', a_tensor.mean())\n", + "\n", + "print('\\nScalar mean:', a_tensor.mean().item())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Indexing\n", + "\n", + "Indexing and slicing is exactly the same in PyTorch as in NumPy.\n", + "\n", + "Adding an empty dimension can be done using `None`." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tensor:\n", + "tensor([[1., 2., 3.],\n", + " [4., 5., 6.]])\n", + "\n", + "Slice of the Tensor:\n", + "tensor([[1., 3.],\n", + " [4., 6.]])\n", + "\n", + "Added dimension: torch.Size([2, 1, 3])\n" + ] + } + ], + "source": [ + "a_tensor = torch.Tensor([[1, 2, 3], [4, 5, 6]])\n", + "\n", + "print('Tensor:')\n", + "print(a_tensor)\n", + "\n", + "print('\\nSlice of the Tensor:')\n", + "print(a_tensor[:, 0:3:2])\n", + "\n", + "b_tensor = a_tensor[:, None, :]\n", + "print('\\nAdded dimension:', b_tensor.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Automatic differentiation with autograd\n", + "\n", + "One of the main features of PyTorch is autograd. This provides automatic differentiation, which is very useful for backpropagation!\n", + "\n", + "Instead of computing all the gradients manually like last week, we let autograd do the work.\n", + "\n", + "Can you manually check the following result?\n", + "\n", + "*$\\frac{\\partial y}{\\partial x} = 2\\ ( 3\\ ( x + 4 ) )\\ 3 = 18 x + 72 = 36 + 72 = 108$*" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "x: 2.0\n", + "\n", + "Auto gradient dy/dx: 108.0\n" + ] + } + ], + "source": [ + "# Specify that x wants the gradient to be computed\n", + "x = torch.full((1,), 2,dtype=float, requires_grad=True)\n", + "\n", + "print('x:', x.item())\n", + "\n", + "y = ((x + 4) * 3) ** 2\n", + "\n", + "# Ask autograd to compute the gradients\n", + "y.backward()\n", + "\n", + "print('\\nAuto gradient dy/dx:', x.grad.item())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Coding an MLP with PyTorch\n", + "\n", + "Let us have a look at how we can implement and train a multi-layer perceptron in PyTorch in only a few steps. In this example, we will re-implement the multi-layer perceptron (MLP) seen in the previous lab for multi-class classification:\n", + "\n", + "$$x^{(0)} \\xrightarrow{W^{(1)},b^{(1)}} s^{(1)} \\xrightarrow{\\sigma} x^{(1)} \\xrightarrow{W^{(2)},b^{(2)}} s^{(2)}$$\n", + "$$x^{(0)} \\xrightarrow{Linear_1} s ^{(1)} \\xrightarrow{tanh} x^{(1)} \\xrightarrow{Linear_2} s^{(2)}$$\n", + "\n", + "As you can see, there is a little difference here: we do not have a sigmoid at the output anymore! This is because we will use a different loss that is better suited for classification tasks: cross entropy. Since we do not have to compute gradients manually, we are not limited to simple losses like MSE.\n", + "\n", + "PyTorch provides a collection of classes and functions, contained in `torch.nn` ([doc](https://pytorch.org/docs/stable/nn.html)) that allows us to write our code in an efficient and elegant way. We start by creating a MLP class that inherits from `torch.nn.Module` ([doc](https://pytorch.org/docs/stable/generated/torch.nn.Module.html)). The class is composed of two parts:\n", + "\n", + "The standard way of building models in PyTorch is to define a class that inherits from `torch.nn.Module` ([doc](https://pytorch.org/docs/stable/generated/torch.nn.Module.html)), which is the base class for constructing all neural networks. \n", + "\n", + "The main elements of our computational graph must be created in the `__init__()` function of our class. In our case, these include the two [Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layers and the [hyperbolic tangent](https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html) activation function. When instantiating the class `Linear` we need to specify the input and output dimesions. For the first layer the input dimension is the number of features in our data (2), while the output dimension is an arbitrary number of hidden neurons (50); for the second layer the input dimension is equal to the output of the previous layer, while the output dimension is the number of possible classes (3).\n", + "\n", + "The forward pass is then defined with the method `forward`, which defines the order of computations on the input data `x`.\n", + "\n", + "**Reminder about classes:**\n", + "* the `self` keyword represents the current object when used in a class\n", + "* the `__init__` function is called when an object is created\n", + "* the `super().__init__()` calls the `__init__` function of the parent class, which `nn.Module` in this case\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "class MLP(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " \n", + " # Create the two linear layers\n", + " self.linear1 = nn.Linear(2, 50)\n", + "\n", + " self.linear2 = nn.Linear(50, 3)\n", + " \n", + " # And the activation\n", + " self.activation1 = nn.Tanh()\n", + " \n", + " def forward(self, x):\n", + " \n", + " # L1 -> s1 -> L2\n", + " x = self.linear1(x)\n", + "\n", + " x = self.activation1(x)\n", + "\n", + " x = self.linear2(x)\n", + " \n", + " return x" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once the model architecture is defined, we can instantiate the class and print the properties of the obtained object." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "MLP(\n", + " (linear1): Linear(in_features=2, out_features=50, bias=True)\n", + " (linear2): Linear(in_features=50, out_features=3, bias=True)\n", + " (activation1): Tanh()\n", + ")\n" + ] + } + ], + "source": [ + "# Instantiate an object of class MLP\n", + "model = MLP()\n", + "\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will train this MLP on the same dataset as last week." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "ename": "FileNotFoundError", + "evalue": "[Errno 2] No such file or directory: 'data.npz'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[11], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m data \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mload(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdata.npz\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 3\u001b[0m circle_X \u001b[38;5;241m=\u001b[39m data[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcircle_X\u001b[39m\u001b[38;5;124m'\u001b[39m]\n\u001b[1;32m 4\u001b[0m circle_y \u001b[38;5;241m=\u001b[39m data[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcircle_y\u001b[39m\u001b[38;5;124m'\u001b[39m]\n", + "File \u001b[0;32m/usr/local/Caskroom/mambaforge/base/envs/pyee311/lib/python3.11/site-packages/numpy/lib/npyio.py:405\u001b[0m, in \u001b[0;36mload\u001b[0;34m(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)\u001b[0m\n\u001b[1;32m 403\u001b[0m own_fid \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m 404\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 405\u001b[0m fid \u001b[38;5;241m=\u001b[39m stack\u001b[38;5;241m.\u001b[39menter_context(\u001b[38;5;28mopen\u001b[39m(os_fspath(file), \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mrb\u001b[39m\u001b[38;5;124m\"\u001b[39m))\n\u001b[1;32m 406\u001b[0m own_fid \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m 408\u001b[0m \u001b[38;5;66;03m# Code to distinguish from NumPy binary files and pickles.\u001b[39;00m\n", + "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data.npz'" + ] + } + ], + "source": [ + "data = np.load('data.npz')\n", + "\n", + "circle_X = data['circle_X']\n", + "circle_y = data['circle_y']\n", + "\n", + "plt.figure(figsize=(7, 7))\n", + "plt.scatter(circle_X[:,0], circle_X[:,1], c=circle_y)\n", + "plt.axis('square')\n", + "plt.grid()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As always, we start by preprocessing the dataset. We split it in train and test.\n", + "\n", + "Note that we do not need one-hot encoding like last week, as we use cross entropy instead of MSE.\n", + "\n", + "To be rigorous, we can only use the train set to compute normalization parameters.\n", + "\n", + "For use with PyTorch, we then convert the data to float Tensors." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train set shape: torch.Size([800, 2])\n" + ] + } + ], + "source": [ + "# Split in train and test sets\n", + "train_input, test_input, train_target, test_target = train_test_split(circle_X, circle_y, test_size=0.2, random_state=0)\n", + "\n", + "# Normalize input \n", + "mu, std = train_input.mean(0), train_input.std(0)\n", + "train_input = (train_input - mu) / std\n", + "test_input = (test_input - mu) / std\n", + "\n", + "# Convert the sets to Tensors\n", + "train_input = torch.from_numpy(train_input).float()\n", + "test_input = torch.from_numpy(test_input).float()\n", + "\n", + "train_target = torch.from_numpy(train_target)\n", + "test_target = torch.from_numpy(test_target)\n", + "\n", + "print('Train set shape:', train_input.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Training the model\n", + "\n", + "Before training the model, we must still instantiate an optimizer and a loss function.\n", + "\n", + "The optimizer will take care of implementing backpropagation, in this case we will use a simple Stochastic Gradient Descent (SGD).\n", + "\n", + "When creating the optimizer, we tell it that it must update all the parameters of our model, and we set the learning rate for gradient descent." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "optimizer = optim.SGD(model.parameters(), lr=1e-1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For the loss function, we use cross entropy as it is very efficient for classification problems. We do not need to specify any additional parameters here." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "criterion = nn.CrossEntropyLoss()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, we define the size of the training batches. Each data batch contains a limited number of samples instead of the full dataset.\n", + "\n", + "Using batches for training allows to reduce the memory usage and speed up the computations.\n", + "\n", + "In this example, we define the batch to contain 50 samples.\n", + "\n", + "We also create an empty list that we will use to save and plot the training loss." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "batch_size = 50\n", + "\n", + "train_loss = []" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is finally time to implement our training loop!\n", + "\n", + "The outer loop iterates over epochs: each time we go over the full training dataset.\n", + "\n", + "The inner loop iterates over the batches in the dataset.\n", + "\n", + "**The core of the loops is the following process:**\n", + "\n", + "1. We reset the gradients of the optimizer as we don't want to accumulate them over time\n", + "\n", + "2. We compute predictions using the forward pass on the batch\n", + "\n", + "3. We compute the loss between the predictions and the targets\n", + "\n", + "4. We compute the gradient of the loss\n", + "\n", + "5. We update the parameters of our model\n", + "\n", + "At the end of each epoch, we save the training loss to generate the final graph. It shows if the network is learning correctly.\n", + "\n", + "Note that if you run the following cell multiple times without creating a new model (and optimizer), the training will continue from where it stopped." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "7a5f653ace5d43b4a51d9837a8c46a79", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Epochs: 0%| | 0/1000 [00:00" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "torch.manual_seed(42)\n", + "\n", + "for epoch in trange(1000, desc='Epochs'):\n", + " epoch_loss = 0\n", + "\n", + " for batch_idx in range(train_input.shape[0] // batch_size):\n", + "\n", + " # Create the batch\n", + " X_batch = train_input[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + " y_batch = train_target[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + " \n", + " # 1. Reset the gradients\n", + " optimizer.zero_grad()\n", + "\n", + " # 2. Compute the predictions\n", + " predict = model(X_batch)\n", + "\n", + " # 3. Compute the loss\n", + " loss = criterion(predict, y_batch)\n", + " \n", + " # 4. Compute the loss gradient\n", + " loss.backward()\n", + " \n", + " # 5. Update the model parameters\n", + " optimizer.step()\n", + "\n", + " # Keep track of the total loss\n", + " epoch_loss += loss.item() * X_batch.shape[0]\n", + "\n", + " # Save the total epoch loss, averaged per sample\n", + " train_loss.append(epoch_loss / train_input.shape[0])\n", + "\n", + "\n", + "plt.figure(figsize=(10, 7))\n", + "plt.plot(train_loss)\n", + "plt.title('Training loss')\n", + "plt.xlabel('Epoch')\n", + "plt.ylabel('Loss')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have trained the model, we want to measure its performance.\n", + "\n", + "Because of the architecture of the model and the loss function, our network outputs 3 values for each sample. We must keep only the class with the highest value for our prediction.\n", + "\n", + "Compute the accuracy of the model over the training and test set. Do they match last week's results?" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training accuracy: 88.50%\n", + "Test accuracy: 85.50%\n" + ] + } + ], + "source": [ + "# Code here\n", + "\n", + "predict = model(train_input)\n", + "predict = predict.argmax(1)\n", + "\n", + "accuracy = ((predict == train_target) * 1.0).mean()\n", + "\n", + "predict_t = model(test_input)\n", + "predict_t = predict_t.argmax(1)\n", + "\n", + "accuracy_t = ((predict_t == test_target) * 1.0).mean()\n", + "\n", + "print('Training accuracy: {:.02f}%'.format(accuracy * 100))\n", + "print('Test accuracy: {:.02f}%'.format(accuracy_t * 100))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Convolutional Neural Network - Image Classification\n", + "\n", + "In this section, we are going to investigate a binary classification problem using a CNN.\n", + "\n", + "We have an annotated dataset of images with two classes, of which some examples are shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "ExecuteTime": { + "end_time": "2020-04-30T08:05:29.567476Z", + "start_time": "2020-04-30T08:05:28.367329Z" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAArkAAAHhCAYAAABqa6azAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAn50lEQVR4nO3df7AlZH0m+OdL324aWtSI6CKgaEVNjIOY9DAhpGYTxxFEo+5ka0cT3apZKySTOGUKM65OUkmc7KZSM1UutbVaFomu+WG0HH9MEiMgKSXGBJVGWxRQQ1iMgBlQkd9Cd/PuH/dgLkx333Nv33Pec97+fKq6+t5zT5/36dv99H3uuafPrdZaAABgJMf0DgAAAFvNyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5C64qvrNqvqj3jmA6egsLA99HZuRuwCq6meqak9V3VNV36iqS6rqx3vnSpKqOr2qPlFV91XVl6vqhb0zQW86C8tDX49eRm5nVXVhkouS/HaSJyd5apK3J3l5x1hrvTfJ55OcmORXk3ygqk7qGwn60VlYHvp6dDNyO6qqxyX5j0l+qbX2odbava21fa21P2ut/ftD/Jr/UlX/UFV3VtUnq+qH1rzt/Kq6rqrurqpbqupXJpc/sao+UlXfqapvV9VfVdW6f/ZV9awkP5zkN1pr97fWPpjki0l+eit+/7BsdBaWh75i5PZ1dpKdST68gV9zSZJnJnlSks8lec+at70zyc+31k5I8twkH59c/oYkNyc5Kaufyf6HJC1JqurtVfX2Q5z1Q0lubK3dveayL0wuh6ORzsLy0Nej3ErvAEe5E5N8s7W2f9pf0Fp718MvV9VvJrmjqh7XWrszyb4kz6mqL7TW7khyx+Sq+5KcnORprbUbkvzVmtv7xcMc95gkdz7qsjuTnDJtXhiMzsLy0NejnHty+/pWkidW1VSfbFTVtqr6nar6u6q6K8lNkzc9cfLzTyc5P8nXquovq+rsyeX/OckNST5WVTdW1ZumzHdPksc+6rLHJrn7INeFo4HOwvLQ16OckdvXlUm+m+QVU17/Z7L6YPkXJnlcktMnl1eStNauaq29PKtfZvmvSd4/ufzu1tobWmvPSPJTSS6sqn8xxXnXJnlGVZ2w5rLnTS6Ho5HOwvLQ16OckdvR5Msfv57kbVX1iqo6vqq2V9WLq+o/HeSXnJDkgax+dnp8Vv+3aJKkqnZU1c9OvqyyL8ldSQ5M3vbSqvr+qqo1lx+YIt9Xk+xN8htVtbOq/qckZyT54BH8tmFp6SwsD33FyO2stfbWJBcm+bUktyf5epLXZfWzxEf7gyRfS3JLkuuSfPpRb39NkpsmX2b5hSSvnlz+zCR/kdUvjVyZ5O2ttSuSpKreUVXvOEzEVybZndXHHv1Okv+5tXb7hn6TMBCdheWhr0e3aq31zgAAAFvKPbkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwZvJtfXfUsW1nds3ipqd3/HF9z0+S++7vevzjf2jq72Q4M0/a9mDvCPnq353YO0LuvvfWb7bWTuqd42AWoa/tccd3PX8R1L7+z3TzrO//Vu8IC+Hqax5Y3L4ec1w7buXR3yRrvtq+fV3PT5KHntD336xjvn1v1/OTZNdz+v+bccpK352THL6vMxm5O7Mr/2yqb/YxO/Xcf9L1/CRpV32x6/kv+0D/D1i/9Piv946Qc//V/9o7Qv7iyl//Wu8Mh7IIff3u/3hW1/OTpA70/YCx8/bvdj0/SS79kz/sHWEhbDv5hoXt63Erj82PPfmVXTPsv+XWrucnyd3n/mjX809436OfQnf+fuSPH+odIb/95Gt6RzhsXz1cAQCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOFON3Ko6r6q+UlU3VNWbZh0K2Dx9heWhrzA7647cqtqW5G1JXpzkOUleVVXPmXUwYOP0FZaHvsJsTXNP7llJbmit3dhaezDJ+5K8fLaxgE3SV1ge+gozNM3IPSXJ19e8fvPkskeoqguqak9V7dmXB7YqH7Ax+grLY8N9ffCh++cWDpbdNCO3DnJZ++8uaO3i1tru1tru7Tn2yJMBm6GvsDw23Ncdxxw3h1gwhmlG7s1JTlvz+qlJbp1NHOAI6SssD32FGZpm5F6V5JlV9fSq2pHklUn+dLaxgE3SV1ge+goztLLeFVpr+6vqdUkuS7Itybtaa9fOPBmwYfoKy0NfYbbWHblJ0lr7aJKPzjgLsAX0FZaHvsLs+I5nAAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDWZnFjdaxO7LtqU+fxU1P7UB1PT5Jctmte7uef+5Tzux6fpL8yI039Y6QfPqa3gkWWu08NttO//6uGXb+2We7np8k+160u+v5l/7JH3Y9P0le/OJX9Y6QSy55b+8IC63t25f9t9zaNcMxZ/xA1/OTZMfdB7qe/6s37u16fpL88pf+de8I+bkn/E3vCIflnlwAAIZj5AIAMBwjFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOGsO3Kr6l1VdVtVfWkegYAjo7OwPPQVZmeae3LfneS8GecAts67o7OwLN4dfYWZWHfkttY+meTbc8gCbAGdheWhrzA7HpMLAMBwVrbqhqrqgiQXJMnOlcdu1c0CM6CvsDwe0dcc3zkNLI8tuye3tXZxa213a233jm3HbdXNAjPwiL6u+KAJi2xtX7fn2N5xYGl4uAIAAMOZ5inE3pvkyiTPrqqbq+q1s48FbJbOwvLQV5iddR+T21p71TyCAFtDZ2F56CvMjocrAAAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4Ri5AAAMx8gFAGA4K7O52UqO6buf7zv1+K7nJ8lbbn9O7wjd/cYzfqR3hNz+b8/uHSF5+wd6Jzik9t0HcuArN/SO0d2xn7qud4Tu6mu39o6Qn/zffq53hCT/e+8Ah/aY49LOPLNrhIf+em/X85Pk+P3P6nr+pXee0fX8JDn5F+/pHSEvfc0be0dIcuEh3+KeXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4aw7cqvqtKr6RFVdX1XXVtXr5xEM2Dh9heWiszA7K1NcZ3+SN7TWPldVJyS5uqoub61dN+NswMbpKywXnYUZWfee3NbaN1prn5u8fHeS65OcMutgwMbpKywXnYXZ2dBjcqvq9CTPT/KZmaQBtoy+wnLRWdha0zxcIUlSVY9J8sEkv9xau+sgb78gyQVJsnPlsVsWENi4DfU1x885HfBoh+vsI/p67OM6pIPlNNU9uVW1Pavle09r7UMHu05r7eLW2u7W2u4d23zQhF422tftOXa+AYFHWK+zj+jr9l3zDwhLappnV6gk70xyfWvtrbOPBGyWvsJy0VmYnWnuyT0nyWuSvKCq9k5+nD/jXMDm6CssF52FGVn3MbmttU8lqTlkAY6QvsJy0VmYHd/xDACA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOCszudVKsrJtJjc9rf/lty7ten6S/Lvv+1rX88/NmV3PT5Jvvfbs3hGycl/vBKznslv39o6Qm/ff0/X8c5/y413PX3Vn7wC541nbe0dILukd4DDuuT/113t7p+juwHVf7Xr+53/8hK7nJ8klf/vnvSMshG2/fei3uScXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4Ri5AAAMx8gFAGA4Ri4AAMNZd+RW1c6q+mxVfaGqrq2qt8wjGLBx+grLRWdhdlamuM4DSV7QWrunqrYn+VRVXdJa+/SMswEbp6+wXHQWZmTdkdtaa0numby6ffKjzTIUsDn6CstFZ2F2pnpMblVtq6q9SW5Lcnlr7TMzTQVsmr7CctFZmI2pRm5r7UBr7cwkpyY5q6qe++jrVNUFVbWnqvY8eOC+LY4JTGujfd2XB+aeEfhH63VWX2FzNvTsCq217yS5Isl5B3nbxa213a213Tu2Hb816YBNm7av23PsvKMBB3GozuorbM40z65wUlU9fvLycUlemOTLM84FbIK+wnLRWZidaZ5d4eQkv19V27I6it/fWvvIbGMBm6SvsFx0FmZkmmdXuCbJ8+eQBThC+grLRWdhdnzHMwAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4azM4kaf+aw78tHL3j+Lm57auU85s+v5SXLBLTf0jtDdfSdX7wg57f/4m94RFtqzzrgvl122t2uGRejrNy78sa7nn5z+f09XnnF67wh50p57e0dYaA89/vjc/xNndc1w3J98tuv5i+CSv/3r3hEW4t/NN//dNb0jHJZ7cgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwnKlHblVtq6rPV9VHZhkIOHL6CstDX2E2NnJP7uuTXD+rIMCW0ldYHvoKMzDVyK2qU5O8JMnvzTYOcKT0FZaHvsLsTHtP7kVJ3pjkoUNdoaouqKo9VbXn9m8d2IpswOZcFH2FZXFRNtDX/Q/cO7dgsOzWHblV9dIkt7XWrj7c9VprF7fWdrfWdp904rYtCwhMT19heWymryvH7ppTOlh+09yTe06Sl1XVTUnel+QFVfVHM00FbJa+wvLQV5ihdUdua+3NrbVTW2unJ3llko+31l4982TAhukrLA99hdnyPLkAAAxnZSNXbq1dkeSKmSQBtpS+wvLQV9h67skFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABjOSu8As3LM836wd4S87JS+5x9zxg/0DZDkqZfc2TtCWu8AC+7ulnzyu71T9Hfae2/sev595/3TrucnSS69qneCVN8/hoV3zHfuy3F/8tneMbq7+1//aOcEezufvxh+6+f+Te8ISd58yLe4JxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAw1mZ5kpVdVOSu5McSLK/tbZ7lqGAzdNXWC46C7Mx1cid+MnW2jdnlgTYSvoKy0VnYYt5uAIAAMOZduS2JB+rqqur6oKDXaGqLqiqPVW15/ZvHdi6hMBGbaivd+or9HbYzq7t67480CEeLKdpH65wTmvt1qp6UpLLq+rLrbVPrr1Ca+3iJBcnye7n7WxbnBOY3ob6+uwz9BU6O2xn1/b1sfUEfYUpTXVPbmvt1snPtyX5cJKzZhkK2Dx9heWiszAb647cqtpVVSc8/HKSFyX50qyDARunr7BcdBZmZ5qHKzw5yYer6uHr/3Fr7dKZpgI2S19huegszMi6I7e1dmOS580hC3CE9BWWi87C7HgKMQAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4VRrbetvtOr2JF87gpt4YpJvblEcGZb3/JEyPK21dtJWhNlq+irDYBn0dX0j/BnJIMPDDtnXmYzcI1VVe1pru2Xom6H3+TIsh0V4/8ggw6Kcvwx6v496ny/D0ZPBwxUAABiOkQsAwHAWdeRe3DtAZFiE8xMZlsEivH9kWCVD//OXQe/3Ue/zExkeNnSGhXxMLgAAHIlFvScXAAA2baFGblWdV1VfqaobqupNnTK8q6puq6ovdTr/tKr6RFVdX1XXVtXrO2TYWVWfraovTDK8Zd4Z1mTZVlWfr6qPdDr/pqr6YlXtrao9PTIsst6d7d3XSYaundXXR5yvr4ehr/37OsmwEJ09Gvq6MA9XqKptSb6a5F8muTnJVUle1Vq7bs45/nmSe5L8QWvtufM8e3L+yUlObq19rqpOSHJ1klfM8/1QVZVkV2vtnqranuRTSV7fWvv0vDKsyXJhkt1JHttae2mH829Ksru11vt5BBfOInS2d18nGbp2Vl8fcf5N0deD0tfvZfAx9h9zDN/XRbon96wkN7TWbmytPZjkfUlePu8QrbVPJvn2vM9dc/43Wmufm7x8d5Lrk5wy5wyttXbP5NXtkx9z/2yoqk5N8pIkvzfvs5lK98727uskQ9fO6itT0tf07+vk3O6dPVr6ukgj95QkX1/z+s2Z81+8RVNVpyd5fpLPdDh7W1XtTXJbkstba3PPkOSiJG9M8lCHsx/Wknysqq6uqgs65lhEOvsovTqrr9+jr4emr49ylH+MvShHQV8XaeTWQS5bjMdSdFBVj0nywSS/3Fq7a97nt9YOtNbOTHJqkrOqaq5fWqqqlya5rbV29TzPPYhzWms/nOTFSX5p8uU2VunsGj07q6/fo6+Hpq9rHM0fY4+mvi7SyL05yWlrXj81ya2dsnQ1eYzOB5O8p7X2oZ5ZWmvfSXJFkvPmfPQ5SV42eczO+5K8oKr+aM4Z0lq7dfLzbUk+nNUv+bFKZycWpbP6qq+Hoa8Ti9LXpFtnj5q+LtLIvSrJM6vq6VW1I8krk/xp50xzN3lA+juTXN9ae2unDCdV1eMnLx+X5IVJvjzPDK21N7fWTm2tnZ7Vvwsfb629ep4ZqmrX5D8mpKp2JXlRkm7/K3gB6Wz6d1ZfV+nruvQ1/fs6ydC1s0dTXxdm5LbW9id5XZLLsvpA8Pe31q6dd46qem+SK5M8u6purqrXzjnCOUlek9XPrPZOfpw/5wwnJ/lEVV2T1X8YL2+tdXmKkc6enORTVfWFJJ9N8uettUs7Z1oYi9DZBehr0r+z+rpKXw9DX7+nd18TnU3m1NeFeQoxAADYKgtzTy4AAGwVIxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkLriq+s0e31Ma2BydheWhr2MzchdAVf1MVe2pqnuq6htVdUlV/XjvXElSVb9VVV+sqv1V9Zu988Ai0FlYHvp69DJyO6uqC5NclOS3s/q9nJ+a5O1JXt4x1lo3JHljkj/vHQQWgc7C8tDXo5uR21FVPS7Jf0zyS621D7XW7m2t7Wut/Vlr7d8f4tf8l6r6h6q6s6o+WVU/tOZt51fVdVV1d1XdUlW/Mrn8iVX1kar6TlV9u6r+qqqm+rNvrf1+a+2SJHdvwW8ZlprOwvLQV4zcvs5OsjPJhzfway5J8swkT0ryuSTvWfO2dyb5+dbaCUmem+Tjk8vfkOTmJCdl9TPZ/5CkJUlVvb2q3n4Evwc4mugsLA99Pcqt9A5wlDsxyTdba/un/QWttXc9/PLk8Tt3VNXjWmt3JtmX5DlV9YXW2h1J7phcdV+Sk5M8rbV2Q5K/WnN7v3jkvw04augsLA99Pcq5J7evbyV5YlVN9clGVW2rqt+pqr+rqruS3DR50xMnP/90kvOTfK2q/rKqzp5c/p+z+rifj1XVjVX1pq37LcBRRWdheejrUc7I7evKJN9N8oopr/8zWX2w/AuTPC7J6ZPLK0laa1e11l6e1S+z/Nck759cfndr7Q2ttWck+akkF1bVv9ia3wIcVXQWloe+HuWM3I4mX/749SRvq6pXVNXxVbW9ql5cVf/pIL/khCQPZPWz0+Oz+r9FkyRVtaOqfnbyZZV9Se5KcmDytpdW1fdXVa25/MA0GSd5dmb178pKVe2sqm2b/13D8tJZWB76ipHbWWvtrUkuTPJrSW5P8vUkr8vqZ4mP9gdJvpbkliTXJfn0o97+miQ3Tb7M8gtJXj25/JlJ/iLJPVn9zPbtrbUrkqSq3lFV7zhMxN9Ncn+SVyX51cnLr9nI7xFGorOwPPT16Fattd4ZAABgS7knFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGM5Mvq3vju272s4dj5/FTU+t7v9u1/OTpPczV9SOHV3PT5J9j93eO0K27ev/DCL33HnLN1trJ/XOcTA76ti2M7u6Ztj35L7nJ0k91Pf8le/2/3v6rKd/s3eE3LsAz/jz5S8+uNh9rb59edY/ua/r+Yvgq9cc3zsCE3fnjkP2dSYjd+eOx+dHn/vzs7jpqdUX/7br+Uny0Hf7Du2VU57a9fwk+W8vPKV3hOz6h6mek3umPvVnb/xa7wyHsjO78s86f3Oef/jZH+t6fpKs3Nd3XH3fVx/oen6S/MV73tU7Qq5+4MHeEXLW6X+/uH2tXfnRlXO7Zrj0sj1dz18E5z7lzN4RmPiL9oFD9tXDFQAAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhjPVyK2q86rqK1V1Q1W9adahgM3TV1ge+gqzs+7IraptSd6W5MVJnpPkVVX1nFkHAzZOX2F56CvM1jT35J6V5IbW2o2ttQeTvC/Jy2cbC9gkfYXloa8wQ9OM3FOSfH3N6zdPLnuEqrqgqvZU1Z59++/dqnzAxmy8r3lgbuGAR9h4X5u+wrSmGbl1kMvaf3dBaxe31na31nZvX9l15MmAzdh4X3PsHGIBB7Hxvpa+wrSmGbk3JzltzeunJrl1NnGAI6SvsDz0FWZompF7VZJnVtXTq2pHklcm+dPZxgI2SV9heegrzNDKeldore2vqtcluSzJtiTvaq1dO/NkwIbpKywPfYXZWnfkJklr7aNJPjrjLMAW0FdYHvoKs+M7ngEAMBwjFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADGdlJrd63/1pV31xJjc9rdb19MWw/+9v6R0hJ/7e3/eOwDr2P2lX/turfqxrhntPeajr+Umy7f7qen49dGzX85PkJWf/VO8IefBdvRMkyf/VO8AhPfT9O3Lf207rmuG8l/9g1/OT5NI/+cOu53/rtWd3PT9JTnznlb0j5Ou/1vdjR5Lktz5wyDe5JxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAw1l35FbVu6rqtqr60jwCAUdGZ2F56CvMzjT35L47yXkzzgFsnXdHZ2FZvDv6CjOx7shtrX0yybfnkAXYAjoLy0NfYXa27DG5VXVBVe2pqj378sBW3SwwA2v7uv/+e3vHAQ7jEX298/7ecWBpbNnIba1d3Frb3VrbvT3HbtXNAjOwtq8rx+3qHQc4jEf09XHH9Y4DS8OzKwAAMBwjFwCA4UzzFGLvTXJlkmdX1c1V9drZxwI2S2dheegrzM7Keldorb1qHkGAraGzsDz0FWbHwxUAABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIaz0jvArLzgi/f2jpCn7fhm1/P/32c/rev5SXLZrXt7R1gI207uneDQVm67N0/+v/+ma4Yn/+gZXc9Pknz6mq7HL0JXzr/iX/WOkMt/8EO9I2Rb7wCH8dSd387/8+z3ds3wsz9xYdfzk+Tcp5zZ9fwTc2XX85Nk20kn9Y6QJ1x/oHeEw3JPLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcNYduVV1WlV9oqqur6prq+r18wgGbJy+wnLRWZidlSmusz/JG1prn6uqE5JcXVWXt9aum3E2YOP0FZaLzsKMrHtPbmvtG621z01evjvJ9UlOmXUwYOP0FZaLzsLsbOgxuVV1epLnJ/nMTNIAW0ZfYbnoLGytqUduVT0myQeT/HJr7a6DvP2CqtpTVXv25YGtzAhskL7CcjlcZ9f29TvffqhPQFhCU43cqtqe1fK9p7X2oYNdp7V2cWttd2tt9/Ycu5UZgQ3QV1gu63V2bV8f/wRPigTTmubZFSrJO5Nc31p76+wjAZulr7BcdBZmZ5pPCc9J8pokL6iqvZMf5884F7A5+grLRWdhRtZ9CrHW2qeS1ByyAEdIX2G56CzMjgf3AAAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMBwjFwCA4Ri5AAAMZ2UWN/qMM+7Jez7617O46am96P/8la7nJ8m/+Xcf7Xr+ytNO63p+kvzEa/9p7wi54p2/2zsC6/n0Nb0TdHf+V87vHSEf/csP9Y6Qc59yZu8ISW7oHeCQbrnusXnTGS/qmuEpd/1N1/NZ9dEvXN47wkLY9oFDv809uQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAwzFyAQAYzrojt6p2VtVnq+oLVXVtVb1lHsGAjdNXWC46C7OzMsV1HkjygtbaPVW1PcmnquqS1tqnZ5wN2Dh9heWiszAj647c1lpLcs/k1e2TH22WoYDN0VdYLjoLszPVY3KraltV7U1yW5LLW2ufmWkqYNP0FZaLzsJsTDVyW2sHWmtnJjk1yVlV9dxHX6eqLqiqPVW151vfemiLYwLT2mhf9+WBuWcE/tF6nV3b1wcf+m6XjLCMNvTsCq217yS5Isl5B3nbxa213a213See6EkboLdp+7o9x847GnAQh+rs2r7uOGZnj2iwlKZ5doWTqurxk5ePS/LCJF+ecS5gE/QVlovOwuxM8+wKJyf5/araltVR/P7W2kdmGwvYJH2F5aKzMCPTPLvCNUmeP4cswBHSV1guOguz48GzAAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDWZnFjd6+//hcfMcPz+Kmp3bSO67sen6SfOQd39f1/Mtu/bOu5yfJuU85s3eE/H/77ukdYaHVju1Z+R9O7Zph/9dv7np+klx2696u5+/+jbO7np8keUvvAKynHXgoB+66q3eM7h48d3fX83dctqfr+clifHzt/e/metyTCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhTD1yq2pbVX2+qj4yy0DAkdNXWB76CrOxkXtyX5/k+lkFAbaUvsLy0FeYgalGblWdmuQlSX5vtnGAI6WvsDz0FWZn2ntyL0ryxiQPzS4KsEUuir7Csrgo+gozse7IraqXJrmttXb1Ote7oKr2VNWee+94cMsCAtPbTF8fPHD/nNIBa22mr/vywJzSwfKb5p7cc5K8rKpuSvK+JC+oqj969JVaaxe31na31nbv+r4dWxwTmNKG+7pj23Hzzgis2nBft+fYeWeEpbXuyG2tvbm1dmpr7fQkr0zy8dbaq2eeDNgwfYXloa8wW54nFwCA4axs5MqttSuSXDGTJMCW0ldYHvoKW889uQAADMfIBQBgOEYuAADDMXIBABiOkQsAwHCMXAAAhmPkAgAwHCMXAIDhGLkAAAzHyAUAYDhGLgAAw1mZxY3efd0x+cszjpvFTU/tmDN+oOv5SVK33Nb1/Jec/VNdz1/19d4B8gtP+/HeEZJ8oHeAQ2r79uXAN/6hb4azn9f1/FV7u55+4u9e2fX8JHn20/5t7wg5Pf3fDxzeytOf1jtCctmevsffurfr+Uly7lPO7B1h4bknFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADMfIBQBgOEYuAADDWZnmSlV1U5K7kxxIsr+1tnuWoYDN01dYLjoLszHVyJ34ydbaN2eWBNhK+grLRWdhi3m4AgAAw5l25LYkH6uqq6vqgoNdoaouqKo9VbVnXx7YuoTARm2sr01fobPDdtbHV9icaR+ucE5r7daqelKSy6vqy621T669Qmvt4iQXJ8lj6wlti3MC09tYX4/RV+jssJ318RU2Z6p7cltrt05+vi3Jh5OcNctQwObpKywXnYXZWHfkVtWuqjrh4ZeTvCjJl2YdDNg4fYXlorMwO9M8XOHJST5cVQ9f/49ba5fONBWwWfoKy0VnYUbWHbmttRuTPG8OWYAjpK+wXHQWZsdTiAEAMBwjFwCA4Ri5AAAMx8gFAGA4Ri4AAMMxcgEAGI6RCwDAcIxcAACGY+QCADAcIxcAgOEYuQAADKdaa1t/o1W3J/naEdzEE5N8c4viyLC854+U4WmttZO2IsxW01cZBsugr+sb4c9IBhkedsi+zmTkHqmq2tNa2y1D3wy9z5dhOSzC+0cGGRbl/GXQ+33U+3wZjp4MHq4AAMBwjFwAAIazqCP34t4BIsMinJ/IsAwW4f0jwyoZ+p+/DHq/j3qfn8jwsKEzLORjcgEA4Egs6j25AACwaQs1cqvqvKr6SlXdUFVv6pThXVV1W1V9qdP5p1XVJ6rq+qq6tqpe3yHDzqr6bFV9YZLhLfPOsCbLtqr6fFV9pNP5N1XVF6tqb1Xt6ZFhkfXubO++TjJ07ay+PuJ8fT0Mfe3f10mGhejs0dDXhXm4QlVtS/LVJP8yyc1JrkryqtbadXPO8c+T3JPkD1prz53n2ZPzT05ycmvtc1V1QpKrk7xinu+Hqqoku1pr91TV9iSfSvL61tqn55VhTZYLk+xO8tjW2ks7nH9Tkt2ttd7PI7hwFqGzvfs6ydC1s/r6iPNvir4elL5+L4OPsf+YY/i+LtI9uWcluaG1dmNr7cEk70vy8nmHaK19Msm3533umvO/0Vr73OTlu5Ncn+SUOWdorbV7Jq9un/yY+2dDVXVqkpck+b15n81Uune2d18nGbp2Vl+Zkr6mf18n53bv7NHS10Uauack+fqa12/OnP/iLZqqOj3J85N8psPZ26pqb5LbklzeWpt7hiQXJXljkoc6nP2wluRjVXV1VV3QMcci0tlH6dVZff0efT00fX2Uo/xj7EU5Cvq6SCO3DnLZYjyWooOqekySDyb55dbaXfM+v7V2oLV2ZpJTk5xVVXP90lJVvTTJba21q+d57kGc01r74SQvTvJLky+3sUpn1+jZWX39Hn09NH1d42j+GHs09XWRRu7NSU5b8/qpSW7tlKWryWN0PpjkPa21D/XM0lr7TpIrkpw356PPSfKyyWN23pfkBVX1R3POkNbarZOfb0vy4ax+yY9VOjuxKJ3VV309DH2dWJS+Jt06e9T0dZFG7lVJnllVT6+qHUlemeRPO2eau8kD0t+Z5PrW2ls7ZTipqh4/efm4JC9M8uV5Zmitvbm1dmpr7fSs/l34eGvt1fPMUFW7Jv8xIVW1K8mLknT7X8ELSGfTv7P6ukpf16Wv6d/XSYaunT2a+rowI7e1tj/J65JcltUHgr+/tXbtvHNU1XuTXJnk2VV1c1W9ds4Rzknymqx+ZrV38uP8OWc4OcknquqarP7DeHlrrctTjHT25CSfqqovJPlskj9vrV3aOdPCWITOLkBfk/6d1ddV+noY+vo9vfua6Gwyp74uzFOIAQDAVlmYe3IBAGCrGLkAAAzHyAUAYDhGLgAAwzFyAQAYjpELAMBwjFwAAIZj5AIAMJz/H4gI5Idep1zHAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "data_X = data['data_X']\n", + "data_y = data['data_y']\n", + "\n", + "plt.figure(figsize=(12, 8))\n", + "\n", + "for i in range(2):\n", + " class_X = data_X[data_y == i]\n", + " for j in range(3):\n", + " plt.subplot(2, 3, 3 * i + j + 1)\n", + " plt.title('Class: {}'.format(i))\n", + " plt.imshow(class_X[j])\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Model architecture\n", + "\n", + "To tackle this binary classification problem, we want to design a simple CNN with a single convolutional layer followed by a fully connected layer.\n", + "\n", + "More precisely, the architecture is the following:\n", + "\n", + "X $\\rightarrow$ [Conv2d](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) $\\rightarrow$ [ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) $\\rightarrow$ [MaxPool2d](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html) $\\rightarrow$ [Flatten](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html) $\\rightarrow$ [Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) $\\rightarrow$ [Sigmoid](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) $\\rightarrow$ y\n", + "\n", + "For the convolutional layer, we choose a kernel of size 3 with a single output channel.\n", + "\n", + "As a reminder, the pooling layer will reduce the dimension of the array by keeping only the maximum value by zones. We use a kernel of size 2 for the pooling.\n", + "\n", + "The flatten layer is needed because the output of the convolutional layer (and the pooling layer) is a 2D image, but fully connected (linear) layers only accept vectors as inputs.\n", + "\n", + "You need to compute the right input size for the linear layer. For this, you need to first check the dimension of the input images in the cell below.\n", + "\n", + "Use that information to compute the desired size for the linear output layer of your model. \n", + "\n", + "When performing a convolution with valid entries only (no padding of the input) on a $W \\times H$ image with a kernel of size $w \\times h$, the output will be of size $(W - w + 1) \\times (H - h + 1)$.\n", + "\n", + "The data then goes into a pooling layer, which reduces its dimension by N, with N being the size of the pooling kernel." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input shape: (6, 6)\n", + "Convolution output size: 4\n", + "Pooling output size: 2\n", + "Linear input size: 4\n" + ] + } + ], + "source": [ + "# Code here\n", + "\n", + "shape = data_X.shape[-2:]\n", + "\n", + "print('Input shape:', shape)\n", + "\n", + "out_conv = shape[0] - 3 + 1\n", + "\n", + "print('Convolution output size:', out_conv)\n", + "\n", + "out_pool = out_conv // 2\n", + "\n", + "print('Pooling output size:', out_pool)\n", + "\n", + "in_linear = out_pool ** 2\n", + "\n", + "print('Linear input size:', in_linear)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can now implement the model in the following cell. As you can see, the forward function uses `squeeze` before returning the result. This is to remove any unnecessary empty dimensions in the output.\n", + "\n", + "Hint: Use the following classes for your implementation\n", + "- `nn.Conv2d`\n", + "- `nn.ReLU`\n", + "- `nn.MaxPool2d`\n", + "- `nn.Flatten`\n", + "- `nn.Linear`\n", + "- `nn.Sigmoid`\n", + "\n", + "Take a look back at the [torch.nn documentation](https://pytorch.org/docs/stable/nn.html) to figure out which parameters to use." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "class Net(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + "\n", + " ########################################\n", + " # Code here\n", + "\n", + " self.conv = nn.Conv2d(1, 1, 3)\n", + "\n", + " self.relu = nn.ReLU()\n", + "\n", + " self.maxpool = nn.MaxPool2d(2)\n", + "\n", + " self.flatten = nn.Flatten()\n", + "\n", + " self.linear = nn.Linear(4, 1)\n", + "\n", + " self.sigmoid = nn.Sigmoid()\n", + "\n", + " ########################################\n", + "\n", + "\n", + " def forward(self, x):\n", + "\n", + " ########################################\n", + " # Code here\n", + " \n", + " x = self.conv(x)\n", + "\n", + " x = self.relu(x)\n", + "\n", + " x = self.maxpool(x)\n", + "\n", + " x = self.flatten(x)\n", + "\n", + " x = self.linear(x)\n", + "\n", + " x = self.sigmoid(x)\n", + " \n", + " ########################################\n", + " \n", + " return x.squeeze()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Processing the data\n", + "\n", + "Start with the preprocessing of the data.\n", + "\n", + "Normalize the datasets (remember, use only the training information), then convert them to float Tensors.\n", + "\n", + "The targets must also be converted to float Tensors for the cross entropy loss." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "X_train, X_test, y_train, y_test = train_test_split(data_X, data_y, test_size=0.2, random_state=42)\n", + "\n", + "\n", + "################################################\n", + "# Code here\n", + "\n", + "x_mean = X_train.mean()\n", + "x_std = X_train.std()\n", + "\n", + "X_train = (X_train - x_mean) / x_std\n", + "X_test = (X_test - x_mean) / x_std\n", + "\n", + "X_train = torch.from_numpy(X_train).float()\n", + "X_test = torch.from_numpy(X_test).float()\n", + "\n", + "y_train = torch.from_numpy(y_train).float()\n", + "y_test = torch.from_numpy(y_test).float()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Setting up the training\n", + "\n", + "We must now create the different objects required to train the model.\n", + "\n", + "Since we are now dealing with a binary classification problem, we will use the binary cross entropy loss implemented with the `nn.BCELoss` class.\n", + "We will use the stochastic gradient descent optimizer as in the first part of the lab.\n", + "\n", + "Complete the following cell, with an imposed learning rate of 0.1 and a batch size of 20.\n", + "\n", + "The `losses` variable should be an empty list to save the training loss later.\n", + "\n", + "Since we do not reset the model and optimizer in the following cell, the training will continue further every time it is launched if the previous cell is not called again." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], + "source": [ + "# Code here\n", + "\n", + "model = Net()\n", + "\n", + "criterion = nn.BCELoss()\n", + "\n", + "optimizer = optim.SGD(model.parameters(), lr=1e-1)\n", + "\n", + "batch_size = 20\n", + "\n", + "training_loss = []" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Training the model\n", + "\n", + "You can now fill in the training loop. Remember the steps we listed above!\n", + "\n", + "Carefully check the documentation of Conv2d for the expected shape of the input data.\n", + "\n", + "At the end of the training, you should see a plot showing the evolution of the loss." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "432d211e8e5d439589c25ef8c9c8e5a2", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Epochs: 0%| | 0/10 [00:00" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "torch.manual_seed(42)\n", + "\n", + "for epoch in trange(10, desc='Epochs'):\n", + " epoch_loss = 0\n", + " for batch_idx in trange(X_train.shape[0] // batch_size, desc='Batches', leave=None):\n", + " \n", + " ####################################################\n", + " # Code here\n", + "\n", + " X_batch = X_train[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + " y_batch = y_train[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + "\n", + " X_batch = X_batch[:, None, ...]\n", + "\n", + " optimizer.zero_grad()\n", + "\n", + " predict = model(X_batch)\n", + "\n", + " loss = criterion(predict, y_batch)\n", + "\n", + " loss.backward()\n", + " \n", + " optimizer.step()\n", + "\n", + " epoch_loss += loss.item() * X_batch.shape[0]\n", + "\n", + " training_loss.append(epoch_loss / X_train.shape[0])\n", + "\n", + "\n", + "plt.figure(figsize=(10, 7))\n", + "plt.plot(training_loss, label='Training')\n", + "plt.title('Losses')\n", + "plt.xlabel('Epoch')\n", + "plt.ylabel('Loss')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Computing the accuracy\n", + "\n", + "How well does the model perform? Compute its accuracy over the training and test sets.\n", + "\n", + "You can run the training cell again to further improve the performance." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training accuracy: 73.75%\n", + "Test accuracy: 59.00%\n" + ] + } + ], + "source": [ + "# Code here\n", + "\n", + "predict = model(X_train[:, None, ...])\n", + "predict = predict > 0.5\n", + "\n", + "accuracy = ((predict == y_train) * 1.0).mean()\n", + "\n", + "predict_t = model(X_test[:, None, ...])\n", + "predict_t = predict_t > 0.5\n", + "\n", + "accuracy_t = ((predict_t == y_test) * 1.0).mean()\n", + "\n", + "print('Training accuracy: {:.02f}%'.format(accuracy * 100))\n", + "print('Test accuracy: {:.02f}%'.format(accuracy_t * 100))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Monitoring the test loss\n", + "\n", + "It is always important to keep track of the test loss as well, particularly when the performance on test and training sets is different.\n", + "\n", + "Copy the two cells under *Setting up the training* and *Training the model*, and modify them to also compute and plot the loss on the test set. This should be computed at the end of every epoch.\n", + "\n", + "You do not need to use batches for computing the test loss." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [], + "source": [ + "# Code here\n", + "\n", + "model = Net()\n", + "\n", + "criterion = nn.BCELoss()\n", + "\n", + "optimizer = optim.SGD(model.parameters(), lr=1e-1)\n", + "\n", + "batch_size = 20\n", + "\n", + "training_loss = []\n", + "test_loss = []" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "df380188a4644f6fb00c8397d4e99060", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Epochs: 0%| | 0/10 [00:00" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "torch.manual_seed(42)\n", + "\n", + "for epoch in trange(10, desc='Epochs'):\n", + " epoch_loss = 0\n", + " for batch_idx in trange(X_train.shape[0] // batch_size, desc='Batches', leave=None):\n", + " \n", + " ####################################################\n", + " # Code here\n", + "\n", + " X_batch = X_train[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + " y_batch = y_train[batch_idx * batch_size : (batch_idx+1) * batch_size]\n", + "\n", + " X_batch = X_batch[:, None, ...]\n", + "\n", + " optimizer.zero_grad()\n", + "\n", + " predict = model(X_batch)\n", + "\n", + " loss = criterion(predict, y_batch)\n", + "\n", + " loss.backward()\n", + " \n", + " optimizer.step()\n", + "\n", + " epoch_loss += loss.item() * X_batch.shape[0]\n", + "\n", + " training_loss.append(epoch_loss / X_train.shape[0])\n", + "\n", + " # Compute the test loss\n", + " predict_t = model(X_test[:, None, ...])\n", + "\n", + " loss_t = criterion(predict_t, y_test)\n", + "\n", + " test_loss.append(loss_t.item())\n", + "\n", + "\n", + "plt.figure(figsize=(10, 7))\n", + "plt.plot(training_loss, label='Training')\n", + "plt.plot(test_loss, label='Test')\n", + "plt.title('Losses')\n", + "plt.xlabel('Epoch')\n", + "plt.ylabel('Loss')\n", + "plt.legend()\n", + "plt.show()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.3" + }, + "toc": { + "base_numbering": 1, + "nav_menu": {}, + "number_sections": true, + "sideBar": true, + "skip_h1_title": false, + "title_cell": "Table of Contents", + "title_sidebar": "Contents", + "toc_cell": false, + "toc_position": {}, + "toc_section_display": true, + "toc_window_display": false + }, + "varInspector": { + "cols": { + "lenName": 16, + "lenType": 16, + "lenVar": 40 + }, + "kernels_config": { + "python": { + "delete_cmd_postfix": "", + "delete_cmd_prefix": "del ", + "library": "var_list.py", + "varRefreshCmd": "print(var_dic_list())" + }, + "r": { + "delete_cmd_postfix": ") ", + "delete_cmd_prefix": "rm(", + "library": "var_list.r", + "varRefreshCmd": "cat(var_dic_list()) " + } + }, + "types_to_exclude": [ + "module", + "function", + "builtin_function_or_method", + "instance", + "_Feature" + ], + "window_display": false + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}