\n Overfitting is one of the common problems that occur during the training of deep learning models. Overfitting refers to the phenomenon where a model is too closely fitted to the training data, leading to a decreased ability to generalize to new data. Therefore, many researchers and engineers strive to prevent overfitting through various methods. One of these methods is ‘Early Stopping.’\n <\/p>\n
\n Early stopping is a technique that monitors the training process of a model and stops the training when the performance on validation data does not improve. This method prevents overfitting by stopping the training when the model performs poorly on validation data, even if it has learned successfully from the training data.\n <\/p>\n
\n Early stopping fundamentally observes the validation loss or validation accuracy during model training and stops the training if there is no performance improvement for a certain number of epochs. At this point, the optimal model parameters are saved, allowing the use of this model after training is completed.\n <\/p>\n
\n Here, we will implement early stopping through a simple example of training an image classification model using PyTorch. In this example, we will use the MNIST dataset to train a model that recognizes handwritten digits.\n <\/p>\n
pip install torch torchvision matplotlib numpy<\/code><\/pre>\nCode Example<\/h3>\n
\n Below is a PyTorch code example with early stopping applied.\n <\/p>\n
import torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torchvision\nimport torchvision.transforms as transforms\nfrom torch.utils.data import DataLoader\n\n# Hyperparameter settings\ninput_size = 28 * 28 # MNIST image size\nnum_classes = 10 # Number of classes to classify\nnum_epochs = 20 # Total number of training epochs\nbatch_size = 100 # Batch size\nlearning_rate = 0.001 # Learning rate\n\n# Load MNIST dataset\ntransform = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize((0.5,), (0.5,))\n])\n\ntrain_dataset = torchvision.datasets.MNIST(root='.\/data', train=True, transform=transform, download=True)\ntrain_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)\n\ntest_dataset = torchvision.datasets.MNIST(root='.\/data', train=False, transform=transform, download=True)\ntest_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)\n\n# Define a simple neural network model\nclass SimpleNN(nn.Module):\n def __init__(self):\n super(SimpleNN, self).__init__()\n self.fc1 = nn.Linear(input_size, 128)\n self.fc2 = nn.Linear(128, num_classes)\n\n def forward(self, x):\n x = x.view(-1, input_size) # Reshape image dimensions\n x = torch.relu(self.fc1(x)) # Activation function\n x = self.fc2(x)\n return x\n\n# Initialize model, loss function, and optimizer\nmodel = SimpleNN()\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=learning_rate)\n\n# Initialize variables for early stopping\nbest_loss = float('inf')\npatience, trials = 5, 0 # Stop training if no performance improvement for 5 trials\ntrain_losses, val_losses = [], []\n\n# Training loop\nfor epoch in range(num_epochs):\n model.train() # Switch model to training mode\n running_loss = 0.0\n\n for images, labels in train_loader:\n optimizer.zero_grad() # Reset gradients\n outputs = model(images) # Model predictions\n loss = criterion(outputs, labels) # Calculate loss\n loss.backward() # Compute gradients\n optimizer.step() # Update weights\n\n running_loss += loss.item()\n\n avg_train_loss = running_loss \/ len(train_loader)\n train_losses.append(avg_train_loss)\n\n # Validation step\n model.eval() # Switch model to evaluation mode\n val_loss = 0.0\n\n with torch.no_grad(): # Disable gradient computation\n for images, labels in test_loader:\n outputs = model(images)\n loss = criterion(outputs, labels)\n val_loss += loss.item()\n\n avg_val_loss = val_loss \/ len(test_loader)\n val_losses.append(avg_val_loss)\n\n print(f'Epoch [{epoch+1}\/{num_epochs}], Train Loss: {avg_train_loss:.4f}, Valid Loss: {avg_val_loss:.4f}')\n\n # Early stopping logic\n if avg_val_loss < best_loss:\n best_loss = avg_val_loss\n trials = 0 # Reset performance improvement record\n torch.save(model.state_dict(), 'best_model.pth') # Save best model\n else:\n trials += 1\n if trials >= patience: # Stop training if no improvement for patience\n print(\"Early stopping...\")\n break\n\n# Evaluate performance on test data\nmodel.load_state_dict(torch.load('best_model.pth')) # Load best model\nmodel.eval() # Switch model to evaluation mode\ncorrect, total = 0, 0\n\nwith torch.no_grad():\n for images, labels in test_loader:\n outputs = model(images)\n _, predicted = torch.max(outputs.data, 1) # Select class with maximum probability\n total += labels.size(0)\n correct += (predicted == labels).sum().item()\n\nprint(f'Accuracy of the model on the test images: {100 * correct \/ total:.2f}%')<\/code><\/pre>\nCode Explanation<\/h2>\n
\n The above code represents the process of training a simple neural network model using the MNIST dataset. First, we import the necessary libraries and load the MNIST dataset. Then, we define a simple neural network composed of two fully connected layers.\n <\/p>\n
\n After that, at each epoch, we calculate the training loss and validation loss and stop the training if there is no improvement in the validation loss through the early stopping logic. Finally, we evaluate the model’s performance by calculating the accuracy on the test data.\n <\/p>\n
Conclusion<\/h2>\n
\n Early stopping is a useful technique for optimizing the performance of deep learning models. It helps prevent overfitting and leads to the generation of an optimal model. In this tutorial, we demonstrated how to implement early stopping using PyTorch to solve the MNIST classification problem. We encourage you to apply early stopping techniques to various deep learning problems based on this.\n <\/p>\n
References<\/h2>\n