\n The importance of Natural Language Processing (NLP) in the fields of Artificial Intelligence (AI) and Machine Learning is increasing day by day. At the center of this is the Hugging Face<\/strong> Transformer library. This library makes it easy to use various NLP models, especially with the advantage of being able to easily apply pre-trained models. In this course, we will teach you how to split training and validation datasets using the Hugging Face Transformer library.\n<\/p>\n \n The first step is to prepare the dataset to be used. Generally, a labeled dataset is required to solve NLP problems. In this example, we will use the IMDb Movie Reviews Dataset<\/strong> to train a model that classifies positive and negative reviews. This dataset is widely used and consists of the text of movie reviews and their corresponding labels (positive\/negative).\n<\/p>\n \n You can download the IMDb dataset using the above code. The \n You can check the structure of the downloaded dataset. The dataset is divided into training (train), testing (test), and validation (validation) sets.\n<\/p>\n \n In general, it is important to split the data into several parts to train a model in machine learning. Typically, the training data and validation data are split, where the training data is used to train the model, and the validation data is used to evaluate its performance. In this case, we will extract a portion of the training data to use as validation data.\n<\/p>\n \n The above code uses the \n You can now check the number of training and validation samples. This information helps to determine whether our data has been properly split.\n<\/p>\n \n After splitting the dataset, we need to prepare the model. Hugging Face’s Transformer library<\/strong> provides a variety of pre-trained models, allowing us to choose a model suitable for our needs.\n<\/p>\n \n We prepare the BERT model using \n We tokenize the training and validation data using the \n To train the model, we can manipulate the data in batches using PyTorch’s DataLoader<\/strong>. We will also set the optimizer and loss function to train the model.\n<\/p>\n \n A new dataset class is defined inheriting from the \n We train the model using the AdamW optimization algorithm. The total loss is calculated and output for each epoch. In this example, training is done for 3 epochs.\n<\/p>\n \n After training the model, we need to evaluate its performance on the validation data. This will help us determine how well the model generalizes.\n<\/p>\n1. Preparing the Dataset<\/h2>\n
1.1 Downloading the Dataset<\/h3>\n
python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"imdb\")\n<\/code><\/pre>\nload_dataset<\/code> function is one available in the Hugging Face datasets library, which allows you to easily download various public datasets.\n<\/p>\n1.2 Checking the Dataset Structure<\/h3>\n
python\nprint(dataset)\n<\/code><\/pre>\n2. Splitting the Dataset<\/h2>\n
2.1 Splitting Training and Validation Data<\/h3>\n
python\nfrom sklearn.model_selection import train_test_split\n\ntrain_data = dataset['train']\ntrain_texts = train_data['text']\ntrain_labels = train_data['label']\n\ntrain_texts, val_texts, train_labels, val_labels = train_test_split(\n train_texts,\n train_labels,\n test_size=0.1, # Using 10% as validation set\n random_state=42\n)\n<\/code><\/pre>\ntrain_test_split<\/code> function to split the training data into 90% and 10%. Since test_size=0.1<\/code> is set, 10% of the original training data is chosen as validation data. The random_state<\/code> parameter ensures the consistency of the split.\n<\/p>\n2.2 Checking the Split Data<\/h3>\n
python\nprint(\"Number of training samples:\", len(train_texts))\nprint(\"Number of validation samples:\", len(val_texts))\n<\/code><\/pre>\n3. Preparing the Hugging Face Transformer Model<\/h2>\n
3.1 Selecting a Pre-trained Model<\/h3>\n
python\nfrom transformers import BertTokenizer, BertForSequenceClassification\n\nmodel_name = \"bert-base-uncased\"\ntokenizer = BertTokenizer.from_pretrained(model_name)\nmodel = BertForSequenceClassification.from_pretrained(model_name)\n<\/code><\/pre>\nBertTokenizer<\/code> and BertForSequenceClassification<\/code>. This model is suitable for text classification tasks and uses the pre-trained version called “bert-base-uncased.”\n<\/p>\n3.2 Tokenizing the Data<\/h3>\n
python\ntrain_encodings = tokenizer(train_texts, truncation=True, padding=True, return_tensors='pt')\nval_encodings = tokenizer(val_texts, truncation=True, padding=True, return_tensors='pt')\n<\/code><\/pre>\ntokenizer<\/code>. truncation=True<\/code> handles inputs that exceed length limits, and padding=True<\/code> ensures all inputs are of equal length.\n<\/p>\n4. Training the Model<\/h2>\n
4.1 Preparing the Data Loader<\/h3>\n
python\nimport torch\nfrom torch.utils.data import DataLoader, Dataset\n\nclass IMDbDataset(Dataset):\n def __init__(self, encodings, labels):\n self.encodings = encodings\n self.labels = labels\n\n def __getitem__(self, idx):\n item = {key: val[idx] for key, val in self.encodings.items()}\n item['labels'] = torch.tensor(self.labels[idx])\n return item\n\n def __len__(self):\n return len(self.labels)\n\ntrain_dataset = IMDbDataset(train_encodings, train_labels)\nval_dataset = IMDbDataset(val_encodings, val_labels)\n\ntrain_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=16, shuffle=False)\n<\/code><\/pre>\nDataset<\/code> class, and we use DataLoader<\/code> for batch processing. A batch size of 16 is used.\n<\/p>\n4.2 Setting Up Model Training<\/h3>\n
python\nfrom transformers import AdamW\n\noptimizer = AdamW(model.parameters(), lr=5e-5)\n\nmodel.train()\nfor epoch in range(3): # Total 3 epochs\n total_loss = 0\n for batch in train_loader:\n optimizer.zero_grad()\n outputs = model(**batch)\n loss = outputs.loss\n total_loss += loss.item()\n loss.backward()\n optimizer.step()\n print(f\"Epoch: {epoch + 1}, Loss: {total_loss \/ len(train_loader)}\")\n<\/code><\/pre>\n5. Evaluating the Model<\/h2>\n
5.1 Defining the Model Evaluation Function<\/h3>\n
python\nfrom sklearn.metrics import accuracy_score\n\ndef evaluate_model(model, val_loader):\n model.eval()\n all_labels = []\n all_preds = []\n\n with torch.no_grad():\n for batch in val_loader:\n outputs = model(**batch)\n preds = outputs.logits.argmax(dim=-1)\n all_labels.extend(batch['labels'].tolist())\n all_preds.extend(preds.tolist())\n\n accuracy = accuracy_score(all_labels, all_preds)\n return accuracy\n\naccuracy = evaluate_model(model, val_loader)\nprint(\"Validation Accuracy:\", accuracy)\n<\/code><\/pre>\n