In the field of deep learning and natural language processing (NLP), the Hugging Face<\/strong>‘s The Now let\u2019s practically set up the parameters needed for training using The code above is an example of creating a After setting the training parameters, you can use the The code above performs the following steps:<\/p>\n Though there are various arguments in TrainingArguments, let\u2019s look at a few commonly used configurations:<\/p>\n If you encounter memory limitations that make it difficult to train with large batches during model training, you can use gradient accumulation. For example, if the batch size is set to 32 and you accumulate gradients over 4 batches, the total effective batch size will be 128.<\/p>\n If your GPU supports Mixed Precision Training, it can accelerate training and reduce memory usage. In this case, you can add the You can configure early stopping to prevent unnecessary training if there is no improvement in performance. This should be combined with In this course, we thoroughly explained how to set up the To train deep learning models more effectively, it is important to make good use of the various parameters in If you have any further questions or would like to know more, please leave a comment, and we will be happy to respond.<\/p>\nTransformers<\/code> library is a very useful tool. In this course, we will explain in detail the TrainingArguments<\/code> class used in Hugging Face’s Trainer<\/code> API, how to configure it, and provide actual code examples.<\/p>\nWhat is TrainingArguments?<\/h2>\n
TrainingArguments<\/code> class is used to define various hyperparameters and settings for model training. This class allows you to set multiple arguments that include training, validation, and logging requirements.<\/p>\nMain Parameters of TrainingArguments<\/h3>\n
\n
output_dir<\/code>: The directory path where model checkpoints will be saved.<\/li>\nnum_train_epochs<\/code>: Sets how many times to iterate through the entire training dataset.<\/li>\nper_device_train_batch_size<\/code>: The batch size to use per device (e.g., GPU).<\/li>\nlearning_rate<\/code>: Sets the learning rate.<\/li>\nevaluation_strategy<\/code>: Sets the evaluation strategy. For example, options like “epoch” or “steps” are available.<\/li>\nlogging_dir<\/code>: The directory path where log files will be saved.<\/li>\nweight_decay<\/code>: Applies regularization using weight decay.<\/li>\nsave_total_limit<\/code>: Limits the maximum number of checkpoints to be saved.<\/li>\n<\/ul>\nSetting Up TrainingArguments<\/h2>\n
TrainingArguments<\/code>. The example code below describes how to use this class and the role of each parameter.<\/p>\nPython Example Code<\/h3>\n
from transformers import TrainingArguments\n\n# Create TrainingArguments object\ntraining_args = TrainingArguments(\n output_dir='.\/results', # Directory path to save checkpoints\n num_train_epochs=3, # Number of epochs to train\n per_device_train_batch_size=16, # Batch size to use on each device\n per_device_eval_batch_size=64, # Batch size to use for evaluation\n learning_rate=2e-5, # Learning rate\n evaluation_strategy=\"epoch\", # Evaluation strategy\n logging_dir='.\/logs', # Directory to save log files\n weight_decay=0.01, # Weight decay\n save_total_limit=2 # Maximum number of saved checkpoints\n)\n\nprint(training_args)\n<\/code><\/pre>\nCode Explanation<\/h3>\n
TrainingArguments<\/code> object. Let\u2019s take a closer look at each parameter:<\/p>\n\n
output_dir='.\/results'<\/code>: Specifies the folder where the model checkpoints will be saved after training.<\/li>\nnum_train_epochs=3<\/code>: Trains the model by iterating through the entire dataset 3 times.<\/li>\nper_device_train_batch_size=16<\/code>: Uses a batch of 16 samples for training on each device.<\/li>\nper_device_eval_batch_size=64<\/code>: Processes 64 samples in a batch for evaluation on each device.<\/li>\nlearning_rate=2e-5<\/code>: Sets the learning rate at the start of training.<\/li>\nevaluation_strategy=\"epoch\"<\/code>: Configures the model to be evaluated after each epoch ends.<\/li>\nlogging_dir='.\/logs'<\/code>: Directory to save training logs.<\/li>\nweight_decay=0.01<\/code>: Applies 1% weight decay to prevent model overfitting.<\/li>\nsave_total_limit=2<\/code>: Limits the maximum number of checkpoints being saved to 2.<\/li>\n<\/ul>\nIntegrating TrainingArguments with the Trainer API<\/h2>\n
Trainer<\/code> API to train your model. Below is an example showing how to integrate the Trainer<\/code> class with TrainingArguments<\/code>.<\/p>\nfrom transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification, AutoTokenizer\n\n# Load model and tokenizer\nmodel = AutoModelForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Prepare training and evaluation datasets (example is omitted)\ntrain_dataset = ...\neval_dataset = ...\n\n# Create Trainer object\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset\n)\n\n# Train the model\ntrainer.train()\n<\/code><\/pre>\nCode Explanation<\/h3>\n
\n
AutoModelForSequenceClassification<\/code>.<\/li>\nAutoTokenizer<\/code>.<\/li>\nTrainer<\/code> object, which takes the model, training arguments, training dataset, and evaluation dataset.<\/li>\ntrainer.train()<\/code> to start the model training.<\/li>\n<\/ul>\nCommon Configurations for TrainingArguments<\/h2>\n
1. Gradient Accumulation<\/h3>\n
training_args = TrainingArguments(\n per_device_train_batch_size=8,\n gradient_accumulation_steps=4, # Accumulate gradients over 4 batches\n)\n<\/code><\/pre>\n2. Mixed Precision Training<\/h3>\n
fp16=True<\/code> setting.<\/p>\ntraining_args = TrainingArguments(\n fp16=True, # Mixed precision training\n)\n<\/code><\/pre>\n3. Early Stopping<\/h3>\n
EarlyStoppingCallback<\/code>.<\/p>\nfrom transformers import EarlyStoppingCallback\n\ntrainer = Trainer(\n ...\n callbacks=[EarlyStoppingCallback(early_stopping_patience=3)], # Stop if no improvement for 3 epochs\n)\n<\/code><\/pre>\nConclusion<\/h2>\n
TrainingArguments<\/code> class in Hugging Face’s Transformers library. You can optimize model training through various hyperparameters.<\/p>\nTrainingArguments<\/code>. We hope you find the optimal hyperparameters through experimentation, continuously improving the model’s performance.<\/p>\n