Category: Using Hugging Face

Using Hugging Face Transformers, BART Inference Result Decoding

The advancement of deep learning has brought innovations to the field of Natural Language Processing (NLP), among which the BART (Bidirectional and Auto-Regressive Transformers) model demonstrates excellent performance in various tasks such as text summarization, translation, and generation.

1. Introduction to the BART Model

BART is a modified transformer model developed by Facebook, designed with two main use cases in mind: (1) text generation and (2) text restoration. BART has an encoder-decoder structure, where the encoder compresses the input text into a hidden state, and the decoder generates output text based on this state.

2. Key Ideas of BART

BART operates in two main steps:

  • Forward Transformation: It transforms the input text in various ways to enable the model to respond to diverse information.
  • Reverse Generation: It generates natural sentences that match the original content based on the transformed input.

3. Installing the Hugging Face Transformers Library

The Hugging Face transformers library provides various pre-trained models, including the BART model. First, we will install the library:

pip install transformers

4. Using the BART Model

First, let’s load the BART model and prepare some example data. Here is the basic usage:

from transformers import BartTokenizer, BartForConditionalGeneration

# Load BART model and tokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

# Input text
input_text = "Deep learning is a core technology of modern artificial intelligence."
input_ids = tokenizer.encode(input_text, return_tensors='pt')

# Model prediction
summary_ids = model.generate(input_ids)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

print("Summary result:", summary)

5. Understanding Inference Result Decoding

The model.generate() method executed in the above code calls the decoder of the BART model to generate a summary. This format is the output of the input sequence processed by BART and is in a tokenized form. It must be decoded to be converted back into a human-readable natural language form.

5.1. Decoding Process

The decoding process mainly occurs through the tokenizer.decode() method. During this process, the following important points should be considered:

  • Removing Special Tokens: Models like BART may include tokens used for special purposes during training (e.g., , , ). We set skip_special_tokens=True to remove these.
  • Sentence Connection: The tokens generated after decoding may often appear without spaces. It is necessary to perform tasks to separate them into natural sentences.

5.2. Various Decoding Techniques

BART supports various decoding techniques. Some of them are as follows:

  • Greedy Search: Selects the word with the highest probability.
  • Beam Search: Considers multiple unexplored paths to generate the final output.
  • Sampling: Randomly selects the next word to generate more creative outputs.

6. Practical Examples with BART

Now let’s perform summarization for several inserted texts. The example code below can help in understanding:

sample_texts = [
    "The definition of deep learning is a kind of machine learning based on artificial neural networks.",
    "The BART model works well for various NLP tasks such as text summarization, translation, and generation.",
    "Hugging Face shares various pre-trained models to help users easily utilize NLP models."
]

for text in sample_texts:
    input_ids = tokenizer.encode(text, return_tensors='pt')
    summary_ids = model.generate(input_ids)
    summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
    print("Summary:", summary)

7. Conclusion

The Hugging Face BART model demonstrates efficient and powerful performance across various NLP tasks. Through this tutorial, we understood the basic usage of the model and practiced with real example codes. These models continue to evolve, and their potential applications in the NLP field are limitless.

We encourage you to gain more experience through various tutorials and practices and to implement them in your own projects.

© 2023 Hugging Face Transformers Utilization Course

huggingface transformer training course, BART inference

With the advancement of machine learning, particularly in natural language processing (NLP), transformer models have shown innovative results in word embedding, sentence generation, and various other tasks. Among them, BART (Bidirectional and Auto-Regressive Transformers) has garnered attention as a model that demonstrates excellent performance across multiple NLP tasks such as text generation, summarization, and translation.

Introduction to BART

BART is a model developed by Facebook AI Research (Fair) that fundamentally combines two architectures: the Encoder-Decoder structure and modified language modeling. BART is trained by symmetrically transforming input text to restore the original text from noisy text. Thanks to this characteristic, BART is suitable for performing various language tasks that require adaptability, such as sentence summarization, translation, and question answering.

Main Features of BART

  • Bidirectional Encoder: BART’s encoder can consider context information from both directions, thanks to the bidirectionality of the Transformer model.
  • Auto-Regressive Decoder: The decoder operates by considering all previous words to predict the next word.
  • Noise Removal: The model is trained by randomly masking or transforming parts of the text to remove this noise.

Hugging Face Transformers Library

The Hugging Face Transformers library provides an API that allows easy use of various transformer models, such as BART. The advantages of this library include:

  • A variety of pre-trained models available
  • An easy and intuitive API
  • Advanced feature support for various NLP tasks

Installation Instructions

To install the Transformers library, use the following pip command:

pip install transformers

Example of Using the BART Model

This section will explain how to perform text summarization using the BART model. Below is a simple example code:

Example Code

from transformers import BartTokenizer, BartForConditionalGeneration

# Load the BART model and tokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

# Text to summarize
text = """
Deep learning is a subset of machine learning that is based on artificial neural networks.
It is used for various applications such as image recognition, natural language processing,
and more. Deep learning allows computers to learn from data in a hierarchical manner,
enabling them to achieve high accuracy in various tasks.
"""

# Tokenize the text and input it to the summarization model
inputs = tokenizer(text, return_tensors='pt', max_length=1024, truncation=True)

# Generate the summary
summary_ids = model.generate(inputs['input_ids'], max_length=50, min_length=25, length_penalty=2.0, num_beams=4, early_stopping=True)

# Decode the summarized text
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print("Summary:", summary)

Code Explanation

  1. Import Libraries: Import the BART model and tokenizer from the `transformers` library.
  2. Loading the Model and Tokenizer: Retrieve the pre-trained model and tokenizer from `facebook/bart-large-cnn`.
  3. Input Text: Set the long text to be summarized.
  4. Tokenizing: Tokenize the input text and convert it to a tensor.
  5. Generating Summary: Use the `generate` method to create the summary.
  6. Output Result: Decode and print the generated summary.

Explanation of Control Parameters

The above code allows for adjusting the quality of the summary through various control parameters. Each parameter plays the following role:

  • max_length: The maximum length of the generated summary.
  • min_length: The minimum length of the generated summary.
  • length_penalty: A penalty for the length of the summary, giving lower scores for longer outputs to adjust the length.
  • num_beams: The number of beams to use in beam search, with higher values exploring more candidates.
  • early_stopping: Determines whether to stop the process when the optimal summary is generated.

Various Applications of BART

BART can be used for various NLP tasks beyond summarization. Here are some key use cases of BART:

1. Machine Translation

BART is effectively used in translation tasks to convert input text into another language, allowing users to perform translations from the source language to the target language.

2. Question Answering

BART demonstrates strong performance in generating answers to given questions.

3. Text Generation

BART can also be used to generate high-quality text in free-form text generation tasks.

Evaluating Model Performance

Various metrics can be used to evaluate the performance of the BART model. The ROUGE metric is commonly used. ROUGE measures the similarity between machine-generated summaries and human summaries, providing various metrics such as F1 score and recall.

Calculating ROUGE Scores

Below is an example of how to calculate ROUGE scores using Python:

from rouge import Rouge

# Human summary and model summary
human_summary = "Deep learning is a subset of machine learning."
model_summary = summary  # The model summary generated above

# Create a ROUGE evaluator object
rouge = Rouge()

# Calculate ROUGE scores
scores = rouge.get_scores(model_summary, human_summary)
print("ROUGE Scores:", scores)

Conclusion

The BART model is a highly useful tool applicable to various natural language processing tasks such as effective text summarization, translation, and question answering. It can be easily used through the Hugging Face Transformers library, and many researchers and developers are leveraging it to achieve innovative results in the field of NLP. Through this tutorial, I hope you gain an understanding of the basic concepts of BART and its usage, and acquire experience in performing text summarization.

References

Using Hugging Face Transformers, Setting Up the BART Library, and Loading Pre-trained Models

What is BART (Bidirectional and Auto-Regressive Transformers)?

BART is a deep learning model designed for Natural Language Processing (NLP) tasks, developed by Facebook AI Research. BART has an encoder-decoder structure and can be effectively used for various NLP tasks. In particular, BART demonstrates excellent performance in tasks such as text summarization, translation, question answering, and document generation.

What is the Hugging Face Transformers library?

Hugging Face’s Transformers library is a Python library that makes it easy to use various pre-trained language models. This library supports not only BART but also various models such as BERT, GPT-2, and T5. Additionally, it provides advanced APIs and tools for model usage and training.

1. Setting up the BART library

1.1. Environment setup

To use BART, you first need to install Python and the Hugging Face Transformers library. You can install it using the command below.

pip install transformers torch

Executing the above command will complete the installation of the Transformers library and PyTorch.

1.2. Loading the pre-trained model

The method to load a pre-trained BART model is as follows. First, you need to import the BART model and tokenizer from the Transformers library. You can implement this with the code below.


from transformers import BartTokenizer, BartForConditionalGeneration

# Load BART model and tokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')

The code above loads a pre-trained BART model called ‘facebook/bart-large’. This allows you to perform advanced natural language generation tasks.

2. Text Summarization using BART

We will proceed with the text summarization task using BART. Let’s look at the process of converting a long input sentence into a short summary through the example below.

2.1. Preparing the example data


text = """
Deep learning is a field of machine learning that uses artificial neural networks to enable computers to learn from data.
Deep learning is used in various fields such as image recognition, natural language processing, and speech recognition, 
and has experienced rapid advancements in recent years, particularly due to the combination of large amounts of data and powerful computing power.
"""

2.2. Preprocessing the input text

The input text must be encoded into a format that the model can understand. This can be done using the code below.


inputs = tokenizer(text, return_tensors='pt', max_length=1024, truncation=True)

2.3. Generating the summary through the model

Using the preprocessed input data, we generate the summary through the model.


summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=50, early_stopping=True)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)

The code above generates a summary using the BART model and prints the result. The num_beams parameter is a setting for beam search, where a higher value can be set for better summary results.

3. Fine-tuning BART

To further enhance the model’s performance, fine-tuning can be performed. Fine-tuning refers to the process of retraining a pre-trained model using a specific dataset.

3.1. Preparing the dataset

To perform fine-tuning, training and validation datasets must be prepared. The code below shows the process of preparing an example dataset.


# Example dataset
train_data = [
    {"input": "Deep learning is a field of machine learning.", "target": "Deep learning"},
    {"input": "Natural language processing technology is advancing.", "target": "Natural language processing"},
]

train_texts = [item["input"] for item in train_data]
train_summaries = [item["target"] for item in train_data]

3.2. Converting the dataset to tensors

It is also necessary to convert the training data into tensors so that they can be input into the BART model. This can be done as follows.


train_encodings = tokenizer(train_texts, truncation=True, padding=True, return_tensors='pt')
train_labels = tokenizer(train_summaries, truncation=True, padding=True, return_tensors='pt')['input_ids']
train_labels[train_labels == tokenizer.pad_token_id] = -100  # Set padding mask

3.3. Initializing the Trainer and performing fine-tuning

Using the Trainer from the Transformers library, the model can be fine-tuned.


from transformers import Trainer, TrainingArguments

# Set training arguments
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=2,
    save_total_limit=2,
)

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_encodings,  # Training dataset
)

# Start model fine-tuning
trainer.train()

The code above shows the process of fine-tuning the model. The num_train_epochs setting determines how many times the training loop runs.

Conclusion

The BART model can be effectively used for various natural language processing tasks and can be conveniently accessed through Hugging Face’s Transformers library. In this tutorial, we learned the process from setting up the BART model to generating summaries and fine-tuning.

For more details and additional examples, please refer to the official documentation of Hugging Face.

Transformers Course Using Hugging Face, Loading MLM Pipeline with ALBERT

1. Introduction

Recently, in the field of Natural Language Processing (NLP), pre-trained models have shown excellent performance across various tasks. Among them, ALBERT (A Lite BERT) is a lightweight BERT model proposed by Google. ALBERT reduces model size through parameter sharing and matrix decomposition techniques, allowing it to achieve high performance with fewer resources. In this course, we will explore in depth how to load ALBERT using Hugging Face’s Transformers library and set up a Masked Language Modeling (MLM) pipeline.

2. Overview of ALBERT

ALBERT is a model based on the structure of BERT, with the following key features:

  • Parameter Sharing: Shares all parameters across layers to reduce model size.
  • Matrix Decomposition: Reduces memory usage by decomposing a large embedding matrix into two smaller matrices.
  • Deeper Model: Allows the use of deeper architectures to enhance performance.

ALBERT demonstrates superior performance in various NLP tasks compared to BERT, especially leaving remarkable results despite a reduction in the number of parameters.

3. Environment Setup

To utilize ALBERT, you first need to install Hugging Face’s Transformers library. This library simplifies the loading and use of NLP models. You can install Transformers and torch using the following command: 

            !pip install transformers torch

After installation is complete, import the necessary libraries. 

            
import torch
from transformers import AlbertTokenizer, AlbertForMaskedLM

4. Loading ALBERT Model and Tokenizer

The ALBERT model is provided in a pre-trained form, making it easy to load. The following steps illustrate how to load ALBERT’s tokenizer and MLM model. 

            
# Load ALBERT model and tokenizer
model_name = 'albert-base-v2'
tokenizer = AlbertTokenizer.from_pretrained(model_name)
model = AlbertForMaskedLM.from_pretrained(model_name)

Running the above code will automatically download and load the ALBERT model and its corresponding tokenizer from Hugging Face’s model hub.

5. Overview of Masked Language Modeling (MLM)

Masked Language Modeling is a task that involves predicting masked words in a text. ALBERT is designed to perform this task effectively. Through MLM, the model learns from a vast amount of language data, enabling it to understand syntactic and semantic patterns.

6. Building the MLM Pipeline

The pipeline for performing MLM includes the following steps:

  • Preprocessing the input sentence
  • Masking words within the sentence
  • Making predictions using the model
  • Analyzing the results

Let’s take a closer look at this process below.

6.1 Preprocessing the Input Sentence

First, define the sentence to be input into the model and tokenize it to match the ALBERT model. The tokenizer splits the sentence into tokens and converts them into integer indices. Below is the process of preprocessing the input sentence. 

            
# Define the input sentence
input_sentence = "Hugging Face is opening the future of NLP."

# Tokenize the sentence
input_ids = tokenizer.encode(input_sentence, return_tensors='pt')
print("Input IDs:", input_ids)

6.2 Masking Words within the Sentence

For MLM, some words in the sentence are masked. The masked target is selected randomly. The following code masks one token randomly. 

            
import random

# Mask one token randomly
masked_index = random.randint(1, input_ids.size(1)-1)  # Exclude 0 as it is the [CLS] token
masked_input_ids = input_ids.clone()
masked_input_ids[0, masked_index] = tokenizer.mask_token_id

print("Masked Input IDs:", masked_input_ids)

6.3 Making Predictions Using the Model

Input the masked sentence into the model to predict the masked token. This is done by passing it through the model and retrieving the results. 

            
# Make predictions using the model
with torch.no_grad():
    outputs = model(masked_input_ids)
    
predictions = outputs[0]
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.decode(predicted_index)

print("Predicted Token:", predicted_token)

6.4 Analyzing the Results

Replace the masked token with the predicted token to check the results. 

            
# Replace the masked token with the predicted token
input_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
input_tokens[masked_index] = predicted_token

output_sentence = tokenizer.convert_tokens_to_string(input_tokens)
print("Output Sentence:", output_sentence)

7. Conclusion

We learned how to perform Masked Language Modeling using innovative models like ALBERT. We discovered how to easily load and utilize models through Hugging Face’s Transformers library and systematically learned from basic concepts to application methods. Through these techniques, you may contribute to the development of more advanced applications in the field of natural language processing.