Using Hugging Face Transformers, Importing BERT Pre-trained Model into the Pipeline

1. Introduction

Recently, the BERT (Bidirectional Encoder Representations from Transformers) model has been playing a crucial role in the fields of artificial intelligence and natural language processing (NLP). BERT is a transformer-based pre-trained model developed by Google, demonstrating state-of-the-art performance across various NLP tasks. In this course, we will learn how to easily utilize the BERT model using Python and the Hugging Face Transformers library.

2. Installing the Hugging Face Transformers Library

First, we need to install the Hugging Face Transformers library. You can install it using the Python package manager pip.

pip install transformers

3. Understanding the BERT Model

BERT enables better natural language understanding by comprehending the bidirectional context of input sentences. In other words, it can read sentences from left to right or right to left, allowing for richer contextual information.

4. Loading the BERT Pre-trained Model

Loading the BERT model using the Hugging Face library is very straightforward. Below is the basic code to load the BERT model.


from transformers import BertTokenizer, BertForMaskedLM
import torch

# Load BERT model and tokenizer
model_name = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForMaskedLM.from_pretrained(model_name)

# Example sentence
text = "Hello, my name is [MASK]."
inputs = tokenizer(text, return_tensors="pt")

# Prediction
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits
    predicted_index = torch.argmax(predictions, dim=-1)
    predicted_token = tokenizer.decode(predicted_index[0, inputs['input_ids'].tolist().index(tokenizer.mask_token_id)])

print(f"Predicted word: {predicted_token}")

4.1 Code Explanation

The above code consists of the following steps:

  • BertTokenizer: Converts the given text into a tensor format suitable for the BERT model.
  • BertForMaskedLM: Loads the BERT model. This model is suitable for language modeling, particularly masked language modeling.
  • inputs: Encodes the input sentence into tensor format.
  • outputs: Passes the input to the model to generate prediction logits.
  • predicted_index: Extracts the index of the token with the highest probability.
  • Final Predicted Word: Converts the index back into an actual word.

5. Text Classification Using BERT

The BERT model can also be easily applied to text classification tasks. The following code shows a simple example of using BERT to analyze the sentiment of a given text.


from transformers import BertTokenizer, BertForSequenceClassification

# Load sentiment classification model
model_name = 'nlptown/bert-base-multilingual-uncased-sentiment'
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)

# Example sentence
sentence = "I love using Hugging Face Transformers!"
inputs = tokenizer(sentence, return_tensors="pt")

# Prediction
with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    predicted_class = torch.argmax(logits, dim=-1)

print(f"Predicted sentiment class: {predicted_class.item()}")

5.1 Text Classification Code Explanation

The above code performs sentiment classification and follows these steps:

  • Loads the nlptown/bert-base-multilingual-uncased-sentiment model to perform multilingual sentiment classification tasks.
  • Converts the input sentence into tensor format using the tokenizer.
  • Calculates logits by providing the input to the model.
  • Selects the class with the highest value from the logits and outputs the predicted sentiment class.

6. Summary

It can be seen that the BERT model can perform various tasks in the field of natural language processing. By using the Hugging Face library, these models can be easily utilized, and their performance can be further improved through experimentation. In the future, fine-tuning or extending to other tasks can also be considered.

7. References

  • Devlin, J. et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  • Hugging Face Transformers Documentation. Link