{"id":36163,"date":"2024-11-01T09:46:17","date_gmt":"2024-11-01T09:46:17","guid":{"rendered":"http:\/\/atmokpo.com\/w\/?p=36163"},"modified":"2024-11-01T09:46:17","modified_gmt":"2024-11-01T09:46:17","slug":"hugging-face-transformers-tutorial-mobile-bert-inference-using-the-last-hidden-layer","status":"publish","type":"post","link":"https:\/\/atmokpo.com\/w\/36163\/","title":{"rendered":"Hugging Face Transformers Tutorial, Mobile BERT Inference Using the Last Hidden Layer"},"content":{"rendered":"

<\/p>\n

In recent years, deep learning-based models have gained popularity and made significant advancements in the field of Natural Language Processing (NLP). Among them, Hugging Face<\/strong>\u2019s transformer models are popular due to their ease of use and performance. In particular, Mobile BERT<\/strong> is a lightweight version of the BERT model, designed to be effectively used in mobile environments. In this course, we will introduce how to extract the output of the last hidden layer using the Mobile BERT model.<\/p>\n

1. What is Mobile BERT?<\/h2>\n

Mobile BERT is a model released by Google as a lightweight version of BERT. The BERT model is based on two main components: Encoder<\/strong> and Decoder<\/strong>, and Mobile BERT has optimized the Encoder to be lightweight so that it can be used on various mobile devices. Mobile BERT has 4 times fewer parameters and applies various techniques to enhance computational efficiency.<\/p>\n

2. Installing the Hugging Face Library<\/h2>\n

To use the Hugging Face transformer model, you first need to install the required libraries. You can use the command below to install the libraries.<\/p>\n

pip install transformers torch<\/code><\/pre>\n
3. Loading the Mobile BERT Model<\/h2>\n
Once the model is installed, you can load the Mobile BERT model. Here is a basic code snippet.<\/p>\n
from transformers import MobileBertTokenizer, MobileBertModel\n\n# Load Mobile BERT model and tokenizer\ntokenizer = MobileBertTokenizer.from_pretrained('google\/mobilebert-uncased')\nmodel = MobileBertModel.from_pretrained('google\/mobilebert-uncased')<\/code><\/pre>\n
4. Preprocessing Input Data<\/h2>\n
The data input to the Mobile BERT model must be in text format and should be converted into the appropriate format through the tokenizer. Here\u2019s how to preprocess the input sentence.<\/p>\n
# Define input sentence\ninput_text = \"Try using Hugging Face's transformer!\"\n\n# Tokenize the sentence and convert to indices\ninputs = tokenizer(input_text, return_tensors='pt')<\/code><\/pre>\n
5. Inference via the Model<\/h2>\n
Once preprocessing is complete, the data can be input to the Mobile BERT model to obtain the output of the last hidden layer. The output can be computed using the model’s forward<\/code> method.<\/p>\n
with torch.no_grad():\n    outputs = model(**inputs)\n\n# The output of the last hidden layer is stored in outputs[0].\nlast_hidden_states = outputs.last_hidden_state\nprint(last_hidden_states.shape)  # (batch size, sequence length, hidden size)<\/code><\/pre>\n
6. Interpreting Results<\/h2>\n
The output of the last hidden layer is returned as a 3-dimensional tensor. The first dimension of the tensor is the batch size, the second dimension is the sequence length (the number of words in the sentence), and the third dimension is the dimension of the hidden layer. For example, if the batch size is 1, the sequence length is 10, and the hidden layer dimension is 768, the shape of the output will be (1, 10, 768).<\/p>\n
7. Application Example: Extracting Embedding Vectors<\/h2>\n
The output of the last hidden layer can be used as the embedding vectors for each word. These vectors can be utilized for various NLP tasks.<\/p>\n
# Extract embedding vector of the first word\nword_embedding = last_hidden_states[0, 0, :]  # (768,)\nprint(word_embedding.shape)<\/code><\/pre>\n
8. Summary<\/h2>\n
In this post, we explored how to utilize the Mobile BERT model using the Hugging Face transformer library. We covered data preprocessing, inference processes, and how to obtain the sequence output of the last hidden layer. These methods can be employed in various NLP applications and are being used in many research and industrial fields.<\/p>\n
9. References<\/h2>\n
You can find more detailed information at the following links:<\/p>\n