In the field of deep learning, natural language processing (NLP) plays a very important role. In particular, the BERT (Bidirectional Encoder Representations from Transformers) model is widely used in the field of NLP.
\n In this course, we will explain how to install the Mobile BERT model using Hugging Face’s Transformers library and how to load the pre-trained model.
\n Mobile BERT is a lightweight BERT model, which has the advantage of being efficiently usable on mobile devices.<\/p>\n
The Hugging Face Transformers library is a Python library that helps you easily use various state-of-the-art NLP models.
\n Through this library, you can load various pre-trained models such as BERT, GPT-2, T5, and apply them to various NLP tasks. It also provides APIs for user-customized training.<\/p>\n
Mobile BERT is a lightweight BERT model developed by Google. Traditional BERT models are pre-trained on large-scale datasets and show strong performance, but
\n their large size poses constraints for use on mobile devices or embedded systems.
\n In contrast, Mobile BERT is designed to reduce size while maintaining as much performance as possible. Thanks to this characteristic, Mobile BERT is being utilized in various NLP tasks.<\/p>\n
To use Mobile BERT, you first need to install the Hugging Face Transformers library and other necessary libraries.
\n You can install the required libraries using pip with the following command:<\/p>\n
pip install transformers torch<\/code><\/pre>\nOnce installation is completed with the command above, you will be ready to use Mobile BERT in your Python environment.<\/p>\n
\nNote:<\/strong> If you haven\u2019t installed PyTorch, you need to do so. If you are using a GPU that supports CUDA,
\n choose the appropriate version of PyTorch for CUDA from the official website and install it.\n <\/div>\n4. Loading Pre-trained Models<\/h2>\n
Now, let’s load the Mobile BERT model. The Hugging Face Transformers library provides several classes to easily use pre-trained models.<\/p>\n
4.1 Code Example<\/h3>\n
The following code loads Mobile BERT:<\/p>\n
from transformers import MobileBertTokenizer, MobileBertForSequenceClassification\nimport torch\n\n# Load Mobile BERT model and tokenizer\nmodel_name = \"google\/mobilebert-uncased\"\ntokenizer = MobileBertTokenizer.from_pretrained(model_name)\nmodel = MobileBertForSequenceClassification.from_pretrained(model_name)\n\n# Sentence to test\ninput_text = \"The Hugging Face transformer is very useful!\"\n\n# Tokenize the input sentence and convert to tensor\ninputs = tokenizer(input_text, return_tensors=\"pt\")\n\n# Input the data into the model and predict\nwith torch.no_grad():\n logits = model(**inputs).logits\n\npredicted_class = torch.argmax(logits, dim=-1).item()\nprint(f\"Predicted class: {predicted_class}\")<\/code><\/pre>\n4.2 Code Explanation<\/h3>\n
Examining each element of the code, we have:<\/p>\n
\nfrom transformers import MobileBertTokenizer, MobileBertForSequenceClassification<\/code>:
\n Loads the Mobile BERT model and tokenizer.<\/li>\nmodel_name = \"google\/mobilebert-uncased\"<\/code>: Sets the name of the pre-trained model to use.<\/li>\ntokenizer = MobileBertTokenizer.from_pretrained(model_name)<\/code>: Initializes the tokenizer for the model.<\/li>\nmodel = MobileBertForSequenceClassification.from_pretrained(model_name)<\/code>: Initializes the model.
\n At this point, the model is suitable for sentence classification tasks.<\/li>\ninputs = tokenizer(input_text, return_tensors=\"pt\")<\/code>: Tokenizes the input sentence and converts it to a PyTorch tensor.<\/li>\nwith torch.no_grad():<\/code>: Configures to not track gradients of tensors for better memory efficiency.<\/li>\nlogits = model(**inputs).logits<\/code>: Retrieves the predictions made by the model.<\/li>\npredicted_class = torch.argmax(logits, dim=-1).item()<\/code>: Selects the class with the highest probability among the predicted classes.<\/li>\n<\/ul>\n5. A Practical Example Using Mobile BERT<\/h2>\n
Let’s take a look at an example of performing sentence classification using the Mobile BERT model.
\n The approach is to classify whether the given sentence is positive or negative.<\/p>\n
5.1 Preparing the Dataset<\/h3>\n
First, we need to prepare reliable data. For example, a movie review dataset is divided into positive and negative reviews.
\n This can be used to train the model. Let’s write code to load and preprocess the data.<\/p>\n
import pandas as pd\n\n# Load sample data (5 positive reviews, 5 negative reviews)\ndata = {\n \"text\": [\n \"I really love this movie.\", \n \"It's the best movie!\", \n \"It was really moving.\", \n \"A perfect masterpiece.\", \n \"This movie touched my heart.\",\n \"This is a waste of time.\", \n \"It's bad and boring.\", \n \"I was really disappointed.\", \n \"Never watch it.\", \n \"This movie is the worst.\"\n ],\n \"label\": [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]\n}\n\ndf = pd.DataFrame(data)\nprint(df.head())<\/code><\/pre>\n5.2 Training the Model<\/h3>\n
Now we move on to the process of training the model with the data. We can write a simple training loop to train the model.
\n However, here we will not delve into the details of the training process, but we will proceed with a simple transfer learning using the provided data.<\/p>\n
from torch.utils.data import DataLoader, Dataset\n\nclass CustomDataset(Dataset):\n def __init__(self, texts, labels, tokenizer):\n self.texts = texts\n self.labels = labels\n self.tokenizer = tokenizer\n\n def __len__(self):\n return len(self.labels)\n\n def __getitem__(self, idx):\n text = self.texts[idx]\n label = self.labels[idx]\n encoding = self.tokenizer(text, return_tensors='pt', padding='max_length', truncation=True, max_length=128)\n return {'input_ids': encoding['input_ids'].flatten(), 'attention_mask': encoding['attention_mask'].flatten(), 'label': torch.tensor(label, dtype=torch.long)}\n\ndataset = CustomDataset(df['text'].values, df['label'].values, tokenizer)\ndataloader = DataLoader(dataset, batch_size=2, shuffle=True)\n\n# Simple training loop\nfor epoch in range(3):\n for batch in dataloader:\n model.train()\n outputs = model(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'], labels=batch['label'])\n loss = outputs.loss\n loss.backward()\n # Optimizer Step etc are omitted\n print(f\"Epoch {epoch + 1}, Loss: {loss.item()}\")<\/code><\/pre>\n5.3 Predictions and Evaluation<\/h3>\n
After the model is trained, we can make predictions on new sentences. Let\u2019s verify this with the following example:<\/p>\n
test_text = \"This movie is very good.\"\ntest_inputs = tokenizer(test_text, return_tensors=\"pt\")\n\nwith torch.no_grad():\n test_logits = model(**test_inputs).logits\n\ntest_predicted_class = torch.argmax(test_logits, dim=-1).item()\nprint(f\"Predicted class for the test sentence '{test_text}': {test_predicted_class}\")<\/code><\/pre>\n6. Conclusion<\/h2>\n
In this course, we explored how to install the Mobile BERT model using the Hugging Face Transformers library,
\n load a pre-trained model, and perform sentence classification tasks. Mobile BERT is a lightweight model, making it useful in mobile environments or resource-constrained settings.
\n We encourage further research into its applicability to various NLP tasks.<\/p>\n
If you found this course helpful, please share it with others! If you have any additional materials or questions, feel free to leave a comment.<\/p>\n
<\/body><\/p>\n","protected":false},"excerpt":{"rendered":"