!pip install transformers torch numpy pandas<\/code><\/pre>\n2. Preparing the Data<\/h3>\n
First, we need to prepare a dataset for the multi-class classification problem. As an example, let’s create a simple dataframe with text and labels.<\/p>\n
import pandas as pd\n\ndata = {\n 'text': [\n 'I like natural language processing.',\n 'PyTorch and TensorFlow are popular.',\n 'Deep learning is a field of machine learning.',\n 'Conversational AI is gaining a lot of attention.',\n 'Text classification is an important task.'\n ],\n 'label': [0, 1, 1, 2, 0]\n}\n\ndf = pd.DataFrame(data)<\/code><\/pre>\n3. Data Preprocessing<\/h3>\n
Prepare the data in the format required by the BERT model. We use the BERT Tokenizer to tokenize the text and generate input IDs and attention masks.<\/p>\n
from transformers import BertTokenizer\n\ntokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n\n# Tokenization and generating input IDs and attention masks\ndef encode_data(text):\n return tokenizer(text, padding='max_length', truncation=True, max_length=128, return_tensors='pt')\n\nencoded_texts = [encode_data(text)['input_ids'] for text in df['text']]\nattention_masks = [encode_data(text)['attention_mask'] for text in df['text']]<\/code><\/pre>\n4. Splitting the Dataset<\/h3>\n
We split the data into training and validation sets. Here, we will use 80% of the data for training and 20% for validation.<\/p>\n
from sklearn.model_selection import train_test_split\n\nX_train, X_val, y_train, y_val = train_test_split(\n df['text'],\n df['label'],\n test_size=0.2,\n random_state=42\n)<\/code><\/pre>\n5. Creating Data Loaders<\/h3>\n
Using PyTorch’s DataLoader, we create data loaders for batch processing.<\/p>\n
import torch\nfrom torch.utils.data import Dataset, DataLoader\n\nclass TextDataset(Dataset):\n def __init__(self, texts, labels):\n self.texts = texts\n self.labels = labels\n\n def __len__(self):\n return len(self.texts)\n\n def __getitem__(self, idx):\n text = encode_data(self.texts[idx])\n return {\n 'input_ids': text['input_ids'].squeeze(),\n 'attention_mask': text['attention_mask'].squeeze(),\n 'labels': torch.tensor(self.labels[idx])\n }\n\ntrain_dataset = TextDataset(X_train.tolist(), y_train.tolist())\nval_dataset = TextDataset(X_val.tolist(), y_val.tolist())\n\ntrain_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)<\/code><\/pre>\n6. Loading the Model<\/h3>\n
Load the pre-trained BERT model from Hugging Face’s Transformers library. We will add a classifier here to address the multi-class classification problem.<\/p>\n
from transformers import BertForSequenceClassification\n\nmodel = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=3)<\/code><\/pre>\n7. Training the Model<\/h3>\n
To train the model, we set up a loss function and optimization algorithm, and create a simple training loop.<\/p>\n
from transformers import AdamW\nfrom tqdm import tqdm\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel.to(device)\n\noptimizer = AdamW(model.parameters(), lr=1e-5)\n\n# Model training\nmodel.train()\nfor epoch in range(3): # Number of epochs\n for batch in tqdm(train_loader):\n input_ids = batch['input_ids'].to(device)\n attention_mask = batch['attention_mask'].to(device)\n labels = batch['labels'].to(device)\n\n optimizer.zero_grad()\n outputs = model(input_ids, attention_mask=attention_mask, labels=labels)\n loss = outputs.loss\n loss.backward()\n optimizer.step()\n\n print(f'Epoch {epoch+1} Loss: {loss.item()}')<\/code><\/pre>\n8. Validation and Performance Evaluation<\/h3>\n
We evaluate the model’s performance using the validation data. Here we measure the accuracy.<\/p>\n
model.eval()\ncorrect = 0\ntotal = 0\n\nwith torch.no_grad():\n for batch in val_loader:\n input_ids = batch['input_ids'].to(device)\n attention_mask = batch['attention_mask'].to(device)\n labels = batch['labels'].to(device)\n\n outputs = model(input_ids, attention_mask=attention_mask)\n _, predicted = torch.max(outputs.logits, dim=1)\n\n total += labels.size(0)\n correct += (predicted == labels).sum().item()\n\naccuracy = correct \/ total\nprint(f'Accuracy: {accuracy:.2f}') <\/code><\/pre>\n9. Conclusion<\/h3>\n
In this tutorial, we learned how to utilize a pre-trained BERT model for multi-class classification problems using the Hugging Face Transformers library. BERT demonstrates powerful performance, making it applicable to many natural language processing problems you may want to analyze. In real projects, you should achieve optimal results through various experiments and tuning processes. Transformer models are rapidly advancing, so continuous learning is necessary.<\/p>\n
If you have any further questions, feel free to ask!<\/p>\n
<\/body><\/p>\n","protected":false},"excerpt":{"rendered":"
Loading a Pre-trained BERT Model for Multi-class Classification BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing model proposed by Google that utilizes a bidirectional Transformer architecture for contextual understanding. BERT can be applied to various natural language processing tasks through pre-training and fine-tuning stages. In this tutorial, we will introduce how to … \ub354 \ubcf4\uae30 “Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[108],"tags":[],"class_list":["post-36185","post","type-post","status-publish","format-standard","hentry","category---en"],"yoast_head":"\nUsing Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/atmokpo.com\/w\/36185\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\" \/>\n<meta property=\"og:description\" content=\"Loading a Pre-trained BERT Model for Multi-class Classification BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing model proposed by Google that utilizes a bidirectional Transformer architecture for contextual understanding. BERT can be applied to various natural language processing tasks through pre-training and fine-tuning stages. In this tutorial, we will introduce how to … \ub354 \ubcf4\uae30 "Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/atmokpo.com\/w\/36185\/\" \/>\n<meta property=\"og:site_name\" content=\"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\" \/>\n<meta property=\"article:published_time\" content=\"2024-11-01T09:46:28+00:00\" \/>\n<meta name=\"author\" content=\"root\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@bebubo4\" \/>\n<meta name=\"twitter:site\" content=\"@bebubo4\" \/>\n<meta name=\"twitter:label1\" content=\"\uae00\uc4f4\uc774\" \/>\n\t<meta name=\"twitter:data1\" content=\"root\" \/>\n\t<meta name=\"twitter:label2\" content=\"\uc608\uc0c1 \ub418\ub294 \ud310\ub3c5 \uc2dc\uac04\" \/>\n\t<meta name=\"twitter:data2\" content=\"4\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/\"},\"author\":{\"name\":\"root\",\"@id\":\"https:\/\/atmokpo.com\/w\/#\/schema\/person\/91b6b3b138fbba0efb4ae64b1abd81d7\"},\"headline\":\"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification\",\"datePublished\":\"2024-11-01T09:46:28+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/\"},\"wordCount\":343,\"publisher\":{\"@id\":\"https:\/\/atmokpo.com\/w\/#organization\"},\"articleSection\":[\"Using Hugging Face\"],\"inLanguage\":\"ko-KR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/\",\"url\":\"https:\/\/atmokpo.com\/w\/36185\/\",\"name\":\"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\",\"isPartOf\":{\"@id\":\"https:\/\/atmokpo.com\/w\/#website\"},\"datePublished\":\"2024-11-01T09:46:28+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/atmokpo.com\/w\/36185\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/atmokpo.com\/w\/36185\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\ud648\",\"item\":\"https:\/\/atmokpo.com\/w\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/atmokpo.com\/w\/#website\",\"url\":\"https:\/\/atmokpo.com\/w\/\",\"name\":\"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/atmokpo.com\/w\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/atmokpo.com\/w\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/atmokpo.com\/w\/#organization\",\"name\":\"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\",\"url\":\"https:\/\/atmokpo.com\/w\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/atmokpo.com\/w\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/atmokpo.com\/w\/wp-content\/uploads\/2024\/11\/logo.png\",\"contentUrl\":\"https:\/\/atmokpo.com\/w\/wp-content\/uploads\/2024\/11\/logo.png\",\"width\":400,\"height\":400,\"caption\":\"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8\"},\"image\":{\"@id\":\"https:\/\/atmokpo.com\/w\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/bebubo4\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/atmokpo.com\/w\/#\/schema\/person\/91b6b3b138fbba0efb4ae64b1abd81d7\",\"name\":\"root\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/atmokpo.com\/w\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/708197b41fc6435a7ce22d951b25d4a47e9e904270cb1f04682d4f025066f80c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/708197b41fc6435a7ce22d951b25d4a47e9e904270cb1f04682d4f025066f80c?s=96&d=mm&r=g\",\"caption\":\"root\"},\"sameAs\":[\"http:\/\/atmokpo.com\/w\"],\"url\":\"https:\/\/atmokpo.com\/w\/author\/root\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/atmokpo.com\/w\/36185\/","og_locale":"ko_KR","og_type":"article","og_title":"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","og_description":"Loading a Pre-trained BERT Model for Multi-class Classification BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing model proposed by Google that utilizes a bidirectional Transformer architecture for contextual understanding. BERT can be applied to various natural language processing tasks through pre-training and fine-tuning stages. In this tutorial, we will introduce how to … \ub354 \ubcf4\uae30 \"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification\"","og_url":"https:\/\/atmokpo.com\/w\/36185\/","og_site_name":"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","article_published_time":"2024-11-01T09:46:28+00:00","author":"root","twitter_card":"summary_large_image","twitter_creator":"@bebubo4","twitter_site":"@bebubo4","twitter_misc":{"\uae00\uc4f4\uc774":"root","\uc608\uc0c1 \ub418\ub294 \ud310\ub3c5 \uc2dc\uac04":"4\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/atmokpo.com\/w\/36185\/#article","isPartOf":{"@id":"https:\/\/atmokpo.com\/w\/36185\/"},"author":{"name":"root","@id":"https:\/\/atmokpo.com\/w\/#\/schema\/person\/91b6b3b138fbba0efb4ae64b1abd81d7"},"headline":"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification","datePublished":"2024-11-01T09:46:28+00:00","mainEntityOfPage":{"@id":"https:\/\/atmokpo.com\/w\/36185\/"},"wordCount":343,"publisher":{"@id":"https:\/\/atmokpo.com\/w\/#organization"},"articleSection":["Using Hugging Face"],"inLanguage":"ko-KR"},{"@type":"WebPage","@id":"https:\/\/atmokpo.com\/w\/36185\/","url":"https:\/\/atmokpo.com\/w\/36185\/","name":"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification - \ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","isPartOf":{"@id":"https:\/\/atmokpo.com\/w\/#website"},"datePublished":"2024-11-01T09:46:28+00:00","breadcrumb":{"@id":"https:\/\/atmokpo.com\/w\/36185\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/atmokpo.com\/w\/36185\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/atmokpo.com\/w\/36185\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\ud648","item":"https:\/\/atmokpo.com\/w\/en\/"},{"@type":"ListItem","position":2,"name":"Using Hugging Face Transformers, Loading Pre-trained BERT Model for Multi-class Classification"}]},{"@type":"WebSite","@id":"https:\/\/atmokpo.com\/w\/#website","url":"https:\/\/atmokpo.com\/w\/","name":"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","description":"","publisher":{"@id":"https:\/\/atmokpo.com\/w\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/atmokpo.com\/w\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/atmokpo.com\/w\/#organization","name":"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8","url":"https:\/\/atmokpo.com\/w\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/atmokpo.com\/w\/#\/schema\/logo\/image\/","url":"https:\/\/atmokpo.com\/w\/wp-content\/uploads\/2024\/11\/logo.png","contentUrl":"https:\/\/atmokpo.com\/w\/wp-content\/uploads\/2024\/11\/logo.png","width":400,"height":400,"caption":"\ub77c\uc774\ube0c\uc2a4\ub9c8\ud2b8"},"image":{"@id":"https:\/\/atmokpo.com\/w\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/bebubo4"]},{"@type":"Person","@id":"https:\/\/atmokpo.com\/w\/#\/schema\/person\/91b6b3b138fbba0efb4ae64b1abd81d7","name":"root","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/atmokpo.com\/w\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/708197b41fc6435a7ce22d951b25d4a47e9e904270cb1f04682d4f025066f80c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/708197b41fc6435a7ce22d951b25d4a47e9e904270cb1f04682d4f025066f80c?s=96&d=mm&r=g","caption":"root"},"sameAs":["http:\/\/atmokpo.com\/w"],"url":"https:\/\/atmokpo.com\/w\/author\/root\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/posts\/36185","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/comments?post=36185"}],"version-history":[{"count":1,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/posts\/36185\/revisions"}],"predecessor-version":[{"id":36186,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/posts\/36185\/revisions\/36186"}],"wp:attachment":[{"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/media?parent=36185"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/categories?post=36185"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/atmokpo.com\/w\/wp-json\/wp\/v2\/tags?post=36185"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}