\ucd5c\uadfc \uc790\uc5f0\uc5b4 \ucc98\ub9ac(NLP) \ubd84\uc57c\uc5d0\uc11c \ud2b8\ub79c\uc2a4\ud3ec\uba38(Transformer) \uae30\ubc18 \ubaa8\ub378\ub4e4\uc774 \ub6f0\uc5b4\ub09c \uc131\ub2a5\uc744 \ubcf4\uc774\uba70 \uac01\uad11\ubc1b\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uadf8 \uc911\uc5d0\uc11c\ub3c4 \uad6c\uae00\uc774 \uac1c\ubc1c\ud55c BigBird<\/strong>\ub294 \ub300\uaddc\ubaa8 \ubb38\uc11c \uc774\ud574\uc640 \uae34 \uc2dc\ud000\uc2a4\ub97c \ucc98\ub9ac\ud558\uae30 \uc704\ud55c \ud601\uc2e0\uc801\uc778 \uc544\ud0a4\ud14d\ucc98\uc785\ub2c8\ub2e4. \ubcf8 \uac15\uc88c\uc5d0\uc11c\ub294 \ud5c8\uae45\ud398\uc774\uc2a4\uc758 \ud2b8\ub79c\uc2a4\ud3ec\uba38 \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \ud65c\uc6a9\ud558\uc5ec BigBird \ubaa8\ub378\uc744 \uc124\uc815\ud558\uace0, \uc0ac\uc804\ud559\uc2b5\ub41c \ubaa8\ub378\uc744 \ubd88\ub7ec\uc624\ub294 \ubc29\ubc95\uc744 \uc54c\uc544\ubcf4\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n BigBird\ub294 Transformer \ubaa8\ub378\uc758 \ud655\uc7a5\uc73c\ub85c \uc124\uacc4\ub41c \ubaa8\ub378\ub85c, \ud2b9\ud788 \uae34 \uc2dc\ud000\uc2a4 \ub370\uc774\ud130\ub97c \ud6a8\uc728\uc801\uc73c\ub85c \ucc98\ub9ac\ud560 \uc218 \uc788\ub3c4\ub85d \ub9cc\ub4e4\uc5b4\uc84c\uc2b5\ub2c8\ub2e4. \uc804\ud1b5\uc801\uc778 Transformer \ubaa8\ub378\uc740 \uc785\ub825 \uc2dc\ud000\uc2a4\uc758 \uae38\uc774\uc5d0 \uc81c\ud55c\uc774 \uc788\uc5b4 \ucd5c\ub300 512\ud1a0\ud070 \uc815\ub3c4\uc758 \ud14d\uc2a4\ud2b8\ub9cc\uc744 \ucc98\ub9ac\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ubc18\uba74 BigBird\ub294 \ud76c\uc18c \uc8fc\uc758(sparse attention) \uba54\ucee4\ub2c8\uc998\uc744 \uc0ac\uc6a9\ud558\uc5ec \uc774 \ud55c\uacc4\ub97c \uadf9\ubcf5\ud569\ub2c8\ub2e4. \uc774\ub294 \ubb38\uc11c \uc694\uc57d, \uc9c8\uc758 \uc751\ub2f5, \ud14d\uc2a4\ud2b8 \ubd84\ub958 \ub4f1 \ub2e4\uc591\ud55c NLP \uc791\uc5c5\uc5d0 \uc720\uc6a9\ud569\ub2c8\ub2e4.<\/p>\n BigBird \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uae30 \uc704\ud574 Python \ud658\uacbd\uc744 \uc124\uc815\ud574\uc57c \ud569\ub2c8\ub2e4. \ub2e4\uc74c\uc758 \ub2e8\uacc4\uc5d0 \ub530\ub77c \uc124\uce58\ub97c \uc9c4\ud589\ud569\ub2c8\ub2e4.<\/p>\n Python 3.6 \uc774\uc0c1\uc774 \ud544\uc694\ud569\ub2c8\ub2e4. \uc544\ub798\uc758 \uba85\ub839\uc5b4\ub85c Python\uacfc pip\ub97c \uc124\uce58\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4:<\/p>\n \uc544\ub798 \uba85\ub839\uc5b4\ub97c \uc0ac\uc6a9\ud558\uc5ec \ud5c8\uae45\ud398\uc774\uc2a4\uc758 \ud2b8\ub79c\uc2a4\ud3ec\uba38 \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc124\uce58\ud569\ub2c8\ub2e4:<\/p>\n BigBird \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uae30 \uc704\ud574 \ucd94\uac00\uc801\uc778 \ub77c\uc774\ube0c\ub7ec\ub9ac\ub3c4 \uc124\uce58\ud574\uc57c \ud569\ub2c8\ub2e4:<\/p>\n \uc774\uc81c \ubaa8\ub4e0 \uc124\uc815\uc774 \ub05d\ub0ac\uc73c\ub2c8, BigBird \ubaa8\ub378\uc744 \ubd88\ub7ec\uc640\uc11c \uc0ac\uc6a9\ud560 \uc900\ube44\uac00 \uc644\ub8cc\ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ub97c \uc704\ud574 Hugging Face\uc758 BigBird \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud55c \ud14d\uc2a4\ud2b8 \uc694\uc57d \uc608\uc81c\ub97c \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uc544\ub798\uc758 \ucf54\ub4dc\ub97c \ucc38\uace0\ud558\uc138\uc694:<\/p>\n \uc704 \ucf54\ub4dc\uc5d0\uc11c \uc774\uc81c \uc0ac\uc804\ud559\uc2b5\ub41c \ubaa8\ub378\uc744 \uae30\ubc18\uc73c\ub85c \ud2b9\uc815 \ub370\uc774\ud130\uc14b\uc5d0 \ub300\ud574 \ucd94\uac00 \ud6c8\ub828\ud558\ub294 \ubc29\ubc95\uc744 \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uc544\ub798\ub294 \uac04\ub2e8\ud55c \ud6c8\ub828 \ub8e8\ud2f4\uc744 \ubcf4\uc5ec\uc8fc\ub294 \ucf54\ub4dc\uc785\ub2c8\ub2e4:<\/p>\n \uc704 \ucf54\ub4dc\uc5d0\uc11c\ub294 \uc774\ubc88 \uac15\uc88c\uc5d0\uc11c\ub294 \ud5c8\uae45\ud398\uc774\uc2a4 \ud2b8\ub79c\uc2a4\ud3ec\uba38 \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc0ac\uc6a9\ud558\uc5ec BigBird \ubaa8\ub378\uc744 \uc124\uc815\ud558\uace0 \uc0ac\uc804\ud559\uc2b5\ub41c \ubaa8\ub378\uc744 \ubd88\ub7ec\uc624\ub294 \ubc29\ubc95\uc744 \uc54c\uc544\ubcf4\uc558\uc2b5\ub2c8\ub2e4. BigBird\ub294 \uae34 \uc785\ub825 \uc2dc\ud000\uc2a4\uc5d0 \ub300\ud55c \ucc98\ub9ac\ub97c \ud6a8\uc728\uc801\uc73c\ub85c \uc218\ud589\ud560 \uc218 \uc788\ub294 \uac15\ub825\ud55c \ub3c4\uad6c\uc785\ub2c8\ub2e4. \ub2e4\uc591\ud55c NLP \uc791\uc5c5\uc5d0 \uc801\uc6a9\ud558\uc5ec \uc2e4\uc9c8\uc801\uc73c\ub85c \uc131\ub2a5\uc744 \uac1c\uc120\ud560 \uc218 \uc788\uc73c\uba70, \ubaa8\ub378\uc744 fine-tuning\ud558\uc5ec \ud2b9\uc815 \uc791\uc5c5\uc5d0 \ucd5c\uc801\ud654\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n \ub525\ub7ec\ub2dd\uc744 \ud65c\uc6a9\ud55c \ub2e4\uc591\ud55c \ud504\ub85c\uc81d\ud2b8\uc5d0\uc11c BigBird\uc640 \uac19\uc740 \ubaa8\ub378\uc744 \uc5b4\ub5bb\uac8c \ud65c\uc6a9\ud560 \uc218 \uc788\ub294\uc9c0 \uacc4\uc18d\ud574\uc11c \ud0d0\uad6c\ud558\uace0 \ub098\uc544\uac00\uc2dc\uae30\ub97c \ubc14\ub78d\ub2c8\ub2e4. \ucd94\uac00\uc801\uc778 \uc790\ub8cc\ub098 \uc9c8\ubb38\uc774 \ud544\uc694\ud558\uc2dc\uba74 \ub313\uae00\ub85c \ub0a8\uaca8 \uc8fc\uc138\uc694! \uac10\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n <\/body><\/p>\n","protected":false},"excerpt":{"rendered":"1. BigBird\ub780 \ubb34\uc5c7\uc778\uac00?<\/h2>\n
1.1 BigBird\uc758 \uc8fc\uc694 \ud2b9\uc9d5<\/h3>\n
\n
2. \ud658\uacbd \uc124\uc815<\/h2>\n
2.1 Python \ubc0f pip \uc124\uce58<\/h3>\n
sudo apt update\nsudo apt install python3 python3-pip<\/code><\/pre>\n2.2 \ud5c8\uae45\ud398\uc774\uc2a4 \ud2b8\ub79c\uc2a4\ud3ec\uba38 \ub77c\uc774\ube0c\ub7ec\ub9ac \uc124\uce58<\/h3>\n
pip install transformers<\/code><\/pre>\n2.3 \ucd94\uac00 \ub77c\uc774\ube0c\ub7ec\ub9ac \uc124\uce58<\/h3>\n
pip install torch<\/code><\/pre>\n3. \uc0ac\uc804\ud559\uc2b5 \ubaa8\ub378 \ubd88\ub7ec\uc624\uae30<\/h2>\n
transformers<\/code> \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc774\uc6a9\ud569\ub2c8\ub2e4.<\/p>\n3.1 \ud14d\uc2a4\ud2b8 \uc694\uc57d<\/h3>\n
from transformers import BigBirdTokenizer, BigBirdForSequenceClassification\n\n# \ud1a0\ud06c\ub098\uc774\uc800\uc640 \ubaa8\ub378 \ubd88\ub7ec\uc624\uae30\ntokenizer = BigBirdTokenizer.from_pretrained('google\/bigbird-roberta-base')\nmodel = BigBirdForSequenceClassification.from_pretrained('google\/bigbird-roberta-base')\n\n# \uc785\ub825 \ud14d\uc2a4\ud2b8\ntext = \"\ub525\ub7ec\ub2dd\uc740 \uae30\uacc4 \ud559\uc2b5\uc758 \ud55c \ubd84\uc57c\ub85c, \uc778\uacf5 \uc2e0\uacbd\ub9dd\uc744 \ud65c\uc6a9\ud55c \ubc29\ubc95\uc785\ub2c8\ub2e4. \ub370\uc774\ud130\uc5d0\uc11c \ud328\ud134\uc744 \ud559\uc2b5\ud558\uace0, \uc774\ub97c \uae30\ubc18\uc73c\ub85c \uc608\uce21 \ubc0f \uacb0\uc815\uc744 \ub0b4\ub9ac\ub294 \ub370 \uc0ac\uc6a9\ub429\ub2c8\ub2e4.\"\n\n# \ud14d\uc2a4\ud2b8\ub97c \ud1a0\ud070\ud654\ud558\uace0 \ud150\uc11c\ub85c \ubcc0\ud658\ninputs = tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True)\n\n# \ubaa8\ub378 \uc608\uce21\noutputs = model(**inputs)\nlogits = outputs.logits\npredicted_class = logits.argmax().item()\n\nprint(f\"\uc608\uce21\ub41c \ud074\ub798\uc2a4: {predicted_class}\")<\/code><\/pre>\n\ucf54\ub4dc \uc124\uba85<\/h4>\n
BigBirdTokenizer<\/code>\uc640 BigBirdForSequenceClassification<\/code> \ud074\ub798\uc2a4\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc0ac\uc804\ud559\uc2b5\ub41c BigBird \ubaa8\ub378\uacfc \ud1a0\ud06c\ub098\uc774\uc800\ub97c \ubd88\ub7ec\uc635\ub2c8\ub2e4.<\/p>\n\n
from_pretrained<\/code> \uba54\uc11c\ub4dc\ub97c \ud1b5\ud574 Google\uc758 pre-trained BigBird \ubaa8\ub378\uc744 \ubd88\ub7ec\uc635\ub2c8\ub2e4.<\/li>\ntokenizer<\/code>\ub97c \uc0ac\uc6a9\ud558\uc5ec \ud14d\uc2a4\ud2b8\ub97c \ud150\uc11c\ub85c \ubcc0\ud658\ud569\ub2c8\ub2e4.<\/li>\n3.2 \ubaa8\ub378 \ud6c8\ub828\ud558\uae30<\/h3>\n
from transformers import Trainer, TrainingArguments\nfrom datasets import load_dataset\n\n# \ub370\uc774\ud130\uc14b \ubd88\ub7ec\uc624\uae30 (\uc608: IMDB \uac10\uc131 \ubd84\uc11d \ub370\uc774\ud130\uc14b)\ndataset = load_dataset('imdb')\n\n# \ud6c8\ub828 \uc778\uc790 \uc124\uc815\ntraining_args = TrainingArguments(\n output_dir='.\/results',\n num_train_epochs=1,\n per_device_train_batch_size=8,\n save_steps=10_000,\n save_total_limit=2,\n)\n\n# Trainer \uac1d\uccb4 \uc0dd\uc131\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=dataset['train'],\n eval_dataset=dataset['test'],\n)\n\n# \ubaa8\ub378 \ud6c8\ub828\ntrainer.train()<\/code><\/pre>\n\ucf54\ub4dc \uc124\uba85<\/h4>\n
datasets<\/code> \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc0ac\uc6a9\ud558\uc5ec IMDB \uac10\uc131 \ubd84\uc11d \ub370\uc774\ud130\uc14b\uc744 \ubd88\ub7ec\uc635\ub2c8\ub2e4. \uc774 \ub370\uc774\ud130\uc14b\uc744 \uae30\ubc18\uc73c\ub85c BigBird \ubaa8\ub378\uc758 \ud6c8\ub828\uc744 \uc218\ud589\ud569\ub2c8\ub2e4:<\/p>\n\n
TrainingArguments<\/code>\ub85c \ud6c8\ub828\uc758 \uc5ec\ub7ec \uac00\uc9c0 \uc124\uc815(epochs, \ubc30\uce58 \uc0ac\uc774\uc988 \ub4f1)\uc744 \uc815\ud569\ub2c8\ub2e4.<\/li>\nTrainer<\/code> \ud074\ub798\uc2a4\ub97c \ud1b5\ud574 \ud6c8\ub828\uacfc \ud3c9\uac00\ub97c \uc218\ud589\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/li>\n<\/ul>\n4. \uc694\uc57d<\/h2>\n