\n1. \uc11c\ub860<\/h2>\n
\ub525\ub7ec\ub2dd \ubd84\uc57c\ub294 \ucd5c\uadfc \uba87 \ub144 \ub3d9\uc548 \uac00\ud30c\ub978 \ubc1c\uc804\uc744 \uc774\ub904\ub0c8\uc73c\uba70, \ud2b9\ud788 \uc790\uc5f0\uc5b4 \ucc98\ub9ac(NLP) \ubd84\uc57c\uc5d0\uc11c \ub9ce\uc740 \uc8fc\ubaa9\uc744 \ubc1b\uace0 \uc788\uc2b5\ub2c8\ub2e4. \ubcf8 \uae00\uc5d0\uc11c\ub294 Hugging Face\uc758 Transformers \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \ud65c\uc6a9\ud558\uc5ec BigBird \ubaa8\ub378\ub85c \ud14d\uc2a4\ud2b8 \ucd94\ub860\uc744 \uc218\ud589\ud558\ub294 \ubc29\ubc95\uc744 \ub2e4\ub8e8\uaca0\uc2b5\ub2c8\ub2e4. BigBird\ub294 \uae34 \ubb38\uc11c\uc758 \uc758\ubbf8\ub97c \uc774\ud574\ud558\uace0 \ucc98\ub9ac\ud558\ub294 \ub370 \uac15\uc810\uc744 \uac00\uc9c4 \ubaa8\ub378\ub85c, \ud2b9\ud788 \uae38\uc774\uac00 \uae34 \uc785\ub825\uc744 \ucc98\ub9ac\ud558\uae30 \uc704\ud574 \uc124\uacc4\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n<\/section>\n\n2. BigBird \ubaa8\ub378 \uc18c\uac1c<\/h2>\n
BigBird\ub294 \uad6c\uae00\uc5d0\uc11c \uac1c\ubc1c\ud55c \ubaa8\ub378\ub85c, Transformer\uc758 \ud55c\uacc4\ub97c \uadf9\ubcf5\ud558\uae30 \uc704\ud574 \uace0\uc548\ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uae30\uc874\uc758 Transformer \ubaa8\ub378\uc740 \uc785\ub825 \uae38\uc774\uac00 \uae38\uc5b4\uc9c8\uc218\ub85d \uacc4\uc0b0 \ube44\uc6a9\uc774 \uae30\ud558\uae09\uc218\uc801\uc73c\ub85c \uc99d\uac00\ud558\ub294 \ubb38\uc81c\uac00 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. BigBird\ub294 \ud76c\uc18c\uc131(sparsity)\uc744 \ud65c\uc6a9\ud558\uc5ec \uc774\ub7ec\ud55c \ubb38\uc81c\ub97c \ud574\uacb0\ud558\uace0, 4,096 \ud1a0\ud070 \uc774\uc0c1\uc758 \uae34 \ubb38\uc11c\ub3c4 \ud6a8\uacfc\uc801\uc73c\ub85c \ucc98\ub9ac\ud560 \uc218 \uc788\ub294 \ub2a5\ub825\uc744 \uac16\ucd94\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n
2.1. BigBird\uc758 \uad6c\uc870<\/h3>\n
BigBird\uc758 \uad6c\uc870\ub294 \uae30\uc874 Transformer \ubaa8\ub378\uc758 \uc5b4\ud150\uc158 \uba54\ucee4\ub2c8\uc998\uc744 \ud655\uc7a5\ud558\uc5ec, \ubd80\ubd84\uc801\uc778 \uc5b4\ud150\uc158\ub9cc \uc218\ud589\ud568\uc73c\ub85c\uc368 \uc131\ub2a5\uc744 \ub192\uc600\uc2b5\ub2c8\ub2e4. \ub354 \uad6c\uccb4\uc801\uc73c\ub85c, BigBird\ub294 \ub2e4\uc74c\uc758 \uc138 \uac00\uc9c0 \uc5b4\ud150\uc158 \ud328\ud134\uc744 \uc870\ud569\ud558\uc5ec \uc0ac\uc6a9\ud569\ub2c8\ub2e4.<\/p>\n
\n- \uc804\uc5ed \uc5b4\ud150\uc158 (Global Attention): \ud2b9\uc815 \uc785\ub825 \ud1a0\ud070\uc5d0 \ub300\ud574 \ubaa8\ub4e0 \ub2e4\ub978 \ud1a0\ud070\uacfc\uc758 \uc0c1\uad00\uad00\uacc4\ub97c \ud559\uc2b5\ud569\ub2c8\ub2e4.<\/li>\n
- \ub85c\uceec \uc5b4\ud150\uc158 (Local Attention): \uc778\uc811\ud55c \ud1a0\ud070 \uac04\uc758 \uad00\uacc4\ub97c \ud559\uc2b5\ud569\ub2c8\ub2e4.<\/li>\n
- \ub79c\ub364 \uc5b4\ud150\uc158 (Random Attention): \ubb34\uc791\uc704\ub85c \uc120\ud0dd\ub41c \ud1a0\ud070\uacfc\uc758 \uad00\uacc4\ub97c \ud559\uc2b5\ud569\ub2c8\ub2e4.<\/li>\n<\/ul>\n<\/section>\n\n
3. Hugging Face \uc124\uce58 \ubc0f \uae30\ubcf8 \uc138\ud305<\/h2>\n
Hugging Face\uc758 Transformers \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc0ac\uc6a9\ud558\uae30 \uc704\ud574\uc11c\ub294 \uba3c\uc800 \ud544\uc694\ud55c \ud328\ud0a4\uc9c0\ub97c \uc124\uce58\ud574\uc57c \ud569\ub2c8\ub2e4. \uc544\ub798 \uba85\ub839\uc5b4\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc124\uce58\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4:<\/p>\n
pip install transformers torch<\/code><\/pre>\n\uc774\uc81c \ubaa8\ub378\uc744 \ub85c\ub4dc\ud558\uace0 \ub370\uc774\ud130\ub97c \uc900\ube44\ud558\ub294 \uacfc\uc815\uc744 \uc2dc\uc791\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n<\/section>\n\n4. BigBird \ubaa8\ub378 \ub85c\ub529 \ubc0f \ub370\uc774\ud130 \uc900\ube44<\/h2>\n
BigBird \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uc5ec \ucd94\ub860\ud558\uae30 \uc704\ud55c \ucf54\ub4dc \uc608\uc81c\uc785\ub2c8\ub2e4. \uba3c\uc800 \ud544\uc694\ud55c \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc784\ud3ec\ud2b8\ud558\uace0 BigBird \ubaa8\ub378\uacfc \ud1a0\ud06c\ub098\uc774\uc800\ub97c \ucd08\uae30\ud654\ud558\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n
\nimport torch\nfrom transformers import BigBirdTokenizer, BigBirdForSequenceClassification\n\n# \ubaa8\ub378 \ubc0f \ud1a0\ud06c\ub098\uc774\uc800 \ucd08\uae30\ud654\ntokenizer = BigBirdTokenizer.from_pretrained('google\/bigbird-pegasus-large-arxiv')\nmodel = BigBirdForSequenceClassification.from_pretrained('google\/bigbird-pegasus-large-arxiv')\n\n# \uc608\uc2dc \uc785\ub825 \ud14d\uc2a4\ud2b8\ntext = \"\ub525\ub7ec\ub2dd\uc740 \uae30\uacc4 \ud559\uc2b5\uc758 \ud55c \ubd84\uc57c\ub85c...\"\n\n# \uc785\ub825 \ud14d\uc2a4\ud2b8 \ud1a0\ud070\ud654\ninputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True)\n <\/code><\/pre>\n\uc704 \ucf54\ub4dc\uc5d0\uc11c\ub294 Hugging Face\uc758 Transformers \ub77c\uc774\ube0c\ub7ec\ub9ac\uc5d0\uc11c BigBird \ubaa8\ub378\uacfc \ud574\ub2f9 \ud1a0\ud06c\ub098\uc774\uc800\ub97c \ub85c\ub4dc\ud588\uc2b5\ub2c8\ub2e4. BigBirdTokenizer<\/code>\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc785\ub825 \ud14d\uc2a4\ud2b8\ub97c \ud1a0\ud070\ud654\ud558\uace0, \ubaa8\ub378\uc758 \uc785\ub825\uc73c\ub85c \ubcc0\ud658\ud569\ub2c8\ub2e4.<\/p>\n<\/section>\n\n5. \ubaa8\ub378\uc758 \ucd94\ub860 \uc218\ud589\ud558\uae30<\/h2>\n
\ubaa8\ub378\uc744 \ud1b5\ud574 \uc785\ub825 \ud14d\uc2a4\ud2b8\uc758 \uc608\uce21\uac12\uc744 \uc0dd\uc131\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc544\ub798 \ucf54\ub4dc\uc5d0\uc11c\ub294 \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uc5ec \ucd94\ub860\uc744 \uc218\ud589\ud558\ub294 \ubc29\ubc95\uc744 \ubcf4\uc5ec\uc90d\ub2c8\ub2e4.<\/p>\n
\n# \ubaa8\ub378\uc744 \ucd94\ub860 \ubaa8\ub4dc\ub85c \uc804\ud658\nmodel.eval()\n\n# \ucd94\ub860 \uc218\ud589\nwith torch.no_grad():\n outputs = model(**inputs)\n logits = outputs.logits\n\n# \uc608\uce21 \ud655\ub960\npredictions = torch.nn.functional.softmax(logits, dim=-1)\npredicted_class = torch.argmax(predictions)\n\nprint(f\"\uc608\uce21\ub41c \ud074\ub798\uc2a4: {predicted_class.item()}, \ud655\ub960: {predictions.max().item()}\")\n <\/code><\/pre>\n\uc704 \ucf54\ub4dc\uc5d0\uc11c \ubaa8\ub378\uc744 \ud3c9\uac00 \ubaa8\ub4dc\ub85c \uc804\ud658\ud558\uace0, \uc785\ub825 \ud14d\uc2a4\ud2b8\uc5d0 \ub300\ud55c \ucd94\ub860\uc744 \uc218\ud589\ud55c \ud6c4 \uc608\uce21\ub41c \ud074\ub798\uc2a4\uc640 \ud574\ub2f9 \ud655\ub960\uc744 \ucd9c\ub825\ud569\ub2c8\ub2e4.<\/p>\n<\/section>\n\n6. \uacb0\ub860<\/h2>\n
BigBird \ubaa8\ub378\uc740 \uae34 \uc785\ub825 \ud14d\uc2a4\ud2b8\uc5d0 \ub300\ud55c \uc790\uc5f0\uc5b4 \ucc98\ub9ac \uc791\uc5c5\uc5d0\uc11c \ub6f0\uc5b4\ub09c \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. Hugging Face\uc758 Transformers \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \ud65c\uc6a9\ud558\uba74 \ubaa8\ub378\uc744 \uc190\uc27d\uac8c \ub85c\ub4dc\ud558\uace0 \ucd94\ub860\uc744 \uc218\ud589\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc624\ub298 \ub2e4\ub8ec \ub0b4\uc6a9\uc744 \ud1b5\ud574 BigBird \ubaa8\ub378\ub85c \ud14d\uc2a4\ud2b8 \ubd84\ub958 \uc791\uc5c5\uc744 \uc218\ud589\ud558\ub294 \uae30\ubcf8\uc801\uc778 \ubc29\ubc95\uc744 \uc775\ud788\uc168\uae30\ub97c \ubc14\ub78d\ub2c8\ub2e4. \ucd94\uac00\uc801\uc73c\ub85c \ub2e4\uc591\ud55c \ub370\uc774\ud130\uc14b\uacfc \ud0dc\uc2a4\ud06c\uc5d0 \ub530\ub77c BigBird \ubaa8\ub378\uc758 \ud65c\uc6a9\uc744 \ud655\uc7a5\ud574 \ub098\uac00\uc2e4 \uc218 \uc788\uae30\ub97c \ubc14\ub78d\ub2c8\ub2e4.<\/p>\n<\/section>\n