{"id":36207,"date":"2024-11-01T09:46:39","date_gmt":"2024-11-01T09:46:39","guid":{"rendered":"http:\/\/atmokpo.com\/w\/?p=36207"},"modified":"2024-11-01T09:46:39","modified_gmt":"2024-11-01T09:46:39","slug":"using-hugging-face-transformers-course-source-language-m2m100-tokenization","status":"publish","type":"post","link":"https:\/\/atmokpo.com\/w\/36207\/","title":{"rendered":"Using Hugging Face Transformers Course, Source Language M2M100 Tokenization"},"content":{"rendered":"

<\/p>\n

With the development of deep learning, Natural Language Processing (NLP) has undergone significant changes. In particular, Hugging Face’s transformers library has established itself as a powerful tool for NLP tasks. In this course, we will introduce the multilingual translation and tokenization process using the M2M100 model.<\/p>\n

Overview of the M2M100 Model<\/h2>\n

The M2M100 (Multilingual To Multilingual Translation) model is a multilingual model that supports direct translation between more than 100 languages. Existing translation models used an indirect translation method that translated from the source language to an intermediate language (e.g., English) and then converted to the target language. The M2M100 overcomes this limitation by enabling direct conversion among multiple languages, significantly improving translation efficiency between various language pairs.<\/p>\n

What is Tokenization?<\/h2>\n

Tokenization is the process of dividing the input text into smaller units called tokens. After converting it into a list format, unique indices are assigned to each token. Tokenization is an essential process in NLP and is necessary before inputting text data into the model.<\/p>\n

Environment Setup<\/h2>\n

Before proceeding with the course, you need to install the required libraries. Specifically, we will install transformers<\/code> and torch<\/code>. You can install them with the following command:<\/p>\n

\n        pip install transformers torch<\/code>\n    <\/pre>\n
Loading the Tokenizer<\/h2>\n
To load the tokenizer for the M2M100 model, we will use the M2M100Tokenizer<\/code> class provided by the transformers<\/code> library.<\/p>\n
\n        \nimport torch\nfrom transformers import M2M100Tokenizer\n\n# Load the tokenizer for the M2M100 model\ntokenizer = M2M100Tokenizer.from_pretrained(\"facebook\/m2m100_418M\")\n        <\/code>\n    <\/pre>\n
Tokenization Process<\/h2>\n
Now we are ready to tokenize the text. Below is an example of tokenizing the sentence ‘Hello, everyone!’.<\/p>\n
\n        \n# Input text\ntext = \"Hello, everyone!\"\n\n# Tokenizing the text\nencoded_input = tokenizer(text, return_tensors=\"pt\")\n\n# Output tokens and indices\nprint(\"Tokenized tokens:\", tokenizer.convert_ids_to_tokens(encoded_input['input_ids'][0]))\nprint(\"Token indices:\", encoded_input['input_ids'])\n        <\/code>\n    <\/pre>\n
Tokenization Result<\/h3>\n
The output generated after running the above code shows how the input text has been tokenized and indexed. You can check the actual values of the tokens using the convert_ids_to_tokens<\/code> method.<\/p>\n
Multilingual Translation<\/h2>\n
Using the tokenized data, we can perform multilingual translation. I will show you an example of translating Korean to English using the M2M100 model.<\/p>\n
\n        \nfrom transformers import M2M100ForConditionalGeneration\n\n# Load the M2M100 model\nmodel = M2M100ForConditionalGeneration.from_pretrained(\"facebook\/m2m100_418M\")\n\n# Korean text\ntext = \"Hello, everyone!\"\nencoded_input = tokenizer(text, return_tensors=\"pt\")\n\n# Translation\ntranslated_tokens = model.generate(**encoded_input, forced_bos_token_id=tokenizer.get_lang_id(\"en\"))\n\n# Translation result\ntranslated_text = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)\nprint(\"Translation result:\", translated_text[0])\n        <\/code>\n    <\/pre>\n
Interpretation of the Translation Result<\/h3>\n
You can check if the Korean sentence has been accurately translated into English using the code above. The generate<\/code> method generates the translated result based on the input token data.<\/p>\n
Conclusion<\/h2>\n
In this course, we explored the multilingual tokenization and translation process using Hugging Face’s M2M100 model. The progress in the field of natural language processing will continue, and using such tools will enable better communication across various languages. I hope that interest and research in NLP and deep learning will continue in the future.<\/p>\n
References<\/h2>\n