\n Recently, with the advancement of artificial intelligence, significant progress has been made in the field of natural language processing. Among them, \n M2M100 is a model for multilingual translation that supports direct conversion between multiple languages. This model excels in handling ‘various \n To use the M2M100 model, you first need to install the Hugging Face Transformers<\/strong> library and related dependencies. You can install it using the \n To use the model, you must first load the M2M100 model. The following code is an example of loading the model and tokenizer and implementing a simple function for translation.\n<\/p>\n \n The code above works as follows:<\/p>\n \n Now, let’s test the translation functionality. The example below demonstrates translating a Chinese sentence into English.\n<\/p>\n \n When the above code is executed, the output will be the English sentence “Hello, World!”. The M2M100 model can effectively translate even languages \n One of the powerful features of the M2M100 model is its support for multiple languages. The example below performs translation between various languages \n Running the code above will output translations between several languages. The important point is that the M2M100 model can translate various languages \n To evaluate the quality of translations, the BLEU (Bilingual Evaluation Understudy) score can be used. The BLEU score quantitatively measures the \n A BLEU score close to 0 indicates poor translation, while a score close to 1 indicates high quality of translation. \n Hugging Face’s M2M100 model is a model that has achieved innovative advancements in the field of multilingual translation. \n In the future, try experimenting with more languages and complex sentences to further improve this model’s performance and find ways to leverage it. The Hugging Face Transformers library can be applied to various NLP tasks, so feel free to apply it to different projects.\n<\/p>\n <\/body><\/p>\n","protected":false},"excerpt":{"rendered":"
\n Hugging Face<\/strong>\u2019s Transformers<\/strong> library has established itself as a tool that helps easily utilize various language models. In this course, we will
\n explain in detail how to implement automatic translation between Chinese and English using the M2M100<\/strong> model with Hugging Face.\n<\/p>\n1. Introduction to the M2M100 Model<\/h2>\n
\n languages’ and supports over 100 languages, offering the advantage of performing direct translations without going through an intermediate language, unlike traditional translation systems.\n<\/p>\n2. Installation and Setup<\/h2>\n
\n pip<\/code> command as shown below.\n<\/p>\npip install transformers torch<\/code><\/pre>\n3. Loading the Model and Implementing the Translation Function<\/h2>\n
\nfrom transformers import M2M100ForConditionalGeneration, M2M100Tokenizer\n\n# Load model and tokenizer\nmodel_name = \"facebook\/m2m100_418M\"\ntokenizer = M2M100Tokenizer.from_pretrained(model_name)\nmodel = M2M100ForConditionalGeneration.from_pretrained(model_name)\n\ndef translate(text, source_language, target_language):\n tokenizer.src_lang = source_language\n encoded_input = tokenizer(text, return_tensors=\"pt\")\n generated_tokens = model.generate(**encoded_input, forced_bos_token_id=tokenizer.get_lang_id(target_language))\n return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]\n<\/code><\/pre>\n3.1 Explanation of the Translation Function<\/h3>\n
\n
tokenizer.src_lang<\/code>: Sets the source language.<\/li>\ntokenizer()<\/code>: Tokenizes the input text.<\/li>\nmodel.generate()<\/code>: Performs translation based on the tokenized input.<\/li>\ntokenizer.batch_decode()<\/code>: Decodes the generated tokens and returns the translated text.<\/li>\n<\/ol>\n4. Translation Examples<\/h2>\n
\n# Sentence to be translated\ntext = \"\u4f60\u597d\uff0c\u4e16\u754c\uff01\" # Hello, World!\nsource_lang = \"zh\" # Chinese\ntarget_lang = \"en\" # English\n\n# Perform translation\ntranslated_text = translate(text, source_lang, target_lang)\nprint(f\"Translation result: {translated_text}\")\n<\/code><\/pre>\n4.1 Interpretation of the Results<\/h3>\n
\n with relatively complex sentence structures.\n<\/p>\n5. Multilingual Translation Examples<\/h2>\n
\n including Korean, French, and Spanish.\n<\/p>\n\n# Multilingual translation test\nsamples = [\n {\"text\": \"\uc5ec\ub7ec \uc5b8\uc5b4\ub97c \uc9c0\uc6d0\ud558\ub294 \ubaa8\ub378\uc785\ub2c8\ub2e4.\", \"source\": \"ko\", \"target\": \"en\"}, # Korean to English\n {\"text\": \"Bonjour le monde!\", \"source\": \"fr\", \"target\": \"ko\"}, # French to Korean\n {\"text\": \"\u00a1Hola Mundo!\", \"source\": \"es\", \"target\": \"ja\"}, # Spanish to Japanese\n]\n\nfor sample in samples:\n translated = translate(sample[\"text\"], sample[\"source\"], sample[\"target\"])\n print(f\"{sample['text']} ({sample['source']}) -> {translated} ({sample['target']})\")\n<\/code><\/pre>\n5.1 Multilingual Translation Results<\/h3>\n
\n directly without going through an intermediate language.\n<\/p>\n6. Performance Evaluation<\/h2>\n
\n similarity between the generated translation and the reference translation. The following is the process to calculate the BLEU score.\n<\/p>\n\nfrom nltk.translate.bleu_score import sentence_bleu\n\n# Reference translation and system translation\nreference = [\"Hello\", \"World\"]\ncandidate = translated_text.split()\n\n# Calculate BLEU score\nbleu_score = sentence_bleu([reference], candidate)\nprint(f\"BLEU score: {bleu_score:.4f}\")\n<\/code><\/pre>\n6.1 Interpretation of Performance Evaluation<\/h3>\n
\n Various examples and reference translations can be used to evaluate the translation performance across multiple languages.\n<\/p>\n7. Conclusion<\/h2>\n
\n In this course, we explored a basic example of automatic translation between Chinese and English using the M2M100 model. This model is capable of direct language conversion, allowing translations between various languages without an intermediate language.\n<\/p>\n