Transformers Course Using Hugging Face, M2M100 Translation Result Decoding

Recent advancements in artificial intelligence and natural language processing (NLP) are occurring at an astonishing pace, and machine translation is receiving significant attention as one of the key areas. Among these, Hugging Face’s Transformers library helps researchers and developers easily access the latest models. In this article, we will conduct a translation task using the M2M100 model and explore decoding the output in depth with explanations and example code.

1. What are Hugging Face Transformers?

Hugging Face (Transformers) is a library that offers a variety of pre-trained natural language processing models, making them easy to use. It includes various models like Bert, GPT, T5, and particularly multilingual models such as M2M100 that support translation between multiple languages.

2. Introduction to the M2M100 Model

M2M100 (Many-to-Many 100) is a multilingual machine translation model developed by Facebook AI Research that supports direct translation between 100 languages. Previous translation systems focused on one-directional translation for specific languages, but M2M100 has the ability to translate directly between any language combination.
The advantages of this model include:

  • Direct translation between various languages
  • Improved quality of machine translation
  • Trained on vast amounts of data, possessing a high generalization ability

3. Installing the Library and Setting Up the Environment

To use the M2M100 model, you must first install the required libraries. A Python environment must be set up, and it can be installed with the following command:

pip install transformers torch

4. Using the M2M100 Model

4.1 Loading the Model

Now, let’s load the M2M100 model and prepare to carry out translation tasks. Below is the code to load the model. 


from transformers import M2M100Tokenizer, M2M100ForConditionalGeneration

# Loading tokenizer and model
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")
model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")

4.2 Defining the Translation Function

Next, we will create a simple translation function to implement the functionality of translating a given input sentence into a specific language. In this example, we will translate an English sentence into Korean. 


def translate_text(text, target_lang="ko"):
    # Tokenizing the input sentence
    tokenizer.src_lang = "en"  # Setting input language
    encoded_input = tokenizer(text, return_tensors="pt")
    
    # Translating through the model
    generated_tokens = model.generate(**encoded_input, forced_bos_token_id=tokenizer.get_lang_id(target_lang))
    
    # Decoding and returning the result
    return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]

4.3 Translation Example

Now, let’s use the translation function. Below is an example of translating the sentence “Hello, how are you?” into Korean. 


source_text = "Hello, how are you?"
translated_text = translate_text(source_text, target_lang="ko")
print(translated_text)  # Output: "안녕하세요, 잘 지내세요?"

5. Decoding the Translation Output

By decoding the translation output, we can convert the tokens generated by the model into natural language. The M2M100 model has the ability to handle outputs generated in multiple languages.
Let’s delve deeper into this with a more in-depth example.

5.1 Implementing the Decoding Function

A decoding function is also needed to carefully handle the tokens obtained from the translation. This helps ensure the format of the model’s output and improve the quality of the translation through additional post-processing. 


def decode_output(generated_tokens, skip_special_tokens=True):
    # Decoding tokens and returning the result string
    return tokenizer.batch_decode(generated_tokens, skip_special_tokens=skip_special_tokens)

5.2 Example of Decoding Results

Let’s decode the list of generated tokens to check the translation results. The example below shows the procedure of decoding the result after the translation is completed. 


# Getting the generated tokens
generated_tokens = model.generate(**encoded_input, forced_bos_token_id=tokenizer.get_lang_id("ko"))

# Decoding and printing the result
decoded_output = decode_output(generated_tokens)
print(decoded_output)  # Output: ["안녕하세요, 잘 지내세요?"]

6. Optimizing Results

Translation results may vary based on context or specific meanings. To optimize this, various parameters can be adjusted, or the model can be retrained for improvement. Additionally, adjusting the maximum output length or various random seeds can enhance the quality of the results.

6.1 Optional Parameter Adjustments

The model’s generate method can be adjusted with various parameters:

  • max_length: Maximum token length to generate
  • num_beams: Number of beams for beam search (improving diversity in decoding)
  • temperature: Adjusting the randomness of generation (values between 0-1)

# Example of additional parameter settings
generated_tokens = model.generate(
    **encoded_input,
    forced_bos_token_id=tokenizer.get_lang_id("ko"),
    max_length=40,
    num_beams=5,
    temperature=0.7
)

6.2 Comparing Results Before and After Optimization

This is a method to evaluate the model’s performance by comparing results before and after optimization. Please choose the settings that best fit your application.

7. Summary and Conclusion

In this article, we explored how to perform machine translation using Hugging Face’s M2M100 model and how to decode the output results. Thanks to advancements in deep learning and NLP technologies, we have established a foundation for easily communicating across various languages.

These technologies and tools will be utilized in the development of various applications in the future, fundamentally changing the way we work. We encourage you to use these tools to tackle even more meaningful projects.

8. References