\n Recent advancements in the field of artificial intelligence are progressing at a remarkable pace. In particular,
\n deep learning models are demonstrating extraordinary performance in the fields of computer vision and natural
\n language processing. Among them, the CLIP (Contrastive Language\u2013Image Pre-Training) model has gained attention as
\n a powerful model capable of understanding and processing text and images simultaneously. In this course, we will
\n provide detailed explanations on how to load and use CLIP-based pre-trained models utilizing the Hugging Face
\n Transformers library.\n <\/p>\n
\n The CLIP model is a model introduced by OpenAI, trained to connect and understand text and images. This model learns
\n from large-scale datasets of text-image pairs, enabling it to generate descriptions for given images or select images
\n that correspond to given text.\n <\/p>\n
\n The core idea of CLIP is “contrastive learning.” This approach ensures that pairs of similar text and images are
\n positioned closely in vector space, while pairs with different content are learned to be far apart. This allows CLIP
\n to exhibit remarkable performance even with unsupervised learning.\n <\/p>\n
\n The Hugging Face Transformers library is a tool that makes it easy to use various models related to natural language
\n processing (NLP). Through this library, users can easily load various pre-trained models and perform tasks such as
\n tokenization and data preprocessing. The CLIP model is also supported by this library.\n <\/p>\n
\n To use the CLIP model, you first need to install the necessary libraries. In a Python environment, you can install
\n Transformers library and related packages using the command below.\n <\/p>\n
pip install transformers torch torchvision<\/code><\/pre>\n5. Loading the CLIP Model<\/h2>\n
\n Now, let’s explain how to load the CLIP model in earnest. The library provides easy access to pre-trained CLIP models.
\n We will look at an example of loading the CLIP model and tokenizer using the Python code below.\n <\/p>\n
from transformers import CLIPProcessor, CLIPModel\n\n# Load CLIP model and processor\nmodel = CLIPModel.from_pretrained('openai\/clip-vit-base-patch16')\nprocessor = CLIPProcessor.from_pretrained('openai\/clip-vit-base-patch16')<\/code><\/pre>\n5.1. Explanation of the Model and Processor<\/h3>\n
\n In the code above, we use the `from_pretrained` method to load the pre-trained CLIP model and processor. The processor
\n serves to process the input text and images, transforming them into a format the model can understand. In other words,
\n it converts images into tensor format and tokenizes the text so that it can be accepted as model input.\n <\/p>\n
6. Input of Images and Text<\/h2>\n
\n The CLIP model can take both images and text as input. The code below demonstrates the process of downloading a random
\n image and inputting it into the model along with its corresponding text.\n <\/p>\n
import requests\nfrom PIL import Image\n\n# Download image\nurl = \"https:\/\/example.com\/sample.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\n# Text input\ntext = \"A sample image description\"<\/code><\/pre>\n6.1. Preparing the Image File<\/h3>\n
\n In the code above, the requests library is used to download the image file. Then, the Pillow library is used to open
\n the image. You can specify the URL of the actual image you want to use for downloading, or you can use an image file
\n stored locally.\n <\/p>\n
7. Inference with the CLIP Model<\/h2>\n
\n Now, let’s input the image and text into the model and proceed with the inference. You can check the model’s output
\n with the following code.\n <\/p>\n
# Preprocess input data\ninputs = processor(text=text, images=image, return_tensors=\"pt\", padding=True)\n\n# Model inference\noutputs = model(**inputs)\n\n# Extract similarity scores\nlogits_per_image = outputs.logits_per_image\nprobs = logits_per_image.softmax(dim=1)\n\nprint(f\"Prediction probability: {probs}\")<\/code><\/pre>\n7.1. Explanation of the Model Inference Process<\/h3>\n
\n After preprocessing the input text and image using the `processor`, we input that information into the model for inference.
\n The logits returned from the model are then converted into a probability distribution using the softmax function to derive
\n the final prediction probability.\n <\/p>\n
8. Example: Utilizing the CLIP Model<\/h2>\n
\n Below is the full code showing how to actually utilize the CLIP model. This code evaluates the similarity of images
\n based on the given text.\n <\/p>\n
import requests\nimport torch\nfrom PIL import Image\nfrom transformers import CLIPProcessor, CLIPModel\n\n# Load the model and processor\nmodel = CLIPModel.from_pretrained('openai\/clip-vit-base-patch16')\nprocessor = CLIPProcessor.from_pretrained('openai\/clip-vit-base-patch16')\n\n# Download image\nurl = \"https:\/\/example.com\/sample.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\n# Text input\ntext = \"A sample image description\"\n\n# Preprocess input data\ninputs = processor(text=text, images=image, return_tensors=\"pt\", padding=True)\n\n# Model inference\nwith torch.no_grad():\n outputs = model(**inputs)\n\n# Extract similarity scores\nlogits_per_image = outputs.logits_per_image\nprobs = logits_per_image.softmax(dim=1)\n\nprint(f\"Prediction probability: {probs}\")<\/code><\/pre>\n9. Conclusion<\/h2>\n
\n In this lecture, we explained how to load a CLIP-based pre-trained model using the Hugging Face Transformers library and
\n evaluate the similarity by inputting image-text pairs. The CLIP model has various applications and can contribute to
\n the development of more advanced AI systems. We encourage you to continue expanding the possibilities of artificial
\n intelligence using various deep learning technologies.\n <\/p>\n
10. References<\/h2>\n