How to Fine-Tune Projection Layer in CLIP Model Using LoRA

To fine-tune the projection layers in a CLIP model using LoRA, you must use the Hugging Face Transformers implementation of CLIP, not the original OpenAI repository. The OpenAI CLIP code defines projection layers as nn.Parameter objects, which the PEFT (Parameter-Efficient Fine-Tuning) library does not support for LoRA. By switching to the Hugging Face CLIPModel, you get standard nn.Linear layers named visual_projection and text_projection that are fully compatible with LoRA. A minimal configuration targeting those two modules yields 18,432 trainable parameters out of 151 million total, a 0.0122% trainable rate.

The Full Answer

Why the OpenAI CLIP Projection Layers Are Invisible to LoRA

When you load a CLIP model using the official OpenAI repository and print its architecture with print(model), you will not see the projection layers listed among the submodules. This is because the OpenAI implementation initializes the projection layers as nn.Parameter objects directly, not as nn.Linear layers. The visual projection layer is stored as visual.proj and the text projection layer as text_projection, but they do not appear as named children of the model.

You can still inspect these parameters by iterating over model.named_parameters(). The following code prints every parameter name and its shape:

for name, param in model.named_parameters():
    print(f'{name}: {param.shape}')

Output:

text_projection: torch.Size([512, 512])
visual.proj: torch.Size([768, 512])
...

This confirms that the parameters exist, but they are not wrapped in a nn.Module subclass that PEFT's LoRA implementation can target. The PEFT library requires that the target modules be instances of nn.Linear (or other supported layer types) so that LoRA adapters can be injected. nn.Parameter objects are raw tensors, not layers, and cannot be adapted with LoRA directly.

Solution: Use the Hugging Face CLIP Implementation

To apply LoRA to the projection layers, you must use the Hugging Face Transformers version of CLIP, which implements both projections as nn.Linear layers with different names. The following code loads the model, configures LoRA to target those layers, and prints the trainable parameter count:

from transformers import CLIPModel
from peft import LoraConfig, get_peft_model

transformers_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")

config = LoraConfig(
    target_modules=["visual_projection", "text_projection"],
)

peft_model = get_peft_model(transformers_model, config)
peft_model.print_trainable_parameters()

Output:

trainable params: 18,432 || all params: 151,295,745 || trainable%: 0.0122

This configuration adds LoRA adapters to both the visual and text projection layers. The default LoRA rank is 8, which gives 18,432 trainable parameters. You can adjust the rank, alpha, and dropout by passing additional arguments to LoraConfig.

Alternative: Modify the OpenAI CLIP Code

If you must use the original OpenAI CLIP repository, you can modify the source code to replace nn.Parameter with nn.Linear layers. This approach is more invasive and requires you to maintain a fork of the repository. The changes involve:

1. Locating the visual projection definition in the CLIP source (typically in clip/model.py).
2. Replacing self.visual.proj = nn.Parameter(...) with self.visual.proj = nn.Linear(...).
3. Doing the same for the text projection layer.
4. Adjusting the forward pass to call the linear layer instead of using the parameter directly.

This approach is not recommended unless you have a specific reason to avoid Hugging Face Transformers. The Hugging Face implementation is more actively maintained and integrates with PEFT.

Step-by-Step: Fine-Tuning with LoRA on Hugging Face CLIP

1. Install the required libraries:

pip install transformers peft torch

2. Load the model and processor:

from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

3. Configure LoRA:

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["visual_projection", "text_projection"],
    lora_dropout=0.1,
    bias="none",
)

peft_model = get_peft_model(model, lora_config)

4. Prepare your dataset. For CLIP fine-tuning, you typically need image-text pairs. Use the processor to tokenize text and preprocess images.

5. Train the model with your preferred optimizer and loss function. Only the LoRA parameters will be updated.

6. Save the adapters:

peft_model.save_pretrained("./clip-lora-projections")

7. Load the adapters later:

from peft import PeftModel

base_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
loaded_model = PeftModel.from_pretrained(base_model, "./clip-lora-projections")