AdaBoosting Text Prompts
for Vision-Language Models

ECCV 2026 Spotlight
Seokhee Jin1,*, Changhwan Sung2,*, Sunung Mun2, Hoyoung Kim3, Jungseul Ok2,†
1KT Corporation   2POSTECH   3National AI Research Lab
*Equal contribution. †Corresponding author.
KT Corporation POSTECH Machine Learning Lab, POSTECH National AI Research Lab
Code arXiv Poster

TL;DR

Soft prompts scale with more shots but remain tied to a single VLM. Hard text prompts transfer across models, but their gains quickly saturate. TPB combines both advantages: it gains +5.80 pp from 1 to 16 shots on the source model and preserves +2.41 pp / +2.51 pp gains after transfer to ViT-L / ViT-H models.

Soft prompting

Scales with shots, tied to one model

More labeled examples help, but the learned prompts cannot be transferred directly to another VLM.

  • Shot-scalable adaptation
  • Direct cross-model transfer

Hard prompting

Transfers across models, saturates with shots

Natural-language prompts work across VLMs, but existing methods gain little from additional shots.

  • Limited shot scalability
  • Direct cross-model transfer

Text Prompt Boosting

Scales with shots and transfers

TPB reweights hard examples and combines weak text-prompt classifiers into a strong ensemble.

  • Shot-scalable adaptation
  • Direct cross-model transfer

Shot Gains That Survive Transfer

On the OpenAI CLIP ViT-B/32 source model, TPB gains 5.80 pp from one to sixteen shots. When the prompt ensembles are transferred to other VLMs without additional tuning, shot-driven gains of 2.41 pp on five ViT-L targets and 2.51 pp on four ViT-H targets remain.

+5.80 pp

1-to-16-shot gain on the ViT-B/32 source model

+2.41 pp

1-to-16-shot gain after transfer from ViT-B/32 to ViT-L targets

+2.51 pp

1-to-16-shot gain after transfer from ViT-B/32 to ViT-H targets

Source model: OpenAI CLIP ViT-B/32

Additional shots widen TPB's lead

Across eleven datasets and three seeds, TPB rises from 67.07 to 72.87 (+5.80 pp), while ProAPO moves from 66.07 to 67.35 (+1.28 pp).

Shot scalability on the source model On OpenAI CLIP ViT-B/32, TPB rises from 67.07 percent at one shot to 72.87 percent at sixteen shots across five shot settings. ProAPO rises from 66.07 to 67.35 percent.

OpenAI CLIP ViT-B/32 to larger VLMs

Shot-driven gains persist after transfer

After direct transfer from OpenAI CLIP ViT-B/32, TPB retains 1-to-16-shot gains of 2.41 points on ViT-L targets and 2.51 on ViT-H targets; ProAPO retains 0.42 and 0.32, respectively.

1-shot base TPB: 1→16 gain ProAPO: 1→16 gain

ViT-L targets

5 models
TPB +2.41 pp79.66 → 82.07
ProAPO +0.42 pp79.59 → 80.01

ViT-H targets

4 models
TPB +2.51 pp81.73 → 84.24
ProAPO +0.32 pp82.07 → 82.39
Full appendix result tables

Source-model results

Dataset-wise Top-1 accuracy across zero-, one-, and sixteen-shot settings.

Corrected full appendix table of dataset-wise shot scalability on OpenAI CLIP RN50
OpenAI CLIP RN50 source model.
Full appendix table of dataset-wise shot scalability on OpenAI CLIP ViT-B/32
OpenAI CLIP ViT-B/32 source model.

Cross-model transfer results

Methods are optimized on the indicated OpenAI CLIP source model and evaluated directly on larger heterogeneous target VLMs.

Full appendix table of target-model-wise transfer from OpenAI CLIP RN50 to ViT-L and ViT-H model families
OpenAI CLIP RN50 to five ViT-L and four ViT-H target models.
Full appendix table of target-model-wise transfer from OpenAI CLIP ViT-B/32 to ViT-L and ViT-H model families
OpenAI CLIP ViT-B/32 to five ViT-L and four ViT-H target models.
Full appendix table of shot-wise transfer to ViT-G and ViT-E model families
OpenAI CLIP ViT-B/32 to four ViT-G targets and EVA-02-CLIP ViT-E.

How Text Prompt Boosting Works

At each boosting round, GPC constructs a class-wise prompt-bank classifier under the current sample weights. Misclassified images receive more weight, directing the next round toward unresolved cases; all rounds are combined into the final ensemble.

1

Compose a Prompt Classifier

Using augmented training views and the current sample weights, GPC selects class-wise banks from CLIP templates, LLM descriptions, and their concatenations.

2

Focus the Next Round on Errors

The classifier is evaluated on the original few-shot set, and misclassified images receive more weight in the next round.

3

Combine the Prompt Classifiers

After M rounds, the class-confidence outputs of all prompt-bank classifiers are aggregated into one final prediction.

Text Prompt Boosting repeatedly reweights mistakes, constructs a new prompt classifier, and aggregates all classifiers
TPB overview. Augmented views inform GPC, errors on the original samples update the next weight distribution, and all prompt-bank classifiers are combined at inference.

From a Miss to the Next Prompt

A prompt classifier correctly recognizes three dogs but misclassifies a lying basset hound
A GPC prompt bank and its predictions in a beagle-versus-basset-hound task.

Beagle vs. basset hound

One missed pose redirects the next round

The current bank relies on floppy ears, low stature, and splayed legs. Those cues classify the other examples, but legs and stature are obscured in the lying basset hound.

  1. 1

    The current weak classifier labels the lying basset hound as a beagle.

  2. 2

    This image receives more weight for the next round.

  3. 3

    The next GPC round prioritizes alternative cues, such as facial wrinkles or snout shape.

A single error that can be diluted in aggregate accuracy becomes a direct objective for the next prompt classifier.

Why Boosting, Not Just More Exposure?

Could the transfer gains come simply from seeing more augmented views? In the 16-shot setting, we hold total exposure fixed at M × a = 200 and trade augmentation factor a for boosting rounds M.

Fixed-exposure ablation

More boosting rounds improve transfer at fixed exposure

Putting most of the budget into augmentation gives the best source result (74.42). Shifting the same budget toward more boosting rounds raises both transfer averages monotonically, reaching 82.24 on ViT-L and 84.49 on ViT-H targets.

With exposure held constant, the transfer trend points to iterative boosting rather than augmentation volume alone.

16-shot mean Top-1 accuracy (%) over ten datasets excluding ImageNet and three random seeds. Target rows also average five ViT-L or four ViT-H VLMs.
Model M/a1/200 M/a5/40 M/a10/20 M/a50/4
Source ViT-B/32 74.42 72.94 73.32 73.36
Target ViT-L avg. 81.54 81.78 82.13 82.24
Target ViT-H avg. 83.12 83.72 83.98 84.49

BibTeX

@inproceedings{jin2026tpb,
  title     = {AdaBoosting Text Prompts for Vision-Language Models},
  author    = {Jin, Seokhee and Sung, Changhwan and Mun, Sunung and Kim, Hoyoung and Ok, Jungseul},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}