AdaBoosting Text Prompts
for Vision-Language Models

ECCV 2026 Spotlight
Seokhee Jin1,*, Changhwan Sung2,*, Sunung Mun2, Hoyoung Kim3, Jungseul Ok2,†
1KT Corporation   2POSTECH   3National AI Research Lab
*Equal contribution. †Corresponding author.
KT Corporation POSTECH Machine Learning Lab, POSTECH National AI Research Lab
Code arXiv Poster

TL;DR

Natural-language prompts transfer across heterogeneous VLMs, but existing text-prompting methods tend to saturate as shots increase. TPB turns prompt construction into a boosting loop: each round focuses on examples the current ensemble still misses. The resulting prompt ensemble scales with supervision on the source model, and its shot-driven gains persist when transferred to other VLMs.

Soft prompt learning

Benefits from more shots, but remains model-specific

Soft prompts improve with additional labeled examples, but remain tied to the VLM on which they were trained.

  • Benefits from additional shots
  • Direct cross-model reuse

Prior text prompting

Transfers across VLMs, but gains little from more shots

Existing text-prompting methods can be transferred to other VLMs, but their performance tends to saturate as more labeled examples are added.

  • Limited gains from more shots
  • Direct cross-model reuse

Text Prompt Boosting

Benefits from more shots and transfers across VLMs

TPB improves with additional labeled examples on the source VLM and retains those gains when transferred to other VLMs.

  • Benefits from additional shots
  • Direct cross-model reuse

Shot Gains That Survive Transfer

On the OpenAI CLIP ViT-B/32 source model, TPB gains 5.80 pp from one to sixteen shots. When the prompt ensembles are transferred to other VLMs without additional tuning, shot-driven gains of 2.41 pp on five ViT-L targets and 2.51 pp on four ViT-H targets remain.

+5.80 pp

1-to-16-shot gain on the ViT-B/32 source model

+2.41 pp

1-to-16-shot gain after transfer from ViT-B/32 to ViT-L targets

+2.51 pp

1-to-16-shot gain after transfer from ViT-B/32 to ViT-H targets

Source model: OpenAI CLIP ViT-B/32

Additional shots widen TPB's lead

Across eleven datasets and three seeds, TPB rises from 67.07 to 72.87 (+5.80 pp), while ProAPO moves from 66.07 to 67.35 (+1.28 pp).

Shot scalability on the source model On OpenAI CLIP ViT-B/32, TPB rises from 67.07 percent at one shot to 72.87 percent at sixteen shots across five shot settings. ProAPO rises from 66.07 to 67.35 percent.

OpenAI CLIP ViT-B/32 to larger VLMs

Shot-driven gains persist after transfer

After direct transfer from OpenAI CLIP ViT-B/32, TPB retains 1-to-16-shot gains of 2.41 points on ViT-L targets and 2.51 on ViT-H targets; ProAPO retains 0.42 and 0.32, respectively.

1-shot base TPB: 1→16 gain ProAPO: 1→16 gain

ViT-L targets

5 models
TPB +2.41 pp79.66 → 82.07
ProAPO +0.42 pp79.59 → 80.01

ViT-H targets

4 models
TPB +2.51 pp81.73 → 84.24
ProAPO +0.32 pp82.07 → 82.39
Full appendix result tables

Source-model results

Dataset-wise Top-1 accuracy across zero-, one-, and sixteen-shot settings.

Full appendix table of dataset-wise shot scalability on OpenAI CLIP RN50
OpenAI CLIP RN50 source model.
Full appendix table of dataset-wise shot scalability on OpenAI CLIP ViT-B/32
OpenAI CLIP ViT-B/32 source model.

Cross-model transfer results

Methods are optimized on the indicated OpenAI CLIP source model and evaluated directly on larger heterogeneous target VLMs.

Full appendix table of target-model-wise transfer from OpenAI CLIP RN50 to ViT-L and ViT-H model families
OpenAI CLIP RN50 to five ViT-L and four ViT-H target models.
Full appendix table of target-model-wise transfer from OpenAI CLIP ViT-B/32 to ViT-L and ViT-H model families
OpenAI CLIP ViT-B/32 to five ViT-L and four ViT-H target models.
Full appendix table of shot-wise transfer to ViT-G and ViT-E model families
OpenAI CLIP ViT-B/32 to four ViT-G targets and EVA-02-CLIP ViT-E.

How Text Prompt Boosting Works

At each boosting round, GPC constructs a class-wise prompt-bank classifier under the current sample weights. Misclassified images receive more weight, directing the next round toward unresolved cases; all rounds are combined into the final ensemble.

1

Compose a Prompt Classifier

Using augmented training views and the current sample weights, GPC selects class-wise banks from CLIP templates, LLM descriptions, and their concatenations.

2

Focus the Next Round on Errors

The classifier is evaluated on the original few-shot set, and misclassified images receive more weight in the next round.

3

Combine the Prompt Classifiers

After M rounds, the class-confidence outputs of all prompt-bank classifiers are aggregated into one final prediction.

Text Prompt Boosting repeatedly reweights mistakes, constructs a new prompt classifier, and aggregates all classifiers
TPB overview. Augmented views inform GPC, errors on the original samples update the next weight distribution, and all prompt-bank classifiers are combined at inference.

From a Miss to the Next Prompt

A prompt classifier correctly recognizes three dogs but misclassifies a lying basset hound
A GPC prompt bank and its predictions in a beagle-versus-basset-hound task.

Beagle vs. basset hound

One missed pose redirects the next round

The current bank relies on floppy ears, low stature, and splayed legs. Those cues classify the other examples, but legs and stature are obscured in the lying basset hound.

  1. 1

    The current weak classifier labels the lying basset hound as a beagle.

  2. 2

    This image receives more weight for the next round.

  3. 3

    The next GPC round prioritizes alternative cues, such as facial wrinkles or snout shape.

A single error that can be diluted in aggregate accuracy becomes a direct objective for the next prompt classifier.

Why Boosting, Not Just More Exposure?

Could the transfer gains come simply from seeing more augmented views? In the 16-shot setting, we hold total exposure fixed at M × a = 200 and trade augmentation factor a for boosting rounds M.

Fixed-exposure ablation

More boosting rounds improve transfer at fixed exposure

Putting most of the budget into augmentation gives the best source result (74.42). Shifting the same budget toward more boosting rounds raises both transfer averages monotonically, reaching 82.24 on ViT-L and 84.49 on ViT-H targets.

With exposure held constant, the transfer trend points to iterative boosting rather than augmentation volume alone.

16-shot mean Top-1 accuracy (%) over ten datasets excluding ImageNet and three random seeds. Target rows also average five ViT-L or four ViT-H VLMs.
Model M/a1/200 M/a5/40 M/a10/20 M/a50/4
Source ViT-B/32 74.42 72.94 73.32 73.36
Target ViT-L avg. 81.54 81.78 82.13 82.24
Target ViT-H avg. 83.12 83.72 83.98 84.49

BibTeX

@inproceedings{jin2026tpb,
  title     = {AdaBoosting Text Prompts for Vision-Language Models},
  author    = {Jin, Seokhee and Sung, Changhwan and Mun, Sunung and Kim, Hoyoung and Ok, Jungseul},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}