[CVPR 2026] Exploring SAM’s Potential in Few-shot Segmentation with Background-centric Prompting
Title: Focus on Background: Exploring SAM’s Potential in Few-shot Medical Image Segmentation with Background-centric Prompting
Authors: Yuntian Bo · Yazhou Zhu · Piotr Koniusz · Haofeng Zhang
Abstract: Conventional few-shot medical image segmentation (FSMIS)
approaches face performance bottlenecks that hinder broader
clinical applicability. Although the Segment Anything Model
(SAM) exhibits strong category-agnostic segmentation
capabilities, its direct application to medical images often
leads to over-segmentation due to ambiguous anatomical
boundaries. In this paper, we reformulate SAM-based
FSMIS as a prompt localization task and propose FoB
(Focus on Background), a background-centric prompt
generator that provides accurate background prompts to
constrain SAM’s over-segmentation. Specifically, FoB
bridges the gap between segmentation and prompt
localization by category-agnostic generation of support
background prompts and directly localizing them in the
query image. To address the challenge of prompt
localization for novel categories, FoB models rich
contextual information to capture foreground-background
spatial dependencies. Moreover, inspired by the inherent
structural patterns of background prompts in medical
images, FoB models this structure as a constraint to
progressively refine background prompt predictions.
Experiments on three diverse medical image datasets
demonstrate that FoB outperforms other baselines by large
margins, achieving state-of-the-art performance on FSMIS
and exhibiting strong cross-domain generalization.