Few-Shot Semantic Segmentation: Methods, Benchmarks, and the Small-Data Frontier

Abstract

Few-shot semantic segmentation asks a model to label every pixel of a novel object class given only a handful of annotated examples of it, sometimes a single one. It is one of the cleaner formulations of the small-data problem, and over six years the field has converged on three working families: matching networks that compare query pixels to support pixels in an embedding space, prototypical methods that compress the support set into one or a few class vectors, and meta-learning that learns an initialisation or an update rule which adapts fast from k examples. This note surveys those families, credits the papers that built them, and reads the PASCAL-5i and COCO-20i benchmarks for what they actually reward. We then make a positioning argument rather than a competitive one: on raster well-log digitisation we did not pick a few-shot method at all. We took the orthogonal route and manufactured the support set with a procedural renderer, which removes the k-shot axis from the problem instead of climbing it. The two answers are not rivals. They are different levers on the same small-data problem, and the survey is the right place to make that distinction precise.

The problem few-shot segmentation is solving

Semantic segmentation is data-hungry in the most expensive way: the label is a per-pixel mask, and a human has to draw it. Collecting a few thousand boxes is tedious; collecting a few thousand dense masks of a new class is a budget line. Few-shot semantic segmentation reframes the task so that, at test time, a model sees a support set of just k labelled examples of a novel class (the k-shot setting, with k often 1 or 5) and must segment that class in a query image it has never been trained on. The training-time trick is episodic: the model is shown many such small tasks during meta-training, each a fresh support-query split over base classes, so it learns to generalise from a few examples rather than to recognise a fixed label set.

The honest framing of the field is that all of these methods solve the small-data problem by borrowing. They assume k real, labelled examples of the novel class exist and can be supplied at inference. The research question is how to squeeze the most generalisation out of those k examples. That is a good question, and the next three sections are about how well each family answers it. But it is worth naming the assumption up front, because the route we took relaxes exactly that assumption, and the contrast only lands if the assumption is on the table.

Family one: matching networks

The matching-network idea predates segmentation and comes from one-shot classification: embed the support examples and the query into a shared metric space, then label the query by a soft nearest-neighbour vote over the support embeddings, with the whole comparison made differentiable end-to-end (Vinyals et al., 2016)Vinyals et al. · 2016Matching Networks for One Shot LearningNeurIPS. The contribution that mattered was not the nearest-neighbour rule, which is old, but that the embedding was trained episodically to make that rule work from one example. Matching networks taught the field to train the way it tests.

Carrying this to segmentation meant making the comparison dense: every query pixel is matched against the support, not just a whole image against a whole image. The first paper to do this end-to-end for masks, and the one that gave the field its standard benchmark, posed one-shot segmentation as conditioning a segmentation branch on a support branch and introduced PASCAL-5i, the four-fold split of PASCAL VOC into base and novel classes that nearly every later method reports on (Shaban et al., 2017)Shaban et al. · 2017One-Shot Learning for Semantic Segmentation (OSLSM)BMVC. Matching-style methods are conceptually simple and strong in the genuinely-one-example regime, but a pixel-by-pixel comparison is sensitive to appearance shift between support and query, which is the seam later families work to close.

Family two: prototypical methods

Prototypical networks made the metric-learning idea cheaper and more robust by collapsing the support set into a single prototype per class, the mean of the support embeddings, and labelling a query point by its distance to those prototypes (Snell et al., 2017)Snell et al. · 2017Prototypical Networks for Few-shot LearningNeurIPS. For segmentation this is a natural fit: pool the masked support features into a foreground prototype and a background prototype, then classify every query pixel by which prototype it is nearer. The first prototype-based segmenter formalised exactly this, masked average pooling to build the class vector (Dong and Xing, 2018)Dong and Xing · 2018Few-Shot Semantic Segmentation with Prototype LearningBMVC.

The line then improved along two axes. PANet added a prototype-alignment regularisation that swaps the roles of support and query and asks the model to segment the support from the query's prototype, which forces the embedding to be symmetric and squeezed more out of the same k examples without extra parameters (Wang et al., 2019)Wang et al. · 2019PANet: Prototype AlignmentICCV. PPNet then argued that one prototype per class throws away intra-class structure and decomposed the class into several part-aware prototypes, which helps when the novel object is non-rigid or appears in varied poses (Liu et al., 2020)Liu et al. · 2020Part-Aware Prototype Network (PPNet)ECCV. Prototypical methods are the workhorse family: cheap at inference, stable, and the place most practical few-shot segmentation systems start.

Family three: meta-learning and feature priors

The third family treats few-shot adaptation as an explicit learning-to-learn problem. The canonical statement is a model-agnostic meta-learner that searches for a network initialisation from which a few gradient steps on k examples reach a good solution for a new task (Finn et al., 2017)Finn et al. · 2017Model-Agnostic Meta-Learning (MAML)ICML. In segmentation the pure-optimisation form is less common than a hybrid: keep the metric-learning backbone, but add a strong, learned prior that guides where the novel class is likely to be. PFENet is the clearest example, building a training-free prior from high-level features and enriching the query representation with it, which lifts the harder folds of the benchmark (Tian et al., 2020)Tian et al. · 2020Prior Guided Feature Enrichment (PFENet)IEEE TPAMI.

The most recent shift this family made is toward dense correlation. HSNet computes a multi-level hypercorrelation between every query and support feature pair and squeezes it with 4D convolutions, treating the support-query relationship as a dense tensor rather than a pooled prototype, which set a strong bar on both PASCAL-5i and COCO-20i (Min et al., 2021)Min et al. · 2021Hypercorrelation Squeeze (HSNet)ICCV. And a useful course-correction came from asking the inverse question: BAM learns what not to segment by training a base-class learner to suppress confident background, which removes a large slice of the false positives that plague few-shot masks (Lang et al., 2022)Lang et al. · 2022Learning What Not to Segment (BAM)CVPR. The trajectory of the family is toward richer, denser use of the same small support set, not toward needing fewer examples.

What the benchmarks reward

Almost every method above reports on two benchmarks, and it is worth being precise about what they measure so the survey does not turn into a leaderboard. PASCAL-5i, from the OSLSM paper, splits twenty PASCAL classes into four folds of five novel classes each; you meta-train on fifteen and evaluate one-shot and five-shot mean-IoU on the held-out five, rotating folds. COCO-20i does the same with eighty COCO classes in four folds of twenty, and it is much harder: smaller objects, more clutter, more classes. Two regularities hold across the literature. First, five-shot beats one-shot, usually by a few mIoU points, because a second through fifth example narrows the appearance gap, which is the whole premise of the k-shot axis. Second, the absolute numbers are modest by full-supervision standards, because the model is being asked to segment a class it has effectively never trained on from a handful of examples. The benchmarks reward methods that extract more from a fixed, small, borrowed support set. That is precisely the axis we chose not to compete on.

Where our route leaves the chart

Here is the positioning claim, stated plainly so it cannot be mistaken for a competitive one. On raster well-log digitisation, the problem is the same small-data problem these methods address: dense per-pixel masks of analogue curves on scanned paper logs are exactly the labels nobody wants to draw, and hand-tracing a curve off a raster is the very work the model is meant to replace. The few-shot answer would be to borrow a few labelled scans per curve type and meta-learn from them. We did something orthogonal. Because a printed log is a deterministic rendering of known source data, we built a procedural renderer that emits a synthetic log together with its pixel-perfect curve masks for free, and trained on those. The final synthetic corpora were 2,000 instances for binary segmentation and 15,000 for the multiclass setting, behind an earlier 20,000-log two-curve dataset, and not one of them was hand-traced.

That is why our marker on the chart above sits off the k-shot axis entirely. Few-shot methods move left and right along k, trying to win more accuracy per borrowed example. The synthetic-first route does not have a k. The support set is not borrowed and counted in shots; it is manufactured and counted in rendered instances. The two approaches are answers to the same question, how do you train a dense segmenter when real labels are scarce, and they are genuinely complementary rather than competing. If real labelled examples exist but are few, the few-shot families are the right tool, and the survey above is a map of which family to reach for. If the image is a rendering of source data you control, manufacturing the support set sidesteps the small-data problem rather than optimising within it. The honest summary is that the field built an excellent set of levers for the borrowed-support world, and our contribution is to point out there is a second world next to it where the support set can be built rather than borrowed.

Conclusion

Few-shot semantic segmentation is a mature, well-credited field with three coherent families and two standard benchmarks. Matching networks taught it to train episodically; prototypical methods made the metric cheap and robust and have become the practical default; meta-learning and dense-correlation priors keep pushing how much can be squeezed from the same few examples. PASCAL-5i and COCO-20i reward exactly that squeezing, and the survey above is a guide to which family to reach for when real-but-scarce labels are what you have. Our own work on raster well-log digitisation does not sit on that map, and the point of this note is that it should not: when the image is a rendering of source data you control, the support set can be manufactured rather than borrowed, and the small-data problem is sidestepped rather than optimised. Credit the prior art for the borrowed-support world it built so well, and recognise the synthetic-first route as the orthogonal lever on the same problem.

Key takeaways

1. 1Few-shot semantic segmentation has three working families: matching networks (dense soft nearest-neighbour over support embeddings, OSLSM and PASCAL-5i), prototypical methods (masked-average-pooled class vectors; PANet alignment, PPNet part-prototypes), and meta-learning plus dense priors (MAML, PFENet, HSNet hypercorrelation, BAM learn-what-not-to-segment). All are credited prior art.
2. 2Every family solves the small-data problem by BORROWING: it assumes k real labelled examples of the novel class exist at inference and competes on extracting more accuracy per borrowed example. The k-shot axis (1-shot vs 5-shot) is the whole game, and five-shot beats one-shot because extra examples narrow the appearance gap.
3. 3PASCAL-5i and COCO-20i reward squeezing a fixed, small, borrowed support set; absolute mIoU stays modest because the class is effectively never trained on. The benchmarks measure the borrowed-support axis, not whether borrowing was necessary.
4. 4Our synthetic-first route is orthogonal, not competitive: on raster well-log digitisation we manufactured the support set with a procedural renderer (2,000 binary and 15,000 multiclass instances, behind a 20,000-log two-curve dataset, zero hand-traced), so k is undefined and the method sits off the k-shot axis entirely.
5. 5Choose by the nature of your scarce labels: if real-but-few, reach for the few-shot families; if the image is a deterministic rendering of source data you control, manufacture the support set and sidestep the small-data problem rather than optimising within it.

References

[1] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, D. Wierstra. Matching Networks for One Shot Learning. NeurIPS 2016. https://arxiv.org/abs/1606.04080

[2] J. Snell, K. Swersky, R. S. Zemel. Prototypical Networks for Few-shot Learning. NeurIPS 2017. https://arxiv.org/abs/1703.05175

[3] C. Finn, P. Abbeel, S. Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (MAML). ICML 2017. https://arxiv.org/abs/1703.03400

[4] A. Shaban, S. Bansal, Z. Liu, I. Essa, B. Boots. One-Shot Learning for Semantic Segmentation (OSLSM). BMVC 2017. https://arxiv.org/abs/1709.03410

[5] N. Dong, E. P. Xing. Few-Shot Semantic Segmentation with Prototype Learning. BMVC 2018. https://www.bmvc2018.org/contents/papers/0255.pdf

[6] K. Wang, J. H. Liew, Y. Zou, D. Zhou, J. Feng. PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment. ICCV 2019. https://arxiv.org/abs/1908.06391

[7] Y. Liu, X. Zhang, S. Zhang, X. He. Part-Aware Prototype Network for Few-Shot Semantic Segmentation (PPNet). ECCV 2020. https://arxiv.org/abs/2007.06309

[8] Z. Tian, H. Zhao, M. Shu, Z. Yang, R. Li, J. Jia. Prior Guided Feature Enrichment Network for Few-Shot Segmentation (PFENet). IEEE TPAMI 2020. https://arxiv.org/abs/2008.01449

[9] J. Min, D. Kang, M. Cho. Hypercorrelation Squeeze for Few-Shot Segmentation (HSNet). ICCV 2021. https://arxiv.org/abs/2104.01538

[10] C. Lang, G. Cheng, B. Tu, J. Han. Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (BAM). CVPR 2022. https://arxiv.org/abs/2203.15712