DR-CIML: Few-shot Object Detection via Base Data Resampling and Cross-iteration Metric Learning

Figures & data

Figure 1. Modifications and improvements of DR-CIML applied to transfer-learning-based FSOD.

Figure 2. Structure of DR-CIML method.

Figure 3. Overview of cross-iteration metric-learning (CIML) algorithm.

Table 1. nAP50 performance of existing FSOD methods on three PASCAL VOC novel splits with K = 1,2,3,5, and 10. “●” represents meta-learning-based methods. “◊” represents transfer-learning-based methods. “–“represents unreported results of other methods.

Method/Shot	Backbone	Novel Split 1					Novel Split 2					Novel Split 3
		1	2	3	5	10	1	2	3	5	10	1	2	3	5	10
● MetaYOLO (Kang et al. 2019)	YOLOv2	14.8	15.5	26.7	33.9	47.2	15.7	15.3	22.7	30.1	40.5	21.3	25.6	28.4	42.8	45.9
● RepMet (Karlinsky et al. 2019)	ResNet-101	26.1	32.9	34.4	38.6	41.3	17.2	22.1	23.4	28.3	35.8	27.5	31.1	31.5	34.4	37.2
● Meta R-CNN (Yan et al. 2019)	ResNet-101	19.9	25.5	35.0	45.7	51.5	10.4	19.4	29.6	34.8	45.4	14.3	18.2	27.5	41.2	48.1
◊ TFA w/cos (Wang et al. 2020)	ResNet-101	39.8	36.1	44.7	55.7	56.0	23.5	26.9	34.1	35.1	39.1	30.8	34.8	42.8	49.5	49.8
◊ CoRPN w/cos (Zhang, Wang, and Forsyth 2020)	ResNet-101	44.4	38.5	46.4	54.1	55.7	25.7	29.5	37.3	36.2	41.3	35.8	41.8	44.6	51.6	49.6
◊ MPSR (Wu et al. 2020)	ResNet-101	41.7	-	51.4	55.2	61.8	24.4	-	39.2	39.9	47.8	35.6	-	42.3	48.0	49.7
● DCNet (Hu et al. 2021)	ResNet-101	33.9	37.4	43.7	51.1	59.6	23.2	24.8	30.6	36.7	46.6	32.3	34.9	39.7	42.6	50.7
● TIP (Li and Li 2021)	ResNet-101	27.7	36.5	43.3	50.2	59.6	22.7	30.1	33.8	40.9	46.9	21.7	30.6	38.1	44.5	50.9
◊ Halluc. (TFA) (Zhang and Wang 2021)	ResNet-101	45.1	44.0	44.7	55.0	55.9	23.2	27.5	35.1	34.9	39.0	30.5	35.1	41.4	49.0	49.3
◊ CGDP+FSCN (Li et al. 2021)	ResNet-50	40.7	45.1	46.5	57.4	62.4	27.3	31.4	40.8	42.7	46.3	31.2	36.4	43.7	50.1	55.6
◊ SRR-FSD (Zhu et al. 2021)	ResNet-101	47.8	50.5	51.3	55.2	56.8	32.5	35.3	39.1	40.8	43.8	40.1	41.5	44.3	46.9	46.4
◊ FSOD-UP (Wu et al. 2021)	ResNet-101	43.8	47.8	50.3	55.4	61.7	31.2	30.5	41.2	42.2	48.3	35.5	39.7	43.9	50.6	53.5
◊ FSCE (Sun et al. 2021)	ResNet-101	44.2	43.8	51.4	61.9	63.4	27.1	29.5	43.5	44.2	50.2	37.2	41.9	47.5	54.6	58.5
◊ FSCE + DR-CIML (ours)	ResNet-101	47.4	48.7	57.3	62.7	65.3	30.7	35.8	47.9	49.8	53.1	37.2	42.2	51.2	55.8	57.4

Table 2. Apply DR-CIML to different baselines on PASCAL VOC split 1 with K = 3, 5, 10. “◊” represents transfer-learning-based methods. “–“represents unreported results of other methods.

Method/Shot	Novel Split 1			Base Split 1
	3	5	10	3	5	10
◊ TFA w/cos (Wang et al. 2020)	44.7	55.7	56.0	79.1	-	78.4
◊ TFA + DR-CIML (ours)	53.2 (+8.5%)	62.7 (+7.0%)	64.0 (+8.0%)	79.2 (+0.1%)	78.3	79.0 (+0.6%)
◊ FSCE (Sun et al. 2021)	51.4	61.9	63.4	74.1	76.6	-
◊ FSCE + DR-CIML (ours)	57.3 (+5.9%)	62.7 (+0.8%)	65.3 (+1.9%)	80.5 (+6.4%)	80.9 (+4.3%)	81.0

Table 3. bAP50 of existing FSOD methods on three PASCAL VOC base splits. “–“represents unreported results of other methods.

Method/Shot	Base Split 1
	3	5	10
● Meta YOLO (Kang et al. 2019)	64.8	-	69.7
● Meta R-CNN (Yan et al. 2019)	64.8	-	67.9
◊ MPSR (Wu et al. 2020)	67.8	-	71.8
● PNPDet (Zhang et al. 2021)	75.5	-	75.5
◊ TFA w/cos (Wang et al. 2020)	79.1	-	78.4
◊ FSCE (Sun et al. 2021)	74.1	76.6	-
Train $D_{base}$ only (Wang et al. 2020)	80.8	80.8	80.8
◊ FSCE + DR-CIML (ours)	80.5	80.9	81.0

Table 4. Evaluation results of existing FSOD methods on two MS COCO novel splits.

Method/Shot	Novel AP50–95		Novel AP75
	10	30	10	30
● MetaYOLO (Kang et al. 2019)	5.6	9.1	4.6	7.6
◊ CoRPN w/cos (Zhang, Wang, and Forsyth 2020)	9.0	13.9	8.3	13.9
● Meta-RCNN (Yan et al. 2019)	8.7	12.4	6.6	10.8
◊ MPSR (Wu et al. 2020)	9.8	14.1	9.7	14.2
◊ TFA w/cos (Wang et al. 2020)	10.0	13.7	9.3	13.4
● QA-FewDet (Han et al. 2021)	10.2	16.5	9.0	15.5
◊ FSCE (Sun et al. 2021)	11.9	16.4	10.5	16.2
◊ SVD (FSCE) (Wu et al. 2021)	12.0	16.2	10.4	15.9
◊ FSCE + DR-CIML (ours)	13.6	17.2	12.0	16.6

Table 5. Ablation for data resample organization and cross-iteration metric learning; results gained from PASCAL VOC split 1. “–” represents unreported results of other methods.

Model	Exp	Data Resample	Metric Learning	Base AP50					Novel AP50
				1	2	3	5	10	1	2	3	5	10
ResNet-101	Exp1	×	CPE	78.9	-	74.1	76.6	-	44.2	43.8	51.4	61.9	63.4
	Exp2	×	CIML	77.5	77.9	74.4	75.1	76.8	45.2	48.5	55.1	63.4	65.0
	Exp3	√	CPE	80.4	80.4	80.1	80.4	81.0	45.2	48.1	57.1	62.7	65.2
	Exp4	√	CIML	80.6	80.6	80.5	80.9	81.0	47.4	48.7	57.3	62.7	65.3

Figure 4. Visual detection results of our method and standard FSCE.

Kang, B., Z. Liu, X. Wang, F. Yu, J. Feng, and T. Darrell 2019. Few-shot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8420–29). doi: 10.1109/ICCV.2019.00851.

 Google Scholar

Karlinsky, L., J. Shtok, S. Harary, E. Schwartz, A. Aides, R. Feris, and A. M. Bronstein 2019. Repmet: Representative-based metric learning for classification and few-shot object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5197–206). doi: 10.1109/CVPR.2019.00534.

 Google Scholar

Yan, X., Z. Chen, A. Xu, X. Wang, X. Liang, and L. Lin 2019. Meta r-cnn: Towards general solver for instance-level low-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9577–86). doi: 10.1109/ICCV.2019.00967.

 Google Scholar

Wang, X., T. E. Huang, T. Darrell, J. E. Gonzalez, and F. Yu. 2020. Frustratingly simple few-shot object detection. arXiv preprint arXiv:2003 06957. doi:10.48550/arXiv.2003.06957.

 Google Scholar

Zhang, W., Y. X. Wang, and D. A. Forsyth. 2020. Cooperating RPN’s improve few-shot object detection. arXiv preprint arXiv:2011 10142. doi:10.48550/arXiv.2011.10142.

 Google Scholar

Wu, J., S. Liu, D. Huang, and Y. Wang 2020. Multi-scale positive sample refinement for few-shot object detection. In European conference on computer vision (pp. 456–72). Springer, Cham. doi: 10.1007/978-3-030-58517-4_27.

 Google Scholar

Hu, H., S. Bai, A. Li, J. Cui, and L. Wang 2021. Dense relation distillation with context-aware aggregation for few-shot object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10185–94). doi: 10.1109/CVPR46437.2021.01005.

 Google Scholar

Li, A., and Z. Li 2021. Transformation invariant few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3094–102). doi: 10.1109/CVPR46437.2021.00311.

 Google Scholar

Zhang, W., and Y. X. Wang 2021. Hallucination improves few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13008–17). doi: 10.1109/CVPR46437.2021.01281.

 Google Scholar

Li, Y., H. Zhu, Y. Cheng, W. Wang, C. S. Teo, C. Xiang, and T. H. Lee 2021. Few-shot object detection via classification refinement and distractor retreatment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15395–403). doi: 10.1109/CVPR46437.2021.01514.

 Google Scholar

Zhu, C., F. Chen, U. Ahmed, Z. Shen, and M. Savvides 2021. Semantic relation reasoning for shot-stable few-shot object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8782–91). doi: 10.1109/CVPR46437.2021.00867.

 Google Scholar

Wu, A., Y. Han, L. Zhu, and Y. Yang 2021. Universal-prototype enhancing for few-shot object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9567–76). doi: 10.1109/ICCV48922.2021.00943.

 Google Scholar

Sun, B., B. Li, S. Cai, Y. Yuan, and C. Zhang 2021. Fsce: Few-shot object detection via contrastive proposal encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7352–62). doi: 10.1109/CVPR46437.2021.00727.

 Google Scholar

Zhang, G., K. Cui, R. Wu, S. Lu, and Y. Tian 2021. Pnpdet: Efficient few-shot detection without forgetting via plug-and-play sub-networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 3823–32). doi: 10.1109/WACV48630.2021.00387.

 Google Scholar

Han, G., Y. He, S. Huang, J. Ma, and S. F. Chang 2021. Query adaptive few-shot object detection with heterogeneous graph convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3263–72). doi: 10.1109/ICCV48922.2021.00325.

 Google Scholar

Wu, A., S. Zhao, C. Deng, and W. Liu. 2021. Generalized and discriminative few-shot object detection via SVD-dictionary enhancement. Advances in Neural Information Processing Systems 34:6353–64.

 Google Scholar