Robust Pseudo-label Learning with Neighbor Relation for Unsupervised Visible-Infrared Person Re-Identification

Published 5/10/2024 by Xiangbo Yin, Jiangming Shi, Yachao Zhang, Yang Lu, Zhizhong Zhang, Yuan Xie, Yanyun Qu

Overview

Unsupervised Visible-Infrared Person Re-identification (USVI-ReID) is a challenging task that aims to match pedestrian images across visible and infrared modalities without any annotations.
Clustered pseudo-label methods have become predominant in USVI-ReID, but the inherent noise in pseudo-labels presents a significant obstacle.
The paper presents a Robust Pseudo-label Learning with Neighbor Relation (RPNR) framework to address the issues with noisy pseudo-labels in USVI-ReID.

Plain English Explanation

The paper tackles the problem of Unsupervised Visible-Infrared Person Re-identification (USVI-ReID), which is the task of matching people across images taken in visible and infrared light without any labeled data. This is a challenging problem because the appearance of a person can change significantly between the two modalities.

Recent methods in USVI-ReID have relied on "pseudo-labels," which are labels automatically generated by the model itself. However, these pseudo-labels can be noisy and inaccurate, which can compromise the performance of the model. To address this, the paper proposes a new framework called Robust Pseudo-label Learning with Neighbor Relation (RPNR).

The key ideas in RPNR are:

Noisy Pseudo-label Calibration: Correcting the noisy pseudo-labels, which can be especially difficult due to the high variation in appearance within the same person across modalities.
Neighbor Relation Learning: Modeling the interactions between all samples to reduce the high intra-class variations, which can help overcome the noise in the pseudo-labels.
Optimal Transport Prototype Matching: Establishing reliable cross-modality correspondences between the visible and infrared images.
Memory Hybrid Learning: Jointly learning modality-specific and modality-invariant information to improve the model's performance.

By incorporating these novel components, the RPNR framework is able to outperform the current state-of-the-art methods in USVI-ReID, achieving an average 10.3% improvement in Rank-1 accuracy on two benchmark datasets.

Technical Explanation

The RPNR framework first introduces a Noisy Pseudo-label Calibration module to correct the noisy pseudo-labels that are inherent in the clustered pseudo-label methods used in USVI-ReID. Due to the high intra-class variations, these noisy pseudo-labels are difficult to calibrate completely.

To address this, the paper proposes a Neighbor Relation Learning module to model the potential interactions between all samples, which can help reduce the high intra-class variations. This, in turn, helps overcome the noise in the pseudo-labels.

Furthermore, the authors devise an Optimal Transport Prototype Matching module to establish reliable cross-modality correspondences between the visible and infrared images. This is crucial for bridging the gap between the two modalities.

Finally, the researchers design a Memory Hybrid Learning module to jointly learn modality-specific and modality-invariant information. This allows the model to capture both the unique characteristics of each modality and the shared features between them, leading to improved performance.

The effectiveness of the RPNR framework is demonstrated through comprehensive experiments on two widely recognized benchmarks, SYSU-MM01 and RegDB. The results show that RPNR outperforms the current state-of-the-art GUR with an average Rank-1 improvement of 10.3%.

Critical Analysis

The paper presents a well-designed and thorough solution to the challenging problem of Unsupervised Visible-Infrared Person Re-identification. The authors have identified a key issue with the existing pseudo-label-based methods, namely the inherent noise in the pseudo-labels, and have proposed a comprehensive framework to address it.

One potential limitation of the RPNR framework is that it may still struggle with cases where the intra-class variations are extremely high, even after the Neighbor Relation Learning module. The paper acknowledges this and suggests that further research is needed to address this issue more effectively.

Additionally, the authors mention that the source code for RPNR will be released soon, which is a positive step towards enabling other researchers to build upon this work. However, it would be helpful if the paper could provide more details on the specific implementation details and hyperparameters used, as this information is crucial for reproducing the results.

Overall, the RPNR framework represents a significant advancement in the field of USVI-ReID, and the ideas presented in this paper could also be applicable to other unsupervised cross-modal matching problems. The critical analysis encourages readers to think carefully about the strengths, limitations, and potential future directions of this research.

Conclusion

The paper presents a novel Robust Pseudo-label Learning with Neighbor Relation (RPNR) framework to address the challenges of Unsupervised Visible-Infrared Person Re-identification (USVI-ReID). By incorporating techniques like Noisy Pseudo-label Calibration, Neighbor Relation Learning, Optimal Transport Prototype Matching, and Memory Hybrid Learning, the RPNR framework is able to outperform the current state-of-the-art methods by a significant margin.

This research represents an important step forward in the field of cross-modal person re-identification, which has various applications in areas like surveillance, security, and assistive technology. The ideas and techniques introduced in this paper could also be adapted to tackle other unsupervised cross-modal matching problems beyond person re-identification.

Overall, the RPNR framework is a compelling and well-executed solution to a challenging problem, and the authors' commitment to releasing the source code will undoubtedly spur further advancements in this field.

Full paper

Loading PDF viewer...

Read original: arXiv:2405.05613

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🤷

Unsupervised Visible-Infrared Person ReID by Collaborative Learning with Neighbor-Guided Label Refinement

De Cheng, Xiaojian Huang, Nannan Wang, Lingfeng He, Zhihui Li, Xinbo Gao

Unsupervised learning visible-infrared person re-identification (USL-VI-ReID) aims at learning modality-invariant features from unlabeled cross-modality dataset, which is crucial for practical applications in video surveillance systems. The key to essentially address the USL-VI-ReID task is to solve the cross-modality data association problem for further heterogeneous joint learning. To address this issue, we propose a Dual Optimal Transport Label Assignment (DOTLA) framework to simultaneously assign the generated labels from one modality to its counterpart modality. The proposed DOTLA mechanism formulates a mutual reinforcement and efficient solution to cross-modality data association, which could effectively reduce the side-effects of some insufficient and noisy label associations. Besides, we further propose a cross-modality neighbor consistency guided label refinement and regularization module, to eliminate the negative effects brought by the inaccurate supervised signals, under the assumption that the prediction or label distribution of each example should be similar to its nearest neighbors. Extensive experimental results on the public SYSU-MM01 and RegDB datasets demonstrate the effectiveness of the proposed method, surpassing existing state-of-the-art approach by a large margin of 7.76% mAP on average, which even surpasses some supervised VI-ReID methods.

11/5/2024

Learning Commonality, Divergence and Variety for Unsupervised Visible-Infrared Person Re-identification

Jiangming Shi, Xiangbo Yin, Yachao Zhang, Zhizhong Zhang, Yuan Xie, Yanyun Qu

Unsupervised visible-infrared person re-identification (USVI-ReID) aims to match specified people in infrared images to visible images without annotations, and vice versa. USVI-ReID is a challenging yet under-explored task. Most existing methods address the USVI-ReID using cluster-based contrastive learning, which simply employs the cluster center as a representation of a person. However, the cluster center primarily focuses on commonality, overlooking divergence and variety. To address the problem, we propose a Progressive Contrastive Learning with Hard and Dynamic Prototypes method for USVI-ReID. In brief, we generate the hard prototype by selecting the sample with the maximum distance from the cluster center. We theoretically show that the hard prototype is used in the contrastive loss to emphasize divergence. Additionally, instead of rigidly aligning query images to a specific prototype, we generate the dynamic prototype by randomly picking samples within a cluster. The dynamic prototype is used to encourage the variety. Finally, we introduce a progressive learning strategy to gradually shift the model's attention towards divergence and variety, avoiding cluster deterioration. Extensive experiments conducted on the publicly available SYSU-MM01 and RegDB datasets validate the effectiveness of the proposed method.

10/25/2024

Unsupervised Visible-Infrared ReID via Pseudo-label Correction and Modality-level Alignment

Yexin Liu, Weiming Zhang, Athanasios V. Vasilakos, Lin Wang

Unsupervised visible-infrared person re-identification (UVI-ReID) has recently gained great attention due to its potential for enhancing human detection in diverse environments without labeling. Previous methods utilize intra-modality clustering and cross-modality feature matching to achieve UVI-ReID. However, there exist two challenges: 1) noisy pseudo labels might be generated in the clustering process, and 2) the cross-modality feature alignment via matching the marginal distribution of visible and infrared modalities may misalign the different identities from two modalities. In this paper, we first conduct a theoretic analysis where an interpretable generalization upper bound is introduced. Based on the analysis, we then propose a novel unsupervised cross-modality person re-identification framework (PRAISE). Specifically, to address the first challenge, we propose a pseudo-label correction strategy that utilizes a Beta Mixture Model to predict the probability of mis-clustering based network's memory effect and rectifies the correspondence by adding a perceptual term to contrastive learning. Next, we introduce a modality-level alignment strategy that generates paired visible-infrared latent features and reduces the modality gap by aligning the labeling function of visible and infrared features to learn identity discriminative and modality-invariant features. Experimental results on two benchmark datasets demonstrate that our method achieves state-of-the-art performance than the unsupervised visible-ReID methods.

4/11/2024

Mutual Information Guided Optimal Transport for Unsupervised Visible-Infrared Person Re-identification

Zhizhong Zhang, Jiangming Wang, Xin Tan, Yanyun Qu, Junping Wang, Yong Xie, Yuan Xie

Unsupervised visible infrared person re-identification (USVI-ReID) is a challenging retrieval task that aims to retrieve cross-modality pedestrian images without using any label information. In this task, the large cross-modality variance makes it difficult to generate reliable cross-modality labels, and the lack of annotations also provides additional difficulties for learning modality-invariant features. In this paper, we first deduce an optimization objective for unsupervised VI-ReID based on the mutual information between the model's cross-modality input and output. With equivalent derivation, three learning principles, i.e., Sharpness (entropy minimization), Fairness (uniform label distribution), and Fitness (reliable cross-modality matching) are obtained. Under their guidance, we design a loop iterative training strategy alternating between model training and cross-modality matching. In the matching stage, a uniform prior guided optimal transport assignment (Fitness, Fairness) is proposed to select matched visible and infrared prototypes. In the training stage, we utilize this matching information to introduce prototype-based contrastive learning for minimizing the intra- and cross-modality entropy (Sharpness). Extensive experimental results on benchmarks demonstrate the effectiveness of our method, e.g., 60.6% and 90.3% of Rank-1 accuracy on SYSU-MM01 and RegDB without any annotations.

7/18/2024