publications

Multi-Object Tracking over humans has improved rapidly with the development of object detection and re-identification. However, multi-actor tracking over humans with similar appearance and non-linear movement can still be very challenging even for the state-of-the-art tracking algorithm. Current motion-based tracking algorithms often use Kalman Filter to predict the motion of an object, however, its linear movement assumption can cause failure in tracking when the target is not moving linearly. And for multi-players tracking over the sports field, because the players in the same team are usually wearing the same color of jersey, making re-identification even harder both in the short term and long term in the tracking process. In this work, we proposed a motion-based tracking algorithm and three post-processing pipelines for three sports including basketball, football, and volleyball, we successfully handle the tracking of the non-linear movement of players on the sports fields. Experimental results, with achieved HOTA of 73.968 on the testing set of ECCV DeeperAction Challenge SportsMOT Dataset, show the effectiveness of the proposed framework and its robustness in different sports scenes.

To improve the generalization of 3D human pose estimators, many existing deep learning based models focus on adding different augmentations to training poses. However, data augmentation techniques are limited to the ”seen” pose combinations and hard to infer poses with rare ”unseen” joint positions. To address this problem, we present CameraPose, a weakly-supervised framework for 3D human pose estimation from a single image, which can not only be applied on 2D-3D pose pairs but also on 2D alone annotations. By adding a camera parameter branch, any in-the-wild 2D annotations can be fed into our pipeline to boost the training diversity and the 3D poses can be implicitly learned by reprojecting back to 2D. Moreover, CameraPose introduces a refinement network module with confidence-guided loss to further improve the quality of noisy 2D keypoints extracted by 2D pose estimators. Experimental results demonstrate that the CameraPose brings in clear improvements on cross-scenario datasets. Notably, it outperforms the baseline method by 3mm on the most challenging dataset 3DPW. In addition, by combining our proposed refinement network module with existing 3D pose estimators, their performance can be improved in cross-scenario evaluation.

Gait recognition, which refers to the recognition or identification of a person based on their body shape and walking styles, derived from video data captured from a distance, is widely used in crime prevention, forensic identification, and social security. However, to the best of our knowledge, most of the existing methods use appearance, posture and temporal feautures without considering a learned temporal attention mechanism for global and local information fusion. In this paper, we propose a novel gait recognition framework, called Temporal Attention and Keypoint-guided Embedding (GaitTAKE), which effectively fuses temporal-attention-based global and local appearance feature and temporal aggregated human pose feature. Experimental results show that our proposed method achieves a new SOTA in gait recognition with rank-1 accuracy of 98.0% (normal), 97.5% (bag) and 92.2% (coat) on the CASIA-B gait dataset; 90.4% accuracy on the OU-MVLP gait dataset.

The Alberta Infant Motor Scale (AIMS) is a well-known assessment scheme that evaluates the gross motor development of infants by recording the number of specific poses achieved. With the aid of the image-based pose recognition model, the AIMS evaluation procedure can be shortened and automated, providing early diagnosis or indicator of potential developmental disorder. Due to limited public infant-related datasets, many works use the SMIL-based method to generate synthetic infant images for training. However, this domain mismatch between real and synthetic training samples often leads to performance degradation during inference. In this paper, we present a CNN-based model which takes any infant image as input and predicts the coarse and fine-level pose labels. The model consists of an image branch and a pose branch, which respectively generates the coarse-level logits facilitated by the unsupervised domain adaptation and the 3D keypoints using the HRNet with SMPLify optimization. Then the outputs of these branches will be sent into the hierarchical pose recognition module to estimate the fine-level pose labels. We also collect and label a new AIMS dataset, which co—tains 750 real and 4000 synthetic infants images with AIMS pose labels. Our experimental results show that the proposed method can significantly align the distribution of synthetic and real-world datasets, thus achieving accurate performance on fine-grained infant pose recognition.

Sensing and perception systems for autonomous driving vehicles in road scenes are composed of four crucial components: object detection, tracking, panoptic segmentation, and 3D localization. While all these components are inherently intertwined, most relevant papers tend to only focus on a subset of these components. We separate the unified video panoptic segmentation tasks into two subtasks: 1) multiobject tracking and segmentation, and 2) semantic segmentation. For the first stage, we propose a unified monocular 3D based framework that effectively tracks detected moving objects over time and estimates their 3D localization information as well as instance segmentation masks from a sequence of 2D images captured from a dash camera on a moving vehicle. Our system contains an RCNN-based Localization for Tracking Network (Loc4Trk-Net). The object association leverages deep pairwise contrastive learning to identify objects in various poses and viewpoints with appearance cues. A straightforward combination of a 3D Kalman filter and the Hungarian algorithm is further utilized for robust instance association via both feature similarity and 3D localization information. For the second stage, we adopt the existing DeepLabV3+ for semantic segmentation and further enhanced the performance with data augmentation using label propagation. Our proposed pipeline achieves an STQ score of 67.55 on the KITTI-STEP dataset as well as the state-of-the-art performance on the KITTI-MOTS leaderboard.

Aerial View Object Classification (AVOC) has started to adopt deep learning approaches with significant success in recent years but limited to optical data. On the other hand, Synthetic Aperture Radar (SAR) has wild aerial view related applications in the remote sensing field. However, SAR has received far less attention due to the special characteristics of the SAR data, which is the long-tailed distribution of the aerial view objects that increase the difficulty of classification. In this paper, we present a two-branch framework, including the cascading expert branch and paralleling expert branch, to tackle the long-tailed distribution of the dataset. Our proposed multi-expert architecture achieves 24.675% and 26.029% in the development phase and testing phase, respectively, in the NTIRE 2021 Multi-modal Aerial View Object Classification Challenge Track 1. The proposed method is proved to possess the effectiveness (top-tier performance among 157 participants) and efficiency (i.e., a lightweight architecture) for the AVOC task.

We propose a novel attention-guided deep learning framework for image classification, with the goal of predicting both image and pixel-level labels for input images. While the training images are with either positive or negative labels, we do not assume that each training image is with annotated pixel-level ground truth information, and thus our method can be realized in such a weakly supervised setting. Our proposed module can be easily combined with standard CNN architectures with no extra parameter needed. With the above advantages, we evaluate our model on a skull CT dataset, and the experimental results confirm the effectiveness and robustness of our approach over recent popular CNN architectures.