Clicky

logo SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory

Cheng-Yen Yang1, Hsiang-Wei Huang1, Wenhao Chai1, Zhongyu Jiang1, Jenq-Neng Hwang1
1University of Washington
arXiv    Code    Raw Results   
1917 (2019), directed by Sam Mendes.

Gameplay from Sekiro: Shadows Die Twice (2019) and Black Myth: Wukong (2024).

Sport scene (NFL, 2024) and choreography scene (ITZY, 2021).

Abstract

The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks but faces challenges in visual object tracking, particularly when managing crowded scenes with fast-moving or self-occluding objects. Furthermore, the fixed-window memory approach in the original model does not consider the quality of memories selected to condition the image features for the next frame, leading to error propagation in videos. This paper introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for visual object tracking. By incorporating temporal motion cues with the proposed motion-aware memory selection mechanism, SAMURAI effectively predicts object motion and refines mask selection, achieving robust, accurate tracking without the need for retraining or fine-tuning. SAMURAI operates in real-time and demonstrates strong zero-shot performance across diverse benchmark datasets, showcasing its ability to generalize without fine-tuning. In evaluations, SAMURAI achieves significant improvements in success rate and precision over existing trackers, with a 7.1% AUC gain on LaSOT-ext and a 3.5% AO gain on GOT-10k. Moreover, it achieves competitive results compared to fully supervised methods on LaSOT, underscoring its robustness in complex tracking scenarios and its potential for real-world applications in dynamic environments.

Embodied agent interface overview.
Figure 1: The overview of our SAMURAI visual object tracker.

Results

Embodied agent interface overview.
Table 1: Zero-shot tracking results on LaSOT, LaSOT-ext, and GOT-10k.

Embodied agent interface overview.
Table 2: Zero-shot tracking results on additional benchmarks: TrackingNet, NFS, and OTB-100.

Embodied agent interface overview.
Table 3: Comparison between proposed SAMURAI and the baseline SAM 2 on LaSOT and LaSOT-ext.

BibTeX

            @misc{yang2024samurai,
                  title={SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory}, 
                  author={Cheng-Yen Yang and Hsiang-Wei Huang and Wenhao Chai and Zhongyu Jiang and Jenq-Neng Hwang},
                  year={2024},
                  eprint={2411.11922},
                  archivePrefix={arXiv},
                  primaryClass={cs.CV},
                  url={https://arxiv.org/abs/2411.11922}, 
            }