近期提出的大型视频基础模型 SAM2 能够在图像和视频中实现任意分割,为多种应用提供了强大的基础模型。然而,许多此类应用场景需在移动电话、笔记本电脑等资源受限设备上运行。本工作旨在提升 SAM2 的移动端适配性,通过知识蒸馏将重型 SAM2 压缩为轻量级模型,从而支持在移动设备上对图像和视频执行任意分割。为此,我们提出超图知识蒸馏(Hypergraphical Knowledge Distillation, HyperKD),将超图概念引入知识蒸馏,以高效建模并迁移 SAM2 所具备的泛化性与全面性知识。HyperKD 包含时间超图蒸馏(Temporal HyperKD)与粒度超图蒸馏(Granularity HyperKD)两部分:前者构建超图以显式建模并提取 SAM2 中可泛化的时序知识,后者构建超图以显式建模并提取 SAM2 中全面的多粒度知识;二者均通过将轻量级学生模型与所构建的超图对齐来完成知识蒸馏。此外,我们提出了 MobileSAM2——一个新型轻量级 SAM2 模型家族,其在模型压缩过程中结合 HyperKD 进行最优架构搜索,以在效率与性能之间取得平衡。大量实验在多个基准数据集上验证了 MobileSAM2 的有效性,并表明其在具身智能(embodied AI)任务中展现出良好的泛化性能。
The recent large video foundation model, SAM2, enables segment anything in both images and videos, serving as a powerful base model for various applications. However, many of such use cases require to operate on resource-constrained devices like mobile phones and laptops. In this work, we aim to make SAM2 more mobile-friendly by distilling the heavyweight SAM2 into a lightweight model, facilitating segment anything in both images and videos on mobile devices. To this end, we propose Hypergraphical Knowledge Distill (HyperKD), which introduces the idea of hypergraph into knowledge distillation, aiming to effectively model and transfer SAM2's generalizable and comprehensive knowledge. HyperKD consists of Temporal HyperKD and Granularity HyperKD that construct hypergraphs to explicitly model and extract the generalizable temporal knowledge and the comprehensive multi-granularity knowledge from SAM2 respectively, which are then distilled into the lightweight student model by aligning it with the constructed hypergraphs. Besides, we present MobileSAM2, a new family of lightweight SAM2 that balances efficiency and effectiveness via searching the best model architectures with HyperKD during model size reduction. Extensive experiments validate MobileSAM2 across multiple benchmarks and show promising generalization performance on embodied AI tasks.