论文
arXiv
GeoAI
GIS
GeoLargeModel
GeoFoundationModel
中文标题
复用而非重训:通过特征蒸馏实现高效、多模态通用的地理空间基础模型
English Title
Reuse, Don't Retrain: Efficient, Modality-Versatile Geospatial Foundation Models through Feature Distillation
Mario Hamon, Pierre Adorni, Minh-Tan Pham
发布时间
2026/10/23 20:48:24
来源类型
preprint
语言
en
摘要
中文对照

地理空间基础模型(GeoFMs)的预训练成本高昂,然而模型系列往往在不同骨干网络规模和传感模态间重复这一过程。我们研究特征蒸馏作为一种更高效地复用现有表示的方法。首先,我们通过补丁级特征匹配,将冻结的 CROMA 和 TerraMind 教师模型蒸馏至更小的 Vision Transformers。在九项 PANGAEA 任务中,蒸馏后的 TerraMind Tiny、Small 和 Base 分别比同规模的已发布对应模型高出 2.76、3.02 和 1.87 个平均 IoU 点;且各自均达到或超过下一更大规模已发布模型的性能。蒸馏后的 CROMA-Base 提升了 2.05 个点。TerraMind-Base 仅需 43 A6000 GPU 小时,约为独立预训练报告成本的 0.5%。其次,我们提出 TEDI,这是一种非配对的多专家方法,结合了用于多光谱图像的 TerraMind 和用于高分辨率 RGB 图像的 DINOv3。独立的补丁嵌入将非配对输入路由至共享 Transformer 主干,从而无需配准数据。TEDI-Large 在我们的 PANGAEA-9 比较中取得了最高的单模型平均值(62.51 mIoU),整体性能优于两个教师模型,同时保留了其模态特定优势。TEDI-Base 以少 72% 的参数达到了 61.75 mIoU。这些结果表明,蒸馏改善了 GeoFM 的精度-计算前沿,并将互补专家整合为单一通用骨干网络。

English Original

Geospatial foundation models (GeoFMs) are costly to pretrain, yet model families often repeat this process across backbone scales and sensing modalities. We investigate feature distillation as a more efficient way to reuse existing representations. First, we distill frozen CROMA and TerraMind teachers into smaller Vision Transformers through patch-level feature matching. Across nine PANGAEA tasks, distilled TerraMind Tiny, Small, and Base outperform their scale-matched released counterparts by 2.76, 3.02, and 1.87 mean IoU points, respectively; each also matches or exceeds the next-larger released model. Distilled CROMA-Base gains 2.05 points. TerraMind-Base required only 43 A6000 GPU-hours, approximately 0.5\% of the reported cost of independent pretraining. Second, we introduce TEDI, an unpaired multi-expert approach combining TerraMind for multispectral imagery and DINOv3 for high-resolution RGB. Separate patch embeddings route unpaired inputs into a shared Transformer trunk, removing the need for co-registered data. TEDI-Large achieves the highest single-model mean in our PANGAEA-9 comparison (62.51 mIoU), outperforming both teachers overall while retaining their modality-specific strengths. TEDI-Base reaches 61.75 mIoU with 72\% fewer parameters. These results show that distillation improves the GeoFM accuracy-compute frontier while consolidating complementary experts into a single versatile backbone.

我的阅读记录

正在加载阅读记录…

元数据
arXiv2610.32566v1
来源arXiv
类型论文
抽取状态raw
关键词
GeoAI
GIS
GeoLargeModel
GeoFoundationModel
cs.CV