遥感大视觉语言模型（RS-LVLMs）已推动地球观测影像的多模态理解进展，但其性能在根本上受限于高分辨率处理：视觉令牌数量随输入线性分辨率呈二次增长，而关键视觉证据本身稀疏，且在扩展后的序列中愈发稀释。现有令牌剪枝方法主要依赖尺度无关的分辨率策略及孤立的重要性线索，难以实现任务对齐的粒度自适应与整体证据保留。为此，我们提出尺度自适应与地理空间证据调制的令牌剪枝方法（SA-GEM），一种即插即用框架，将任务自适应的令牌粒度分配与整体地理空间令牌重要性调制相统一。具体而言，一个轻量级路由器依据查询相关的令牌粒度需求选择分辨率；一个令牌重要性调制器则联合建模任务相关性、空间结构与局部冗余，以保留整体地理空间证据。我们表明，更高分辨率并非普遍有益；一旦达到足够粒度，令牌质量比令牌数量更为关键。在多个基准上的实验表明，SA-GEM 在准确率与推理效率两方面均持续优于现有剪枝方法。在 XLRS-Bench 上，其准确率较 GeoLLaVA-8K 提升 2.3%，总推理速度提升 2.4 倍。

RS-LVLMs have advanced multimodal understanding of Earth observation imagery, yet their performance is fundamentally constrained by high-resolution processing, as visual token counts grow quadratically with linear input resolution while important visual evidence is inherently sparse and increasingly diluted across the expanded sequence. Existing token pruning methods largely rely on scale-agnostic resolution policies and isolated importance cues, limiting task-aligned granularity adaptation and holistic evidence preservation. To address this, we present Scale-Adaptive and Geospatial Evidence-Modulated Token Pruning (SA-GEM), a plug-and-play framework that unifies task-adaptive token granularity allocation with holistic geospatial token importance modulation. Specifically, a lightweight router selects the resolution based on query-dependent token granularity, while a token importance modulator jointly models task relevance, spatial structure, and local redundancy to preserve holistic geospatial evidence. We show that higher resolution is not universally beneficial and, once sufficient granularity is reached, token quality matters more than token quantity. Experiments across various benchmarks demonstrate that SA-GEM achieves consistent gains in both accuracy and efficiency over existing pruning methods. On XLRS-Bench, it surpasses GeoLLaVA-8K by 2.3% in accuracy with a 2.4 times total inference speedup.