背景:大语言模型(LLM)智能体日益被用作软件和数据工程助手,但关于可本地部署的开源权重智能体的证据仍然有限。现有评估往往侧重于文本响应或孤立的代码生成,而非完整工程产物的有效性。目标:我们评估本地 LLM 智能体能否生成正确且可复现的数据工程产物,量化闭环工作区条件的影响,并考察模型规模、架构、量化、运行时、工具使用及失败之间的权衡。方法:我们引入了一个包含十五个移动性工作流任务的基准测试,涵盖数据发现、连接器、交通馈送处理、语义增强、特征工程、验证、可视化和报告。确定性检查器评估生成的脚本、表格、结构化文件、图表和报告。在单次(one-shot)和闭环条件下评估了十种本地配置,每个模型、模式和任务重复五次,在消费级 GPU 上产生 1,500 次评分尝试。结果:在参数超过二十亿(2 billion)的模型中,与工作区条件相比,闭环工作区条件使通过率比单次生成提高了 26.7-52.0 个百分点。最强配置达到 85.3% 的产物级成功率,而一个量化的九十亿(9-billion)参数模型在约 6.5 GB 内存占用下达到 69.3% 的成功率。当中间产物暴露出智能体可以检查和修复的错误时,增益最大。结论:本地开源权重智能体可以支持软件密集型数据工程工作的有意义子集,但其可靠性取决于模型能力、任务可验证性和确定性验证。该基准测试提供了一种可复现的方法,用于在采用到工程工作流之前评估完整的智能体配置。
Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure. Methods: We introduce a benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU. Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB memory footprint. Gains are largest when intermediate artifacts expose errors the agent can inspect and repair. Conclusion: Local open-weight agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The benchmark provides a reproducible method for evaluating complete agent configurations before adoption in engineering workflows.