在构建训练混合数据时，这一区分至关重要。原生记录与罗马化记录提供了不同的脚本暴露度，但可表达相同的底层内容。Token计数描述的是保留的语料库（包括已接受的改写），而非特定训练运行的Token预算。计数还取决于所使用的分词器，因此语料库清单与训练Token计数应分开解读。

This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.

在构建训练混合数据时，这一区别至关重要。原生文字与罗马化记录提供了不同的字符集暴露度，但可能表达相同的底层内容。Token 计数描述的是保留语料库（包括已接受的改写版本），而非特定训练运行的 Token 预算。由于计数依赖于所使用的分词器，因此语料库清单和训练 Token 计数应分开解读。这些增益支持了结合合成与自然数据的印度语言持续预训练（CPT）方案，而其他模型和语言的混合比例选择仍有待进一步研究。配套的印度语言报告利用十二种语言组的首轮已接受数据，考察了源数据混合、合成数据池规模以及字符集覆盖情况。评估完整语料库（包括恢复的改写版本及其他十种语言）仍是后续对照研究的下一步工作。 以下示例首先从数据集卡片中获取仓库 ID，随后发现可用的数据划分并流式传输样本。Hugging Face Datasets 支持流式加载，使读者无需下载整个语料库即可检查记录。一旦发布可用，请固定数据集修订版本以确保实验的可复现性。对于首次训练研究，请在固定的 Token 预算下比较不同混合比例，并记录自然/合成数据以及原生/罗马化文字的占比。跟踪各语言的结果，同时关注英语能力的保持情况以及语言/字符集质量。在创建留出集（held-out splits）时，需考虑跨语言和跨字符集源自同一来源的记录。 建议从少量自有源文档开始。检查翻译质量以及代码、数学公式和格式的保留情况，然后在与母语评审人员共同优化提示词和检查机制后再进行规模化扩展。将生成过程与语料库接受过程视为独立步骤：插件返回候选结果及评分，而你的过滤策略决定哪些记录进入训练语料库。翻译拓宽了对源材料的访问途径，但不能替代原本以印度语言撰写的文本，也无法保证对本地知识和文化的覆盖。我们建议将此语料库与适当来源的自然印度语言文本相结合，并针对预期用例评估所得到的混合效果。 探索语言配置，测试训练混合方案，或将开源方案适配至新领域。分享各语言的结果以及对生成工作流程的改进，有助于社区在此基础上继续推进相关工作。

This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately. These gains support the combined synthetic-and-natural Indic CPT recipe, while leaving the choice of mixture for other models and languages to further study. The companion Indic report examines source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data for the twelve-language group. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step for those controlled studies. The example below asks for the repository ID from the dataset card, then discovers the available splits and streams a sample. Hugging Face Datasets supports streaming so readers can inspect records without downloading the entire corpus. Once the release is available, pin a dataset revision for reproducible experiments. For a first training study, compare mixtures at a fixed token budget and record the natural/synthetic and native/romanized shares. Track per-language results alongside English capability retention and language/script quality. When creating held-out splits, account for records derived from the same source across languages and scripts. Start with a small sample of your own source documents. Inspect translation quality and preservation of code, mathematics, and formatting, then refine the prompts and checks with native-language reviewers before scaling up. Treat generation and corpus acceptance as separate steps: the plugin returns candidates and scores, while your filtering policy determines which records enter the training corpus. Translation broadens access to the source material, but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture. We recommend combining this corpus with appropriately sourced natural Indic text and evaluating the resulting mixture for the intended use case. Explore the language configurations, test a training mixture, or adapt the open recipe to a new domain. Sharing per-language results and improvements to the generation workflow can help the community build on this work.