From 996de62c1593ee8c227138ba1e8fb72ea4f1741d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Jun 2025 02:37:03 +0000 Subject: [PATCH 1/4] Initial plan for issue From a1583e0bd0c47cabcf53ea1ac1c533a6f7524011 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Jun 2025 02:44:21 +0000 Subject: [PATCH 2/4] Add Doge2 architecture information and improve README content Co-authored-by: LoserCheems <124847097+LoserCheems@users.noreply.github.com> --- README.md | 20 ++++++++++++++++++-- README_zh.md | 20 ++++++++++++++++++-- 2 files changed, 36 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index fa46655..cf9f9f6 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,10 @@ English | [简体中文](./README_zh.md) **News**: -* **[2025-3-12]** 🎉We have completed the pre-training of four base models with parameters of **Doge-20M**, **Doge-60M**, **Doge-160M**, and **Doge-320**! +* **[2025-3-12]** 🎉We have completed the pre-training of four base models with parameters of **Doge-20M**, **Doge-60M**, **Doge-160M**, and **Doge-320M**! * **[2025-3-9]** 🎉We released the **SmallThoughts** inference dataset, greatly reducing the cost of inference fine-tuning! * **[2025-2-20]** 🎉 We now support the full training process of **pre-trained Doge-Base**, **instruction fine-tuned Doge-Instruct**, and **reasoning fine-tuned Doge-R1**, please refer to the [guide](./recipes/doge/README.md)! +* **[2025-2-15]** 🚀 We introduce **Doge2**, an enhanced architecture with improved Mixture of Experts design and better scalability! # small-doge @@ -34,6 +35,7 @@ English | [简体中文](./README_zh.md) * The small doge series is extremely lightweight, with the smallest version being about **$\frac{1}{7800}$** the size of GPT3, and strives to make even the most ordinary personal GPU capable of fast inference and even training.🏎️ * We provide full-stage code for dataset preprocessing, pre-training, supervised fine-tuning, reinforcement learning preference alignment, visual multimodal VLM (under development), and inference fine-tuning R1 (under development).🧪 * Standing on the shoulders of giants can see further, we hope that the small doge series of small models can provide researchers with more ideas and contribute to the road to achieving **Embodied Artificial General Intelligence**.🤖 +* We now offer two model architectures: **Doge** (original) and **Doge2** (enhanced with better MoE design and scalability) to meet different research and application needs.✨ > [!TIP] > We hope to use open-source tools and frameworks as much as possible to simplify the process from data processing to model training, so that beginners can easily understand and use.🤗 @@ -53,6 +55,8 @@ This project aims to develop a series of dynamic and fast small models to promot As shown in the figure, the sequence transformation part of the Doge architecture uses `Dynamic Mask Attention`, which can be understood as using self-attention related to value states during training, and using state-space without past state decay during inference, to solve the problem of existing Transformers or SSMs getting lost in long text. The state transformation part of Doge uses `Cross Domain Mixture of Experts`, which consists of dense linear layers and sparse embedding layers, and can additionally increase sparse parameters to continue training from dense weight checkpoints without retraining the entire model, thereby reducing the cost of continuous iteration of the model. In addition, Doge also uses `RMSNorm` and `Residual` with learnable parameters to adapt the gradient range of deep models. +We also provide **Doge2**, an enhanced architecture that builds upon the original Doge design with improved Mixture of Experts capabilities, offering better scalability and performance while maintaining the core efficiency principles. + ## Requirements @@ -90,7 +94,7 @@ pip install -e . ## Quick Start -We have written a [notebook](./examples/notebook.ipynb) and a [training guide](./recipes/doge/README.md) to demonstrate the entire process of dataset processing, model training, and model evaluation. You can also use the models that have been released independently. If you are interested, please read the notebook or training guide in detail, which contains specific steps and details! +We have written a [notebook](./examples/notebook.ipynb) and training guides for [Doge](./recipes/doge/README.md) and [Doge2](./recipes/doge2/) to demonstrate the entire process of dataset processing, model training, and model evaluation. You can also use the models that have been released independently. If you are interested, please read the notebook or training guide in detail, which contains specific steps and details! ## Models Released @@ -164,6 +168,18 @@ Here are the initial learning rates required to continue training at each checkp - Software: Transformers, TRL +### Doge2 (Enhanced Architecture) + +**Doge2** represents our next-generation model architecture with enhanced Mixture of Experts design: + +- **Improved MoE**: Enhanced expert routing with up to 900 experts and 15 experts per token +- **Better Scalability**: Optimized architecture for larger parameter counts while maintaining efficiency +- **Enhanced Performance**: Improved training stability and convergence +- **Extended Vocabulary**: Support for larger vocabulary sizes (up to 49,152 tokens) + +*Doge2 models are currently in active development. Training recipes and model weights will be released soon.* + + ## Expectations > [!IMPORTANT] > - If you find this project helpful, please consider giving it a star ⭐!

diff --git a/README_zh.md b/README_zh.md index 216cc66..114b99e 100644 --- a/README_zh.md +++ b/README_zh.md @@ -23,9 +23,10 @@ **新闻**: -* **[2025-3-12]** 🎉我们已经完成了 **Doge-20M**, **Doge-60M**, **Doge-160M**, **Doge-320** 四种参数规模的Base模型的预训练! +* **[2025-3-12]** 🎉我们已经完成了 **Doge-20M**, **Doge-60M**, **Doge-160M**, **Doge-320M** 四种参数规模的Base模型的预训练! * **[2025-3-9]** 🎉我们发布了 **SmallThoughts** 推理数据集, 大幅降低推理微调的成本! * **[2025-2-20]** 🎉 我们现在已经支持 **预训练的Doge-Base**, **指令微调的Doge-Instruct**, 以及**推理微调的Doge-Reason**的全部训练流程, 请参阅[指南](./recipes/doge/README_zh.md)! +* **[2025-2-15]** 🚀 我们推出了 **Doge2**, 这是一个改进的专家混合架构, 具有更好的可扩展性和性能! # small-doge @@ -33,6 +34,7 @@ * small doge系列极其轻量, 最小版本体积约是 GPT3 的 **$\frac{1}{7800}$**, 力求做到最普通的个人GPU也可快速推理甚至训练.🏎️ * 我们提供了数据集预处理、预训练、监督微调、强化学习偏好对齐的全阶段代码、视觉多模态VLM(正在开发)和推理微调R1(正在开发).🧪 * 站在巨人的肩膀上可以看的更远, 希望small doge系列小模型能为研究者提供更多思路, 为实现**具身通用人工智能**的道路添砖加瓦.🤖 +* 我们现在提供两种模型架构: **Doge** (原始架构) 和 **Doge2** (改进版本，具有更好的专家混合设计和可扩展性) 以满足不同的研究和应用需求.✨ > [!TIP] > 我们希望尽可能使用开源工具和框架来简化从数据处理到模型训练的过程, 以便初学者可以轻松理解和使用.🤗 @@ -51,6 +53,8 @@ 如图所示, Doge 架构的序列变换部分使用了 `Dynamic Mask Attention`, 可以理解为在训练时使用与值状态相关的自注意力, 在推理时使用没有过去状态衰减的状态空间, 以解决现有的 Transformer 或 SSM 在长文本中迷失的问题. Doge 的状态变换部分使用了 `Cross Domain Mixture of Experts`, 由密集线性层和稀疏嵌入层组成, 并可以额外增加稀疏参数, 以从密集权重检查点继续训练而无需重新训练整个模型, 从而降低模型的持续迭代成本. 此外, Doge 还使用了具有可学习参数的 `RMSNorm` 和 `Residual` 来适应深度模型的梯度范围. +我们还提供了 **Doge2**, 这是一个基于原始Doge设计构建的增强架构, 具有改进的专家混合能力, 在保持核心效率原则的同时提供更好的可扩展性和性能. + ## 安装要求 @@ -88,7 +92,7 @@ pip install -e . ## 快速入门 -我们已经编写了一个 [notebook](./examples/notebook.ipynb) 和 [训练指南](./recipes/doge/README.md) 来演示数据集处理、模型训练和模型评估的整个过程. 您还可以独立使用已经发布的模型, 如果感兴趣请详细阅读notebook或训练指南, 里面有具体的步骤和细节！ +我们已经编写了 [notebook](./examples/notebook.ipynb) ([中文版](./examples/notebook_zh.ipynb)) 和针对 [Doge](./recipes/doge/README_zh.md) 和 [Doge2](./recipes/doge2/) 的训练指南来演示数据集处理、模型训练和模型评估的整个过程. 您还可以独立使用已经发布的模型, 如果感兴趣请详细阅读notebook或训练指南, 里面有具体的步骤和细节！ ## 型号发布 @@ -162,6 +166,18 @@ Doge 使用 `wsd_scheduler` 作为训练调度器, 将学习率分为 `warmup`, - 软件: Transformers, TRL +### Doge2 (增强架构) + +**Doge2** 代表我们的下一代模型架构, 具有增强的专家混合设计: + +- **改进的专家混合**: 增强的专家路由, 支持多达900个专家和每个token15个专家 +- **更好的可扩展性**: 针对更大参数数量优化的架构, 同时保持效率 +- **增强的性能**: 改进的训练稳定性和收敛性 +- **扩展的词汇表**: 支持更大的词汇表大小 (多达49,152个tokens) + +*Doge2模型目前正在积极开发中. 训练配方和模型权重将很快发布.* + + ## 期许 > [!IMPORTANT] > - 如果您觉得 small-doge 对你有所帮助, 请给它一颗星星⭐吧!

From 9b24eed93a4e996814912755d31d76b2f563b638 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Jun 2025 02:45:55 +0000 Subject: [PATCH 3/4] Fix alt text typo and standardize dataset links Co-authored-by: LoserCheems <124847097+LoserCheems@users.noreply.github.com> --- README.md | 6 +++--- README_zh.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index cf9f9f6..382a528 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@
- samlldoges + smalldoges
@@ -144,14 +144,14 @@ Here are the initial learning rates required to continue training at each checkp |---|---|---|---|---|---|---| | [Doge-20M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-20M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 8e-4 | 0.25M | bfloat16 | | [Doge-60M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-60M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 6e-4 | 0.25M | bfloat16 | -| [Doge-160M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-160M-Instruct-SFT) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 4e-4 | 0.25M | bfloat16 | +| [Doge-160M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-160M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 4e-4 | 0.25M | bfloat16 | **DPO**: | Model | Training Data | Epochs | Content Length | LR | Batch Size | Precision | |---|---|---|---|---|---|---| | [Doge-20M-Instruct](https://huggingface.co/SmallDoge/Doge-20M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 8e-5 | 0.125M | bfloat16 | | [Doge-60M-Instruct](https://huggingface.co/SmallDoge/Doge-60M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 6e-5 | 0.125M | bfloat16 | -| [Doge-160M-Instruct](https://huggingface.co/SmallDoge/Doge-160M-Instruct) | [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 4e-5 | 0.125M | bfloat16 | +| [Doge-160M-Instruct](https://huggingface.co/SmallDoge/Doge-160M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 4e-5 | 0.125M | bfloat16 | **Evaluation**: | Model | IFEval (Prompt Strict Acc) | MMLU | BBH | ARC | PIQA | HellaSwag | tokens / s on i7-11 CPU | diff --git a/README_zh.md b/README_zh.md index 114b99e..b01c94b 100644 --- a/README_zh.md +++ b/README_zh.md @@ -142,14 +142,14 @@ Doge 使用 `wsd_scheduler` 作为训练调度器, 将学习率分为 `warmup`, |---|---|---|---|---|---|---| | [Doge-20M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-20M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 8e-4 | 0.25M | bfloat16 | | [Doge-60M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-60M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 6e-4 | 0.25M | bfloat16 | -| [Doge-160M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-160M-Instruct-SFT) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 4e-4 | 0.25M | bfloat16 | +| [Doge-160M-Instruct-SFT](https://huggingface.co/SmallDoge/Doge-160M-Instruct-SFT) | [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 2048 | 4e-4 | 0.25M | bfloat16 | **直接优化微调**: | 模型 | 训练数据 | 轮次 | 上下文长度 | 学习率 | 批量大小 | 精度 | |---|---|---|---|---|---|---| | [Doge-20M-Instruct](https://huggingface.co/SmallDoge/Doge-20M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 8e-5 | 0.125M | bfloat16 | | [Doge-60M-Instruct](https://huggingface.co/SmallDoge/Doge-60M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 6e-5 | 0.125M | bfloat16 | -| [Doge-160M-Instruct](https://huggingface.co/SmallDoge/Doge-160M-Instruct) | [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 4e-5 | 0.125M | bfloat16 | +| [Doge-160M-Instruct](https://huggingface.co/SmallDoge/Doge-160M-Instruct) | [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 4e-5 | 0.125M | bfloat16 | **评估**: | 模型 | IFEval (Prompt Strict Acc) | MMLU | BBH | ARC | PIQA | HellaSwag | tokens / s on i7-11 CPU | From 599b26bbf719f5329172b4402748e4e6f16b0a7a Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 7 Jun 2025 02:47:16 +0000 Subject: [PATCH 4/4] Fix Quick Start section links for better accuracy Co-authored-by: LoserCheems <124847097+LoserCheems@users.noreply.github.com> --- README.md | 2 +- README_zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 382a528..bf863e9 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ pip install -e . ## Quick Start -We have written a [notebook](./examples/notebook.ipynb) and training guides for [Doge](./recipes/doge/README.md) and [Doge2](./recipes/doge2/) to demonstrate the entire process of dataset processing, model training, and model evaluation. You can also use the models that have been released independently. If you are interested, please read the notebook or training guide in detail, which contains specific steps and details! +We have written a [notebook](./examples/notebook.ipynb) and a [training guide for Doge](./recipes/doge/README.md) to demonstrate the entire process of dataset processing, model training, and model evaluation. Doge2 training configurations are available in the [doge2 recipes folder](./recipes/doge2/). You can also use the models that have been released independently. If you are interested, please read the notebook or training guide in detail, which contains specific steps and details! ## Models Released diff --git a/README_zh.md b/README_zh.md index b01c94b..539c9fb 100644 --- a/README_zh.md +++ b/README_zh.md @@ -92,7 +92,7 @@ pip install -e . ## 快速入门 -我们已经编写了 [notebook](./examples/notebook.ipynb) ([中文版](./examples/notebook_zh.ipynb)) 和针对 [Doge](./recipes/doge/README_zh.md) 和 [Doge2](./recipes/doge2/) 的训练指南来演示数据集处理、模型训练和模型评估的整个过程. 您还可以独立使用已经发布的模型, 如果感兴趣请详细阅读notebook或训练指南, 里面有具体的步骤和细节！ +我们已经编写了 [notebook](./examples/notebook.ipynb) ([中文版](./examples/notebook_zh.ipynb)) 和 [Doge训练指南](./recipes/doge/README_zh.md) 来演示数据集处理、模型训练和模型评估的整个过程. Doge2训练配置可在 [doge2配方文件夹](./recipes/doge2/) 中找到. 您还可以独立使用已经发布的模型, 如果感兴趣请详细阅读notebook或训练指南, 里面有具体的步骤和细节！ ## 型号发布