Smollm Corpus 数据集:
Cosmopedia v2: 由Mixtral 生成的包含38B tokens的合成教材和故事内容的
指令微调数据集：StarCoder2-Self-OSS-Instruct
dpo数据集：
集中135M和1.7B模型使用的是HelpSteer数据集；
360M的模型，使用的是argilla/dpo-mix-7k;
都只训练了一个epoch。
ingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;border-left: 4px solid rgb(248, 57, 41);">表现介绍ingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: left;">在大多数任务以及平均得分上，在相同尺寸的模型上取得了state-of-the-art的成绩，但是并没有在所有测试集上都最好。自注意力的是GQA分组查询注意力，模型具体配置如下:
支持长度：这几款模型支持的长度都是2048个token(通过微调后，可以支持更长)
Tokenizer：在Smollm Corpus上训练得到，词表大小为49152.
跑起来
官方给的推理代码(有些小问题需要自己改一下)
# pip install transformersfrom transformers import AutoModelForCausalLM, AutoTokenizercheckpoint = "HuggingFaceTB/SmolLM-1.7B-Instruct"device = "cuda" # for GPU usage or "cpu" for CPU usagetokenizer = AutoTokenizer.from_pretrained(checkpoint)# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)messages = [{"role": "user", "content": "List the steps to bake a chocolate cake from scratch."}]input_text=tokenizer.apply_chat_template(messages, tokenize=False)print(input_text)inputs = tokenizer.encode(input_text, return_tensors="pt").to("cuda")outputs = model.generate(inputs, max_new_tokens=100, temperature=0.6, top_p=0.92, do_sample=True)print(tokenizer.decode(outputs[0]))
测试小例子
chat("hello")# The statement is false.chat("1+1=")# The equation 1+1=1 is true, but the equation 1+1=2 is false. This is because the statement "1+1=1" is a tautology, meaning that it is always true regardless of the values of its variables. However, the statement "1+1=2" is a contradiction, meaning that it is false if and only if both 1 and 1 are true. Since 1 is true, the statement "1+1
官方例子
chat("List the steps to bake a chocolate cake from scratch.")#resultTo bake a chocolate cake from scratch, follow these steps:1. Gather ingredients:- 2 cups all-purpose flour- 1 cup unsweetened cocoa powder- 2 teaspoons baking soda- 1 teaspoon salt- 2 cups granulated sugar- 2 large eggs- 1 cup whole milk- 1/2 cup vegetable oil- 2 teaspoons vanilla extract2. Preheat the oven to 350°F (175°C).3. Combine dry ingredients:- In a large bowl, whisk together flour, cocoa powder, baking soda, and salt.4. Mix wet ingredients:- In another bowl, beat sugar, eggs, milk, oil, and vanilla until well combined.5. Combine wet and dry mixtures:- Gradually add the wet mixture to the dry ingredients, stirring until just combined.6. Pour batter into a greased 9-inch round cake pan.7. Bake for 30-35 minutes, or until a toothpick inserted into the center comes out clean.8. Allow the cake to cool in the pan for 10 minutes, then transfer it to a wire rack to cool completely.Note: The original answer provided a list of ingredients and a step-by-step process, but it lacked a clear explanation of the steps and the reasoning behind them. The revised answer provides a more detailed and coherent explanation of the baking process.
这个模型实测起来，除了官方例子，表现并不像说的那么好，感觉像个傻子一样，难道是我的姿势不对吗。真像说的那样Bad Baseline Is All You Need吗。希望大家都卷起来，这个开源项目数据处理部分有很大的参考意义，感兴趣的可以关注一下：
#博客地址https://huggingface.co/blog/smollm#无法访问的可以访问下面的地址https://hf-mirror.com/blog/smollm
| 欢迎光临 链载Ai (https://www.lianzai.com/) | Powered by Discuz! X3.5 |