ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">投完ICML之后火急火燎的入门RL，花了一些时间把RLHF学了。后来在知乎上看到了很多优秀的开源R1复现项目，于是手痒痒啃了下比较火的两个开源项目准备自己实践一下，一个是Huggingface的Open-R1，一个是Logic-RL。由于Logic-RL基于Verl，模型推理和训练过程都是shard到不同显卡上的，Huggingface的GRPOTrainer是单独用一张显卡做vllm推理，所以我在浅尝Open-R1做数学题的训练之后转移到了Logic-RL上面。ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">因为是民科，手上只有四张降了功率的3090，我就拿0.5B的千问做的实验，中间遇到了不少问题。ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">
探索0.5B模型在KK数据集上的强化学习训练
ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin-top: calc(2.33333em);margin-bottom: calc(1.16667em);clear: left;color: rgb(25, 27, 31);letter-spacing: normal;background-color: rgb(255, 255, 255);">ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">我使用了qwen-0.5VB的instruct模型实验，我看logic-rl论文中instruct模型也可以复现出思维链增长。但这里选用instruct模型主要还是因为base模型很难训练。ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin-top: calc(1.90909em);margin-bottom: calc(1.27273em);clear: left;color: rgb(25, 27, 31);letter-spacing: normal;background-color: rgb(255, 255, 255);">1. 多余的reward会让模型变懒ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">我首先是使用了Logic-RL中原来配置的reward规则，在原版规则中，模型学会格式后会得到一定奖励，但很快模型的输出就变得很短
<think>The reasoning process here involves understanding the characteristics of knights and knaves, analyzing the statements made by each person, and determining who is a knight and who is a knave.</think> <answer>Ethan is a knight, David is a knave, Samuel is a knight.</answer><|im_end|>
ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">然后模型几乎不care回答是否正确，输出长度急剧下降到只有几十个的长度。于是我尝试更改规则，仅当模型格式正确且回答正确的时候才有reward，其余情况都是最低分。ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-size: medium;letter-spacing: normal;background-color: rgb(255, 255, 255);">但令人没想到的模型依然只是水一下思考内容，然后在answer tag里输出答案，似乎只要遵循<think></think>的格式就会让模型放弃思考过程。我后来将格式reward中有关<think></think>的要求去掉了，在训练过程中就能顺利保留下来模型的思考过程。ingFang SC', 'Microsoft YaHei', 'Source Han Sans SC', 'Noto Sans CJK SC', 'WenQuanYi Micro Hei', sans-serif;font-optical-sizing: inherit;font-size-adjust: inherit;font-kerning: inherit;font-feature-settings: inherit;font-variation-settings: inherit;margin-top: calc(1.90909em);margin-bottom: calc(1.27273em);clear: left;color: rgb(25, 27, 31);letter-spacing: normal;background-color: rgb(255, 255, 255);">2. 模型很难直接学习3ppl以上的问题
2.**Avery's statement**: If Jacob is a knight, then Avery must be a knave, regrettably (vice versa). Since Jacob is not a knight, there's a contradiction here, indicating amistakein our reasoning. Alternatively, if Jacob is a knight, Avery must be a knave, meaning Jacob cannot be the perfect knight according to her statements.
Recheck
So, the contradiction here comes from Scarlett being a knight. This means the first term is misleading. Let'srecheck the clues.
To determine the identities of each character, we will analyze each statement step by step.
1. Avery's statement: "Zoey is a knight if and only if Aiden is a knave." Therefore, Zoey is a knave if Aiden is a knight. 2. Zoey's statement: "Aria is not a knave." Hence, Aria is a knight. 3. Lily's statement: "Zoey is a knight and Aiden is a knight." Therefore, Zoey is a knight. 4. Evelyn's statement: "Aria is a knave or Lily is a knight." Since Aria is a knight, Evelyn must be a knight. 5. Aria's statement: "Evelyn is a knight or Avery is a knave." Since Avery is a knight, Aria's statement is true, so Aryan is a knight. 6. Aiden's statement: "Avery is a knave." Therefore, Aiden is a knave.
So, the identities of the characters are: - Avery is a knave, - Zoey is a knight, - Lily is a knight, - Evelyn is a knight, - Aria is a knight, - Aiden is a knight.
The final answer is: <answer> (1) Avery is a knave, (2) Zoey is a knight, (3) Lily is a knight, (4) Evelyn is a knight, (5) Aria is a knight, (6) Aiden is a knight. </answer><|im_end|>
模型的回答固定为
总起，一步一步推理。
先重复每个人的话，在后面紧接着做推理。
给出答案。
在<answer> tag里给出答案。
但是，当我们仔细检查推理过程的时候，发现这个推理是错的，但答案是对的。例如第四条，Since Aria is a knight, Evelyn must be a knight，第五条Aryan名字都没出现过。当然最明显的，推理过程得到的结论，例如第六条Aiden is a Knave和答案Aiden is a knight根本也对不上。
