[Example] Clip_B and Clip_V from entropy dynamics#509
[Example] Clip_B and Clip_V from entropy dynamics#509pan-x-c merged 15 commits intoagentscope-ai:mainfrom
Conversation
Summary of ChangesHello @hiyuchang, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the framework's capabilities for fine-tuning Large Language Models using Reinforcement Learning by incorporating advanced entropy dynamics algorithms. It specifically implements the 'Clip_B' method, which refines token-level advantage calculations and introduces a new GRPO variant to manage these computations within the trainer. This integration allows for more nuanced control over the training process, aiming to improve model performance and stability. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
hiyuchang
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Clip_B algorithm, including a new advantage function and an example configuration. The implementation is largely correct, but there are opportunities for improvement. Specifically, the logic for token filtering in the advantage function can be simplified for better readability. There's also an inconsistency in the standard deviation calculation that should be addressed. A significant maintainability concern is the use of a patch file for modifying core trainer behavior; this change should be integrated directly into the source code. Finally, there are minor issues in the example's README and configuration file that should be fixed.
|
/gemini-review |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Clip_B and Clip_V algorithms, which is a valuable addition. However, the implementation has some significant structural issues that should be addressed. The reliance on manually applied patch files and code duplication (e.g., clipv_dp_actor.py) makes the changes difficult to integrate and maintain. A better approach would be to incorporate these changes directly into the core library, using configuration flags to enable or disable the new functionality. This would avoid code duplication and make the examples self-contained. Additionally, there are several areas in the code with duplication and opportunities for simplification that I've pointed out in the specific comments.
|
/unittest-module-trainer |
Summary
Failed Tests
Skipped
Tests
Github Test Reporter by CTRF 💚 |
|
/unittest-module-trainer |
1 similar comment
|
/unittest-module-trainer |
Summary
Skipped
Tests
Github Test Reporter by CTRF 💚 |
2 participants
Description
We add support for algorithms in On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models. Contact: @shuminwang-ai.
Checklist
Please check the following items before code is ready to be reviewed.