docs(longmemeval): clarify script selection and document messages_use config#72
Open
FBISiri wants to merge 1 commit into
Open
docs(longmemeval): clarify script selection and document messages_use config#72FBISiri wants to merge 1 commit into
FBISiri wants to merge 1 commit into
Conversation
… config The longmemeval readme only listed run_lightmem_gpt.py in Quick Start while the root README's Quick Start recommends run_lightmem_qwen.py. First-time users following different entry points ran different scripts without realising both exist and serve different provider backends. Changes: - Quick Start Step 3: list both scripts with clear provider labels (GPT-compatible vs Qwen/DeepSeek/compatible), and note that the root README's default example is run_lightmem_qwen.py - Add 'Key Configuration Parameters' section documenting the messages_use option (user_only / assistant_only / hybrid), including a note clarifying that Table 2 results used 'user_only' -- directly addressing the question raised in issue zjunlp#70 Closes zjunlp#70
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two documentation gaps in
experiments/longmemeval/readme.md.1. Missing
run_lightmem_qwen.pyin Quick StartThe
experiments/longmemeval/readme.mdQuick Start (Step 3) only listed:However, the root
README.mdQuick Start recommends:cd experiments python run_lightmem_qwen.pyBoth scripts exist and both implement the same LongMemEval evaluation pipeline — the only difference is which LLM provider they're configured for. Users following the longmemeval readme were unaware that a Qwen/DeepSeek-compatible alternative existed, causing confusion when the root README pointed to a different script.
Fix: Step 3 now lists both scripts with clear provider labels, and notes that the root
README.mdusesrun_lightmem_qwen.pyas its default example.2. Undocumented
messages_useconfiguration option (closes #70)Both evaluation scripts hard-code
"messages_use": "user_only"in their configuration block. The paper (Section 3.2) describes memory entries as containing{user_i, model_i}pairs, which led to the question in issue #70: which setting was actually used for the reported LongMemEval results in Table 2?Fix: Added a Key Configuration Parameters section documenting all three values:
"user_only""assistant_only""hybrid"A short note clarifies the apparent discrepancy between the paper's description and the script default.
Type of change
Checklist
run_lightmem_gpt.pyandrun_lightmem_qwen.pyexist in the directory and both usemessages_use: "user_only"