feat: add file-based Devin provider to fix ENAMETOOLONG on Windows#5
Open
jb-improving wants to merge 1 commit into
Open
feat: add file-based Devin provider to fix ENAMETOOLONG on Windows#5jb-improving wants to merge 1 commit into
jb-improving wants to merge 1 commit into
Conversation
Add devinProvider.js as a promptfoo custom provider that writes prompts to a temp file and passes them via --prompt-file instead of as a CLI argument. This avoids the Windows ENAMETOOLONG error when prompts exceed the 32k character command-line limit. The provider also uses async spawn and parses token usage from Devin's --export output. The provider entry in promptfooconfig.yaml is added but commented out, ready to be enabled when needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Windows, passing large prompts (>32k characters) as CLI arguments to the Devin CLI causes an ENAMETOOLONG error due to the operating system's command-line length limit. This PR adds a new file-based provider (devinProvider.js) that writes prompts to a temporary file and passes them via the --prompt-file flag, bypassing the command-line length limitation entirely.
Changes
devinProvider.js (new file)
promptfooconfig.yaml (modified)
Design Decisions
Validation
px promptfoo eval --no-cachewith both the exec-based and file-based providers enabledNext Steps