ti.sil/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-12-13 03:15:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs: update evaluation dataset size in finetuning benchmark documentation

Browse source
This commit is contained in:
mrT23 2025-03-04 08:24:48 +02:00
parent d3f83f3069
commit bb24a6f43d
No known key found for this signature in database
GPG key ID: D350490E39D5F5AD
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/docs/finetuning_benchmark/index.md
View file

@ -68,7 +68,7 @@ Here are the prompts, and example outputs, used as input-output pairs to fine-tu
### Evaluation dataset
- For each tool, we aggregated 100 additional examples to be used for evaluation. These examples were not used in the training dataset, and were manually selected to represent diverse real-world use-cases.
- For each tool, we aggregated 200 additional examples to be used for evaluation. These examples were not used in the training dataset, and were manually selected to represent diverse real-world use-cases.
- For each test example, we generated two responses: one from the fine-tuned model, and one from the best code model in the world, `gpt-4-turbo-2024-04-09`.
- We used a third LLM to judge which response better answers the prompt, and will likely be perceived by a human as better response.