Support overriding metric configs - #223
Conversation
via CLI arguments or environment variables.
a61b552 to
89f6cde
Compare
| if config.metric_configs: | ||
| runner.config.metric_configs = config.metric_configs |
There was a problem hiding this comment.
We can debate this. Should it be possible to override metric configs in a rerun? It was useful to me for testing, but it might be too dangerous to accidentally override metric configs.
There was a problem hiding this comment.
I think that's a debate we could have about multiple config fields. Instead, we could consider adding an option to explicitly enable config overrides (or disable reading the config saved on disk), something like:
eva --run-id ... --ignore-previous-config
There was a problem hiding this comment.
i like the --ignore-previous-config idea. For most config values right now I don't think we can override them on re-run, so I wonder if we should enable overriding all of them at the same time and introduce --ignore-previous-config at the same time
2 participants
Support overriding metric configs via CLI arguments or environment variables.
Examples:
EVA_METRIC_CONFIGS='{"faithfulness": {"judge_model": "gpt-5.6-terra", "judge_params": {"reasoning_effort": "none"}}}'EVA_METRIC_CONFIGS__FAITHFULNESS__JUDGE_MODEL=gpt-5.6-terraEVA_METRIC_CONFIGS__FAITHFULNESS__JUDGE_PARAMS__REASONING_EFFORT=none