test: Make a fixture file the only kind of test

[chfast]

Naming a file used to collect one test per fixture in it, which naming a directory could not afford: enumerating fixtures means loading every file, and a release is gigabytes. The two forms counted, filtered and skipped differently, and only one of them scaled.

A file is now one test either way, in both tools. Nothing is read until a test runs, so what --collect-only lists no longer depends on how the path was given.

Peeled out of #1685 to land the behaviour change on its own: a few tests change meaning here, which is worth reading apart from the command that replaces these tools.

[codspeed-hq]

Merging this PR will not alter performance

✅ 129 untouched benchmarks

---

_{Comparing test/one-test-per-file (eea90ca) with master (ab3d68e)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.00%. Comparing base (ab3d68e) to head (eea90ca).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1699      +/-   ##
==========================================
- Coverage   98.01%   98.00%   -0.01%     
==========================================
  Files         180      180              
  Lines       16332    16288      -44     
  Branches     3739     3736       -3     
==========================================
- Hits        16007    15963      -44     
  Misses        247      247              
  Partials       78       78              

Flag	Coverage Δ
eest-develop	87.64% <81.81%> (+0.50%)	⬆️
eest-develop-gmp	25.90% <42.42%> (-0.04%)	⬇️
eest-legacy	16.84% <39.39%> (+0.07%)	⬆️
eest-libsecp256k1	28.08% <42.42%> (-0.05%)	⬇️
eest-stable	87.60% <81.81%> (+0.50%)	⬆️
evmone-unittests	94.28% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
core	96.19% <ø> (ø)
tooling	93.96% <100.00%> (-0.14%)	⬇️
tests	99.81% <ø> (ø)

Files with missing lines	Coverage Δ
test/blockchaintest/blockchaintest.cpp	95.34% <100.00%> (-1.63%)	⬇️
test/statetest/statetest.cpp	96.36% <100.00%> (-1.01%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

The blockchain runner’s new direct-file collection branch lacks dedicated test coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Unifies fixture handling so each file is collected as one test without loading it during collection.

Changes:

• Defers fixture loading until execution.
• Aligns named-file and directory behavior.
• Updates file-level collection and counting expectations.

File summaries

File	Review
test/statetest/statetest.cpp	Implements unified collection; update stale function and filtering comments.
test/integration/evmone-cli/test/statetest/CMakeLists.txt	Updates state-test collection expectations.
test/integration/evmone-cli/test/blockchaintest/CMakeLists.txt	Updates blockchain execution-count expectations.
test/blockchaintest/blockchaintest.cpp	Implements equivalent behavior, but needs direct-file --collect-only coverage asserting test.json appears once.

Review details

Suppressed comments (3)

test/blockchaintest/blockchaintest.cpp:43

• This changes directly named files to one file-level test, but the path option help at lines 59–61 still promises one separate test per fixture inside the file. Update the user-facing description so it matches the new collection semantics.

             }});

test/statetest/statetest.cpp:52

• This changes directly named files to one file-level test, but the path option help at lines 68–70 still tells users that each test inside a named file becomes a separate test. Update the CLI contract together with the behavior.

            }});

test/statetest/statetest.cpp:50

• Filtering now happens inside the file test for directly named files as well as directory entries. The filter_directory integration-test comment still says the direct-file case applies -k at registration, so it now documents a distinction that no longer exists; update that comment with this behavior change.

                    if (selected(test))
                        evmone::test::run_state_test(test, vm, trace, report);

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Fix the integration coverage omission and update the outdated CLI help text.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Lazy-loading coverage and help-text accuracy must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

test/blockchaintest/blockchaintest.cpp:40

• The deleted unloadable-file case leaves this new lazy-loading guarantee untested: all remaining direct-file collection tests use valid JSON, so they still pass if collection accidentally starts parsing files again. Keep not_json.txt as a --collect-only case that must succeed and list the path once, and separately assert that running it fails.

        cases.push_back(
            {file.path.string(), [path = file.path, &vm](evmone::test::TestReport& report) {
                 std::ifstream f{path};

test/blockchaintest/blockchaintest.cpp:60

• The help text's index.json exception applies only while walking a directory. A directly named index.json (or even a non-JSON file) reaches the else branch above and is registered as one test, so this currently documents behavior that the command does not enforce. Distinguish named-file handling from directory discovery.

               "Path to a test file or a directory of them. Every .json file is one test, "
               "except index.json, which is not a test file.")

test/statetest/statetest.cpp:69

• The help text's index.json exception applies only while walking a directory. A directly named index.json (or even a non-JSON file) reaches the else branch above and is registered as one test, so this currently documents behavior that the command does not enforce. Distinguish named-file handling from directory discovery.

               "Path to a test file or a directory of them. Every .json file is one test, "
               "except index.json, which is not a test file.")

test/statetest/statetest.cpp:49

• Named files now use this deferred selected predicate too, but the filter_directory test comment in test/integration/evmone-cli/test/statetest/CMakeLists.txt:99-101 still says filtering happens at registration for the direct-file test above. Update that comment so it no longer describes the removed distinction.

            [path = file.path, selected, &vm, trace](evmone::test::TestReport& report) {
                std::ifstream f{path};
                for (const auto& test : evmone::test::load_state_tests(f))
                {
                    if (selected(test))

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The state-test help text does not accurately describe direct-file behavior.

Review details

Suppressed comments (1)

test/statetest/statetest.cpp:69

• This help text is inaccurate for directly named paths: collect_tests() treats any non-directory—including index.json and the notes.txt fixture exercised below—as one test. Restrict the .json/index.json rule to directory traversal and describe the direct-file behavior, as the blockchain runner does.

               "Path to a test file or a directory of them. Every .json file is one test, "
               "except index.json, which is not a test file.")

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The reviewed changes have no unresolved approval-blocking issues.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

All reviewed changes are covered with no unresolved issues.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Blockchain-runner direct-file ignore behavior lacks dedicated regression coverage.

Review details

Suppressed comments (1)

test/integration/evmone-cli/test/blockchaintest/CMakeLists.txt:104

• The new direct-file --ignore path is only exercised by the statetest integration suite, so this separate blockchain-runner implementation can regress (for example, by using root instead of root.parent_path()) while all blockchain tests still pass. Add the analogous two-file collection case here, ignoring one basename and asserting only the other file is listed.

    PASS_REGULAR_EXPRESSION "^[^\n]*test\\.json\n$"

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced