feat: test metadata reporting and a unit-aware metric engine - #316
Draft
roxblnfk wants to merge 7 commits into
Draft
feat: test metadata reporting and a unit-aware metric engine#316roxblnfk wants to merge 7 commits into
roxblnfk wants to merge 7 commits into
Conversation
Add a sandbox playground that emits ##teamcity[testMetadata] messages for every TeamCity metadata type. A #[TestMetadata] attribute declares the values; a TestMetadataInterceptor (registered by TestMetadataPlugin on the sandbox suite) writes them straight to the stream after the test body runs, nested between testStarted and testFinished like the built-in bench metrics. Covers numbers (a chartable table), text, links, and images/artifacts sourced from both local files and URLs; relative file references resolve against the test directory. Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bench metrics now spell `bench.<case>.<columnGroup>.<column>` instead of the flat `bench.<case>.<metric>`, so a consumer that understands the four-segment `testMetadata` convention rebuilds them as a sortable table with spanning column-group bands — the same grouping the terminal's ASCII table prints: Benchmark setup, Time results, Filtered results, Summary. Every case emits every column so the grid stays rectangular, and the standalone `bench.iterations` counter folds into each row's Benchmark setup group to keep the key depth uniform — a mixed depth would drop a consumer back to a flat list. The unit no longer hides in the leaf name. A value is a `Testo\Core\Metric\Metric` — a generic `Metric<TUnit>` carrying its number in the unit it was measured in. Units are split into family enums (Time, Memory, Percent, Scalar) under a shared `Unit` interface, so a `Metric<Percent>` cannot be handed to a consumer keyed to `Metric<Time>`. Each reporter renders the unit its own way: JUnit — no type attribute — suffixes the flat property name with the native unit (`…mean.us`, `…memory.bytes`); TeamCity converts at its boundary via `TeamcityMetricType` to the four number types it charts, scaling every time unit to milliseconds and every byte unit to bytes. The source keeps native precision for every consumer instead of rounding once at the mapper. Affects the TeamCity `testMetadata` and JUnit `<property>` output; the HTML/JSON structured map is unchanged. The sandbox metadata playground gains ms/bytes/ percent examples. Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A three-level `<prefix>.<row>.<column>` table (no column group) whose values all share one type, so the metadata consumer can plot it as a graph rather than a grid: rows as the x-axis, columns as the series, milliseconds as the y-axis. Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `#[TestMetadata]` attribute and its interceptor move out of the sandbox and into `testo/test`, so any suite can attach `testMetadata` to a test — the interceptor is registered by the default TestPlugin, and the attribute lives beside `#[Test]` as `Testo\Test\TestMetadata`. The interceptor builds the `##teamcity[testMetadata …]` service message itself rather than through the core formatter, so the plugin depends on nothing internal; the format is TeamCity's own stable protocol. The sandbox keeps only the showcase test, now importing the attribute from the plugin, and its bespoke plugin/interceptor are gone. Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `metadata` group to the bench self-test and the sandbox showcase so the metadata-reporting tests can be run as a set, rename the showcase's image entries, point the remote-image example at a reachable asset, and extend the Assert sandbox's logging demo. Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every case is present in the result, in declaration order, so a caller can walk an enum once and fall back to a default where a case carries nothing. Assisted-By: Claude Fable 5.1
…en units refactor(output): build the JUnit property suffix in the writer instead of on the unit The metric engine moves to `Testo\Metric`: `#[Dimension]` names what an enum measures, `#[Factor]` on a case gives its multiplier to the family's base unit, and `Units` reflects both once per enum and does the arithmetic. `Testo\Core\Metric` keeps only the concrete families. A conversion stays integral while every operand is an integer, so `2 KiB` is `2048` bytes rather than `2048.0`. `compact()` derives the most readable unit from the factors alone, so no per-family hook is needed. `Memory` is renamed to its binary units (`KiB`, `MiB`, `GiB`): the factors were already 1024-based, and the SI names misrepresented them. Backing values are the unit symbols, which the JUnit writer now appends itself; the `suffix()` method was a single reporter's concern living in the core contract. Assisted-By: Claude Fable 5.1
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔍 What was changed
#[TestMetadata]intesto/test: a repeatable attribute that attaches atestMetadatavalue (number, text, link, image, artifact) to a test. The interceptor is registered by the defaultTestPlugin, so any suite reports it to TeamCity out of the box. Relative image and artifact paths resolve against the test file's directory.bench.<case>.<columnGroup>.<column>table instead of a flat list, with the same column groups the terminal table prints. TeamCity and JUnit output of bench metrics changes; the HTML and JSON structured map does not.Testo\Metric:#[Dimension]on a unit enum names what it measures,#[Factor]on a case gives its multiplier to the base unit.Unitsreflects both once per enum and does the conversion;Metric::to()andMetric::compact()sit on top. The concrete familiesTime,Memory,Percent,Scalarstay inTesto\Core\Metric.Memoryuses binary units with honest names:Bytes,Kibibytes,Mebibytes,Gibibyteswith symbolsB,KiB,MiB,GiB. The factors were already 1024-based.Reflection::fetchEnumCaseAttributes()returns the attributes of every enum case keyed by case name.How it works
Unitvia theUnitConversiontrait and declares its cases with#[Factor]; the case without one is the base. Conversion is a ratio of factors and stays integral while every operand is an integer.compact()picks the largest unit the value still reaches at least one of, derived from the factors alone, so a family needs no hook of its own.msand memory tobytes, JUnit appends the unit symbol to the flat property name (…mean.us,…memory.B).Why?
A value should carry its unit instead of hiding it in a key name. Reporters were each parsing units back out of leaf names, and the bench mapper rounded once for everyone. With the unit on the
Metric, each consumer renders it natively and the source keeps its precision.Checklist
composer psalmpasses locally;composer test:ci -- --json --type=!benchpasses exceptBenchAttr::rangeSum, which trips the local Xdebug nesting limit on the recursive bench and is unrelated to this branchReview notes
Testo\Metricor#[TestMetadata]; that is left for a follow-up.