Add StarRocks Parquet entries (single, partitioned) and Doris Parquet single variant - #1461
Conversation
… single Three new parquet-in-place entries: - starrocks-parquet / starrocks-parquet-partitioned: same local 1 FE + 1 BE cluster as the starrocks entry, but querying the parquet files in place through a view over the FILES() table function (file:// protocol) instead of loading them into StarRocks storage. The files store EventTime/EventDate as raw epoch ints (no parquet logical types), so the view converts them to DATETIME/DATE — otherwise date predicates silently match nothing; the partitioned files additionally lack the String annotation on BYTE_ARRAY columns, so those are cast back to VARCHAR (the duckdb-parquet binary_as_string equivalent). Validated locally on arm64: all 43 queries pass on both variants, with spot-checked results identical to clickhouse-local over the same files, and the view survives the stop/start cold cycle. - doris-parquet-single: sibling of doris-parquet (which reads the 100 partitioned files) over the single hits.parquet. Unlike the sibling, load moves the file the harness already downloaded into the BE dir instead of downloading a second 14 GB copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The run of Logs:
|
The first PR benchmark run failed at view creation with "Can not build local(): No available backend": the start script waited for a non-empty SHOW BACKENDS column by position, which passes as soon as the backend row exists, so ./load ran create.sql against a still-initializing BE. Wait for Alive=true instead, locating the column by header name (the position shifts between Doris versions), fail loudly if the BE never comes up, and make ./check require an alive BE too so the harness' check loop gates the load and every per-query cold restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The first Fixed in 697d7a5: 🤖 Generated with Claude Code |
…ioned (c6a.4xlarge)
|
Results for Logs:
|
The Alive-gated retry showed the real failure: the 3.0.5 BE never
becomes alive on the current Ubuntu 24.04 benchmark image ("FE is up
but no BE is alive" for the full 15-minute wait). The doris entry
already moved to 4.1.0-rc01; use the same tarball and be.conf cache
settings here. Also make ./check print SHOW BACKENDS status and the
tail of be.out when no BE is alive — the harness surfaces the last
failing ./check stderr on timeout, so a BE startup crash becomes
self-diagnosing in the benchmark log instead of invisible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ioned (c6a.4xlarge)
|
Results for Logs:
|
…ioned (c6a.4xlarge)
|
Results for Logs:
|
…nostics The 4.1.0 run showed the BE registers but never turns Alive — and on 4.x even `SELECT 1` fails with "No backend available as scan node", so the previous ./check aborted before printing its diagnostics, and the start script's FE wait (polling SELECT 1) burned its full timeout before ADD BACKEND ran. - Pin priority_networks = 127.0.0.1/32 in both fe.conf and be.conf so the FE's saved BE address (ADD BACKEND '127.0.0.1:9050') matches the address the BE reports in heartbeats; without it the BE self-detects the instance's private IP and can stay Alive=false forever (the starrocks entry needs the same single-node pin). - Wait for the FE with SHOW FRONTENDS (FE-only metadata) instead of SELECT 1. - ./check now dumps SHOW BACKENDS state, whether the doris_be process is running, and the tail of be.out on every failure path, so the next failing benchmark log is self-diagnosing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions
Error
|
The run of Logs:
|
The last run's diagnostics showed the BE process is not running and the FE heartbeat gets Connection refused, but the be.out tail printed nothing — because `2>/dev/null >&2` points stdout at the already-nulled stderr. Use `>&2 2>/dev/null`. Also capture start_fe/start_be output to files (the harness silences ./start, and a launcher that refuses to start writes only to its own stderr and never creates be.out), and dump the BE log dir listing, be.INFO tail, and dmesg oom/segfault lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions
Error
|
The run of Logs:
|
The captured start_be.sh output finally showed the root cause of every failed run: "Disable swap memory before starting be". cloud-init adds a 16 GB swapfile to every benchmark instance since f236e7b (2026-05-09), and Doris's BE launcher hard-refuses to run while swap is active — this is also why the doris and doris-parquet entries have produced no results since that date. Turn swap off in ./install; parquet-in-place reads the file from disk and doesn't need the swap headroom the swapfile exists for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions
Error
|
Root cause of the Fixed here for 🤖 Generated with Claude Code |
|
The run of Logs:
|
|
Results for Logs:
|
|
The run of Logs:
|
|
The run of Logs:
|
…arrocks-parquet-partitioned (c6a.4xlarge, c6a.metal, c7a.metal-48xl)
|
Results for Logs:
|
…arrocks-parquet-partitioned (c6a.2xlarge, c6a.large, c6a.metal, c6a.xlarge, c8g.4xlarge, c8g.metal-48xl)
|
Results for Logs:
|
…ioned (c8g.4xlarge, c8g.metal-48xl)
|
Results for Logs:
|
|
Results for Logs:
|
1 participant
Adds three parquet-in-place entries:
starrocks-parquet / starrocks-parquet-partitioned
Same local 1 FE + 1 BE cluster as the
starrocksentry, but querying the parquet files in place through a view over theFILES()table function (file://protocol) instead of loading them into StarRocks storage. Two data-format quirks required fixups in the view:EventTimeas raw INT64 epoch seconds andEventDateas raw INT32 epoch days (no parquet logical types), so the view converts them toDATETIME/DATE. Without this, predicates likeEventDate >= '2013-07-14'silently match zero rows.hits.parquet) lack the String annotation on their BYTE_ARRAY columns, so those infer asVARBINARYand string queries fail withError statistic type : varbinary; the view casts them back toVARCHAR— the equivalent of duckdb-parquet'sbinary_as_string=True.Validated locally on an arm64 box with StarRocks 4.0.2: all 43 queries pass on both variants, spot-checked results (including exact
COUNT(DISTINCT)and the minute-truncation query) are identical toclickhouse-localover the same files, and the view survives the stop/start cold cycle the harness performs before every query.doris-parquet-single
Sibling of
doris-parquet(which reads the 100 partitioned files) over the singlehits.parquet, again via thelocal()TVF. One improvement over the sibling:loadmoves the file the harness already downloaded into the BE dir instead of downloading a second 14 GB copy. Not validated locally (Doris ships no arm64 tarball) — the PR benchmark run will cover it.🤖 Generated with Claude Code