Conversation
…achine, fake peer First milestone of the native (Connector/C-free) backend, as an isolated `MySQL.Protocol` module with no DBInterface/Tables dependency: - constants generated from the server headers (`scripts/gen_constants.jl` over my_command.h / mysql_com.h / field_types.h, SHA-256 of the inputs stamped), MariaDB extended capability bits, load-time asserts that guard the documented vendor defects (COM_SET_OPTION = 0x1B); - bounds-checked byte cursor and lenenc/fixed-int codecs; packet reader/writer with 0xFFFFFF chunking, the empty-terminator rule, sequence validation and every `Limits` check applied before a buffer grows; - the command-specific phase machine (`phases.jl`) with a transition log; the test suite asserts every transition row is exercised; - HandshakeV10 / SSLRequest / HandshakeResponse41, capability negotiation, MariaDB version normalization, phase-aware OK/EOF/ERR/LOCAL-INFILE/auth discriminators, column definitions, command/response framing, a `Session` whose transport is replaced in place at STARTTLS; - `FaultTransport` fault injection and a scripted loopback fake peer covering malformed, limit and interruption cases; vendor golden hex vectors; - Reseau TCP/TLS transports only: no `Sockets` dependency (Unix sockets and named pipes are deferred), so the protocol tests run on every CI lane without Docker. `docs/protocol-notes.md` records the sources of truth, the vendor conflicts and the clean-room rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Connection-phase security for the native backend: - `crypto.jl`: RSAES-OAEP(SHA-1, MGF1-SHA-1) through OpenSSL_jll's libcrypto (PEM public key -> EVP_PKEY, explicit padding/digest setup, `k - 42` plaintext cap, every handle freed in `finally`, OPENSSL_cleanse zeroing); - `auth.jl`: mysql_native_password, caching_sha2_password (fast path, full authentication over TLS, RSA password exchange over plain TCP only when the caller opts in via `server_public_key`/`get_server_public_key`), sha256_password, mysql_clear_password (explicit enablement and either `ssl_mode = :verify_identity` or `insecure_cleartext_auth`), auth-switch and more-data rounds under the round/byte limits, an `AuthPolicy` carrying the transport facts, and an optional trace of the exchange shape; - `tls.jl`: `ssl_mode` (`:disabled`/`:preferred`/`:required`/`:verify_ca`/ `:verify_identity`) mapped onto a Reseau TLS config, SNI for DNS names and IP literals only when verification needs them, `starttls!` replacing the session transport in place with no plaintext fallback once SSLRequest is sent; TLS failures surface as `TLSNegotiationError` before authentication (a TLS 1.3 peer may reject the session on the first post-handshake record). Tests: OAEP round trips through a test-side decrypt for 2048/3072/4096-bit keys, nondeterminism, length boundary, malformed/EC keys, a 20k-iteration leak check; scramble verification with the server-side formulas; the policy gates and the caching_sha2 continuation state machine; full exchanges against the fake peer (fast, refused-by-default, TLS cleartext, RSA with key retrieval and local keys, auth switch, unsupported switch, MariaDB native, wrong password, sha256, cleartext gating). Test PKI under `test/protocol/certs/` (`gen.sh`). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e lanes The opt-in `MySQL.Native` layer on top of `Protocol`: - `options.jl`: `ConnectOptions` with the keyword truth table (removed, deprecated and deferred keywords fail or warn explicitly), the `ssl_*` conflict table (`resolve_ssl_mode`: ssl_ca/ssl_capath escalate the default to :verify_ca, ssl_verify_server_cert to :verify_identity, ssl_enforce to :required, an explicit ssl_mode wins, contradictions are errors), `tls_version`, option files (`[client]` + group, quoting, unknown keys ignored, `!include` rejected, world-writable and `.mylogin.cnf` skipped with a warning), opt-in `MYSQL_TCP_PORT`, utf8mb4-only charset; - `reaper.jl`: finalizer-free reclamation — a dropped `Handle` enqueues its `ReapEntry` via a CAS on an `@atomic` state and the timer/`reap_now!`/ `atexit` reaper closes the transport exactly once under a global lock; - `connect.jl`: dial, greeting, STARTTLS per ssl_mode, authentication, the utf8mb4 bootstrap contract (SET NAMES only when session tracking does not already report utf8mb4) and `init_command`, all under one absolute `connect_timeout` deadline that is cleared once the session is READY; `close!` sends COM_QUIT and retires the reaper entry. Tests: the ssl_mode matrix against a Reseau-TLS fake peer (preferred with and without server TLS, disabled, required against a TLS-less server, verify_ca against a self-signed chain with no fallback, verify_identity with IP SAN, DNS-only certificate, ssl_server_name override, mutual TLS on TLS 1.2, 1.3 and the mixed-version path, a coalescing peer, stalls at the greeting, the TLS handshake and the auth reply under connect_timeout, init_command); the options truth table, conflict table and option files; reaper exactly-once and descriptor-count tests; and live lanes (`MYSQL_NATIVE_IMAGES`, default mysql:8.4 + mariadb:11.4) proving fast vs full vs RSA caching_sha2, the auth switch to native accounts, sha256 over TLS and RSA, access-denied errors and the utf8mb4 bootstrap against real servers. The mixed-version mutual-TLS case needs Reseau with JuliaServices/Reseau.jl#150. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, reaper locking - crypto: keep the PEM bytes alive for the whole lifetime of the memory BIO (BIO_new_mem_buf only borrows the buffer) and accept non-Vector byte windows by copying; preserve the error-text buffer while reading it; - auth: wipe the packet writer's frame buffer and every reply buffer after the handshake response and each continuation send, so a cleartext or full-auth password does not linger until the next packet; name the trace events after the branch that ran (`:full_auth_cleartext`/`:full_auth_rsa`); - tls: a failing close of the half-built TLS wrapper can no longer mask the handshake error; bracketed IPv6 hosts reach the verifier without brackets; - reaper: stats updated under the queue lock; the timer/atexit setup runs under a ReentrantLock instead of the finalizer-side spinlock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Describe per-operation transport timeouts and the current reentrant statement-reaping lock. Co-Authored-By: Codex <codex@openai.com>
Execute every plan section 4.2 surface against both backends and assert exact line coverage of the contract table. Co-Authored-By: Codex <codex@openai.com>
Reject integer limits that cannot fit their storage type and saturate transport deadlines instead of wrapping at large timeout values. Co-Authored-By: Codex <codex@openai.com>
Reject an outer executemultiple advance from a task that does not own the active streaming cursor before it can stale rows or drain the wire response. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Some servers reply to COM_SET_OPTION with a single 0xFE byte. Treat that legacy form as an EOF response while retaining the current session status. Co-Authored-By: Codex <codex@openai.com>
Binary DATE values use only the zero-length and four-byte forms. Reject DATETIME-sized DATE payloads instead of ignoring their time fields. Co-Authored-By: Codex <codex@openai.com>
Fail while reading result metadata instead of decoding MYSQL_TYPE_VECTOR as a string. The connection now faults with a protocol error before any row data is used. Co-Authored-By: Codex <codex@openai.com>
Treat named_pipe=nothing like the C backend default instead of rejecting it during native option validation. Co-Authored-By: Codex <codex@openai.com>
A re-prepare can reduce the parameter count. Clear retained chunks that no longer name a valid parameter so the next execute cannot index past its parameter tuple. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Validate all length-encoded column identifier fields before native callers can convert a name to Symbol. This keeps hostile metadata failures inside the protocol error contract and faults the session deterministically. Co-Authored-By: Codex <codex@openai.com>
Document every finding fixed during the independent M6 review, the exact validation results, review decisions, deferred external gates, and the final CLEAN verdict. Co-Authored-By: Codex <codex@openai.com>
Run the section 8.9 correctness, limit, and allocation workload whenever Docker is available. MYSQL_PERF_GATES now gates only timing ratios, as documented, and explicit policy tests cover CI, opt-in, local, and no-Docker plans. Co-Authored-By: Codex <codex@openai.com>
Record the recovered section 8.9 correctness gates, their final bounds-checked results, and the CI policy regression as finding 29 while retaining the CLEAN verdict. Co-Authored-By: Codex <codex@openai.com>
Document that an empty host or localhost (Unix) / "." (Windows) selects the deferred local transport and needs protocol=:tcp for a local TCP connection, and relocate Codex's M6 cross-review report under docs/reviews/ alongside M3/M4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #243 +/- ##
==========================================
- Coverage 71.68% 69.52% -2.17%
==========================================
Files 10 35 +25
Lines 1275 4046 +2771
==========================================
+ Hits 914 2813 +1899
- Misses 361 1233 +872 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The branch adds the Windows CI lane, which ran the suite on Windows for the first time and exposed two pre-existing portability issues: - Clong is Int32 on Windows x64 (Int64 on Linux/macOS), so YEAR maps to unsigned(Clong) = UInt32 there; the schema/value assertions hardcoded UInt64. The native mapping is correct (it preserves 1.x's Clong-based YEAR); the test now uses unsigned(Clong). - Windows runners ship the Docker CLI but only Windows-container mode, so pulling the Linux mysql:8.4 image fails. docker_available() now returns false on Windows so the live lanes, §8.9 gates, and Connector/C integration tests skip there (macOS already skips: no CLI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex's revision ran the §8.9 native-vs-Connector/C correctness gates (1M-row scans, 64 MiB blob, 100k executemany) in the PR test job. Under CI's coverage instrumentation that pushed the Linux Julia-1 lane past the 60-minute cap (it was cancelled mid-run; nightly finished at 47 min). Value parity is already asserted in the PR live lanes by the two-backend compat manifest, and the dedicated perf job (MYSQL_PERF_GATES=1) runs the full §8.9 block. So gate the whole §8.9 block behind MYSQL_PERF_GATES under CI (opt-in), keep it on for local runs, and raise the test-job timeout to 90 min for headroom. Native line coverage without §8.9 is 96.89% (gate 85%), so the coverage check is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
julia-runtest instruments coverage on every matrix lane, and running the Docker live lanes + leak soak under instrumentation on a shared runner blew the time budget (the Julia-1.12 Linux lane was cancelled at the cap). Coverage is only needed for the §8.15 gate, and the serverless fake-peer suite alone covers 96.15% of src/Protocol+src/Native. So: - test matrix runs with coverage: false (fast lanes; still runs serverless + live lanes + Connector/C integration), timeout back to 60m. - a new job runs the serverless suite under --code-coverage=@src (no Docker, no soak), enforces the 85% native gate, and feeds codecov. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…trix serverless-only The cross-platform test matrix ran the full Pkg.test, whose Docker integration (native live lanes on mysql:8.4/mariadb:11.4, the GC-thrash leak soak, and the Connector/C tests) hung the Julia 1.12 runtime on shared runners: the Julia-1 and nightly Linux lanes produced no test output for 60 minutes and were cancelled, while a Docker-less runner finished the same lane in 4 minutes. (The soak's exposure to a Julia 1.12+ GC-runtime flake is a known risk.) - runtests.jl gates the live lanes and the Connector/C integration behind run_integration = docker_available() && MYSQL_INTEGRATION != "0"; the serverless protocol suite always runs. Verified: MYSQL_INTEGRATION=0 -> 1528/1528 serverless-only in ~1 min. - The test matrix (all OSes x 1.10/1/nightly) sets MYSQL_INTEGRATION=0 -> fast, reliable, server-free; timeout back to 30m. - A new `integration` job runs the Docker tests on ubuntu + Julia 1.10 (stable GC under the soak), timeout 90m. - codecov.yml marks codecov's project/patch statuses informational: the `coverage` job uploads native serverless coverage (the Connector/C backend is exercised by integration but not instrumented in that upload), and the real coverage gate is scripts/check_native_coverage.jl (>=85%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review artifacts of the M3-M6 rounds; they don't belong in the package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removes MariaDB Connector/C entirely: src/api (3.5k lines of C wrappers,
enums, and handle lifetimes), the C driver layer (execute/prepare/load),
and the MariaDB_Connector_C_jll / Libdl dependencies. The former
MySQL.Native driver layer is promoted to the package root:
MySQL.Connection, Statement, Cursor{binary,buffered} (TextCursor /
BinaryCursor, TextRow / BinaryRow), MySQL.ping / escape /
escape_identifier / send_long_data! / reset_statement! / ConnectOptions.
The value types move to the top level (MySQL.Bit, MySQL.DateAndTime,
MySQL.juliatype) and the protocol errors get their 1.x names back as
aliases: MySQL.Error / MySQL.StmtError / MySQL.MySQLError.
Deliberate 2.0 cleanups beyond the documented native-backend fixes:
only a leading mysql:// host prefix is stripped, port=0 means 3306,
conn.port is an Int, enum-valued options are Symbols, option values are
validated against closed type sets, and lastrowid on a SELECT cursor
reports 0. A bare scalar remains accepted as single-parameter params.
The test suite follows: the dual-backend compat manifest becomes a
native-only golden behavior manifest (25 preserve-row goldens captured
against mysql:8.4 from the final dual-backend runs), the §8.9 gates keep
correctness/allocation/limit assertions and print a timing report (the
C-ratio gates retire; see bench/), and the 1.x integration suite runs
against the native client.
Internals are --trim=safe-clean for juliac: concrete Transport and auth
plugin unions (which also lets the reaper drop its invokelatest — no
transport close method can postdate the timer's world), typed option
extraction, Val-parametrized cursor construction, named runtime
callbacks with registered entrypoints, and the user-supplied local-infile
handler routed through the runtime's generic-dispatch entry.
Docs: the migration guide is rewritten as a 1.x → 2.0 upgrade guide
(API mapping table, upgrade checklist, behavior tables, option value
types, static-compilation notes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test/mysql_trim_workload.jl drives the main entrypoints — connect (handshake, auth, charset bootstrap), buffered and streaming text execute, prepared statements, one-shot parameterized execute, ping, escape, close — against a scripted in-process MySQL server on a loopback Reseau listener, consuming rows through the schema-typed accessor. The harness (test/trim_compile_tests.jl, wired into Pkg.test on Julia 1.12+) builds it with juliac --trim=safe in a temp environment, requires zero verifier errors and warnings, runs the executable, and asserts its output; MYSQL_RUN_TRIM_TESTS=0 skips it. Known trim caveats (documented in the migration guide): task bodies must be entrypoint-registered named functions, and Reseau's deadline-armed waits (connect/read/write timeouts) hang in trimmed builds, so the workload connects without timeouts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bench/run.jl compares the checkout (native) against MySQL@1 (MariaDB Connector/C) on the same Docker fixture the §8.9 gates use, one child process per environment, and prints a ratio table; bench/README.md records reference numbers from the final dual-backend branch (scans at 0.97-1.09x of Connector/C, round-trip-bound paths at 0.58-0.81x on macOS with a ~35µs/command latency gap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
quinnj
changed the title
…rip round) Commands are framed directly into the packet output buffer (start_command!/finish_command! with an automatic chunked fallback for >=16 MiB payloads) instead of building one or two intermediate payload vectors per command; COM_STMT_EXECUTE lets the driver append the parameter block in place. Repeated prepared executes compare the cached type signature without materializing it and reuse a statement-owned scratch, and — when the execute-time metadata matches the statement's cache, the overwhelmingly common case — the cursor now aliases the statement's schema containers (which are replaced, never mutated, on a schema change). Result-less cursors share constant empty containers, the name->index Dict is built lazily on first name-based access, buffered rows are copied without a per-row SubArray, and executes without per-call decode overrides reuse the connection's ResultOptions. Same-fixture cross-driver bench (bench/run.jl, macOS ARM): 100k executemany 0.76x -> 1.38x of Connector/C, text/tiny scans 1.13x, 10k round trips 0.58x -> 0.85x; COM_PING 6 -> 3 allocs, SELECT 1 43 -> 35, prepared execute 22. In-suite: 10k round trips 2.57s -> 1.60s, 100k executemany 24.5s -> 14.0s. The residual round-trip gap is transport task-wakeup latency, not per-command CPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The entrypoint registrations for the reaper/finalizer/bind-resolver callbacks were silently ineffective: jl_add_entrypoint calls from package top level run in the precompile process, and from __init__ they never reach the juliac driver's entrypoint list either (established empirically while fixing the same class in Reseau; see Reseau PR #151). Every runtime-dispatched callback now gets a static call edge instead — a direct call guarded on the never-true-but-unfoldable TRIM_CALL_EDGE Ref, with the callee @noinline so the standalone specialization the runtime dispatches to is actually emitted. The reaper also moves from Timer(callback) — Base dispatches timer callbacks from an internal closure task a trimmed build cannot run — to a named reaper_loop task that waits on a plain repeating Timer (woken from the event loop's C side). Two latent issues the traced bodies exposed are fixed on the way: reap_now!'s stats update no longer goes through a closure that boxed the counter (also drops the allocation under the SpinLock the 2.0 design review flagged), and the bind resolver's channel element is a concrete struct (tuple types are covariant, so Tuple{Bool, Any} is abstract and its put! was not statically resolvable). The trim workload now proves the chain end to end inside the trimmed executable: it abandons a connection, forces GC, and asserts the finalizer parked the transport and the timer-woken reaper closed it. connect_timeout stays out of the workload until a Reseau release carries the #151 dial fix; read/write deadlines are verified working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires Reseau's new opt-in bounded direct wait (Reseau PR #152) through a connect option: with direct_wait_ms > 0, a task waiting for the server's response first blocks its own OS thread in a bounded poll(2) before parking on the central poller, so readiness arrives as a single kernel wake instead of two — measured ~10-15us less per round trip against Dockerized mysql:8.4 (COM_PING ~150 -> ~135us, 20k-row executemany -13%) at the cost of other tasks scheduled on that thread waiting up to the budget. Off by default; capped at 10s; applied to the final transport after STARTTLS and on reconnect. Requires a Reseau providing TCP.set_direct_wait! (newer than 1.4.1) — requesting it on an older Reseau is an ArgumentError; the option-validation tests are Reseau-version independent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MySQL.jl 2.0: native Julia wire-protocol client
This replaces the MariaDB Connector/C backend with a from-scratch implementation of the
MySQL client/server wire protocol in Julia —
MySQL.Protocol(framing, handshake, auth,TLS, text + binary protocols) on Reseau.jl
transports — and removes the C library entirely.
MySQL.Connectionis the nativeclient; there is no dual-backend window. Every
libmariadbccall, its dynamic pluginloading, and its C handle lifetimes are gone, along with the crash classes they caused
(#220, #236, #240, #208, #206).
Migration: docs/src/migration.md
— upgrade checklist, the
MySQL.API→ 2.0 name mapping, and the full behavior tables.Most DBInterface/Tables code runs unchanged; the notable breaks are
MySQL.APIremoval(
MySQL.Error/MySQL.StmtError/MySQL.Bitat top level),multi_statementsdefaultingto
false(1.x enabled it via anif/elseifbug), Symbol-valuedssl_mode/protocoloptions, and strict validation of unknown/removed options.
Testing (1893 tests):
mysql:8.4andmariadb:11.4plus a golden behavior manifest:every §4.2 compatibility row asserts a recorded value, captured from the pre-removal
dual-backend runs that proved native ≡ Connector/C
(≤ String/Vector cols + 1 per row) assert in CI; wall-clock timings are reported, and
bench/keeps the cross-driver comparison againstMySQL@1repeatable--trim=safeworkload:Pkg.test(Julia 1.12+) compilestest/mysql_trim_workload.jl— connect, buffered + streaming text queries, preparedstatements, one-shot parameterized execute, ping, escape against an in-process scripted
server — with zero trim-verifier errors and runs the executable
Performance vs Connector/C on the same servers (macOS ARM; Linux CI holds the 0.75x
gates): binary scans 1.09x, tiny/NULL rows 1.06x, 64 MiB blob 1.02x, text scan 0.97x;
round-trip-bound paths trail on macOS (COM_PING 151µs vs 116µs) — the per-command latency
work is queued as a follow-up round along with the remaining copy/allocation findings.
Deferred (clear errors, planned for 2.x): Unix sockets / named pipes, compression, server
cursors, query attributes, MariaDB ed25519/PARSEC auth, pooling, cancellation.
🤖 Generated with Claude Code