3.x: add Meson adapt for Cloudberry - #1940
Draft
tuhaihe wants to merge 13 commits into
Draft
Conversation
The PG16 merge brought in 266 meson.build files verbatim from upstream, with no Cloudberry awareness at all: no GP options, no GP source files, no GP catalogs, and several upstream targets that Cloudberry does not build. This wires up enough of it to configure, build, install and initdb a working single-node Cloudberry with meson. Build options and configuration: - 24 GP options in meson_options.txt, defaults matching configure.ac - GP_VERSION / GP_MAJORVERSION / GP_VERSION_NUM computed via getversion, without writing a VERSION file into the source tree - ~20 GP config macros (USE_ORCA, USE_INTERNAL_FTS, FAULT_INJECTOR, ...) - libcurl, libbz2 and libuv detection Backend: - 21 new meson.build files for cdb, fts, task, crypto, the AO/AOCS/bitmap access methods, resgroup and the other GP utils subdirectories - GP sources added to 21 existing meson.build files - 40+ GP catalog headers and 5 .dat files registered; gp_version_at_initdb.dat generated from its template next to a build-dir copy of its header, since genbki.pl derives each .dat path from its .h path - system_views_gp.sql generated, plus cdb_schema.sql and the GP catalog SQL - libpq frontend sources compiled into the backend for QD<->QE communication - libpqwalreceiver built as a normal backend object, not a separate module Divergences from upstream that meson was still assuming: - geqo is not built by Cloudberry (not in optimizer/Makefile SUBDIRS) - gen_node_support.pl is unused; NodeTag is hand-maintained in nodes.h and copyfuncs.funcs.c / copyfuncs.switch.c are committed to the repository - ecpg is not built (commented out of src/interfaces/Makefile SUBDIRS) - man pages cannot be built; stylesheet-man.xsl is absent from the tree - libpq needs an explicit -DFRONTEND, as its sources include c.h rather than postgres_fe.h so the same files can also be compiled into the backend Extensions: - contrib/interconnect, required as a preload library by initdb - the eight always-on gpcontrib extensions array_userfuncs.c included ../../catalog/pg_type_d.h, which only resolves in an in-place build; corrected to the normal include path. ORCA, gpfdist, pxf, gpMgmt, the flag-gated gpcontrib extensions and the test suites are not wired up yet. Assisted-by: Claude Code
ORCA is 945 C++ translation units spread over 34 near-identical leaf Makefiles that only emit objfiles.txt. In meson the whole tree collapses to two files, built as static libraries and linked whole into the backend: - C++ is enabled before the LLVM block, gated on the orca option, because upstream only enables it for the JIT - xerces-c is probed with a library check plus a header check honouring extra_include_dirs / extra_lib_dirs, and scoped to the ORCA targets rather than leaked into the global LIBS the way configure does - cpp_std=c++14 as an option, where configure appends -std=c++14 to CXX - the strict -Werror -Wextra -Wpedantic triple is kept on Linux and dropped on darwin, matching gporca.mk - postgres gets link_language: 'cpp', since src/backend/Makefile links it with $(CXX) and meson would otherwise pick the C driver libpq_binddomain() used ldir without declaring it. That code only compiles under ENABLE_NLS, which Cloudberry does not build, so it had never been reached; the declaration is restored from upstream anyway. Verified: 3048 targets build, initdb succeeds, gp_opt_version() reports "GPOPT version: 4.0.0, Xerces version: 3.3.0", and a query planned by ORCA returns the expected rows. Assisted-by: Claude Code
…sions - gpfdist, with apr located through apr-1-config rather than pkg-config. configure uses the config tool (--with-apr-config) for a reason: the system apr-1.pc on macOS points at an include directory that does not exist. Adds an apr_config option mirroring --with-apr-config. libevent is required, libyaml optional and gating -DGPFXDIST, and gfile.c keeps its per-file -DFRONTEND. - pxf_fdw, with libcurl as a dependency instead of a hardcoded -lcurl - the debug-extension set (gp_debug_numsegments, gp_inject_fault, gp_replica_check including its python script, reject_partition_fullscan), zstd (MODULE_big name differs from the directory name) and orafce src/Makefile.global.in's Cloudberry @variables@ are now filled in. They are defined by configure via AC_SUBST and consumed by recursive make -- for example gpcontrib/Makefile reads enable_gpcloud / enable_pxf / with_diskquota / enable_debug_extensions, and src/bin/gpfdist/Makefile reads have_yaml / EVENT_LIBS / apr_*. The meson build does not use them itself, but Makefile.global is installed as part of PGXS, so out-of-tree extensions built with USE_PGXS=1 still read them; leaving them empty would silently change their behaviour. meson setup no longer warns about missing substitutions. Assisted-by: Claude Code
Builds Cloudberry with meson in the same rocky9 build container the autoconf workflows use, installs it, and runs a smoke test. This does not replace build-cloudberry.yml: until the two paths are proven equivalent the autoconf build stays authoritative, and this exists to keep the meson files from rotting and to surface divergence early. The workflow header maps each flag of the reference configure line in coverity.yml onto its meson option, and records which ones are not reachable from meson yet (gpcloud, pax, mapreduce, diskquota, gp_stats_collector) and that --with-pythonsrc-ext was dropped deliberately. The smoke test lives in devops/build/automation/cloudberry/scripts so it can be run by hand against any install prefix. It checks the installed binaries and GP extensions, that the generated catalog data is present, that initdb succeeds, and then in single-user mode that the GP catalogs exist, that the gp_* views generated from system_views_gp.in were created, that gp_opt_version() reports a working ORCA, and that an append-only table actually stores and returns rows. Assisted-by: Claude Code
Four levels of recursive make that exist almost entirely to copy files collapse into per-directory install_data calls. The 0755 / 0644 split is inconsistent in the autoconf build (base.py is installed as a script while its siblings are data, and so on), so it is reproduced file by file rather than with a blanket install_subdir. - 32 utilities into bindir, 11 into sbindir, gppylib and its eight subpackages into libdir/python/gppylib, the gpcheckcat / gpconfig / gpssh module directories and bin/lib into bindir - the two C programs: stream into bindir/lib with the backward-compatible bindir/stream/stream symlink, and ifaddrs into libexecdir, which is where gp_bash_functions.sh looks for it - the six gpdemo scripts, which live in gpAux/gpdemo but are installed from gpMgmt/bin - gp_bash_version.sh generated from its template; configure.ac does this with an inline sed because the placeholder is ##version## rather than @Version@, and writes the result into the source tree putversion rewrites $Revision$ in ~48 files *after* installing them, which meson cannot do. subst_version.py performs the same substitution on the way into the build directory instead, so the installed file is already correct. It is applied to exactly the files the Makefiles apply putversion to; the two that are left untouched here are untouched by the autoconf build as well. Not ported: --with-pythonsrc-ext (dropped by decision - it downloads psutil, PyYAML and PyGreSQL from PyPI during install), and the behave test targets. Assisted-by: Claude Code
Everything left that the autoconf build produces and meson did not: the frontend binaries under src/bin (gpfts, gpnetbench, pg_alterckey), and the gpcontrib components that need more than a PGXS-shaped extension -- gpmapreduce, gp_stats_collector, gpcloud with gpcheckcloud, and diskquota. Three needed something other than a straight translation. gpmapreduce's generated scanner and parser hardcode their own output names with %output= and %option outfile=, which override the -o meson passes, so bison and flex run with the build directory as their working directory. gp_stats_collector generates C++ from protobuf, so protoc runs from the extension root: its -I has to be a textual prefix of every input path, and the sources import "protos/xxx.proto". diskquota is a cmake project in tree. Rather than shell out to cmake from meson -- two build systems, two configurations, one of them invisible to ninja -- its sources are listed directly, which is what the rest of this port does with everything else. libpostgres.so comes with them, and with the symbol map the Makefiles generate: the shared backend must not re-export libpq's symbols, or an extension linked against libpq gets the backend's copies. gen_symbol_map.py derives the list from src/interfaces/libpq/exports.txt, as the Makefile does, and emits a version script on Linux and an -unexported_symbols_list on darwin. Assisted-by: Claude Code
PAX ships a standalone CMake project driven from its Makefile. The meson source list mirrors the groups in src/cpp/cmake/pax.cmake for the default configuration (USE_PAX_CATALOG=ON, USE_MANIFEST_API=OFF, VEC_BUILD off), which is 93 files. Note the mixed extensions: one source is .cpp where the rest are .cc, and missing it produced a module that built cleanly but failed to load with an undefined pax::tools::OrcDumpReader::Dump(). Two pieces of code generation: - protoc, declared in a nested storage/proto/meson.build so the generated headers land in a storage/proto directory inside the build tree, which is how the sources include them. CMake instead writes them back into the source tree. - the extension SQL, which cmake/pax.cmake produces by compiling tools/gen_sql.c and redirecting its stdout into pax-cdbinit--1.0.sql in the source tree; meson captures the output instead. yyjson is not needed: it is only linked when USE_MANIFEST_API is on and USE_PAX_CATALOG is off, which is the opposite of the defaults. tabulate is needed unconditionally (used by a UDF), so PAX requires that submodule. pg_waldump also needs PAX when it is enabled: rmgrdesc.c includes "paxc_desc.h" under USE_PAX_STORAGE, and the Makefile symlinks paxc_desc.[ch] in from contrib/pax_storage. Verified end to end: the pax access method is registered, and a table created with USING pax stores and returns rows. CI now enables gpcloud, mapreduce, diskquota and pax as well, and the smoke test checks the additional binaries and the generated PAX SQL. gp_stats_collector remains off there for the reason recorded in the workflow. Assisted-by: Claude Code
Two places where the meson build did more than configure.ac asks for. NLS: upstream's meson exposes an nls option; Cloudberry's configure.ac has had --enable-nls deliberately removed since 2015, and the translation catalogues have gone stale in the years since. The option now defaults to disabled and errors if turned on, rather than offering a build nobody maintains. Warnings: configure.ac keeps -Wdeclaration-after-statement commented out, because "GPDB code is full of declarations after statement", and adds -Wno-unused-but-set-variable. meson had neither, which is the whole difference between 2263 warnings and 10 on the same tree. The meson floor moves to 0.61 here, for install_symlink. That turns out to be the wrong trade and is settled later in the series; see "meson: fixes from installing and running the result". Assisted-by: Claude Code
Mirrors build-cloudberry.yml's shape -- check-skip, build, test, report -- so
that test jobs can be added alongside cluster-test later.
The build matrix is the part that earns its keep today. Most of the bugs found
while porting were in option combinations rather than in any single
configuration: C++ being enabled only for ORCA, PAX requiring the shared
backend, extensions gated on flags never exercised together. Three
configurations cover that surface:
full every component this branch supports
minimal everything optional off, which catches conditional
guards that only work when enabled
no-shared-backend -Dshared_postgres_backend=false, a path nothing else
covers; PAX comes off with it since PAX links against
libpostgres.so
cluster-test goes past what the smoke test can reach. meson-smoke-test.sh only
runs single-user mode, so it never starts a segment. The new script brings up a
real multi-segment cluster with gpinitsystem and then checks that data actually
distributes (1000 rows across segments, not just 1000 rows), that a plan
contains a Motion node, that ORCA plans a distributed join, and that ao_row and
pax tables work distributed.
That also makes it the first test of two things the build alone cannot prove:
the gpMgmt install is complete enough to run gpinitsystem, and dropping
--with-pythonsrc-ext is workable -- psutil, PyGreSQL and PyYAML are installed
from python-dependencies.txt instead of being vendored.
Two checks are new in the build job: meson_version feature warnings are fatal,
since they are cheap to fix and easy to let accumulate, and the compiler
warning count is reported to the step summary, because a jump there usually
means the meson flags have drifted from configure.ac.
Regression suites are still not run. src/test/regress and friends are
make-driven and include src/Makefile.global from the source tree, which only a
configure run produces, so they cannot execute against a meson-only tree.
meson also now installs $prefix/cloudberry-env.sh, which gpMgmt/Makefile
generates and which demo_cluster.sh sources; without it the cluster test could
not run, and an install was not usable the way an autoconf one is.
Assisted-by: Claude Code
None of these were visible from a build. Every one came out of installing the tree and starting a real cluster on it, and each would have shipped a build that compiles and links and then does not work. libdir. meson derives it from the platform, which is lib64 on RHEL. cloudberry-env.sh hardcodes PYTHONPATH=$GPHOME/lib/python and LD_LIBRARY_PATH=$GPHOME/lib, so gppylib and libpostgres.so landed where nothing looks for them. Every file present, install unusable. PG_VERSION_STR. gpMgmt parses select version() and insists on "(Apache Cloudberry <version> build <build>)". With upstream's string, gpstop dies on a healthy cluster with "too many tokens in version". gpMgmt files. Ten the autoconf build installs and this port did not, nine of them Python modules, so nothing failed to build: gppylib/commands/base.py, all of gppylib/util, gppylib/programs/gppkg.py, gppylib/system/ComputeCatalogUpdate.py, gppylib/gpMgmttest, lib's crashreport.gdb, and the foreign-key JSON gpcheckcat reads, which is generated from the same headers as postgres.bki. gpMgmt Python modules. --with-pythonsrc-ext downloads psutil, PyGreSQL and PyYAML mid-build, which has no honest meson equivalent, so it was dropped. What should not have gone with it is the install: the Makefile's half of that option only copies what is already in gpMgmt/bin/ext, and without the same conditional copy there is no supported way to get the modules into an install at all. meson floor. Raising it to 0.61 for install_symlink turned roughly thirty-five deprecations in upstream's own meson files into warnings on every configure -- meson only reports a deprecated API once the project claims a version that has it deprecated. Three symlinks are not worth that, nor worth diverging from upstream on the one line every future merge touches. The floor goes back to upstream's value and the symlinks are made by an install script. Per-directory warning flags. Five directories relax a warning in their own Makefile. Two need more than an extra c_args: meson compiles the whole backend as one target, so src/backend/task moves into a static library that carries the flag, and plpgsql, where the Makefile scopes -Wno-array-bounds to pl_exec.o alone, gets a library for that one file. Assisted-by: Claude Code
The workflow that came with the port built one configuration and ran a fixed smoke test. Everything here is a response to something the first real runs turned up. An install-parity job. The port lists installed files by hand, so a file added to a Makefile and not to the meson.build beside it is silently dropped. Ten such files went unnoticed until gpstop died on a cluster that had come up cleanly; no build catches that, and a test catches only the fraction it exercises. The check is static, finishes in seconds, and reports before the build it would not have been caught by. A configuration-aware smoke test. The old one asserted the full configuration's binaries and GUCs, so every other matrix entry failed for doing what it was told -- missing bin/gpmapreduce, and a FATAL from optimizer=on in a build without ORCA. The expected set now comes from meson-info, so one test serves the whole matrix, which is what makes having a matrix worth anything. gpcheckcat in the cluster test. It is the only thing that reads the foreign-key JSON, so an install missing that file is silent everywhere else. The PyPI downloads move to the front of the job and are cached. They were happening after a forty-minute build, which put the network on the critical path at the worst moment; the Makefile's download target also pip-installs wheel and Cython when absent, and they are absent from the build image, so those go in with meson and ninja instead. Nothing between meson setup and the end of the build reaches the network now, and an offline build only has to stage the three tarballs. Building src/common's frontend library on its own, before the full build. That directory is compiled both ways, and a missing generated-header dependency there fails or not depending on scheduling -- one red matrix entry and one green one on the same commit. Also: the matrix drops to two opposite configurations rather than three, and the workflow triggers on main only, pg16-ci having been a branch in a personal fork. Assisted-by: Claude Code
curl without -f treats an HTTP error as success: it writes the error body to
the output file and exits 0. The guard around each download then only asks
whether the file exists, so a single 404 or bad gateway leaves a broken
tarball that every later run skips over as already downloaded. The build
fails much later, in tar, with nothing pointing at the cause.
$ curl -sSL .../psutil-99.99.99.tar.gz -o t # 404
$ echo $?; wc -c < t
0
0
Add -f so the request fails, --retry 3 so a transient one does not, and make
each guard -s rather than -f so a zero-byte leftover is retried instead of
trusted. curl leaves the partial file behind, and --remove-on-error needs a
curl newer than rocky8 ships, so the guard is the portable place to fix this.
Verified in the build container: a 404 now stops the target with curl's exit
status and leaves no file, and with no route to PyPI a zeroed tarball fails
the target rather than silently persisting.
Assisted-by: Claude Code
src/common is compiled twice, once with -DFRONTEND. percentrepl.c and
kmgr_utils.c include utils/builtins.h unguarded, and that pulls in
utils/fmgrprotos.h, which is generated for the backend and is not a declared
dependency of anything frontend. Nothing orders the two, so ninja is free to
compile the frontend copy first:
../src/include/utils/builtins.h:21:10: fatal error:
utils/fmgrprotos.h: No such file or directory
Which is exactly what happened, and only sometimes: on one commit the full
matrix entry went green and the minimal one went red, on the same runner
image, differing only in how much else there was to schedule. make never
showed it because every Makefile there depends on submake-generated-headers,
so the ordering is forced.
Everything the offending headers declare -- GpIdentity, terminal_fd, pg_ltoa
-- is used only from #ifndef FRONTEND blocks, so the includes belong inside
one. kmgr_utils.c also included postgres.h unconditionally, which a FRONTEND
translation unit must not do at all; it uses nothing from either header.
Verified as a deterministic before/after in a cold build directory, building
just the two frontend objects: both fail on the old sources, both compile on
the new ones.
Assisted-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions