gpfdist supports reading LZO-compressed files. - #1887
Conversation
|
Before moving forward with the current PR, would you have the bandwidth to take another look at apache/cloudberry#1226? The two PRs follow a similar operational model, and the previous PR has already been open for quite some time. It might be better to bring that one to completion first and then proceed with this PR. This would present a more consistent development process and help build confidence among the contributors and reviewers. |
|
I validated this feature in the development environment previously and confirmed it works properly. Back then, Cloudberry’s CI dependencies were undergoing adjustments. Later on, I was occupied with urgent work tasks. Currently, there are assessment requirements for community contributions, hence I intend to contribute the newly developed features to Cloudberry. |
|
The CI of another candidate item I reviewed also shows a failure. Please help review it again. Follow-up work will include gradual optimization to support exporting table data as LZO files via gpfdist. |
| } | ||
| #endif | ||
|
|
||
| #ifdef USE_LZO |
| # for name in `seq 1 1000`; \ | ||
| # do \ | ||
| # head -100 data/gpfdist2/lineitem.tbl >> data/gpfdist2/lineitem.tbl.long.tmp; \ | ||
| # done |
There was a problem hiding this comment.
why All of these lines are commented out ？
There was a problem hiding this comment.
The comments describe the steps I followed to construct the dataset.
The reason lineitem.tbl.long.lzo is checked in as a pre-generated binary file is that it requires the lzop command to generate. I'm not sure whether the Apache project's CI/CD pipeline has lzop installed in its build environment. If it doesn't, dynamically generating this file during test execution would cause the build to fail. That's why I opted to provide it as a pre-generated artifact for now.
There was a problem hiding this comment.
I noticed that lineitem.tbl.long.lzo appears to be a binary file. Per Apache project policies, binary files are generally not allowed in the repository. Would it be possible to generate this test data dynamically during the test execution instead of checking it in? This would help keep the repository source-only and reduce its size.
Please let me know if there's a practical way to achieve this, or if there are any constraints I'm not aware of. Happy to discuss alternatives.
There was a problem hiding this comment.
Thanks for raising this concern!
The reason lineitem.tbl.long.lzo is checked in as a pre-generated binary file is that it requires the lzop command to generate. I'm not sure whether the Apache project's CI/CD pipeline has lzop installed in its build environment. If it doesn't, dynamically generating this file during test execution would cause the build to fail. That's why I opted to provide it as a pre-generated artifact for now.
4 participants
Fixes #ISSUE_Number
What does this PR do?
This PR adds LZO compressed file format read/write support to the
gpfdistutility in Apache Cloudberry.Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
Scenario 1: High-Volume Data Loading
LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio.
Why this matters: For time-critical ETL jobs, LZO can reduce load times by 2-5x compared to heavier compression algorithms, while still saving ~50% storage space compared to uncompressed files.
Scenario 2: Hadoop Ecosystem Integration
Many Hadoop ecosystem tools natively support LZO compression:
Scenario 3: Storage Efficiency with Parallel Performance
LZO is splittable, meaning multiple gpfdist instances (or segments) can read different portions of the same file simultaneously. This enables near-linear scalability with additional segments.
User-facing changes:
example
The --with-lzo configure option must be specified at compile time.
./configure --with-lzo --prefix=/home/cdblzo
Dependencies:
#1886 (reply in thread)
Checklist
References
Technical Documentation
Cloudberry Documentation
Related Projects
Additional Context
CI Skip Instructions