Skip to content

AGENTS.md

Latest commit

 

History

History
111 lines (83 loc) · 7.3 KB

AGENTS.md

File metadata and controls

111 lines (83 loc) · 7.3 KB
Raw

PyIceberg — Agent Instructions

Project conventions, architecture, and coding patterns synthesized from PyIceberg developers.

Security Model

When assessing potential vulnerabilities or calibrating automated security findings, use SECURITY-THREAT-MODEL.md as the authoritative detailed description of this repository's security boundaries, trust assumptions, and non-boundaries.

Architecture

PyIceberg is a pure-Python library — it has no separate engine modules. The code lives under pyiceberg/, organized by concern rather than by engine:

  • schema.py, types.py, transforms.py, partitioning.py, conversions.py: The table spec core — schemas, types, partition specs, and value conversions. Must stay engine- and catalog-agnostic.
  • table/: Table abstraction, metadata (metadata.py), snapshots, refs, sort orders, and the transaction/update machinery (table/update/). The commit path lives here.
  • catalog/: Catalog implementations — rest/, hive, glue, dynamodb, sql, bigquery_metastore, memory, noop. New catalogs subclass the base Catalog in catalog/__init__.py. Catalog-specific assumptions must not leak into table/ or the spec core.
  • io/: The FileIO abstraction over storage. pyarrow.py and fsspec.py are the two backends. Never hard-code a storage SDK where FileIO exists.
  • expressions/: The expression DSL and its visitors (predicate binding, projection, evaluation).
  • avro/, manifest.py: Manifest and Avro read/write — performance-sensitive, partly accelerated by Cython.
  • cli/: The pyiceberg command-line interface (Click + Rich).
  • utils/: Shared helpers (deprecated.py, concurrent.py, config.py, bin_packing.py, singleton.py, etc.). Check here before writing new utility code.

Coding Conventions

Style & Typing

  • Formatting and linting are enforced by ruff via prek (pre-commit): line length 130, double-quoted strings, isort with pyiceberg/tests as first-party. Run make lint — ruff autofixes most issues.
  • Full type annotations are required (mypy runs in strict mode: disallow_untyped_defs, no_implicit_optional, warn_unused_ignores). Avoid Any types.
  • Docstrings follow the project's pydocstyle config; one-line summaries on public functions. No personal pronouns in comments.
  • Use domain-specific exceptions from pyiceberg/exceptions.py for Iceberg-specific error conditions. For invalid arguments/values, ValueError is acceptable. Don't raise bare Exception.
  • Comments should be succinct and follow the same style of comments found in the rest of the codebase.

Dependencies

  • Large/integration libraries must be optional extras in pyproject.toml, not core dependencies.

Documentation

  • Any external URL mentioned in the docs should have <!-- markdown-link-check-disable-next-line --> on the previous line.

Testing

  • Bias towards adding tests to existing files, rather than creating new files.
  • Use existing test fixtures when possible.
  • We have a strong bias towards integration testing over mocks. Mocks should be avoided whenever possible and should only be used if similar, existing tests are using mocks.

Required CI and Merge Queue

  • Keep the required_status_checks.contexts list in .asf.yaml synchronized whenever a required job/check is added, renamed, or removed. Each entry is a job/check context name, not a workflow filename.
  • Verify that every required context reports for both pull_request and merge_group. Its producer workflow must run on both events.
  • When a workflow uses an aggregate required job, such as python-ci-required, keep if: always() and its needs list in sync with every job whose result should block merging.
  • Do not use pull_request path filters in workflows that produce required contexts. A skipped workflow does not report its required context, which blocks pull requests and causes Merge Queue entries to time out.

Commands

  • Install / set up dev env: make install (installs uv, syncs all extras, builds Cython, installs pre-commit hooks)
  • Run unit tests: make test
  • Run a subset: make test PYTEST_ARGS="-v -k <pattern>"
  • Integration tests (Spark/Docker): make test-integration (rebuild with make test-integration-rebuild)
  • Cloud storage suites: make test-s3 / make test-adls / make test-gcs
  • Lint & format: make lint
  • Docs preview / build: make docs-serve / make docs-build
  • Clean build artifacts: make clean
  • Use a specific Python: prefix with PYTHON=3.12, e.g. PYTHON=3.12 make install

PR & Commit Conventions

  • Ensure that there are no existing PRs for this feature before beginning development.
  • One concern per PR. Keep unrelated formatting/import churn out of feature PRs.
  • Keep the first version of a PR minimal; defer optimizations and edge cases to follow-ups.
  • Commit messages explain the what and why, not line-by-line implementation. Be as succinct as possible.
  • The Apache License header is required on every new source file (enforced by ./dev/check-license).
  • Run make lint and make test before pushing; CI runs both plus the lockfile check.

GitHub Write Approval

  • Never modify GitHub state without explicit user approval for that specific action in the current session; treat GitHub as read-only by default.
  • Ask first before any GitHub write: show the exact target and action, including the full proposed content when applicable, then wait for explicit approval. This includes creating, editing, or deleting issues, pull requests, comments, reviews, labels, and branches, as well as merges, releases, and workflow runs.
  • Never treat a request to investigate, implement, fix, draft, or prepare as approval to write to GitHub, or infer, reuse, or bypass approval.

Boundaries

  • Never add a hard (non-optional) dependency without discussion — keep heavy/integration libraries as optional extras.
  • Never edit uv.lock by hand — regenerate it with uv lock.
  • Never reach for a storage SDK (boto3, s3fs, gcsfs, adlfs) outside its FileIO backend module.
  • Never break the public API or remove a public method without a @deprecated cycle.
  • Never leak catalog- or engine-specific assumptions into the spec core (schema.py, types.py, table/metadata.py).
  • Never commit secrets, credentials, or cloud tokens (integration tests can delete catalog data — never point them at production).
  • Ask first before adding any third-party dependency or promoting internal (_-prefixed) APIs to public.