Data Commons

Data Commons is an open source semantic graph database for modeling, querying, and analyzing interconnected data.

Data Commons powers datacommons.org, Google's open knowledge graph that connects public data across domains like demographics, economics, health, and education.

Getting Started

• Deploying or operating a DCP instance? If you are an instance owner, data steward, or administrator looking to install and run your own Data Commons instance, follow the Data Commons Platform User Guide.
• Developing or contributing to the codebase? If you are an engineer contributing to DCP, start with the hands-on Developer Onboarding Codelab for a guided tutorial deploying and testing an instance, consult the Developer Guide for workbench recipes and monorepo topology, and review CONTRIBUTING.md for pull request guidelines.

Prerequisites

Before you begin, ensure you have the following installed:

• Python 3.11 or higher
• uv (Python project manager)
• A Google Cloud Platform (GCP) project with Cloud Spanner enabled
• A Cloud Spanner instance and database (using Google Standard SQL) for storing the knowledge graph

Deploying Data Commons Platform In GCP

Use the CLI to scaffold a Terraform deployment directory:

git clone https://github.com/datacommonsorg/datacommons
cd datacommons/
uv run datacommons admin init

The command will prompt for:

• GCP project id
• Instance name
• Data Commons API key

It then creates a new folder with main.tf, terraform.tfvars, and a deployment README.md.

From the generated folder:

terraform init
terraform plan
terraform apply

Once infrastructure is deployed, initialize the database and trigger data ingestion using the CLI.

You can run these commands directly from your Terraform deployment directory (where state outputs are detected automatically), or from anywhere by passing the --project-id and --instance-name flags:

Option A: Run from your Terraform directory

# Run from inside your deployment folder (e.g. cd prod/)
uv run datacommons admin init-db

# Trigger data ingestion
uv run datacommons admin ingest start --imports <import_name>

Option B: Run from anywhere (Remote GCS State)

# Run from any directory or CI/CD runner without local Terraform files
uv run datacommons admin --project-id my-gcp-project --instance-name prod init-db

# Trigger data ingestion
uv run datacommons admin --project-id my-gcp-project --instance-name prod ingest start --imports <import_name>

Documentation & Guides

User Documentation (DCP Instance Owners and Administrators)

If you are an instance owner, data steward, or administrator looking to install, configure, and manage a Data Commons instance:

• Data Commons Platform User Guide: Master operational manual covering platform deployment, schema modeling, data ingestion, and instance administration.
• CLI Reference & Cheatsheet: Full command reference and operational CLI cheatsheet for instance management, database migrations, and data ingestions.

Developer Documentation (Platform Contributors)

If you are an engineer contributing to the codebase, creating new services, or tuning platform internals:

Guide	Target Audience	Purpose
Developer Onboarding Codelab	New Developers	Hands-on zero-to-hero onboarding tutorial deploying and testing an instance on GCP.
Developer Guide	Monorepo Developers	Monorepo package layout, uv workspace linking, local development recipes, and testing strategy.
Platform Architecture	All Contributors	4-repository architecture, container roles, and complete serving and ingestion data flows.
Terraform Stack Architecture	Infrastructure Developers	Module hierarchy, variable propagation pipelines, and cross-module IAM wiring.
Admin CLI Architecture	CLI Contributors	Admin CLI internals, scaffolding regex contract, and state-driven operation choreography.
Contributing Guide	Pull Request Authors	Code quality standards, formatting, linting, and unit test requirements.
Platform Release Guide	Release Managers	Official 3-stage release candidate workflow and PyPI lockstep publishing procedures.
Documentation Standards	All Readers	Platform documentation architecture and writing standards.
Infrastructure Cheatsheet	Operators & Contributors	Operational cheatsheet for infra/dcp/: commands, inputs, and outputs.