diff --git a/docs.json b/docs.json index 2fe23e34..7ed88e49 100644 --- a/docs.json +++ b/docs.json @@ -158,6 +158,7 @@ "semantic-layer/dimensions", "semantic-layer/metrics", "semantic-layer/joins", + "semantic-layer/nested-and-repeated-columns", "semantic-layer/explores", "semantic-layer/parameters", "semantic-layer/sql-variables", diff --git a/images/semantic-layer/nested-and-repeated-columns/non-repeated-record-yaml-mapping.png b/images/semantic-layer/nested-and-repeated-columns/non-repeated-record-yaml-mapping.png new file mode 100644 index 00000000..2b177888 Binary files /dev/null and b/images/semantic-layer/nested-and-repeated-columns/non-repeated-record-yaml-mapping.png differ diff --git a/images/semantic-layer/nested-and-repeated-columns/repeated-record-yaml-mapping.png b/images/semantic-layer/nested-and-repeated-columns/repeated-record-yaml-mapping.png new file mode 100644 index 00000000..2605c941 Binary files /dev/null and b/images/semantic-layer/nested-and-repeated-columns/repeated-record-yaml-mapping.png differ diff --git a/semantic-layer/dimensions.mdx b/semantic-layer/dimensions.mdx index 530de677..860e27a0 100644 --- a/semantic-layer/dimensions.mdx +++ b/semantic-layer/dimensions.mdx @@ -2173,6 +2173,8 @@ dimensions: ### Parsing JSON columns +BigQuery `STRUCT` and `ARRAY` fields use the configuration described in [Nested and repeated columns](/semantic-layer/nested-and-repeated-columns). + Usually you'll want to add `hidden:true` for the main JSON dimension since raw JSON is not useful in charts. diff --git a/semantic-layer/joins.mdx b/semantic-layer/joins.mdx index 5630c9c0..d20c9529 100644 --- a/semantic-layer/joins.mdx +++ b/semantic-layer/joins.mdx @@ -1369,6 +1369,9 @@ LIMIT 500 There are a few situations where Lightdash doesn't currently handle inflated metrics: + + For the `Repeated columns … are unnested together` query warning, see [Nested and repeated columns](/semantic-layer/nested-and-repeated-columns#query-warnings). + When a metric in one table references a dimension from a joined table, Lightdash's fanout handling may not correctly deduplicate the results. diff --git a/semantic-layer/nested-and-repeated-columns.mdx b/semantic-layer/nested-and-repeated-columns.mdx new file mode 100644 index 00000000..21665179 --- /dev/null +++ b/semantic-layer/nested-and-repeated-columns.mdx @@ -0,0 +1,219 @@ +--- +title: "Nested and repeated columns" +description: "Define BigQuery STRUCT and ARRAY fields in the Lightdash semantic layer" +doc-type: reference +tag: "Beta" +--- + + + Beta Available for BigQuery projects. On Lightdash Cloud, ask Lightdash support to enable it for your organization. For self-hosted deployments, see [Deploy and generate](#deploy-and-generate). [What Beta means](/support/feature-maturity-levels). + + +BigQuery columns can contain non-repeated `RECORD` values (`STRUCT`), repeated records (`ARRAY`), or repeated scalar values (`ARRAY`). Define record leaves with dotted column names in dbt YAML. Lightdash keeps leaves from non-repeated records on the model and exposes repeated values as virtual tables joined with `UNNEST`. + +Use [How column shapes map to fields](#how-column-shapes-map-to-fields) for a visual overview and [Define nested columns in YAML](#define-nested-columns-in-yaml) for the complete configuration. Refer to [Virtual tables](#virtual-tables), [Query behavior and grain](#query-behavior-and-grain), [Query warnings](#query-warnings), [Pre-aggregates](#pre-aggregates), and [Known limitations](#known-limitations) as needed. + +## How column shapes map to fields + +### Non-repeated RECORD (`STRUCT`) + +A leaf below a non-repeated record is an ordinary dimension on the model with its dotted name. In this `orders` model, `- name: customer.address.city` exposes only the `city` leaf. Add separate entries for `customer.customer_id`, `customer.name`, or `customer.address.street` if you also want them as fields. + + + Orders table with a customer record containing an address record, mapped to the dotted YAML column customer.address.city + + +The `customer.address.city` dimension has the SQL path `orders.customer.address.city` and the field ID `orders_customer__address__city`. Lightdash gets its type from the warehouse catalog, so `meta.dimension.type` is optional. An explicitly declared type takes precedence over the catalog type. + +### REPEATED RECORD (`ARRAY`) + +A repeated column with at least one listed leaf becomes a virtual table. In this `orders` model, `line_items.sku` becomes the `sku` dimension on `orders__line_items`. The `price` leaf is not exposed because it is not listed in YAML. + + + Orders table with three line item records and their array positions, mapped to line_items and line_items.sku entries in YAML + + +The `- name: line_items` container entry is optional. Include it when you want to set the virtual table's `description` or `meta.dimension.label`. Every virtual table also gets an `offset` dimension for the array position, so this example exposes `sku` and `offset` on `orders__line_items`. + +### Repeated scalar values (`ARRAY`) + +An array of scalars has no leaves to list. Add the array column itself to dbt YAML. For example, if the `orders` model has a `tags` column with the type `ARRAY`, add: + +```yaml +columns: + - name: tags +``` + +Lightdash creates the `orders__tags` virtual table with a string dimension named `value` and the number dimension `offset`. Reference the value as `orders__tags.value` in filters and pre-aggregates or as `${orders__tags.value}` in custom SQL. + +## Define nested columns in YAML + +Add one YAML entry for each record leaf or scalar array you want to expose. Write a leaf's full path from the top-level column, separating each level with a dot—for example, `customer.address.city`. Lightdash gets the structure from your BigQuery table, so you do not need to specify whether each level is a `STRUCT` or `ARRAY` in YAML. + +Only record leaves and scalar arrays listed in YAML become fields. The example below extends the `orders` model from the diagrams by exposing `line_items.price` and the `tags` scalar array. The warehouse table has one row per order, a `customer` record, a repeated `line_items` record, and a `tags` scalar array: + + +```yaml dbt v1.10+ and Fusion +models: + - name: orders + config: + meta: + primary_key: order_id + columns: + - name: order_id + - name: status + # Non-repeated RECORD leaf: an ordinary dotted dimension on orders + - name: customer.address.city + # Optional metadata for the orders__line_items virtual table + - name: line_items + description: One row per item in the order + # Leaves below the REPEATED RECORD belong to orders__line_items + - name: line_items.sku + - name: line_items.price + config: + meta: + metrics: + total_revenue: + type: sum + # A repeated scalar becomes the value dimension on orders__tags + - name: tags +``` + +```yaml dbt v1.9 and earlier +models: + - name: orders + meta: + primary_key: order_id + columns: + - name: order_id + - name: status + # Non-repeated RECORD leaf: an ordinary dotted dimension on orders + - name: customer.address.city + # Optional metadata for the orders__line_items virtual table + - name: line_items + description: One row per item in the order + # Leaves below the REPEATED RECORD belong to orders__line_items + - name: line_items.sku + - name: line_items.price + meta: + metrics: + total_revenue: + type: sum + # A repeated scalar becomes the value dimension on orders__tags + - name: tags +``` + + +See [Deploy and generate](#deploy-and-generate) for how `lightdash generate` handles nested and repeated columns. + +## Virtual tables + +A repeated record becomes a virtual table when at least one of its leaves is listed in YAML. A scalar array becomes a virtual table when the array column itself is listed. Lightdash joins each virtual table with `UNNEST`. + +A leaf belongs to the virtual table of its deepest repeated ancestor. For example, `line_items.sku` becomes the `sku` dimension on `orders__line_items`. There is no nesting-depth limit: if each line item also contains a repeated `discounts` record, `line_items.discounts.code` belongs to `orders__line_items__discounts`. + +### Names and field references + +- **Table name:** `__`. Nested virtual-table names continue the chain, as in `orders__line_items__discounts`. The separator is two underscores. +- **Sidebar label:** `: `. A nested label continues the chain, as in `Orders: Line items: Discounts`. +- **Field ID:** `_`, as in `orders__line_items_sku`. +- **YAML reference:** use the virtual-table name as the table prefix. For example, use `${orders__line_items.price}` in metric SQL and `orders__line_items.sku` in filters and pre-aggregates. + +Every virtual table also has a number dimension named `offset`, which is the element's zero-based position in its array. + +If a generated virtual-table name clashes with any other table in the Explore, that Explore fails to compile and the error names the conflicting table. + +### Join behavior + +Lightdash joins a virtual table with a left join and `ON TRUE`. The relationship is one-to-many, so a parent with an empty or `NULL` array keeps its row with `NULL` leaf values. A virtual table has no primary key and cannot declare one; its grain is the parent row multiplied by the array element. + +When a model is joined into another Explore under an alias, or joined more than once, its virtual-table names and labels follow that alias. For example, aliases named `online_orders` and `store_orders` produce `online_orders__line_items` and `store_orders__line_items`, labeled `Online orders: Line items` and `Store orders: Line items`. + +## Generated SQL + +For a query that selects `customer.address.city`, `line_items.sku`, and total revenue, Lightdash generates one `UNNEST` join: + +```sql +SELECT + `orders`.customer.address.city AS `orders_customer__address__city`, + `orders__line_items`.sku AS `orders__line_items_sku`, + SUM(`orders__line_items`.price) AS `orders__line_items_total_revenue` +FROM `my-project`.`analytics`.`orders` AS `orders` +LEFT OUTER JOIN UNNEST(`orders`.line_items) AS `orders__line_items` WITH OFFSET AS `orders__line_items__offset` + ON TRUE +GROUP BY 1, 2 +``` + +## Query behavior and grain + +The query grain follows the fields included in the query: + +- A virtual table is joined only when one of its fields is selected, filtered, or sorted, just like any other joined table. If the query uses no repeated leaf, Lightdash does not add `UNNEST`, and the query stays at the model's grain. +- When the query uses a field from a virtual table, the join introduces one row per array element before Lightdash groups the results by the selected dimensions. A parent with an empty or `NULL` array contributes one row with `NULL` virtual-table fields. +- Metrics defined on the model pass through Lightdash's existing primary-key deduplication. They remain correct at element grain when the model declares a `primary_key`. +- Metrics defined on a repeated leaf or scalar array are calculated at element grain. +- A filter on a repeated leaf keeps only matching element rows. Parents with no matching element are excluded, and Lightdash does not restore the other elements from a parent after one element matches. +- Grand totals in the results table drop dimensions. If a query's only repeated fields are dimensions, its grand total is calculated at model grain. + +## Query warnings + +The **Query warnings** icon next to **Run query** shows warnings about combinations that can inflate metrics. + +- When two repeated columns that are not nested inside one another are selected together, Lightdash shows this warning once per query and only for the deepest virtual tables. For example, if `orders` also has a repeated `shipments` column, selecting fields from both `line_items` and `shipments` shows: + + > Repeated columns "orders__line_items" and "orders__shipments" are unnested together, so each row pairs their elements and metrics can be inflated. + + A nested chain such as `line_items` and `line_items.discounts` does not trigger this warning. + +- When a metric on a virtual table is queried with a deeper or sibling unnest, Lightdash shows: + + > Metric "Total revenue" could be inflated by another unnested repeated column. + +- The existing **could be inflated due to join relationships** warning still applies to model metrics that Lightdash cannot deduplicate, including metrics on a model without a `primary_key`. + +## Pre-aggregates + +Reference virtual-table fields with the virtual-table name as their table prefix: + + +```yaml dbt v1.10+ and Fusion +models: + - name: orders + config: + meta: + pre_aggregates: + - name: revenue_by_sku + dimensions: + - orders__line_items.sku + metrics: + - orders__line_items.total_revenue +``` + +```yaml dbt v1.9 and earlier +models: + - name: orders + meta: + pre_aggregates: + - name: revenue_by_sku + dimensions: + - orders__line_items.sku + metrics: + - orders__line_items.total_revenue +``` + + +## Deploy and generate + +Explores compile in the CLI during [`lightdash deploy`](/workflow/cli/deploy), and the CLI reads the nested-column setting from the Lightdash server. Before deploying, [update the Lightdash CLI](/workflow/cli/install#updating-the-lightdash-cli). If the CLI cannot reach the server or read the setting, it compiles with nested-column support disabled and silently drops repeated leaves from the Explore. + +On self-hosted deployments, add `unnest-repeated-columns` to `LIGHTDASH_ENABLE_FEATURE_FLAGS`; see [Feature flags](/self-host/customize-deployment/environment-variables#feature-flags). + +[`lightdash generate`](/workflow/cli/generate) does not generate an entry for a container column or any leaf below a repeated column. Add those entries to the generated YAML by hand. + +## Known limitations + +- Only BigQuery is supported. +- To access an array element by index, such as `line_items[0].sku`, define a dimension with custom `sql`. +- Record and array containers are not selectable as fields. A scalar array exposes its elements through the virtual table's `value` dimension. +- Selecting two repeated columns that are not nested inside one another multiplies their rows. See [Query warnings](#query-warnings). +- SQL Runner and virtual views type nested columns but do not expand them. diff --git a/semantic-layer/pre-aggregates.mdx b/semantic-layer/pre-aggregates.mdx index 9eb15250..f6dbb872 100644 --- a/semantic-layer/pre-aggregates.mdx +++ b/semantic-layer/pre-aggregates.mdx @@ -187,7 +187,7 @@ See [Filtered pre-aggregates](/semantic-layer/pre-aggregates#filtered-pre-aggreg ### Dimensions from joined tables -Pre-aggregates support dimensions from joined tables. Reference them by their full name (for example, `customers.first_name`) in the `dimensions` list. +Pre-aggregates support dimensions from joined tables. Reference them by their full name (for example, `customers.first_name`) in the `dimensions` list. For virtual tables created from repeated BigQuery columns, see [Pre-aggregates with nested and repeated columns](/semantic-layer/nested-and-repeated-columns#pre-aggregates). ## Filtered pre-aggregates