Get list of validation results for each rule/check?

[reinholdsson]

Hi,

Would it be possible to add support for returning (optionally) a dictionary or similar structure that maps each rule/check to its ValidationResult instance?

We have a use case for integrating dataframely's schema validation with Dagster asset checks. To enable this, we need a way to programmatically access each rule's validation result so we can dynamically map them to corresponding asset checks.

Currently, the validation output may look like:

ValidationError: 4 rules failed validation:
 - 'reasonable_bathroom_to_bedrooom_ratio' failed for 1 rows
 - 'minimum_zip_code_count' failed for 2 rows
 * Column 'zip_code' failed validation for 1 rules:
   - 'min_length' failed for 1 rows
 * Column 'num_bedrooms' failed validation for 1 rules:
   - 'nullability' failed for 2 rows

Hence, it would be great to have an API to obtain validation results in a structured mapping, e.g.:

{
    "reasonable_bathroom_to_bedrooom_ratio": ValidationResult(...),
    "minimum_zip_code_count": ValidationResult(...),
    "zip_code_min_length": ValidationResult(...),
    ...,
}

This would allow us to:

• Dynamically map each check to a Dagster asset check.
• Retrieve per-rule validation details programmatically, not just from the textual summary.

Thanks! :)

[AndreasAlbertQC]

Hey Thomas Falk (@reinholdsson) ! Good suggestion, I think we can definitely do a better job of exposing this info. As of today, you can:

1. run .filter instead of .validate whenever there is any chance of failure.
2. This returns a FilterResult, which contains a FailureInfo

FailureInfo contains the information you are looking for, albeit only in a private member _lf, which is a dataframe where the rows are the rows you filtered and the columns represent the checks (true = pass, false = fail, None=not evaluated). In our own internal code that uses dataframely, we have so far often converged on using this dataframe to infer the info we need. Clearly, that already hints we should either just make it public as is or enhance it a little and then make it public.
Oliver Borchert (@borchero) Daniel Elsner (@delsner) wdyt?

[reinholdsson]

Excellent, thanks!

I suppose MySchema.filter(df, cast=True).failure.counts() combined with MySchema._validation_rules(with_cast=True) gives me the info I need for my use case (although it would still be nice to have the validation results available as a public method).

[borchero]

We should definitely improve the usability of FailureInfo; there's been a bunch of questions/requests around this (see also #122, #124, #190). I think all of these issues should give us enough of an idea what common use cases look like and we should extend the public API accordingly.

[AndreasAlbertQC]

Hey Thomas Falk (@reinholdsson) : In #275, we have now exposed the per-row check information in FailureInfo.details(). Does that cover your use case?

[reinholdsson]

Andreas Albert (@AndreasAlbertQC) thanks you for looking into this, it's definitely an improvement. It doesn't exactly cover my use case at the moment since I only need to construct a dictionary with both the failed and passed validations (see example below), but it will be useful if/when I need to get more detailed validation results (e.g. counts of valid/invalid/unknown per rule). I got what I need now, so you may close this issue if you want, thanks!

rules = HouseSchema._validation_rules(with_cast=True)
failed_rules = HouseSchema.filter(df, cast=True, eager=True).failure.counts()
validation_results = {i: failed_rules.get(i) is None for i in rules}
validation_results

# => 
{
  'reasonable_bathroom_to_bedrooom_ratio': False,
  'num_bathrooms_ok': True,
  'zip_code|nullability': True,
  'zip_code|min_length': False,
  'num_bedrooms|nullability': False,
  'num_bedrooms|max': True,
  'num_bathrooms|nullability': True,
  'price|nullability': True,
  'price|inf': True,
  'price|nan': True,
  'zip_code|dtype': True,
  'num_bedrooms|dtype': True,
  'num_bathrooms|dtype': True,
  'price|dtype': True
}

Below is an example of how it's reported as asset checks within Dagster:

[AndreasAlbertQC]

Sweet, thanks for explaining Thomas Falk (@reinholdsson)! I am not very familiar with the details of dagster, but I wonder if the type of integration work you are doing would also be useful to others. I think it's a little too detailed to directly integrate new code into the dataframely package, but I think it would be cool to showcase this kind of integration in the docs. Would you be interested in contributing a docs page giving a simpe example of how to use dataframely with dagster? Assuming that this can be done with a reasonable amount of explainable code + a few screenshots, I think it would be neat.

[danielcrane]

I also feel like exposing a method that returns a dictionary with the validation results (incl. rows that failed the check) with one key per rule as described in the initial post by Thomas Falk (@reinholdsson) could be useful.

It's definitely already possible to do this, but could be worth considering making a more user-friendly way of doing this if the demand is high enough.

[borchero]

Daniel Crane (@danielcrane) these days, dataframely provides FilterResult.details(). Does that do what you need?

[ozanozbeker]

I just came across this thread. I ended up in the same place from my own work: I wanted Dataframely rules to show up as Dagster asset checks, hit exactly what Thomas Falk (@reinholdsson) describes here, and have been building it out publicly as dagster-dataframely. The README serves as the user guide right now but I'm working on a docs site, the following are pulled from it:

One decorator attaches a schema to a Dagster asset:

import dagster as dg
import dataframely as dy
import polars as pl

import dagster_dataframely as dd


class Orders(dy.Schema):
    order_id = dy.String(primary_key=True)
    amount = dy.Float64(nullable=False, min=0.0)


@dd.dataframely_asset(schema=Orders)
def orders(raw_orders: pl.DataFrame) -> pl.DataFrame:
    return raw_orders.select("order_id", "amount")

@dg.multi_asset is the mechanism underneath, but the vocabulary is @dg.asset's. That single declaration gets you:

• every rule as its own asset check with its own pass/fail history, behind a blocking check that compares columns and dtypes first, so a shape drift stops the run before any row is filtered
• the Columns tab filled from the schema before the asset has ever run: dtypes, nullability, uniqueness, the primary key, and the remaining constraints rendered from the column arguments
• failing rows optionally routed to a sibling asset instead of failing the run, carrying details() with one column per rule, which is the part feat: Expose a user-friendly version of FailureInfo._lf in FailureInfo.details() #275 made much easier
• rule docstrings as the check descriptions, which is Feature Request: Rule metadata #208

Andreas Albert (@AndreasAlbertQC), I'd be glad to help out with the docs page as well. The README already has the worked examples, so it would mostly be picking the smallest one and cutting it down to schema, asset, and what the checks look like in the UI, rather than writing something new.

Thomas Falk (@reinholdsson) if you have a chance, I'd really like you to try the package and tell me where it falls short. Your screenshots suggest you have real schemas to point at it, which is exactly the feedback I'm looking for. I'm using the package at my job currently but the more the merrier. Issues, requests and disagreements all welcome.

One thing worth flagging for the maintainers: my package leans on _validation_rules, RuleFactory.validation_fn and Rule.expr. Each is covered by a characterization test on my side, so an upstream change surfaces as one named failure rather than silent breakage, but you may want to know they're load-bearing for someone downstream.

[reinholdsson]

Ozan Ozbeker (@ozanozbeker) really interesting! Thanks for sharing. I will have a closer look. Here are some of my example code:

https://gist.github.com/reinholdsson/6b8f07408360ab669904874fe57edcb9

mostly with help from Claude 🙄 not sure if it works with the later versions of dataframely