Feature Request: Add an argument to `FailureInfo.write_parquet(only_invalid_rules: bool = False)`

[DeflateAwning]

When dealing with very large schemas (e.g., 50 string columns, all with min and max lengths), the failure info parquet file ends up very large/wide. It's challenging to open, and challenging to sort horizontally through.

It'd be helpful if there was a way to only write the debugging rule output columns that were responsible for at least one failing row (i.e., only the columns with min length violations).

All the core columns from the dataframe being validated would still be included - this feature request is only scoped to the additional columns like id|min_length, id|max_length, etc.

[delsner]

Your suggestion implies that we'd loose information about which rules were successfully applied when persisting the FailureInfo object when loading the FailureInfo object again

But if I understand your request correctly, FailureInfo.counts() is what you're interested in? Since it just outputs a mapping of failing rules to failure count, you could persist it using json.dump(failures.count(), fp).

[DeflateAwning]

Your suggestion implies that we'd loose information about which rules were successfully applied

I don't follow. My request is only about writing out to the parquet file. My suggestion is to implement this as the public-facing API - you can do whatever you need internally to have the data available for subsequent calls.

But if I understand your request correctly, FailureInfo.counts() is what you're interested in?

No, that's not what my request says. I want a parquet file, which shows only the relevant columns. I don't care about a summary of the counts - I care about scrolling through the failing rows and quickly finding quick rule(s) cause it to fail.