Feature Request: Rule metadata

[danielcrane]

It'd be great if rules could be provided metadata, similarly to how columns can, to be able to pass for example a description of the rule that could be programmatically accessed from elsewhere.

i.e. @dy.rule(metadata={'description': 'xxx'})

Alternatively, allowing the rule's docstring to override the Rule class docstring could also work -- i.e. via one of the following solutions: https://stackoverflow.com/a/68733424 (which would allow one to do Schema.rule_func.__doc__ to get the docstring).

[borchero]

Out of curiosity: what's your use case for the metadata argument?

[danielcrane]

Thanks for your response!

Frankly right now the only use-case I have is for adding a description to the rule.

This would be useful to allow me to for example (programmatically) walk through the Schema's rules and generate a catalogue of rules with descriptions.

I'm currently storing descriptions in a config file, which works, but it'd be more elegant to be able to store it in-line with the function itself.

The alternative I mentioned (using functools to propagate the doc from the validation function) would work fine too for my specific use-case.

[ozanozbeker]

Hey Daniel Crane (@danielcrane), I've done something similar:

For a single rule, this works without functools:

class Orders(dy.Schema):
    order_id = dy.String(primary_key=True)
    amount = dy.Float64(nullable=False, min=0.0)

    @dy.rule()
    def paid_orders_have_amount(cls) -> pl.Expr:
        """Paid orders must carry a positive amount."""
        return cls.amount.col > 0


Orders.paid_orders_have_amount.validation_fn.__doc__
# 'Paid orders must carry a positive amount.'

To walk through all rules in a schema, you can do something like my _naming.rule_description:

import inspect

import dataframely as dy
from dataframely._rule import RuleFactory


def rule_description(schema: type[dy.Schema], rule_name: str) -> str | None:
    """A rule's docstring, or None for a rule with no place to carry one."""
    factory = getattr(schema, rule_name, None)
    if not isinstance(factory, RuleFactory):
        return None
    return inspect.getdoc(factory.validation_fn)

descriptions = {
    rule: rule_description(Orders, rule)
    for rule in Orders._validation_rules(with_cast=False)
}
# {'paid_orders_have_amount': 'Paid orders must carry a positive amount.',
#  'primary_key': None,
#  'order_id|nullability': None,
#  'amount|nullability': None,
#  'amount|min': None,
#  'amount|inf': None,
#  'amount|nan': None}

The Nones are from the column arguments, which have no function to hang a docstring on. The isinstance check is mainly for primary_key: getattr(Orders, "primary_key") gives Schema.primary_key which is a method of dy.Schema and gives an AttributeError without the check.

[danielcrane]

Thanks for your insight Ozan Ozbeker (@ozanozbeker) .

I've ended up making a decorator that wraps @dy.rule() where I can give the rule a description, severity, and a few other things -- it's turned out to be quite useful in my validation results visualisation (having different severity levels is especially useful).