[Feature Request] Add support for clipping values during type enforcement

[svengiegerich]

Is your feature request related to a problem? Please describe.

Currently, when enforcing a schema where data exceeds the range of the target dtype (e.g., a value of 260 for a uint8 column), the operation may fail or lead to silent overflows. In production environments, it is often preferable to bound these values rather than allowing the pipeline to crash or produce corrupted data.

Describe the solution you'd like

I would like to see an option—perhaps a parameter like out_of_bounds="clip"—within the schema enforcement logic. When enabled, any value exceeding the maximum or minimum of the target numeric type would be clipped to that type's limit.

Example Scenario

Target Dtype: uint8 (Range: 0 to 255)

Input Value: 260

Expected Result (with clipping): 255

Describe alternatives you've considered

The current alternative is to manually call .clip() on the DataFrame before validation, but this duplicates logic that could be handled more efficiently during the schema enforcement/casting phase.

[AndreasAlbertQC]

hey Sven (@svengiegerich)!

In production environments, it is often preferable to bound these values rather than allowing the pipeline to crash or produce corrupted data.

I'm not sure I follow entirely: silently clipping values that don't fit a dtype sounds exactly like data corruption, leading to hard-to-predict downstream behavior. I'd be a little apprehensive about exposing this as a standard option because I would have a hard time recommending this approach to anyone. Why would you prefer to accept possible clipping instead of just increasing the size of your dtype?

[svengiegerich]

hey @svengiegerich!

In production environments, it is often preferable to bound these values rather than allowing the pipeline to crash or produce corrupted data.

I'm not sure I follow entirely: silently clipping values that don't fit a dtype sounds exactly like data corruption, leading to hard-to-predict downstream behavior. I'd be a little apprehensive about exposing this as a standard option because I would have a hard time recommending this approach to anyone. Why would you prefer to accept possible clipping instead of just increasing the size of your dtype?

I understand your reasoning and why you're hesitant to make this the default behavior. I’m approaching this from a slightly different angle: I don't want to silently clip data, but I would value having an explicit option to clip (e.g., across multiple columns simultaneously).

My reasoning follows standard data science practices for clipping and winsorizing. To provide a concrete use case: in a production feature DataFrame validated with dy where all features are durations, an out-of-range value in a single column isn't always a "deal-breaker." Since it’s just one part of a larger feature battery, I’d prefer to clip the value rather than discard the entire row.

Why would you prefer to accept possible clipping instead of just increasing the size of your dtype?

By not increasing, I can save memory and document the value range more accurately, implying I don't care about the exact values of outliers.

[AndreasAlbertQC]

I see, I was not really thinking of this in terms of a deliberate transformation. You use case makes more sense to me now. In that case, you would also want to min and max constraints to be clippable, right? In case you want to clip to something that's not a dtype boundary

[Hannoma]

I also would be really interested in this feature!
What about adding Schema.clip_to class method, so that you must explicitly run this transformation before casting or validating?

[borchero]

This feature really opens a can of worms 😅 if we do that, we'd probably also sooner or later need to support rounding of floats, truncation of strings, ... currently I'm not particularly keen on committing to that 😅