Worker: ensure that final_state gets redacted

[josephjclark]

Turns out there's a long-standing bug where large dataclips moving between the inner engine child process and the main worker thread are NOT properly redacted.

I don't know how or why, or how I didn;t notice, but it seems we only redact state.data and ignore other keys. This is crazy.

So what's happening right now in production is:

• Cron jobs run once a day
• Some of them have multiple leaves in their final state - so the have a state objec like { step1: { data }, step2: { data }, rather than just { data }
• That final dataclip should be redacted if it exceeds 10 mb
• But it is not!!!
• The engine rather meekly just tries to scrub the datakey, which is honestly not doing much

I think we sort of escaped this because the engine notes that the state is redacted, and then the worker refusses to send it back to lightning. So it sort of works itself out.

But the run-complete event does not do this. So final_state was being sent unredacted through the websocket. Any final state object bigger than 11mb causes lightning's cowboy to drop the frame and disconnect the worker, resulting in the whole run getting lost.

These multi-leaf cron jobs have a tendency to bloat state over time. And now we're seeing some of these large objects causing real problems.

Fixes #1517

There are two fixes here:

• Fix redaction on dataclips so that the whole state object is replaced, not just the data key
• If a final state is redacted, we log a warning. It's not much but it might serve to explain to users why their workflows are failing.

QA Notes

This is easily reproduced.

Check out this workflow on staging: https://app.staging.openfn.org/projects/b387f5a4-a3bb-4bb8-8093-f99436d09d92/w/ace8117e-dcfa-481f-b341-ea06978c366d?step=05155006-7d8c-462e-b215-449f52c450b2#log

It has two leaf nodes which both return an object of, uh 7mb (6 mb should do it but it doesn't seem to trigger, eh whatever).

Each output dataclip is small enough that it gets returned to the app (see the output for each step and you;ll find the state objects). But large enough that when they're both aggregated into a single final_state, they blow the cowboy limit and the run will be lost.

This workflow runs great locally with this fix in place.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy