[GLM 5.3 Tests] B300 profile and profiled Tinker client

[hershg]

Changes

• A fixed-input GSPO benchmark with one shared SDK implementation. Start at run_client.run(): prepare → initial publish/sample → reference, forward/backward, optimizer, publish/sample → checkpoint/unload. The server script only configures and starts the service.
• Extend existing profiling hooks: raw CPU/CUDA/memory traces, OOM export, separate optimizer dispatch/trace-processing metrics, Megatron forward/backward timers, and optional vLLM capture. Fix profiler-config conversion and SDK tokenizer/LoRA metadata.

Testing

uv run --isolated --no-project --with tinker==0.24.1 --with pytest --with httpx \
  python -m pytest --confcutdir=examples/tinker/glm53/tests examples/tinker/glm53/tests -q

41 passed; Ruff, Black and secret checks pass.

• Cleanup 930706b3: all 13 SDK operations are structurally equivalent after parameter renaming/types and equivalent context-manager flattening. Runtime and model-config diffs against 9fe0b11d are empty. Existing runs remain pinned; their receipts are not relabeled as this commit.
• GPU 9fe0b11d: Qwen3-8B completed warmup + two measured updates, checkpoint and unload. Six inference traces contain CUDA/memory events; optimizer dispatch is distinct from trace processing. Actual vLLM application-construction regression: old code fails, fixed code passes 24 tests.
• Earlier GLM-5.3 32K profiling completed on 97e14ca4. New fully instrumented GLM runs are still in progress. Real allocator-OOM trace export/recovery was tested separately on fb3a4ecf.

This benchmark proves mechanics/timing, not learning or numerical parity. Short samples do not qualify full-context inference; the 256K profiles remain candidates. Raw optimizer API duration still includes trace processing—use its separate metrics when interpreting performance.

[hershg]

note - for 256k 2node, try PP2, and trade off TP for CP (TP 4 CP 2, instead of TP 8 CP 1)

(point agent at skyrl repo .claude/skills/parallelism-strategies/SKILL.md pp picker - could be useful, might be pre-training centric rn though)

[hershg]

Reviewed the parallelism skill. The two-node disaggregated profile has 8 trainer GPUs and already uses TP4/CP2/PP1. TP4/CP2/PP2 would require 16 trainer GPUs (three total nodes including inference), so a two-node PP2 candidate must change another axis, e.g. TP2/CP2/PP2 or TP4/CP1/PP2, and recheck expert layout. Leaving this experimental suggestion open rather than silently changing the benchmark target; no new 256K qualification is claimed.

[gemini-code-assist]

Code Review

This pull request introduces full-context profiling capabilities for GLM models through SkyRL's Tinker API, adding configuration profiles, server and client scripts, and comprehensive tests. It also implements a decorator to flush active torch profiler traces upon encountering a CUDA OutOfMemoryError. The review feedback suggests two important robustness improvements: safely accessing the profiler attribute using getattr in the OOM decorator to avoid masking the original error with an AttributeError, and adding a sleep delay in the client's model unloading loop when handling 408 status codes to prevent a tight busy-polling loop.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking defects identified.

Diagram

sequenceDiagram
    participant C as Tinker client
    participant A as SkyRL API
    participant P as Policy workers
    participant I as Inference engine
    C->>A: create LoRA model
    A->>P: initialize policy and start profiler
    C->>A: forward reference batch
    A->>P: forward-only scoring
    P-->>C: old-policy logprobs
    C->>A: forward_backward GSPO batch
    A->>P: forward and backward
    C->>A: optimizer step
    A->>P: apply update and advance profiler
    C->>A: publish weights
    A->>I: synchronize adapter
    C->>I: sample
    C->>A: save checkpoint and unload
    A->>P: stop profiler on runtime teardown

Loading

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit fb3a4ec. Configure here.}