WAR for the cuBLAS grouped gemm issue

[ptrendx]

Description

cuBLAS 13.7 introduced a silent data corruption issue in one of the grouped gemm algorithms on B300 and Rubin. This PR disables this algorithm from being selected.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

Please list the changes introduced in this PR:

• Change A
• Change B

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[greptile-apps]

Greptile Summary

The PR adds a targeted workaround for a cuBLAS 13.7 NVFP4 grouped-GEMM correctness issue on SM103 and SM107.

• Detects affected cuBLAS and GPU configurations.
• Requests multiple heuristic candidates and rejects algorithms using the unsafe 768xAUTO stage configuration.
• Preserves the existing single-result selection behavior on unaffected configurations.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete correctness, compatibility, or security regressions identified.

The workaround is limited to affected NVFP4 grouped GEMMs on the specified cuBLAS version and architectures, while existing selection behavior remains intact elsewhere.

Important Files Changed

Filename	Overview
transformer_engine/common/gemm/cublaslt_grouped_gemm.cu	Adds narrowly gated filtering of the affected cuBLAS grouped-GEMM algorithm without changing algorithm selection on unaffected configurations.

_{Reviews (1): Last reviewed commit: "WAR for the cuBLAS issue" | Re-trigger Greptile}

[ksivaman]

LGTM

[ptrendx]

/te-ci

[ptrendx]

/te-ci