CondonFM Skills

.github/workflows/request-nvskills-ci.yml

@@ -0,0 +1,26 @@
+name: Request NVSkills CI
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request:
+    types: [opened, reopened, synchronize, ready_for_review]
+  push:
+
+jobs:
+  request:
+    if: >
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'issue_comment' &&
+        github.event.issue.pull_request &&
+        startsWith(github.event.comment.body, '/nvskills-ci')) ||
+      (github.event_name == 'push' &&
+        github.actor == (vars.NVSKILLS_SIGNATURE_PUSH_ACTOR || 'nv-skills-ci[bot]') &&
+        startsWith(github.event.head_commit.message, vars.NVSKILLS_SIGNATURE_COMMIT_TITLE || 'Attach NVSkills validation signatures'))
+    permissions:
+      contents: read
+      pull-requests: read
+      statuses: read
+    uses: NVIDIA/skills/.github/workflows/team-request.yml@main
+    secrets:
+      NVSKILLS_CI_DISPATCH_TOKEN: ${{ secrets.NVSKILLS_CI_DISPATCH_TOKEN }}

requirements.txt

@@ -1,3 +1,6 @@
+# xFormers CUDA wheels are published on the PyTorch index.
+--extra-index-url https://download.pytorch.org/whl/cu124
+
 # --------- pytorch --------- #
 torch==2.5.1
 torchvision==0.20.1
@@ -22,9 +25,10 @@ pre-commit==4.0.1      # hooks for applying linters on commit
 rich==13.9.4            # beautiful text formatting in terminal
 pytest==8.1.1  # tests
 sh==2.2.2  # for running bash commands in some tests (linux/macos only)
+python-dotenv==1.0.1
 transformers==4.54.1
 polars==1.12.0
-xformers==0.0.28.post3 --index-url https://download.pytorch.org/whl/cu124
+xformers==0.0.28.post3
 ninja==1.11.1.1
 einops==0.8.0
 ipython-autotime==0.3.2

skills/codonfm-embed/BENCHMARK.md

@@ -0,0 +1,119 @@
+# Skill Benchmark: codonfm-embed
+
+> **Overall verdict: NEUTRAL — One or more dimensions remain below PASS**
+
+Live evaluation did not show a material gain or regression. Collect more evidence or improve the skill before making a publication decision.
+
+## Evaluation Metadata
+
+- Skill: `codonfm-embed`
+- Evaluation date: 2026-09-18
+- Evaluator version: `1.5.6`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
+- Tasks: 2 evaluation tasks (2 positive)
+- Dataset digest: `sha256:13e6b2a6ffaa03dba4985cf78c69ce0dfef33331f5fba362890e08c6cf187db2` (skill-evaluator-dataset-snapshot/1)
+- Attempts per task: 3
+- Environment: `k8s-sandbox`
+- Tier 2 evidence: required for publication
+- Tier 3 evidence: required for publication
+
+Each task attempt ran in its own isolated sandbox pod.
+
+## What This Report Answers
+
+The three-tier evaluation checks whether the skill:
+
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
+
+## Results at a Glance
+
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | 91.0% — baseline ran, but no comparable score was available; uplift unavailable | 81.2% — baseline ran, but no comparable score was available; uplift unavailable |
+| Security | 100.0% → 100.0% (±0.0 points) | 50.0% → 50.0% (±0.0 points) |
+| Correctness | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
+| Discoverability | 99.0% — baseline ran, but no comparable score was available; uplift unavailable | 75.0% — baseline ran, but no comparable score was available; uplift unavailable |
+| Effectiveness | 75.0% → 81.3% (+6.3 points) | 100.0% → 87.5% (-12.5 points) |
+| Efficiency | 74.8% — baseline ran, but no comparable score was available; uplift unavailable | 93.4% — baseline ran, but no comparable score was available; uplift unavailable |
+
+**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
+
+Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
+
+## Token Usage
+
+Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
+
+| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
+|---|---|---:|---:|---:|---:|---|
+| claude-code | All cases | 1,038,604 | 2,307,979 | -1,269,375 | -55.00% | skill 2/2; base 2/2 |
+| claude-code | codonfm-embed-001 | 812,527 | 1,597,312 | -784,785 | -49.13% | skill 1/1; base 1/1 |
+| claude-code | codonfm-embed-002 | 226,077 | 710,667 | -484,590 | -68.19% | skill 1/1; base 1/1 |
+| codex | All cases | 480,684 | 991,001 | -510,317 | -51.50% | skill 2/2; base 2/2 |
+| codex | codonfm-embed-001 | 255,871 | 619,189 | -363,318 | -58.68% | skill 1/1; base 1/1 |
+| codex | codonfm-embed-002 | 224,813 | 371,812 | -146,999 | -39.54% | skill 1/1; base 1/1 |
+| ALL AGENTS | Dataset aggregate | 1,519,288 | 3,298,980 | -1,779,692 | -53.95% | skill 4/4; base 4/4 |
+
+Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.
+
+## Tier Status
+
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 7 finding(s) |
+| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |
+| Tier 3 | Live agent evaluation | **NEUTRAL** | 2 agent(s); 2 task(s) |
+
+## Findings and Observations
+
+<details>
+<summary>Show detailed findings and successful checks</summary>
+
+- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/codonfm-embed/SKILL.md`)
+- **LOW** QUALITY/quality_discoverability: Description very long (373 chars, recommend 50-150) (`skills/codonfm-embed/SKILL.md`)
+- **LOW** QUALITY/quality_discoverability: No '## Purpose' section (`skills/codonfm-embed/SKILL.md`)
+- **LOW** QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/codonfm-embed/SKILL.md`)
+- **LOW** QUALITY/quality_reliability: No limitations documented (`skills/codonfm-embed/SKILL.md`)
+- 2 additional finding(s) are available in the full evaluation artifacts.
+
+</details>
+
+## Scoring Methodology
+
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
+
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |
+
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
+
+Signals present in this run:
+
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
+- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
+
+</details>
+
+## Freshness
+
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

skills/codonfm-embed/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: codonfm-embed
+description: Extract frozen CLS embeddings from public CodonFM Encodon checkpoints for coding-sequence property modeling. Use when a user explicitly asks for CodonFM or Encodon embeddings, or wants Encodon features for translation-efficiency, expression, or mRNA-stability modeling. Support Encodon embedding_prediction only; do not claim Decodon embedding support in public CodonFM v1.
+metadata:
+  author: "NVIDIA BioNeMo <bionemofeedback@nvidia.com>"
+---
+
+# Extract public Encodon embeddings
+
+Extract one frozen CLS vector per coding sequence. This workflow writes
+embeddings only; it does not automatically train a downstream regressor.
+
+## Instructions
+
+Resolve the sequence CSV, checkpoint, and output directory from the request and
+available files. Validate inputs before extraction. Execution requires the
+project's ML dependencies and a compatible NVIDIA GPU. If a required resource
+is unavailable, complete the available preparation and return the command with
+that prerequisite identified. When extraction is requested and resources are
+ready, execute and verify the embedding arrays. A request for preparation ends
+with the inputs and command. If no sequences were supplied, report the required
+inputs. For Decodon, inspect the [public parser](../../src/runner.py) and
+[model configuration](../../src/config.py), explain the missing implementation,
+and finish without attempting installation or model development.
+
+For a demonstration use `nvidia/NV-CodonFM-Encodon-80M-v1`, revision
+`399ca9fe17b57941a7bebc6788033919b417413c`, file
+`NV-CodonFM-Encodon-80M-v1.safetensors` with sibling `config.json`.
+Reuse an existing checkpoint or download it when needed for the requested work.
+Preserve a user's explicit checkpoint choice.
+
+## Preflight and inputs
+
+1. Confirm `src/runner.py`, `src/data/codon_bert_dataset.py`, and
+   `src/inference/encodon.py` exist.
+2. Accept only `encodon_80m`, `encodon_600m`, or `encodon_1b`.
+3. For execution require a `.ckpt`, or `.safetensors` with sibling `config.json`;
+   input preparation can use a planned path.
+4. Require CSV columns `id`, `ref_seq`, `value`, and `split`.
+
+`ref_seq` must be a coding sequence. For extraction-only data, set `value` to
+`0.0` and `split` to `test` on every row. Although the public dataset labels
+`split` optional, its evaluation path calls the test split and fails without
+that column. Normalize sequences to uppercase DNA (`A/C/G/T`) and require
+lengths divisible by three. Sequences longer than `--context_length - 2`
+codons are truncated rather than embedded in full.
+
+## Examples
+
+Set `CODONFM_DATA_PATH` to the sequence CSV, `CODONFM_CHECKPOINT_PATH` to the
+checkpoint, and `CODONFM_RUN_DIR` to your chosen output directory:
+
+```bash
+python -m src.runner eval \
+    --exp_name embed_extract \
+    --model_name encodon_80m \
+    --checkpoint_path "$CODONFM_CHECKPOINT_PATH" \
+    --data_path "$CODONFM_DATA_PATH" \
+    --process_item codon_sequence \
+    --dataset_name CodonBertDataset \
+    --task_type embedding_prediction \
+    --num_nodes 1 \
+    --num_gpus 1 \
+    --num_workers 0 \
+    --val_batch_size 2 \
+    --out_dir "$CODONFM_RUN_DIR" \
+    --predictions_output_dir "$CODONFM_RUN_DIR/predictions"
+```
+
+For preparation requests, inspect the CSV directly against the input schema
+above and report the test-row count and sequence checks. Extra columns are
+allowed; extraction does not require a measured target. This does not require
+the ML runtime. The command above performs extraction when resources are ready.
+
+The existing `--dryrun` optionally builds runtime configuration and skips
+execution. It requires the ML dependencies, can create the prediction directory,
+and does not read the CSV or load weights. Do not use it as evidence that inputs,
+checkpoint compatibility, or embedding quality have been validated.
+
+## Outputs
+
+- `embeddings_merged.npy`: shape `(number_of_rows, hidden_size)`.
+- `ids_merged.npy`: IDs aligned with the embedding rows.
+
+Use the checked-in Encodon notebooks as downstream-model references:
+
+- `notebooks/4-EnCodon-Downstream-Task-riboNN.ipynb`
+- `notebooks/5-EnCodon-Downstream-Task-mRFP-expression.ipynb`
+- `notebooks/6-EnCodon-Downstream-Task-mRNA-stability.ipynb`
+
+Do not reference `notebooks/te_predictor.py`, `notebooks/mfe_predictor.py`, or
+Decodon notebooks because they are absent from public v1.
+
+## Boundaries
+
+- Do not use for Decodon; the public repository has no Decodon model or
+  inference class.
+- Do not claim a benchmark-trained regressor generalizes to a new organism,
+  cell type, or assay without new labeled validation data.
+- Do not invoke this skill for a generic expression-prediction request that
+  does not mention CodonFM or Encodon.

skills/codonfm-embed/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "CodonFM Embeddings"
+  short_description: "Extract public Encodon sequence embeddings"
+  default_prompt: "Use $codonfm-embed to extract Encodon embeddings from my coding-sequence CSV."

skills/codonfm-embed/evals/evals.json

@@ -0,0 +1,38 @@
+{
+  "skill_name": "codonfm-embed",
+  "evals": [
+    {
+      "id": "codonfm-embed-001",
+      "prompt": "Validate the supplied sequences.csv and prepare a public Encodon embedding-extraction command. Explain which rows will be processed and how to associate the output embeddings with sequence IDs. Use the supplied public source and checkpoint metadata.",
+      "files": [
+        "files/codonfm_source.zip",
+        "files/encodon_checkpoint.json",
+        "files/sequences.csv"
+      ],
+      "expected_output": "Two validated test rows and a public embedding_prediction command with the correct embedding/ID output contract.",
+      "assertions": [
+        "The command uses embedding_prediction, codon_sequence, and CodonBertDataset",
+        "The command specifies --checkpoint_path and output/prediction paths appropriate to the chosen working directory",
+        "The agent validates both sequence rows, including value and split=test, and explains that evaluation processes the test split",
+        "The response identifies embeddings_merged.npy and ids_merged.npy and explains their row alignment without fabricating embeddings"
+      ],
+      "expected_skill": "codonfm-embed",
+      "expected_script": null
+    },
+    {
+      "id": "codonfm-embed-002",
+      "prompt": "Does the public CodonFM implementation support extracting Decodon embeddings? Check the supplied source and explain the limitation, if any.",
+      "files": [
+        "files/codonfm_source.zip",
+        "files/encodon_checkpoint.json"
+      ],
+      "expected_output": "The agent identifies the absence of a public Decodon model and inference implementation.",
+      "assertions": [
+        "The agent explains that the supplied public source has no Decodon model/inference implementation and cites inspected files",
+        "The agent does not invent a Decodon command or attempt to implement the missing model"
+      ],
+      "expected_skill": "codonfm-embed",
+      "expected_script": null
+    }
+  ]
+}

skills/codonfm-embed/evals/files/encodon_checkpoint.json

@@ -0,0 +1,33 @@
+{
+  "repo_id": "nvidia/NV-CodonFM-Encodon-80M-v1",
+  "revision": "399ca9fe17b57941a7bebc6788033919b417413c",
+  "model_name": "encodon_80m",
+  "filename": "NV-CodonFM-Encodon-80M-v1.safetensors",
+  "size_bytes": 307351588,
+  "config_filename": "config.json",
+  "config": {
+    "vocab_size": 69,
+    "hidden_size": 1024,
+    "num_hidden_layers": 6,
+    "num_attention_heads": 8,
+    "intermediate_size": 4096,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "attention_probs_dropout_prob": 0.1,
+    "initializer_range": 0.02,
+    "layer_norm_eps": 1e-12,
+    "pad_token_id": 3,
+    "position_embedding_type": "rotary",
+    "classifier_dropout": 0.1,
+    "rotary_theta": 10000.0,
+    "ignore_index": -100,
+    "loss_type": "cross_entropy",
+    "lora": false,
+    "lora_alpha": 32.0,
+    "lora_r": 16,
+    "lora_dropout": 0.1,
+    "finetune_strategy": "full"
+  },
+  "source_url": "https://huggingface.co/nvidia/NV-CodonFM-Encodon-80M-v1/tree/399ca9fe17b57941a7bebc6788033919b417413c",
+  "weights_included": false
+}

skills/codonfm-embed/evals/files/sequences.csv

@@ -0,0 +1,3 @@
+id,ref_seq,value,split
+example_1,ATGGCTGAATTTCCGTAA,0.0,test
+example_2,ATGGCAGAATTTCCGTAA,0.0,test