feat(medcat-llm-components): CPDT-108 Add llm components

[mart-r]

This adds LLM components.

The idea is to add these such that anyone can point this to an LLM and run that as part of the same MedCAT workflow.

I've built this as a separate package. I think that makes the most sense at this stage.

These are not meant as the best implementation that works for everything. They're meant as an example and a quick way to get going.

PS:
There's a few tests, but I do need to run through a real world test as well. But wanted it up for CI.

TODO:

• Run end to end / real tests (e.g with the GPU box)
• Add README + improve documentation
• Add PyPI release path (e.g through trusted publisher)

Example with output for NER done remotely

The code:
https://gist.github.com/mart-r/2dee696a289b16d574750fcd172b24bc

The text:

Patient had nausea and a fever.

The final output:

% python .temp/test_with_local_ollama_ner.py                          
Loading model ../../medcat-v2/.temp/20230227__kch_gstt_trained_model_f76d2121b77c3e9a.zip
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-plugins/llm-comps/.venv/lib/python3.12/site-packages/spacy/util.py:971: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.16). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-plugins/llm-comps/.venv/lib/python3.12/site-packages/spacy/util.py:971: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.16). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
Test BEFORE change
GOT in 2.01109375001397s:
{0: {'acc': 1,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1,
     'cui': '116154003',
     'detected_name': 'patient',
     'end': 7,
     'icd10': [],
     'id': 0,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Patient',
     'source_value': 'Patient',
     'start': 0,
     'type_ids': ['31601201']},
 1: {'acc': 1,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1,
     'cui': '422587007',
     'detected_name': 'nausea',
     'end': 18,
     'icd10': ['R11'],
     'id': 1,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Nausea',
     'source_value': 'nausea',
     'start': 12,
     'type_ids': ['67667581']},
 2: {'acc': 0.7626635672372345,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 0.7626635672372345,
     'cui': '386661006',
     'detected_name': 'fever',
     'end': 30,
     'icd10': ['R50.9'],
     'id': 2,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Fever',
     'source_value': 'fever',
     'start': 25,
     'type_ids': ['67667581']}}
Update NER
NER type before default -> <medcat.components.ner.vocab_based_ner.NER object at 0x16ae006e0>
NER type before llm_ner -> <medcat_llm_components.ner.LLMNER object at 0x16d5be450>
Done with prep - now for the main course
Unable to find 'fever' near [18:23]. Nearest match at [25:30] is 14 chars away (over span_tolerance_total). If this is still correct, raise the tolerance in the config.
GOT in 7.160184249980375s:
{-1: {'acc': 1,
      'context_center': [],
      'context_left': [],
      'context_right': [],
      'context_similarity': 1,
      'cui': '422587007',
      'detected_name': 'nausea',
      'end': 18,
      'icd10': ['R11'],
      'id': -1,
      'meta_anns': {},
      'opcs4': [],
      'pretty_name': 'Nausea',
      'source_value': 'nausea',
      'start': 12,
      'type_ids': ['67667581']}}
All done!

Example with output for linker done remotely

The code:
https://gist.github.com/mart-r/3d539395a5ea0ac039c4bf222132017d

The text:

John presented to the ED with acute SOB

The final output:

% python .temp/test_with_local_ollama_linker.py    
Loading model ../../medcat-v2/.temp/20230227__kch_gstt_trained_model_f76d2121b77c3e9a.zip
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-plugins/llm-comps/.venv/lib/python3.12/site-packages/spacy/util.py:971: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.16). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-plugins/llm-comps/.venv/lib/python3.12/site-packages/spacy/util.py:971: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.16). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
Test BEFORE change
GOT in 1.965135041042231s:
{0: {'acc': 1,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1,
     'cui': '246105001',
     'detected_name': 'presented',
     'end': 14,
     'icd10': [],
     'id': 0,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Presentation',
     'source_value': 'presented',
     'start': 5,
     'type_ids': ['43039974']},
 3: {'acc': 1,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1,
     'cui': '267036007',
     'detected_name': 'sob',
     'end': 39,
     'icd10': ['R06.0'],
     'id': 3,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Dyspnea',
     'source_value': 'SOB',
     'start': 36,
     'type_ids': ['67667581']}}
Update linker
Linker type before default -> <medcat.components.linking.context_based_linker.Linker object at 0x3e996ca40>
Linker type before llm_linker -> <medcat_llm_components.linker.LLMLinker object at 0x3e6ee9940>
Done with prep - now for the main course
GOT in 8.25323154090438s:
{0: {'acc': 1.0,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1.0,
     'cui': '246105001',
     'detected_name': 'presented',
     'end': 14,
     'icd10': [],
     'id': 0,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Presentation',
     'source_value': 'presented',
     'start': 5,
     'type_ids': ['43039974']},
 1: {'acc': 1.0,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1.0,
     'cui': '860914002',
     'detected_name': 'ed',
     'end': 24,
     'icd10': ['F52.2'],
     'id': 1,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Erectile dysfunction',
     'source_value': 'ED',
     'start': 22,
     'type_ids': ['9090192']},
 2: {'acc': 1.0,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1.0,
     'cui': '402355000',
     'detected_name': 'acute',
     'end': 35,
     'icd10': ['T86.0'],
     'id': 2,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Acute graft-versus-host disease',
     'source_value': 'acute',
     'start': 30,
     'type_ids': ['9090192']},
 3: {'acc': 1.0,
     'context_center': [],
     'context_left': [],
     'context_right': [],
     'context_similarity': 1.0,
     'cui': '267036007',
     'detected_name': 'sob',
     'end': 39,
     'icd10': ['R06.0'],
     'id': 3,
     'meta_anns': {},
     'opcs4': [],
     'pretty_name': 'Dyspnea',
     'source_value': 'SOB',
     'start': 36,
     'type_ids': ['67667581']}}
All done!

[tomolopolis]

lgtm - nice - can you do the next stage of actually creating the model pack with a more up to date model and example prompt. gemma-3-1b is suitably small enough for people to run and get a hang of what's going on

[mart-r]

lgtm - nice - can you do the next stage of actually creating the model pack with a more up to date model and example prompt. gemma-3-1b is suitably small enough for people to run and get a hang of what's going on

This is based on the assumption that the actual LLM is running somewhere else (or could be running on the same machine but within some other process). So I can't really "build a model that uses it" without having an openly available endpoint to point at. This will always be up to the user to configure as far as I can tell.

Perhaps there's room for LLMs running alongside medcat, but I thought separating these concerns would be an easier use case.

EDIT:
It might be useful to integrate this direcrtly with medcat-service though - where on the service level you can specify which endpoint / model to use (and potentially credentials). But I don't know if we want to clutter that right now. Not to mention that this implementation might not be the most robust for every LLM setup.