CogStack · shubham-s-agarwal · Sep 3, 2026 · Sep 3, 2026 · Sep 3, 2026 · Sep 3, 2026
diff --git a/docs/docs/cogstack-ai/index.md b/docs/docs/cogstack-ai/index.md
@@ -1,11 +1,60 @@
-# CogStack AI
 
-CogStack AI provides powerful natural language processing tools for healthcare and biomedical text analysis.
+# CogStack Natural Language Processing
 
-## Available Tools
+## Overview
 
-- **[MedCAT](https://docs.cogstack.org/projects/nlp)** - Medical Concept Annotation Tool for Named Entity Recognition and Linking (NER+L) from clinical free-text documents.
+CogStack ecosystem provides a standard set of natural language processing applications that are used either as standalone applications or implemented as RESTful services with uniform API, each running in a Docker container. <br> These NLP applications when used inside the data processing pipeline cover one of the key steps of information extraction. They may include extracting medical concepts from free-text notes using a specific terminology, such as [SNOMED CT](https://en.wikipedia.org/wiki/SNOMED_CT) or using all the terminologies as available in [UMLS](https://www.nlm.nih.gov/research/umls/index.html). Often, more specialised applications will be built on top of the standard set of NLP applications provided in CogStack, utilising both structured and unstructured information tailored to a defined use-case. These custom applications can be further integrated into CogStack and used as a part of standard set of NLP applications.
 
-- **[MedCAT Trainer](https://docs.cogstack.org/projects/medcat-trainer/)** - Interface for building, improving, and customizing Named Entity Recognition and Linking models for biomedical domain text.
+!!! tip
 
-Click on the links above to access the full documentation for each tool.
+    Please see [CogStack using Apache NiFi Deployment Examples](https://github.com/CogStack/CogStack-NiFi/tree/devel/deploy) to see how to integrate NLP services in example data pipelines.
+
+!!! tip
+
+    Apart from being integrated directly in the data processing pipeline, many NLP applications are often used as standalone applications and have a rich set of tools build around them – please see below for more details.
+
+---
+## Tools
+
+### MedCAT - Medical Concept Annotation Tool
+
+One of the key tools is MedCAT – a Medical Concept Annotation Tool that is used for Named Entity Recognition and Linking (NER+L) tasks for clinical concepts from free-text documents.
+
+ MedCAT is based on a light-weight neural network that calculates vector embeddings and that is used for disambiguation and concept detection. MedCAT also uses Deep Learning Language Model that is used for detection of negation, experiencer or any other type of classification. <br/> MedCAT can utilise a concepts dictionary with a vocabulary provided by the end-user that will be used to perform annotate the concepts in the clinical notes. The provided concepts dictionary can be e.g., SNOMED CT terminology or full/ subset of UMLS resource. Apart from providing the vocabulary and concepts dictionary, the underlying MedCAT model can be further trained and fine-tuned for performing context-aware concept disambiguation with additional meta-annotations tasks. MedCAT be run also directly with pre-trained models. <br/> **MedCAT** can be used either as a standalone Python module, as a part of a model trainer application **MedCAT Trainer** or be deployed as a RESTful **MedCAT Service** inside a data processing pipeline. Below are briefly covered possible ways of working with MedCAT. <br/> 
+
+!!! warning
+
+    Please note that we only provide few basic models for MedCAT that have been prepared using open datasets. Some of the used models are restricted by the external licensing of the resource that was used to build it, such as SNOMED CT or UMLS. In such cases, the user needs to apply for an appropriate license – please see: [UMLS licensing](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/license_agreement.html) and [SNOMED CT licensing](http://www.snomed.org/snomed-ct/get-snomed).
+
+!!! important
+
+    When deploying MedCAT into data processing pipelines one may be interested in training and tailoring the MedCAT models as a part of model preparation. This can be done directly by using MedCAT Trainer or MedCAT library working with a corpus of input documents. Such trained model can be in the next step provided into MedCAT Service that will be deployed as a service and used in the data pipeline.
+
+**Key resources:** <br>
+
+- GitHub repository with code and documentation: <https://github.com/CogStack/MedCAT>
+- MedCAT publication: <https://arxiv.org/abs/1912.10166>
+- Tutorial on MedCAT: [MedCAT – Analysing Electronic Health Records](https://towardsdatascience.com/medcat-introduction-analyzing-electronic-health-records-e1c420afa13a) (in a series of articles)
+- PIP repository: <https://pypi.org/project/medcat/>
+
+!!! tip
+
+    The MedCAT Python library is the functional core of MedCAT project. The library is used by MedCAT Trainer when training and updating the models. It is also used within the MedCAT Service that exposes the medical concepts extraction functionality.
+
+### MedCAT Trainer
+
+MedCAT Trainer is an interface for building, improving and customising a given Named Entity Recognition and Linking models for biomedical domain text. <br/> The models trained by MedCAT Trainer can be later used directly with custom Python applications using on MedCAT module. Alternatively, the models can be deployed in data pipelines, e.g. behind a RESTful API via MedCAT Service. <br/> 
+
+**Key resources:** <br/><ul><li><p>GitHub repository with code and documentation: <a class="external-link" data-card-appearance="inline" href="https://github.com/CogStack/MedCATTrainer" rel="nofollow">https://github.com/CogStack/MedCATTrainer</a> </p></li><li><p>MedCAT Trainer publication: <a class="external-link" href="https://arxiv.org/abs/1907.07322" rel="nofollow">https://arxiv.org/abs/1907.07322</a></p></li><li><p>Tutorial on using MedCAT Trainer: <a class="external-link" href="https://towardsdatascience.com/medcattrainer-a-tool-for-inspecting-improving-and-customising-medcat-880a11297ebe" rel="nofollow">MedCATTrainer: A Tool For Inspecting, Improving and Customising MedCAT</a></p></li></ul> <br/> 
+
+## MedCAT Service
+
+MedCAT Service implements a RESTful API over MedCAT module to perform extraction of concepts from provided text. Usually, a single instance of MedCAT Service will be serving a single MedCAT model. Such model can be later deployed in data processing pipelines. The API specification is provided in the sections below.
+
+**Key resources:**
+
+- GitHub repository with code, documentation and use examples: <https://github.com/CogStack/MedCATservice>
+
+!!! tip
+
+    Please note that there is available public MedCAT model trained on MedMentions corpus that can be used to play with.
diff --git a/docs/docs/cogstack-ai/index_.md b/docs/docs/cogstack-ai/index_.md
@@ -0,0 +1,12 @@
+[# CogStack AI
+
+CogStack AI provides powerful natural language processing tools for healthcare and biomedical text analysis.
+
+## Available Tools
+
+- **[MedCAT](https://docs.cogstack.org/projects/nlp)** - Medical Concept Annotation Tool for Named Entity Recognition and Linking (NER+L) from clinical free-text documents.
+
+- **[MedCAT Trainer](https://docs.cogstack.org/projects/medcat-trainer/)** - Interface for building, improving, and customizing Named Entity Recognition and Linking models for biomedical domain text.
+
+Click on the links above to access the full documentation for each tool.
+]()
diff --git a/docs/docs/cogstack-ai/medcat_service_api.md b/docs/docs/cogstack-ai/medcat_service_api.md
@@ -0,0 +1,96 @@
+# MedCAT Service API
+
+### REST API definition
+
+The API defines 3 endpoints, that consume and return data in JSON format:
+
+- *GET* `/api/info` - displays general information about the the NLP application,
+- *POST* `/api/process` - processes the provided single document and returns back the annotations,
+- *POST* `/api/process_bulk` - processes the provided list of documents and returns back the annotations.
+
+#### GET `/api/info`
+
+Returns information about the used NLP application. The returned fields are:
+
+- `name`, `version`, `language` of the underlying NLP application
+- `parameters` – a generic JSON object representing any relevant parameters that have been specified to the application (optional)
+
+#### POST `/api/process`
+
+Returns the annotations extracted from the provided document.
+
+The request message payload JSON consists of following objects
+
+- `content` that represents the single document content to be processed
+- `applicationParams` – a generic JSON object representing NLP application run-time parameters (optional)
+
+The single document processing `content` (\*\*\*) has following keys :
+
+- `text` – the document to be processed
+- `metadata` – a generic JSON object representing any relevant metadata associated with the document that will be consumed by the NLP application (optional)
+- `footer` – a generic JSON object representing a payload footer that will be returned back with the result (optional)
+
+The response message payload JSON consists of an object `result` that has following fields:
+
+- `text` – the input document that was processed (optional)
+- `annotations` – an array of generic JSON annotation objects, not enforcing any schema
+- `metadata` – a metadata associated with the processed document that was reported by the NLP application (optional)
+- `success` – boolean value indicating whether the NLP processing was successful
+- `timestamp` – document processing timestamp
+- `errors` – an array of NLP processor errors (present only in case when `success` is `false`)
+- `footer` – the footer object as provided in the request payload (present only when provided in the request message)
+
+#### POST `/api/process_bulk`
+
+Returns the annotations extracted from a list of documents.
+
+The request message payload JSON consists of following objects
+
+- `content` – an array of documents content to be processed
+- `applicationParams` – a generic JSON object representing NLP application run-time parameters (optional)
+
+Here, the `content` object holds an array of single document content to be processed as defined above in (\*\*\*).
+
+### Example use
+
+!!! tip
+
+    Please see [CogStack using Apache NiFi Deployment Examples](https://github.com/CogStack/CogStack-NiFi/tree/devel/deploy) to see how to deploy example NLP services, i.e. MedCAT with a public MedMentions model.
+
+#### MedCAT
+
+Assuming that the application is running on the `localhost` with the API exposed on port `5000`, one can run:
+
+```bash
+curl -XPOST http://localhost:5000/api/process \
+  -H 'Content-Type: application/json' \
+  -d '{"content":{"text":"The patient was diagnosed with leukemia."}}'
+
+```
+
+and the received result:
+
+```json
+{
+  "result": {
+    "text": "The patient was diagnosed with leukemia.",
+    "annotations": [
+      {
+        "pretty_name": "leukemia",
+        "cui": "C0023418",
+        "tui": "T191",
+        "type": "Neoplastic Process",
+        "source_value": "leukemia",
+        "acc": "1",
+        "start": 31,
+        "end": 39,
+        "info": {},
+        "id": "0",
+        "meta_anns": {}
+      }
+    ],
+    "success": true,
+    "timestamp": "2019-12-03T16:09:58.196+00:00"
+  }
+}
+```
diff --git a/docs/docs/data-engineering/attachments/b5fc6b57-faf2-4747-9e77-eb9adf51d8b3.jpg b/docs/docs/data-engineering/attachments/b5fc6b57-faf2-4747-9e77-eb9adf51d8b3.jpg
diff --git a/docs/docs/data-engineering/changes.md b/docs/docs/data-engineering/changes.md
@@ -0,0 +1,13 @@
+# Cogstack-NiFi
+
+### Major changes from Cogstack-Pipeline
+
+There are some key major changes when using and deploying Apache NiFi as compared with CogStack-Pipeline.
+
+One of the most important changes is the way how defining, configuring and monitoring data flows works. When using CogStack-Pipeline the ingestion jobs were defined in `.properties` files and were having very limited job execution monitoring and troubleshooting possibilities. Apache NiFi implements (an optional) web-based user interface that can be used to define data flows on drag-and-drop fashion with further configuration and monitoring capabilities. The data flow definitions can be saved and exported into XML format and later loaded into other instances of Apache NiFi or just kept under version control.
+
+Each ingestion job that is being run by CogStack-Pipeline also requires a separate CogStack-Pipeline application instance. In Apache NiFi multiple data flows can be run in parallel each being managed by a single, main Apache NiFi data processing engine instance.
+
+Moreover, one of the main limitations of CogStack pipeline has been support only for a document-centric data model for performing ingestion where each ingested record could only contain one document to be processed. Apache NiFi does not enforce document-centric data model and provides flexibility on defining custom data flows and data schemas. Handling multiple documents in a single record or using a patient-centric data model is a matter of tailoring the pipeline and defining or tailoring appropriate schema.
+
+Moreover, fixed ETL operations (implemented as modules in CogStack-Pipeline) can be included as custom ETL scripts or application modules inside a defined Apache NiFi data flow. For example, NLP functionality, such as running [MedCAT](https://github.com/CogStack/MedCATservice) was implemented as external micro-services exposing that expose a REST API and hence can be used directly in the data flow. All the third-party application dependencies are handled by the external services that further allows for separating the responsibilities.
diff --git a/docs/docs/data-engineering/index.md b/docs/docs/data-engineering/index.md
@@ -1,9 +1,57 @@
-# Data Engineering
+# Cogstack-NiFi
 
-CogStack Data Engineering provides tools and frameworks for building scalable, configurable data processing pipelines.
+## What is Cogstack-NiFi
 
-## Available Tools
+CogStack-NiFi is the re-architected version of CogStack-Pipeline that replaces the fixed Spring Batch-based pipeline engine with [Apache NiFi](https://nifi.apache.org/). It focuses on fully configurable and scalable data flows with the data processing engine that is easy to use, deploy and tailor to any site-specific data flow requirements. Apache NiFi also comes in with built-in monitoring, data provenance and security features that puts the operations in better control and reliability.
 
-- **[NiFi](https://docs.cogstack.org/projects/nifi/)** - Apache NiFi integration for CogStack, providing fully configurable and scalable data flows with built-in monitoring, data provenance, and security features.
+At its core, CogStack-NiFi handles the ingestion and harmonisation of data from disparate EHR sources — structured tables, unstructured free text, scanned documents requiring OCR — into a common, queryable format. It extracts, transforms, and loads this data into OpenSearch (or Elasticsearch) indices, and into structured SQL stores, ready for downstream NLP processing and search.
 
-Click on the link above to access the full documentation.
+---
+
+## Apache NiFi
+
+![Apache NiFi logo](./attachments/nifi-logo.png)
+
+![Apache NiFi flow example](./attachments/b5fc6b57-faf2-4747-9e77-eb9adf51d8b3.jpg)
+
+*From the official documentation:* Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. NiFi has a web-based user interface for design, control, feedback, and monitoring of dataflows. It is highly configurable along several dimensions of quality of service, such as loss-tolerant versus guaranteed delivery, low latency versus high throughput, and priority-based queuing. NiFi provides fine-grained data provenance for all data received, forked, joined cloned, modified, sent, and ultimately dropped upon reaching its configured end-state.
+
+Some of the key features of Apache NiFi engine are:
+
+- Highly configurable and extendable
+    - Can build own data processors and modules that can be easily integrated into data pipeline
+    - Enables rapid prototyping, development and effective testing
+    - Data flows can be modified, inspected and troubleshot at runtime
+- Web-based user interface
+    - Seamless experience between design, control, feedback, and monitoring of the data flows
+- Data Provenance
+    - Can track data flow from beginning to end for addressing information governance requirements
+- Security
+    - Support for SSL, SSH, HTTPS, encrypted content, etc.
+    - Multi-tenant authorization and internal authorization/policy management
+
+For a detailed description of Apache NiFi, it's functionality and broad set of features please refer to links to the official documentation provided below.
+
+---
+
+## Useful links
+
+[This guide](https://docs.cogstack.org/projects/nifi/en/latest/) containing the official documentation to Cogstack-Nifi is the next step, please take a look to learn in-depth about it, from introduction, to deploying!
+
+**Cogstack-Nifi resources**
+
+- Official documentation: [https://docs.cogstack.org/projects/nifi/en/latest/](https://docs.cogstack.org/projects/nifi/en/latest/)
+- GitHub: [https://github.com/CogStack/CogStack-NiFi](https://github.com/CogStack/CogStack-NiFi)
+- Documentation with deployment examples: [https://github.com/CogStack/CogStack-NiFi/tree/devel/deploy](https://github.com/CogStack/CogStack-NiFi/tree/devel/deploy)
+- Documentation on available services: [https://github.com/CogStack/CogStack-NiFi/tree/devel/services](https://github.com/CogStack/CogStack-NiFi/tree/devel/services)
+- DockerHub: [https://cloud.docker.com/repository/docker/cogstacksystems/cogstack-nifi](https://cloud.docker.com/repository/docker/cogstacksystems/cogstack-nifi)
+
+**Apache NiFi resources**
+
+- The official website: [https://nifi.apache.org/](https://nifi.apache.org/)
+- The official documentation: [https://nifi.apache.org/docs.html](https://nifi.apache.org/docs.html)
+
+---
+### Example deployment and services
+
+Please see [CogStack-NiFI example deployment with workflow examples](https://github.com/CogStack/CogStack-NiFi/tree/devel/deploy) .
diff --git a/docs/docs/data-engineering/nifi.md b/docs/docs/data-engineering/nifi.md
@@ -0,0 +1,5 @@
+# CogStack NiFi
+
+Apache NiFi integration for CogStack, providing fully configurable and scalable data flows with built-in monitoring, data provenance, and security features.
+
+Click [here](https://docs.cogstack.org/projects/nifi/) to access the full documentation.