FibMem

A benchmark designed to evaluate the Fiduciary Responsibility of agent memory systems.

Project Structure

FibMem/
├── dataset/              # 100 personas, each with dialogues + QA
│   └── <Persona Name>/
│       ├── dialogues_64K.json
│       └── qa.json
├── src/
│   ├── adapters/         # Memory system adapters (mem0, memos, memu, ...)
│   ├── core/             # Pipeline, data models, loaders, answerer, evaluator
│   └── utils/            # Config utilities
├── config/
│   ├── systems/          # Per-system YAML configs
│   └── prompts.yaml      # Prompt templates for answer & evaluate stages
├── cli.py                # Single-user CLI
├── batch_add.py          # Batch upload dialogues to memory systems
└── batch_evaluate.py     # Batch search → answer → evaluate pipeline

Setup

pip install -r requirements.txt
cp .env.template .env  # Fill in API keys

Usage

1. Add dialogues to a memory system

# Single user
python cli.py --system mem0 --dialogue dataset/Arin\ Dev\ Mital/dialogues_64K.json \
    --user-id "Arin Dev Mital" --stages add

# Batch (all personas)
python batch_add.py --systems mem0 memos memu

2. Evaluate

# Batch evaluate all personas
python batch_evaluate.py --systems mem0 memos memu --stages all

# Run stages independently
python batch_evaluate.py --stages search
python batch_evaluate.py --stages answer
python batch_evaluate.py --stages evaluate

3. LLM baseline (no memory system)

python batch_evaluate.py --systems llm --llm-name gpt-4o-mini --stages all

Both batch_add.py and batch_evaluate.py support --resume for checkpoint recovery and --dry-run for previewing operations. Run with --help for full options.

License

See LICENSE.