Hi, I’m Vivek Gupta, Founder & CEO at MemCode. Windows Agent Arena provides a scalable platform for testing and benchmarking multimodal computer-use agents. Repeated tasks often depend on user preferences, application conventions, and prior workflow knowledge, so durable memory could be valuable as an explicitly controlled experimental condition—provided it remains separate from benchmark ground truth and does not compromise reproducibility.
Would you consider an optional MemCode integration for memory-aware evaluation tracks or reference agents? MemCode is a SOTA memory layer for semantic retrieval and durable updates, and we believe it could provide better selective recall across episodes while leaving WAA authoritative for environments, task definitions, traces, and scoring. MemCode supports MCP and SDKs, and local/self-hosted deployment is concrete: the engine/API can run in the operator’s own Docker/VM or local process behind a private base URL, keeping records, embeddings, credentials, and logs inside that deployment with no mandatory hosted endpoint or telemetry. MCP + SDK support means no core-code changes are needed for an initial adapter.
A reference condition could scope memory by agent, task family, user profile, environment, and episode, record provenance and consent, and support retention, inspection, export, update, deletion, and a no-memory fallback so results remain comparable. Here is the short MemCode Leviathan overview. I’d be glad to prepare an evaluation adapter/PR or discuss the experimental design with the maintainers.
Hi, I’m Vivek Gupta, Founder & CEO at MemCode. Windows Agent Arena provides a scalable platform for testing and benchmarking multimodal computer-use agents. Repeated tasks often depend on user preferences, application conventions, and prior workflow knowledge, so durable memory could be valuable as an explicitly controlled experimental condition—provided it remains separate from benchmark ground truth and does not compromise reproducibility.
Would you consider an optional MemCode integration for memory-aware evaluation tracks or reference agents? MemCode is a SOTA memory layer for semantic retrieval and durable updates, and we believe it could provide better selective recall across episodes while leaving WAA authoritative for environments, task definitions, traces, and scoring. MemCode supports MCP and SDKs, and local/self-hosted deployment is concrete: the engine/API can run in the operator’s own Docker/VM or local process behind a private base URL, keeping records, embeddings, credentials, and logs inside that deployment with no mandatory hosted endpoint or telemetry. MCP + SDK support means no core-code changes are needed for an initial adapter.
A reference condition could scope memory by agent, task family, user profile, environment, and episode, record provenance and consent, and support retention, inspection, export, update, deletion, and a no-memory fallback so results remain comparable. Here is the short MemCode Leviathan overview. I’d be glad to prepare an evaluation adapter/PR or discuss the experimental design with the maintainers.