Skip to content

perf: restore lost elasticsearch and redpanda memory limits - #307

Open
dargmuesli wants to merge 2 commits into
mainfrom
fix/swarm/restore-memory-limits
Open

dargmuesli wants to merge 2 commits into
mainfrom
fix/swarm/restore-memory-limits

Conversation

@dargmuesli

Copy link
Copy Markdown
Member

Elasticsearch's heap and Redpanda's memory flags were deliberately capped in 6a6ac65 (2026-07-23), but the dargstack v4 migration restored both compose files from a point in history before that tuning, so the caps were lost silently. Redpanda has since run without any memory limit at all; its Seastar runtime reserves most of the host's memory by default, which is a likely contributor to the reccoom Kafka connectivity errors seen in Sentry on 2026-09-04, 2026-09-07, and 2026-09-13. This restores the original limits verbatim from 6a6ac65, rebased onto the current compose structure: elasticsearch's heap back to 512m/512m with 1024m/512m container limits, redpanda's --memory 512M --smp 1 --reserve-memory 0M with 1024M/512M container limits.

@dargmuesli

Copy link
Copy Markdown
Member Author

Locally load-tested both services against a real Docker Swarm deployment (linux/arm64, not emulated) before merging.

Redpanda: /proc/1/cmdline confirms --smp=1 --memory=512M --reserve-memory=0M are applied. Idle usage: 193.5MiB / 1GiB (18.9%). Comfortable headroom.

Elasticsearch: heap confirmed at 512mb via its own boot log. Sent a single 118.7MB bulk request (matching the exact scenario in the existing http.max_content_length comment: the helpdesk rebuilding its search index) against both limit values for comparison.

  • At limits.memory: 1024m (the value restored verbatim from the original commit): peaked at 1011MiB / 1GiB, 98.69%. Survived, but with no safety margin and zero warning in the Elasticsearch log, since its own circuit breakers watch JVM heap against -Xmx, not container RSS against the cgroup limit. A kernel OOM-kill under this config would be silent from Elasticsearch's perspective.
  • At limits.memory: 1536m: peaked at 1.289GiB / 1.5GiB, 85.93%, ~210MB of headroom.

Bumped the limit to 1536m in the latest commit based on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant