fix: chunk large Docker container lists to prevent WebSocket message loss#304
Open
jaydeep-pipaliya wants to merge 2 commits intofosrl:mainfrom
Open
fix: chunk large Docker container lists to prevent WebSocket message loss#304jaydeep-pipaliya wants to merge 2 commits intofosrl:mainfrom
jaydeep-pipaliya wants to merge 2 commits intofosrl:mainfrom
Conversation
…loss When more than ~20 Docker containers are running, the WebSocket message containing the full container list can be too large and get dropped by intermediary proxies (e.g., Traefik), causing the Docker Container View in Pangolin to show nothing. Split container lists larger than 15 items into multiple chunked messages with chunkIndex/totalChunks metadata. The Pangolin server reassembles chunks before processing. Small lists (<=15) are sent in a single message for backward compatibility with older Pangolin versions. Fixes fosrl/pangolin#2117
Each chunked batch now includes a unique batchId (reusing generateChainId) so the server can distinguish interleaved sends — e.g., a manual fetch and a Docker event firing at the same time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Companion PR for fosrl/pangolin#2117 — Docker Container View not displaying when >20 containers are running.
Problem
gorilla/websocket.WriteJSONserializes the full container list into a single WebSocket text frame. With 55+ containers (each 1-5KB of JSON metadata), the resulting frame can be 55-275KB. Intermediary proxies (Traefik, nginx, Cloudflare tunnels) can silently drop or truncate frames this large, causing the pangolin server to never receive the container data.Solution
Added
sendContainerList()that automatically chunks large container lists:Why
batchId?Two concurrent container sends can happen when a manual fetch request and a Docker event fire at the same time. Without
batchId, their chunks would interleave and corrupt the accumulated data on the server. Each batch gets a unique ID (reusing the existinggenerateChainIdhelper) so the server can track and supersede batches correctly.Why chunk size of 15?
Each container with full metadata (labels, ports, networks) serializes to 1-5KB. 15 containers ≈ 15-75KB per WebSocket frame — safely under common proxy defaults (Traefik default buffer: 64KB, nginx: 64KB). The threshold was chosen to balance message count vs. frame size.
Changes
main.go:sendContainerList()— chunks large lists, passes small lists through unchangedgenerateChainId()for batch IDs — no new dependenciesCompanion PR
Pangolin server: fosrl/pangolin#2817
batchIdtracking, input validation, typed accumulator, 120s TTL on partial chunksTesting
go buildpasseschunkIndex/totalChunks/batchIdbatchId, server handles superseding