Gunicorn/multiprocess with restarting workers, .db files are not cleaned up

[Dominick-Peluso-Bose]

We are using Gunicorn with multiple workers with the Gunicorn max_requests option. We found that memory usage in our containers increases with every worker restart. By reducing the max_requests value so workers restart very frequently, the problem became apparent.

Using the method described in the readme:

from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

While running a performance test, I see the following in the container:

[root@mock-adapter-8568fdcf6d-lz8tx app]# cd /app/prometheus_tmp
[root@mock-adapter-8568fdcf6d-lz8tx prometheus_tmp]# ls
counter_10.db   counter_214.db  counter_319.db  counter_41.db   counter_91.db     histogram_196.db  histogram_296.db  histogram_401.db  histogram_69.db
counter_111.db  counter_222.db  counter_31.db   counter_425.db  counter_97.db     histogram_1.db    histogram_305.db  histogram_409.db  histogram_77.db
counter_117.db  counter_232.db  counter_328.db  counter_433.db  counter_9.db      histogram_202.db  histogram_311.db  histogram_419.db  histogram_84.db
counter_130.db  counter_23.db   counter_338.db  counter_444.db  histogram_10.db   histogram_214.db  histogram_319.db  histogram_41.db   histogram_91.db
counter_138.db  counter_244.db  counter_345.db  counter_450.db  histogram_111.db  histogram_222.db  histogram_31.db   histogram_425.db  histogram_97.db
counter_148.db  counter_251.db  counter_353.db  counter_458.db  histogram_117.db  histogram_232.db  histogram_328.db  histogram_433.db  histogram_9.db
counter_154.db  counter_258.db  counter_361.db  counter_470.db  histogram_130.db  histogram_23.db   histogram_338.db  histogram_444.db
counter_175.db  counter_269.db  counter_36.db   counter_48.db   histogram_138.db  histogram_244.db  histogram_345.db  histogram_450.db
counter_187.db  counter_277.db  counter_374.db  counter_56.db   histogram_148.db  histogram_251.db  histogram_353.db  histogram_458.db
counter_18.db   counter_286.db  counter_384.db  counter_61.db   histogram_154.db  histogram_258.db  histogram_361.db  histogram_470.db
counter_196.db  counter_296.db  counter_401.db  counter_69.db   histogram_175.db  histogram_269.db  histogram_36.db   histogram_48.db
counter_1.db    counter_305.db  counter_409.db  counter_77.db   histogram_187.db  histogram_277.db  histogram_374.db  histogram_56.db
counter_202.db  counter_311.db  counter_419.db  counter_84.db   histogram_18.db   histogram_286.db  histogram_384.db  histogram_61.db

Watching that directory, new files are created with every new worker and old ones are not being cleaned up. Deleting the files in that directory reduce down the memory usage on that container, which then starts building again.

What is the proper way of dealing with these database files? Is this mark_process_dead's responsibility?

And a general question about Prometheus: do we need to keep metrics around after they are collected? Could we wipe our metrics after the collector hits our metrics endpoint? If so, can this be done via the prometheus_client?

[brian-brazil]

This is the expected behaviour, you can only delete these safely when the whole application restarts. I'd suggest increasing max_requests.

[Dominick-Peluso-Bose]

I'd suggest increasing max_requests.

We are using max_requests to deal with some memory leaks introduced with Gunicorn with gevent workers. Not ideal at all.

But even if we didn't have this problem, workers can restart for other reasons. Is it also expected that memory usage would increase like this? Because if so, it seems like eventually over the lifetime of the service, at some point all services using this pattern would run out of memory.

[brian-brazil]

Memory usage for workers should be constant.

[mpitt]

We have the same issue in our uWSGI deployment. We also restart workers every N requests, and they could restart by themselves because of harakiri (if the request takes too long). I don't think we could just keep the same workers forever, some sneaky memory leak might persist and max-requests is widely used to mitigate it.

Is it at all possible that the prometheus client would support this use case? Or should we just look for another solution?

[brian-brazil]

There's not much we can do here if you have high churn I'm afraid.

[Dominick-Peluso-Bose]

We disabled worker restarts and we found that using Gunicorn and Flask, once in a while, we do have unplanned worker restarts infrequently. So for us, we were able to reduce the large memory leak and now have a very slow one that grows over the course of months.

[jkemp101]

Couldn't mark_process_dead() summarize the data into a single "dead" file so that there wasn't a need to leave a bunch of files hanging around? Are the values in each individual file needed after the process dies?

[brian-brazil]

That'd require locking across processes, which we want to avoid, and cause issues if that pid was reused.

[jkemp101]

I think locking across processes in mark_process_dead() and/or collect() would probably be acceptable as long as it doesn't require any locking for "active" processes to record their samples. But of course this would get tricky.

I guess reusing pids is another issue if it mixes samples in the same pid file when it should actually be a true unique file per process. That would cause potential loss of data like a min value for a Gauge.

[bloodearnest]

Hey folks.

We have a workaround for this currently which uses gunicorn's child_exit hook to merge the dead worker's counter/histogram metrics into archive files and deletes the pid file. This has kept our file count and thus scrape response times stable. The attached image shows our scrape times (yellow line) when we deployed our workaround.

Our thinking was that using child_exit avoids the need to lock, as it's run only in the master. We still call the current mark_process_dead() function too, so gauge files are removed (although we are not heavy users of gauges).

Regards pid recycling, I think it is only an issue for gauges, which are still deleted anyway in our experiment. Also, given filenames with the pid in should be deleted very soon after the worker death, the chance of a recycling pid clash is very remote? I may have missed something though...

It seems to be working for us, but I would appreciate any feedback on anything we may have missed.

Our experiment involves a slightly altered version of the MultiProcessCollector.collect() function, which you can see here: https://github.com/canonical-ols/talisker/blob/master/talisker/metrics.py#L193

The changes are documented there, but for convenience are copied here:

This almost verbatim from MultiProcessCollector.collect().

The original collects all results in a format designed to be scraped. We
instead need to collect limited results, in a format that can be written
back to disk. To facilitate this, this version of collect() preserves label
ordering, and does not aggregate the histograms.

Specifically, it differs from the original:

1. it takes its files as an argument, rather than hardcoding '*.db'
2. it does not accumulate histograms
3. it preserves label order, to facilitate being inserted back into an mmap file.

It needs to be kept up to date with changes to prometheus_client as much as
possible, or until changes are landed upstream to allow reuse of collect().

We'd like to create a PR that upstreams some of this work, if you think the approach is sensible? Specifically, we'd like to slightly refactor the collect() method to support preserving label order, allow for not accumulating histograms and not hardcoding '*.db'. This would allow us to reuse it, rather than reimplement it.

Also, if this approach is deemed worthwhile, then we'd be happy to work on a PR to expand to include this aggregation as appropriate.

Thanks for reading!

EDIT: typo

[brian-brazil]

Our thinking was that using child_exit avoids the need to lock, as it's run only in the master.

That's not a good assumption. For example if another worker is started up and gets the old pid, that'll break things. We also need this to work on things that aren't Gunicorn.

For this to work, you'd need a lock of some form.

refactor the collect() method to support preserving label order

Does it not already? That's usually something we have.

allow for not accumulating histograms

That violates the Prometheus text format, histograms must be cumulative.

[jkemp101]

Isn't the current approach already broken if the new worker gets the same PID? How does it currently keep the values for the old and new processes if the PID is the same? Won't the new process start overwriting the old process's values since it uses the same file. Just trying to understand this better to figure out potential workarounds.

[brian-brazil]

Isn't the current approach already broken if the new worker gets the same PID?

No, it's designed to handle it. It reads in the previous values at startup.