I case you’re stuck with Celery for a while: I was hit with this same problem, and solved it by adding a sidecar HTTP server thread to the Python workers that would expose metrics written by the workers into a multithreaded registry. This has been working amazingly well in production for over two years now, and makes it really straightforward to get custom metrics out of a distributed Celery app.