TasQ worker prevents graceful uWSGI shutdown for 60 seconds

[thomaswoeckinger]

Description

Stopping or restarting grommunio-admin-api.service consistently takes about 62 seconds when TasQ is enabled.

uWSGI eventually kills the worker after its 60-second graceful shutdown timeout:

SIGINT/SIGTERM received...killing workers...
worker 1 (...) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
goodbye to uWSGI.

Environment

• grommunio-admin-api: 1.20.45.m49b16fc-70.2
• uWSGI: 2.0.30
• Python: 3.12.14
• Native EL10 installation
• TasQ configuration: enabled, one worker
• TasQ status before shutdown:

{"queued":0,"running":true,"workers":1}

There were no queued TasQ tasks during the test.

Reproduction

1. Start grommunio-admin-api.service with the default TasQ configuration.
2. Run: systemctl stop grommunio-admin-api.service
3. The stop takes approximately 62 seconds.
4. Configure:

tasq:
disabled: true

5. Start the service and stop it again.

With TasQ disabled, the same stop completes in approximately 1.02 seconds.

Python thread stacks during the blocked shutdown

The stacks were collected using uWSGI's py-tracebacker while the service was stopping.

thread_id = uWSGIWorker1Core0
tools/tasq.py line 444 in stop
proc.join(...)
threading.py in join
threading.py in _wait_for_tstate_lock

thread_id = TasQ Worker
tools/tasq.py line 97 in run
self._queued.get()
queue.py in get
threading.py in wait

thread_id = Thread-1 (_process)
tools/tasq.py line 545 in _process
cls._finished.get()
queue.py in get
threading.py in wait

TasQServer.stop() adds an exit control task to _queued and then waits indefinitely at:

https://github.com/grommunio/admin-api/blob/master/tools/tasq.py#L418-L446

During the uWSGI shutdown, the TasQ worker remains blocked in _queued.get():

https://github.com/grommunio/admin-api/blob/master/tools/tasq.py#L95-L100

The exit task is therefore not processed, and proc.join() never returns. uWSGI finally kills the worker after its 60-second mercy timeout.

TasQ is registered as the uWSGI exit callback here:

https://github.com/grommunio/admin-api/blob/master/main.py#L36-L44

Expected behavior

The TasQ worker and clerk threads should receive their exit notifications and terminate promptly, allowing the Admin API service to stop within a few seconds.

Alternatively, the shutdown joins should have a bounded timeout so that a blocked TasQ thread cannot delay every restart or system shutdown by 60 seconds.