Celery Blocked queues error commcareHQ

Developers
1

Hello good people,

Ran into an error celery has blocked queues tried to run solutions provided by the firefighting celery when queues are blocked to no avail. Error message that is being logged in the celerybeat.log is cannot add item to database PANIC fatal region error detected; run recovery.

I have attached a screenshot from reference.

Your assistance will be greatly appreciated.

Fredrick

Celery
Celery3829×1992 361 KB

2

Hi Fredrick

I think this is due to an issue with the Python runtime. I would suggest rebuilding the virtualenv.

I have not tested these steps but I think this should work:

# stop all CommCare services on the machine
sudo supervisor-ctl stop all

# change directory to the current release
cd /home/cchq/www/<env>/current

# rename the current virtualenv
mv python_env-3.9 python_env-3.9.old

# create new virtualenv
virtualenv -p python3.9 python_env-3.9
./python_env-3.9/bin/pip install -r requirements/prod-requirements.txt

# update symlink
ln -s -f python_env-3.9 python_env

# restart services
sudo supervisor-ctl start all

See also: _dbm.error: cannot add item to database (celery 4.3.0, python 3.7) · Issue #5447 · celery/celery · GitHub

3

Thank very much Simon. Unfortunately I am getting additional errors. Due to my urgency I have resorted to tearing down the whole server and rebuilding it from scratch.

Thank you for your assistance. I will try to recreate the issue in a test environment and follow it through because it might happen again and I will need a ready solution.

Regards,