Having couchdb troubles with upgrade to Ubuntu 22.04

Developers
1

This step in the instructions for upgrading to Ubuntu 22.04 is not clear to me:

https://commcare-cloud.readthedocs.io/en/latest/changelog/0071-upgrade_to_ubuntu_22.04.html#prepare

The old machine has couchdb version 2.3.1
The new machine has couchdb version 3.3.1

Is the old machine meant to be upgraded to 3.3.1 before the backup? Or is the 2.3.1 backup going to be compatible / migrated when it's transferred to the new machine?

Thanks in advance!

2

Hi Ed,

The 2.3.1 backup should be entirely compatible with 3.3.1 as well. This is the recommended upgrade procedure. For example in 1.9. Upgrading from prior CouchDB releases — Apache CouchDB® 3.3 Documentation you can see that they recommend simply stopping couch and upgrading it in place, and starting it again. In your case it sounds like it'll be on a new machine, but the same concept holds: the data from 2.3.1 should be compatible with 3.3.1.

As in any major maintenance, make sure you keep a clean copy of the original data in case there are any issues with the upgrade and you need to roll back.

Cheers,
Danny

1 Like
3

Excellent, thanks for confirming that @dannyroberts
I'll give that a go and comment here.

Ed

4

@dannyroberts I'm unable to restore couchDB after a server upgrade. I'm able to do everything to the point of restoring data. PostgreSQL and blobdb seem to restore fine, but after restoring couchDB, check_services reports:

EXCEPTION (Took 0.01s) couch : Service check errored with exception 'HTTPError('500 Server Error: Internal Server Error for url: http://127.0.0.1:35984/commcarehq__apps')'

If I curl that address, I get:

{"error":"internal_server_error","reason":"No DB shards could be opened.","ref":3096826546}

The couchdb server itself appears up, if I curl http://127.0.0.1:35984 I get:

{"couchdb":"Welcome","version":"3.3.1","git_sha":"1fd50b82a","uuid":"97343bbbe857577cb85edd4525cafb0b","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

an ls -l of /opt/data/couchdb shows:

-rw-rw-r--  1 couchdb couchdb 61671 Oct 25 18:43 _dbs.couch
-rw-r--r--  1 couchdb couchdb  8388 Oct 24 19:03 _nodes.couch
-rw-rw-r--  1 couchdb couchdb 20703 Oct 25 18:43 _replicator.couch
-rw-rw-r--  1 couchdb couchdb 12482 Jun 18  2021 _users.couch
drwxrwxr-x 10 couchdb couchdb  4096 Jun 18  2021 shards

and /opt/data/couchdb/shards:

drwxrwxr-x 2 couchdb couchdb 4096 Oct 18 17:51 00000000-1fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:31 20000000-3fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:39 40000000-5fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:55 60000000-7fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:43 80000000-9fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:35 a0000000-bfffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:59 c0000000-dfffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 18 17:47 e0000000-ffffffff

Those subdirs appear fine on the surface, e.g. /opt/data/couchdb2/shards/00000000-1fffffff:

-rw-rw-r-- 1 couchdb couchdb     90311 Oct 19 14:06 _global_changes.1624022307.couch
-rw-r--r-- 1 couchdb couchdb     57539 Oct 24 19:21 _global_changes.1698174227.couch
-rw-rw-r-- 1 couchdb couchdb      8348 Jun 18  2021 _replicator.1624022306.couch
-rw-r--r-- 1 couchdb couchdb      8349 Oct 24 19:03 _replicator.1698174226.couch
-rw-rw-r-- 1 couchdb couchdb      8348 Jun 18  2021 _users.1624022306.couch
-rw-r--r-- 1 couchdb couchdb      8349 Oct 24 19:03 _users.1698174225.couch
-rw-rw-r-- 1 couchdb couchdb     61652 Oct 18 02:16 commcarehq.1624025546.couch
-rw-r--r-- 1 couchdb couchdb     41124 Oct 24 19:17 commcarehq.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 111534286 Oct 10 13:42 commcarehq__apps.1624025549.couch
-rw-r--r-- 1 couchdb couchdb     69840 Oct 24 19:21 commcarehq__apps.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb      8348 Jun 18  2021 commcarehq__auditcare.1624025545.couch
-rw-rw-r-- 1 couchdb couchdb     86222 Jun  9 11:13 commcarehq__domains.1624025549.couch
-rw-r--r-- 1 couchdb couchdb     32964 Oct 24 19:21 commcarehq__domains.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 677601501 Dec 31  2022 commcarehq__fixtures.1624025550.couch
-rw-rw-r-- 1 couchdb couchdb      8348 Jun 18  2021 commcarehq__fluff-bihar.1624025558.couch
-rw-rw-r-- 1 couchdb couchdb      8348 Jun 18  2021 commcarehq__m4change.1624025558.couch
-rw-rw-r-- 1 couchdb couchdb    819418 Oct 15 04:00 commcarehq__meta.1624025550.couch
-rw-r--r-- 1 couchdb couchdb     78032 Oct 24 19:21 commcarehq__meta.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb     98504 Feb  4  2023 commcarehq__receiverwrapper.1624025558.couch
-rw-r--r-- 1 couchdb couchdb     20679 Oct 24 19:21 commcarehq__receiverwrapper.1698175056.couch
-rw-rw-r-- 1 couchdb couchdb    495828 Oct  3 09:54 commcarehq__users.1624025550.couch
-rw-r--r-- 1 couchdb couchdb     61648 Oct 24 19:21 commcarehq__users.1698175054.couch

The couch data was backed up from the old server on Ubuntu 18.04 (with CouchDB 2.x)
The new server is Ubuntu 20.04 (CouchDB 3.x)
This issue appears to be the same as this user experienced: Upgrade to Ubuntu 22.04 couch service error after couchdb restoration - #5 by fchiyenda

Help would be greatly appreciated!