This step in the instructions for upgrading to Ubuntu 22.04 is not clear to me:
https://commcare-cloud.readthedocs.io/en/latest/changelog/0071-upgrade_to_ubuntu_22.04.html#prepare
The old machine has couchdb version 2.3.1
The new machine has couchdb version 3.3.1
Is the old machine meant to be upgraded to 3.3.1 before the backup? Or is the 2.3.1 backup going to be compatible / migrated when it's transferred to the new machine?
Thanks in advance!
Hi Ed,
The 2.3.1 backup should be entirely compatible with 3.3.1 as well. This is the recommended upgrade procedure. For example in 1.9. Upgrading from prior CouchDB releases — Apache CouchDB® 3.3 Documentation you can see that they recommend simply stopping couch and upgrading it in place, and starting it again. In your case it sounds like it'll be on a new machine, but the same concept holds: the data from 2.3.1 should be compatible with 3.3.1.
As in any major maintenance, make sure you keep a clean copy of the original data in case there are any issues with the upgrade and you need to roll back.
Cheers,
Danny
@dannyroberts I'm unable to restore couchDB after a server upgrade. I'm able to do everything to the point of restoring data. PostgreSQL and blobdb seem to restore fine, but after restoring couchDB, check_services reports:
EXCEPTION (Took 0.01s) couch : Service check errored with exception 'HTTPError('500 Server Error: Internal Server Error for url: http://127.0.0.1:35984/commcarehq__apps')'
If I curl that address, I get:
{"error":"internal_server_error","reason":"No DB shards could be opened.","ref":3096826546}
The couchdb server itself appears up, if I curl http://127.0.0.1:35984 I get:
{"couchdb":"Welcome","version":"3.3.1","git_sha":"1fd50b82a","uuid":"97343bbbe857577cb85edd4525cafb0b","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
an ls -l of /opt/data/couchdb shows:
-rw-rw-r-- 1 couchdb couchdb 61671 Oct 25 18:43 _dbs.couch
-rw-r--r-- 1 couchdb couchdb 8388 Oct 24 19:03 _nodes.couch
-rw-rw-r-- 1 couchdb couchdb 20703 Oct 25 18:43 _replicator.couch
-rw-rw-r-- 1 couchdb couchdb 12482 Jun 18 2021 _users.couch
drwxrwxr-x 10 couchdb couchdb 4096 Jun 18 2021 shards
and /opt/data/couchdb/shards:
drwxrwxr-x 2 couchdb couchdb 4096 Oct 18 17:51 00000000-1fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:31 20000000-3fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:39 40000000-5fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:55 60000000-7fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:43 80000000-9fffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:35 a0000000-bfffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 19 17:59 c0000000-dfffffff
drwxrwxr-x 2 couchdb couchdb 4096 Oct 18 17:47 e0000000-ffffffff
Those subdirs appear fine on the surface, e.g. /opt/data/couchdb2/shards/00000000-1fffffff:
-rw-rw-r-- 1 couchdb couchdb 90311 Oct 19 14:06 _global_changes.1624022307.couch
-rw-r--r-- 1 couchdb couchdb 57539 Oct 24 19:21 _global_changes.1698174227.couch
-rw-rw-r-- 1 couchdb couchdb 8348 Jun 18 2021 _replicator.1624022306.couch
-rw-r--r-- 1 couchdb couchdb 8349 Oct 24 19:03 _replicator.1698174226.couch
-rw-rw-r-- 1 couchdb couchdb 8348 Jun 18 2021 _users.1624022306.couch
-rw-r--r-- 1 couchdb couchdb 8349 Oct 24 19:03 _users.1698174225.couch
-rw-rw-r-- 1 couchdb couchdb 61652 Oct 18 02:16 commcarehq.1624025546.couch
-rw-r--r-- 1 couchdb couchdb 41124 Oct 24 19:17 commcarehq.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 111534286 Oct 10 13:42 commcarehq__apps.1624025549.couch
-rw-r--r-- 1 couchdb couchdb 69840 Oct 24 19:21 commcarehq__apps.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 8348 Jun 18 2021 commcarehq__auditcare.1624025545.couch
-rw-rw-r-- 1 couchdb couchdb 86222 Jun 9 11:13 commcarehq__domains.1624025549.couch
-rw-r--r-- 1 couchdb couchdb 32964 Oct 24 19:21 commcarehq__domains.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 677601501 Dec 31 2022 commcarehq__fixtures.1624025550.couch
-rw-rw-r-- 1 couchdb couchdb 8348 Jun 18 2021 commcarehq__fluff-bihar.1624025558.couch
-rw-rw-r-- 1 couchdb couchdb 8348 Jun 18 2021 commcarehq__m4change.1624025558.couch
-rw-rw-r-- 1 couchdb couchdb 819418 Oct 15 04:00 commcarehq__meta.1624025550.couch
-rw-r--r-- 1 couchdb couchdb 78032 Oct 24 19:21 commcarehq__meta.1698175054.couch
-rw-rw-r-- 1 couchdb couchdb 98504 Feb 4 2023 commcarehq__receiverwrapper.1624025558.couch
-rw-r--r-- 1 couchdb couchdb 20679 Oct 24 19:21 commcarehq__receiverwrapper.1698175056.couch
-rw-rw-r-- 1 couchdb couchdb 495828 Oct 3 09:54 commcarehq__users.1624025550.couch
-rw-r--r-- 1 couchdb couchdb 61648 Oct 24 19:21 commcarehq__users.1698175054.couch
The couch data was backed up from the old server on Ubuntu 18.04 (with CouchDB 2.x)
The new server is Ubuntu 20.04 (CouchDB 3.x)
This issue appears to be the same as this user experienced: Upgrade to Ubuntu 22.04 couch service error after couchdb restoration - #5 by fchiyenda
Help would be greatly appreciated!