Elasticsearch issues on an upgraded monolith

Developers
1

I'm trying to upgrade an existing server and all seems well at this point except Elasticsearch.
An observation:
A few documents (e.g.) assume Elasticsearch is listening on localhost - that doesn't seem to be the case with a default Monolith install. Mine is listening on the external ethernet adapter address, e.g. I can list indexes with curl -X GET "http://10.3.0.4:9200/_cat/indices" but not curl -X GET "http://localhost:9200/_cat/indices"

That aside, I'm having trouble in general with elasticsearch scripts. I'm trying to rebuild the scripts - I tried following this: Elasticsearch — CommCareHQ 1.0 documentation
I'm able to delete all indexes with curl -X DELETE http://10.3.0.4:9200/*
Any attempts to recreate indexes results in an error on the elasticsearch section:

I get similar errors from running ./manage.py elastic_sync_multiplexed start ${INDEX_CNAME} as per changelog 75:

https://commcare-cloud.readthedocs.io/en/latest/changelog/0075-reindex-all-indexes-for-es-upgrade.html

2

Hey Ed!

The errors that you linked in the Pastebin are because of the incompatible mappings. As you have seen in the changelog we are upgrading to ES5 so the mapping present in the latest code reflects the ES 5 mappings which are not compatible with ES2.

In the recent past we have updated how we would be managing the mappings on our environment. We are using django migrations to manage them. When you deleted all the indexes it messed up the situation a bit. Now you can create indexes with the current mappings on ES5. Although the migrations contain the information of the state, they can’t be applied again because as per django the mappings are already applied.

If you have backup of the ES2 data including all indexes that were deleted we would recommend to use that backup and get back to the last known good ES2 state.

If not, our recommendation is to upgrade to ES 5 and then populate the indexes.

You can follow the process outlined below, the following steps assume that then environment name is monolith and elasticsearch host is http://10.3.0.4:9200 based on the details shared on the question -

  • Login to control machine, replace with your environment name

    commcare-cloud monolith ssh control

  • Pull the latest changes

    git pull origin master

  • Start downtime

    commcare-cloud monolith downtime start

  • Update environments/monolith/public.yml with following values

    elasticsearch_version: 5.6.16
elasticsearch_download_sha256: 6b035a59337d571ab70cea72cc55225c027ad142fbb07fd8984e54261657c77f.

ELASTICSEARCH_MAJOR_VERSION: 5

ES_APPS_INDEX_MULTIPLEXED: False
ES_CASE_SEARCH_INDEX_MULTIPLEXED: False
ES_CASES_INDEX_MULTIPLEXED: False
ES_DOMAINS_INDEX_MULTIPLEXED: False
ES_FORMS_INDEX_MULTIPLEXED: False
ES_GROUPS_INDEX_MULTIPLEXED: False
ES_SMS_INDEX_MULTIPLEXED: False
ES_USERS_INDEX_MULTIPLEXED: False


ES_APPS_INDEX_SWAPPED: True
ES_CASE_SEARCH_INDEX_SWAPPED: True
ES_CASES_INDEX_SWAPPED: True
ES_DOMAINS_INDEX_SWAPPED: True
ES_FORMS_INDEX_SWAPPED: True
ES_GROUPS_INDEX_SWAPPED: True
ES_SMS_INDEX_SWAPPED: True
ES_USERS_INDEX_SWAPPED: True

  • Apply the updated settings

    commcare-cloud monolith update-config

  • Disable Elasticsearch shard allocation.It should already be disabled but for ensuring that its already disabled.

    curl -X PUT "http://10.3.0.4:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

  • Stop the Monit and Elasticsearch services

    commcare-cloud monolith run-shell-command elasticsearch "service monit stop" -b

commcare-cloud monolith run-shell-command elasticsearch "service elasticsearch stop" -b

  • Install and run the new version of Elasticsearch

    commcare-cloud monolith ansible-playbook deploy_stack.yml --limit=elasticsearch --tags=elasticsearch

  • Verify the permissions, ensure that both user and group are set to elasticsearch

    commcare-cloud  monolith run-shell-command  elasticsearch  "ls -al  /opt/data/elasticsearch-5.6.16" -b

  • Verify Cluster health is green

    curl -XGET "http://10.3.0.4:9200/_cluster/health?pretty"

  • Create indices and populate data in them

    ./manage.py ptop_preindex --reset

  • End Downtime

    commcare-cloud monolith downtime end

  • Verify that Commcare is running fine and functionality dealing with elasticsearch like reports are working properly.

Hope this helps. If you run into any other issues feel free to reach out.

Thanks

1 Like
3

Amit, thank you for the response, I have the original server with the ES 2 data structure intact so I have options. Thanks for the explanation, that makes sense. I'll try these suggestions and provide feedback,
Cheers

Ed