Recreating elasticsearch indexes from scratch - ptop_es_manage?

erobinson · October 29, 2023, 8:12am

I'm trying to recreate indexes for elasticsearch but the ptop_es_manage command seems absent. Is there another command / script I should be using to recreate them?

I was trying:

python manage.py ptop_es_manage --code_red
python manage.py ptop_preindex --reset

as per this: Restoring form data - #8 by Simon_Kelly

Also, If I run the below command to just do forms and cases:

python manage.py ptop_reindexer_v2 sql-case --cleanup

I get:

Are you sure you want to delete the current index (if it exists)? y/n
y
2023-10-29 14:29:40,000 INFO es_utils Initializing elasticsearch index for [case]
2023-10-29 14:29:40,001 INFO [corehq.apps.es.migration_operations] Creating Elasticsearch index: hqcases_2016-03-04
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/manage.py", line 190, in <module>
    main()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/manage.py", line 48, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_
command_line
    utility.execute()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/hqcase/management/commands/ptop_reindexer_v2.py", line 68, in run_from_argv
    super(SubCommand, self).run_from_argv(argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/hqcase/management/commands/ptop_reindexer_v2.py", line 140, in handle
    reindexer.reindex()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 269, in reindex
    prepare_index_for_reindex(self.adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 153, in prepare_index_for_reindex
    initialize_index_and_mapping(adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/es_utils.py", line 35, in initialize_index_and_mapping
    initialize_index(adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/es_utils.py", line 40, in initialize_index
    CreateIndex(
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/es/migration_operations.py", line 91, in run
    manager.index_create(self.name, self.render_index_metadata(
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/es/client.py", line 220, in index_create
    self._es.indices.create(index, metadata)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/client/indices.py", line 109, in create
    return self.transport.perform_request('PUT', _make_path(index),
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/connection/http_urllib3.py", line 110, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/connection/base.py", line 114, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch2.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'No handler for type [keyword] declared on field [owner_id]')

...and:

python manage.py ptop_reindexer_v2 sql-form --cleanup

I receive this error:

Are you sure you want to delete the current index (if it exists)? y/n
y
2023-10-29 14:30:12,146 INFO es_utils Initializing elasticsearch index for [xform]
2023-10-29 14:30:12,146 INFO [corehq.apps.es.migration_operations] Creating Elasticsearch index: xforms_2016-07-07
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/manage.py", line 190, in <module>
    main()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/manage.py", line 48, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/hqcase/management/commands/ptop_reindexer_v2.py", line 68, in run_from_argv
    super(SubCommand, self).run_from_argv(argv)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/hqcase/management/commands/ptop_reindexer_v2.py", line 140, in handle
    reindexer.reindex()
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 269, in reindex
    prepare_index_for_reindex(self.adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 153, in prepare_index_for_reindex
    initialize_index_and_mapping(adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/es_utils.py", line 35, in initialize_index_and_mapping
    initialize_index(adapter)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/ex-submodules/pillowtop/es_utils.py", line 40, in initialize_index
    CreateIndex(
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/es/migration_operations.py", line 91, in run
    manager.index_create(self.name, self.render_index_metadata(
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/corehq/apps/es/client.py", line 220, in index_create
    self._es.indices.create(index, metadata)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/client/indices.py", line 109, in create
    return self.transport.perform_request('PUT', _make_path(index),
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/connection/http_urllib3.py", line 110, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/cchq/www/monolith/releases/2023-10-29_14.00/python_env-3.9/lib/python3.9/site-packages/elasticsearch2/connection/base.py", line 114, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch2.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'No handler for type [text] declared on field [@uiVersion]')

Am I doing something wrong?
Thanks!

Ethan_Soergel · October 30, 2023, 1:48pm

I haven't done ES management in a while so I'll refrain from offering specific advice (please feel free, anyone else reading this).

However, I did recently see this document created, which I think should have answers to your questions:

github.com

dimagi/commcare-hq/blob/master/corehq/apps/es/REINDEX_PROCESS.md

# Elasticsearch Reindex Process

This document outlines the process for reindexing an Elasticsearch index in CommCare HQ. The process would typically involve:
1. Creating a secondary index via migrations. ES migrations can be created by following [this](./README.rst#creating-elasticsearch-index-migrations) guide.
2. Updating [Secondary Index Names](./const.py#L27-L57) with the new secondary index names.
3. Multiplexing the HQ index adapters, so that data is written to both primary and secondary indices simultaneously using ElasticSyncMultiplexer.
4. Reindexing the secondary index using the `elastic_sync_multiplexed` utility.
5. Swapping the indices.
6. Turning off the multiplexer
7. Deleting the older index.
8. Updating [Index Names](https://github.com/dimagi/commcare-hq/blob/26ddc8f18f9a1c60c2aae6e09ecea4c4e6647758/corehq/apps/es/const.py#L27-L57), and setting primary index name to secondary index name and secondary index name to None.


### Prerequisites:

1. Ensure that there is enough free space available in your cluster. The command to estimate disk space required for reindexing is:

```sh
cchq <env> django-manage elastic_sync_multiplexed estimated_size_for_reindex
```

This file has been truncated. show original