SearchParseException[No mapping found for [username.exact]

Hi,

am getting the following error while navigating to web user list on dashboard.

on elasticsearch log the following error displayed

tail -f /opt/data/elasticsearch-2.4.6/logs/covid-es.log

Caused by: SearchParseException[failed to parse search source [{"query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "$
ovid19"}}, {"term": {"domain_memberships.domain.exact": "covid19"}}]}, {"term": {"doc_type": "WebUser"}}, {"term": {"base_doc": "couchuser"}},
{"term": {"is_active": true}}]}, "query": {"match_all": {}}}}, "sort": [{"username.exact": {"order": "asc"}}], "from": 0, "size": 5}]]; neste
d: SearchParseException[No mapping found for [username.exact] in order to sort on];
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:873)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:667)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportActi
on.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAct$
on.java:365)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: SearchParseException[No mapping found for [username.exact] in order to sort on]
at org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:213)
at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:187)
at org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:85)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:856)
... 12 more

and this is the django.log

tail -f /home/cchq/www/echis/log/django.log

return view_func(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/apps/users/views/init.py", line 579, in paginate_web_users
.start(skip).size(limit).run()
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/apps/es/es_query.py", line 220, in run
es_instance_alias=self.es_instance_alias,
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/elastic.py", line 216, in run_query
raise ESError(e)
corehq.elastic.ESError: TransportError(400, 'search_phase_execution_exception')

while all elasticsearch indices showing healthy

curl -XGET 172.31.85.158:9200/_cat/indices

green open smslogs_2020-01-28 5 1 0 0 1.5kb 795b
green open xforms_2016-07-07 5 1 0 0 1.5kb 795b
green open case_search_2018-05-29 5 1 0 0 1.5kb 795b
green open hqgroups_2017-05-29 5 1 0 0 1.5kb 795b
green open report_cases_czei39du507m9mmpqk3y01x72a3ux4p0 5 1 0 0 1.5kb 795b
green open hqapps_2020-02-26 5 1 61 180 5.4mb 2.7mb
green open hqusers_2017-09-07 2 1 5 1 153.5kb 76.7kb
green open hqdomains_2020-02-10 5 1 1 1 59kb 29.5kb
green open report_xforms_20160824_1708 5 1 0 0 1.5kb 795b
green open hqcases_2016-03-04 5 1 0 0 1.5kb 795b

Is this a new environment? It looks like the mapping isn't registered in that index. I'd try resetting it:

./manage.py update_es_mapping hqusers

yes, it's new environement

the command ./manage.py update_es_mapping hqusers brings an error

Confirm that you want to update the mapping for 'hqusers_2017-09-07' [y/n]y
Traceback (most recent call last):
File "manage.py", line 195, in
main()
File "manage.py", line 41, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py", line 36
4, in execute_from_command_line
utility.execute()
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py", line 35
6, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 283, i
n run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 330, i
n execute [8/1920] output = self.handle(*args, **options)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/ex-submodules/pillowtop/management/commands/update_es_mapping.py", line 38, in h
andle
mapping_res = es.indices.put_mapping(index_info.type, {index_info.type: mapping}, index=index_info.index)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 69, in
_wrapped
return func(*args, params=params, **kwargs)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/client/indices.py", line 289,
in put_mapping
'_mapping', doc_type), params=params, body=body)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/transport.py", line 307, in pe
rform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", l
ine 93, in perform_request
self._raise_error(response.status, raw_data)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 105,
in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, 'illegal_argument_exception')

Elasticsearch is returning a 400 error - what version of elasticsearch is running? If it's 1.x, you'll need to upgrade to 2.x as described in this changelog entry. If you're already on 2.4.6, could you try to figure out why Elasticsearch considers that a bad request? I was able to successfully run that command, so I suspect it's a version problem.

Actually, am using elasticsearch 2.4.6 as shown below.

(ansible) ansible@covid0:~/.commcare-cloud/repo$ curl -XGET 172.31.85.158:9200
{
"name" : "es0",
"cluster_name" : "covid-es",
"cluster_uuid" : "KuvZCzbORayP0KakZgjG1Q",
"version" : {
"number" : "2.4.6",
"build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
"build_timestamp" : "2017-07-18T12:17:44Z",
"build_snapshot" : false,
"lucene_version" : "5.5.4"
},
"tagline" : "You Know, for Search"
}

The same version of ES is working without issue on locally deployed server.

Hi Ethan,

issue has been solved. these are the steps i walk through.

  1. i re-install elastic-search cluster.
  2. run chq <env> django-manage ptop_preindex
  3. the above command throws the following error

Traceback (most recent call last):
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/apps/hqcase/management/commands/ptop_preindex.py", line 120, in handle
job.get()
File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get
File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/apps/hqcase/management/commands/ptop_preindex.py", line 52, in do_reindex
FACTORIES_BY_SLUGreindex_command.build().reindex()
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 265, in reindex
_prepare_index_for_reindex(self.es, self.index_info)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/ex-submodules/pillowtop/reindexer/reindexer.py", line 153, in _prepare_index_for_reindex
set_index_reindex_settings(es, index_info.index)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/ex-submodules/pillowtop/es_utils.py", line 119, in set_index_reindex_settings
return ElasticsearchInterface(es).update_index_settings(index, INDEX_REINDEX_SETTINGS)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/corehq/util/es/interface.py", line 20, in update_index_settings
return self.es.indices.put_settings(settings_dict, index=index)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/client/indices.py", line 620, in put_settings
'_settings'), params=params, body=body)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
self._raise_error(response.status, raw_data)
File "/home/cchq/www/echis/releases/2020-04-24_06.56/python_env-3.6/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, 'illegal_argument_exception')

  1. Then i run cchq <env> django-manage run_ptop --all
    at this cmd execution, elasticsearch is updating mapping for app index

[2020-04-24 19:37:08,799][INFO ][cluster.metadata ] [es1] [hqapps_2020-02-26] update_mapping [app]

it works for the time being, please give me a comment if i need to see it further.