Elasticsearch node health in the red

Developers
1

I am having some major issues with elasticsearch

[commcarehq@Kids-Demo ~]# curl 'localhost:9200/_cat/indices?v'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1547 100 1547 0 0 77191 0 --:--:-- --:--:-- --:--:-- 81421
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open hqdomains_2016-08-08 5 0 1 1 21.7kb 21.7kb
red open xforms_2016-07-07 5 0 42216 0 30.2mb 30.2mb
red open hqcases_2016-03-04 5 0 54276 29 16.1mb 16.1mb
red open ledgers_2016-03-15 5 0 0 0 432b 432b
red open pleasereadthis 5 1
red open report_cases_czei39du507m9mmpqk3y01x72a3ux4p0 5 0 0 0 432b 432b
red open smslogs_708c77f8e5fe00286fa5791e9fa7d45f 5 0 0 0 432b 432b
red open case_search_2016-03-15 5 0 0 0 432b 432b
red open hqgroups_20150403_1501 5 0 1 0 3.9kb 3.9kb
red open report_xforms_20160824_1708 5 0 0 0 288b 288b
green open hqusers_2016-09-29 5 0 95 36 387.6kb 387.6kb
red open hqapps_2016-10-20_1835 5 0 3 1 110.5kb 110.5kb

application is still functioning but I am missing data and cases on exports and reports.

Do I blow away each index and resync or can it be done all at the same time?

any thoughts.....

also anyone have any idea what "PLEASEREADTHIS" index is all about?

2

Hi John,

What's in the contents of the pleasereadthis index? It looks like your elasticsearch index might be compromised: https://medium.com/@bromiley/elastic-ransoms-grow-259f33c9a475

Thankfully it can be rebuilt from the primary, but I'd recommend running a careful security check of your deployment before doing so.

Cory

3

Cory, the doc is blank but the index is still there. I can drop it completely as I do not believe this has anything to do with the app.

Almost a year ago we did have some issue with that attack but restored back from a snapshot to remove the damage. maybe we did not get it all.

This is just a single node dev box but i have a lot of work on it ..........How do you restore from primary? can I if this is the primary?

4

Elasticsearch is never the primary for CommCare so unless you've also lost data in postgres / couch / riak it should be possible to rebuild everything. Your plan of blowing away all the indices and resyncing sounds right.

You can either run "./manage.py ptop_reindexer_v2 [datatype] --cleanup" for each affected index / datatype, or manually delete them in ES and then run "./manage.py ptop_preindex" to do it in bulk.