Elasticsearch node health in the red

(John Harper) #1

I am having some major issues with elasticsearch

[commcarehq@Kids-Demo ~]# curl ‘localhost:9200/_cat/indices?v’
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 1547 100 1547 0 0 77191 0 --:–:-- --:–:-- --:–:-- 81421
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open hqdomains_2016-08-08 5 0 1 1 21.7kb 21.7kb
red open xforms_2016-07-07 5 0 42216 0 30.2mb 30.2mb
red open hqcases_2016-03-04 5 0 54276 29 16.1mb 16.1mb
red open ledgers_2016-03-15 5 0 0 0 432b 432b
red open pleasereadthis 5 1
red open report_cases_czei39du507m9mmpqk3y01x72a3ux4p0 5 0 0 0 432b 432b
red open smslogs_708c77f8e5fe00286fa5791e9fa7d45f 5 0 0 0 432b 432b
red open case_search_2016-03-15 5 0 0 0 432b 432b
red open hqgroups_20150403_1501 5 0 1 0 3.9kb 3.9kb
red open report_xforms_20160824_1708 5 0 0 0 288b 288b
green open hqusers_2016-09-29 5 0 95 36 387.6kb 387.6kb
red open hqapps_2016-10-20_1835 5 0 3 1 110.5kb 110.5kb

application is still functioning but I am missing data and cases on exports and reports.

Do I blow away each index and resync or can it be done all at the same time?

any thoughts…

also anyone have any idea what “PLEASEREADTHIS” index is all about?

(Cory Zue) #2

Hi John,

What’s in the contents of the pleasereadthis index? It looks like your elasticsearch index might be compromised: https://medium.com/@bromiley/elastic-ransoms-grow-259f33c9a475

Thankfully it can be rebuilt from the primary, but I’d recommend running a careful security check of your deployment before doing so.


(John Harper) #3

Cory, the doc is blank but the index is still there. I can drop it completely as I do not believe this has anything to do with the app.

Almost a year ago we did have some issue with that attack but restored back from a snapshot to remove the damage. maybe we did not get it all.

This is just a single node dev box but i have a lot of work on it …How do you restore from primary? can I if this is the primary?

(Cory Zue) #4

Elasticsearch is never the primary for CommCare so unless you’ve also lost data in postgres / couch / riak it should be possible to rebuild everything. Your plan of blowing away all the indices and resyncing sounds right.

You can either run “./manage.py ptop_reindexer_v2 [datatype] --cleanup” for each affected index / datatype, or manually delete them in ES and then run “./manage.py ptop_preindex” to do it in bulk.