Elasticsearch: moving unassigned shard to different node

Developers
1

Hi there

after a server restart, most shards were unassigned(i.e 84/94 shards). we just moved unassigned shards to different nodes and works fine (That is All are green).

But late on, whern i look at the data size using a get command, it displays a huge drop.

curl -XGET 172.19.3.41:9200/_cat/indices
green open smslogs_2020-01-28 5 1 0 0 1.5kb 796b
green open case_search_2018-05-29 5 1 8927905 80820 2.1gb 1gb
green open xforms_2016-07-07 5 1 626402 395 3.5gb 1.7gb
green open hqgroups_2017-05-29 5 1 0 0 1.5kb 805b
green open hqapps_2020-02-26 5 1 0 0 1.5kb 795b
green open report_cases_czei39du507m9mmpqk3y01x72a3ux4p0 5 1 0 0 1.5kb 795b
green open hqusers_2017-09-07 2 1 286093 96196 288.8mb 145.8mb
green open hqdomains_2020-02-10 5 1 0 0 1.5kb 795b
green open hqcases_2016-03-04 5 1 3785754 179426 4.8gb 2.4gb
green open report_xforms_20160824_1708 5 1 0 0 1.5kb 795b

For instance, the size of hqcases_2016 and case_search_2018 indices was around 10GB and 60 GB respectively.

But looking into the size of Elasticsearch data and logs on DIsk it's huge and seems data is not lost.

cchq echis run-shell-command elasticsearch "du -sh /opt/data/elasticsearch-2.4.6"

172.19.4.48 | CHANGED | rc=0 >>
129G /opt/data/elasticsearch-2.4.6
172.19.3.41 | CHANGED | rc=0 >>
95G /opt/data/elasticsearch-2.4.6
172.19.4.59 | CHANGED | rc=0 >>
4.8G /opt/data/elasticsearch-2.4.6
172.19.3.40 | CHANGED | rc=0 >>
38G /opt/data/elasticsearch-2.4.6
172.19.4.43 | CHANGED | rc=0 >>
4.5G /opt/data/elasticsearch-2.4.6

Actually, huge data is available on logs than on the data folder.

cchq echis run-shell-command elasticsearch "du -sh /opt/data/elasticsearch-2.4.6/data"
172.19.4.48 | CHANGED | rc=0 >>
2.7G /opt/data/elasticsearch-2.4.6/data
172.19.3.41 | CHANGED | rc=0 >>
244K /opt/data/elasticsearch-2.4.6/data
172.19.3.40 | CHANGED | rc=0 >>
2.1G /opt/data/elasticsearch-2.4.6/data
172.19.4.59 | CHANGED | rc=0 >>
4.1G /opt/data/elasticsearch-2.4.6/data
172.19.4.43 | CHANGED | rc=0 >>
3.3G /opt/data/elasticsearch-2.4.6/data

cchq echis run-shell-command elasticsearch "du -sh /opt/data/elasticsearch-2.4.6/logs"
172.19.4.48 | CHANGED | rc=0 >>
116G /opt/data/elasticsearch-2.4.6/logs
172.19.3.40 | CHANGED | rc=0 >>
36G /opt/data/elasticsearch-2.4.6/logs
172.19.3.41 | CHANGED | rc=0 >>
88G /opt/data/elasticsearch-2.4.6/logs
172.19.4.59 | CHANGED | rc=0 >>
739M /opt/data/elasticsearch-2.4.6/logs
172.19.4.43 | CHANGED | rc=0 >>
1.2G /opt/data/elasticsearch-2.4.6/logs

2

Hi

How did you move the unassigned shards? Also what are your cluster settings:

curl -XGET localhost:9200/_cluster/settings
3

moving shards:

curl -XPOST '172.19.3.41:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "report_xforms_20160824_1708",
"shard" : 4,
"node" : "es3",
"allow_primary" : true
} }]
}'

curl -XGET 172.19.3.41:9200/_cluster/settings
{"persistent":{"cluster":{"routing":{"allocation":{"enable":"all"}}}},"transient":{}}

4

Hi Demisew

The issue is with the allow_primary parameter you passed in. According to the ES docs:

The allow_primary parameter will force a new empty primary shard to be allocated without any data. If a node which has a copy of the original shard (including data) rejoins the cluster later on, that data will be deleted: the old shard copy will be replaced by the new live shard copy.

I think you're options are:

  1. restore a backup and backfill the data by resetting the CommCare pillowtop checkpoints
  2. If you don't have a backup you'll need to rebuild the index from scratch using the ptop_reindexer_v2 command.
5

Oh, that is weird. because i run this , reroute, command several time before to resolve unassigned shards. And works as expected.

we didn't take elasticsearch backup. so that i opted to step 2. we have done rebuild several times before(especially when server restarted). there is either something i missed or the some bugs on the tool.

2nd. why is the log data this much bigger than the data size?