Hi Amit, on step 12, I had to first check the cluster name with
ls /opt/data/elasticsearch-6.8.23/data/
which identified it as:
monolith-es
I could then run the command with the cluster name:
cchq ${ENV} run-shell-command elasticsearch "mv /opt/data/elasticsearch-6.8.23/data/monolith-es/* /opt/data/elasticsearch-6.8.23/data/" -b
That wasn't clear from the instructions.
Step 16 is where I'm now running into issues, it looks like the ES service is not running.
The ES log looks like this:
2025-06-09T19:00:43,492][INFO ][o.e.e.NodeEnvironment ] [es0] using [1] data paths, mounts [[/opt/data (/dev/sda1)]], net usable_space [197.8gb], net total_space [502.8gb], types [ext4]
[2025-06-09T19:00:43,494][INFO ][o.e.e.NodeEnvironment ] [es0] heap size [2gb], compressed ordinary object pointers [true]
[2025-06-09T19:00:43,551][INFO ][o.e.n.Node ] [es0] node name [es0], node ID [1k_6ORgNRK6wPf8kc0_4xA]
[2025-06-09T19:00:43,551][INFO ][o.e.n.Node ] [es0] version[6.8.23], pid[114910], build[default/tar/4f67856/2022-01-06T21:30:50.087716Z], OS[Linux/6.8.0-1028-azure/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/17.0.15/17.0.15+6-Ubuntu-0ubuntu122.04]
[2025-06-09T19:00:43,552][INFO ][o.e.n.Node ] [es0] JVM arguments [-Xms2048m, -Xmx2048m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp, -XX:-HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/opt/data/elasticsearch-6.8.23/logs/heapdump.hprof, -XX:ErrorFile=/opt/data/elasticsearch-6.8.23/logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.path.home=/opt/elasticsearch-6.8.23, -Des.path.conf=/etc/elasticsearch-6.8.23, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2025-06-09T19:00:44,567][INFO ][o.e.p.PluginsService ] [es0] loaded module [aggs-matrix-stats]
[2025-06-09T19:00:44,567][INFO ][o.e.p.PluginsService ] [es0] loaded module [analysis-common]
[2025-06-09T19:00:44,567][INFO ][o.e.p.PluginsService ] [es0] loaded module [ingest-common]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [ingest-geoip]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [ingest-user-agent]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [lang-expression]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [lang-mustache]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [lang-painless]
[2025-06-09T19:00:44,568][INFO ][o.e.p.PluginsService ] [es0] loaded module [mapper-extras]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [parent-join]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [percolator]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [rank-eval]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [reindex]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [repository-url]
[2025-06-09T19:00:44,569][INFO ][o.e.p.PluginsService ] [es0] loaded module [transport-netty4]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [tribe]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-ccr]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-core]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-deprecation]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-graph]
[2025-06-09T19:00:44,570][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-ilm]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-logstash]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-ml]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-monitoring]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-rollup]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-security]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-sql]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-upgrade]
[2025-06-09T19:00:44,571][INFO ][o.e.p.PluginsService ] [es0] loaded module [x-pack-watcher]
[2025-06-09T19:00:44,572][INFO ][o.e.p.PluginsService ] [es0] loaded plugin [analysis-phonetic]
[2025-06-09T19:00:44,703][INFO ][i.n.u.i.PlatformDependent] [es0] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
[2025-06-09T19:00:46,666][INFO ][i.n.u.i.PlatformDependent] [es0] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
[2025-06-09T19:00:46,859][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es0] [controller/114997] [Main.cc@114] controller (64 bit): Version 6.8.23 (Build 31256deab94add) Copyright (c) 2022 Elasticsearch BV
[2025-06-09T19:00:47,103][DEBUG][o.e.a.ActionModule ] [es0] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2025-06-09T19:00:47,288][ERROR][o.e.g.GatewayMetaState ] [es0] failed to read local state, exiting...
java.lang.IllegalStateException: The index [[users-20230524/PeeM1L30RZ-zV2tZzDPyQQ]] was created with version [2.4.6] but the minimum compatible version is [5.0.0]. It should be re-indexed in Elasticsearch 5.x before upgrading to 6.8.23.
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkSupportedVersion(MetaDataIndexUpgradeService.java:126) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:97) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:246) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:89) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:499) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:266) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.23.jar:6.8.23]
[2025-06-09T19:00:47,294][ERROR][o.e.b.Bootstrap ] [es0] Exception
java.lang.IllegalStateException: The index [[users-20230524/PeeM1L30RZ-zV2tZzDPyQQ]] was created with version [2.4.6] but the minimum compatible version is [5.0.0]. It should be re-indexed in Elasticsearch 5.x before upgrading to 6.8.23.
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkSupportedVersion(MetaDataIndexUpgradeService.java:126) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:97) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:246) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:89) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.23.jar:6.8.23]
[2025-06-09T19:00:47,296][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[users-20230524/PeeM1L30RZ-zV2tZzDPyQQ]] was created with version [2.4.6] but the minimum compatible version is [5.0.0]. It should be re-indexed in Elasticsearch 5.x before upgrading to 6.8.23.
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.23.jar:6.8.23]
Caused by: java.lang.IllegalStateException: The index [[users-20230524/PeeM1L30RZ-zV2tZzDPyQQ]] was created with version [2.4.6] but the minimum compatible version is [5.0.0]. It should be re-indexed in Elasticsearch 5.x before upgrading to 6.8.23.
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkSupportedVersion(MetaDataIndexUpgradeService.java:126) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:97) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:246) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:89) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.23.jar:6.8.23]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.23.jar:6.8.23]
... 6 more
[2025-06-09T19:00:47,298][INFO ][o.e.x.m.p.NativeController] [es0] Native controller process has stopped - no new native processes can be started
Not sure why we're getting "The index .... was created with version [2.4.6] but the minimum compatible version is [5.0.0]."
I was under the impression we've been on ES v5 for at least a year.
The /opt/data directory looks like this:
drwxr-xr-x 3 redis root 4096 Jun 9 19:16 redis
drwxrwxr-x 3 nobody nfs 148918272 Jun 9 18:52 blobdb
drwxr-xr-x 5 elasticsearch elasticsearch 4096 Jun 9 18:28 elasticsearch-6.8.23
drwxr-xr-x 5 elasticsearch elasticsearch 4096 Jun 9 18:06 elasticsearch-6.8.23-new-installation
drwxr-xr-x 4 root root 4096 May 17 12:47 home
drwxr-xr-x 5 cchq cchq 4096 Jun 7 2024 formplayer
drwxr-xr-x 4 elasticsearch elasticsearch 4096 Feb 2 2024 elasticsearch-5.6.16-new-installation
drwxrwxrwx 6 root root 4096 Feb 2 2024 backups
drwxr-x--- 5 couchdb couchdb 4096 Aug 9 2023 couchdb2
drwxr-xr-x 3 kafka kafka 4096 Aug 9 2023 kafka
drwxr-xr-x 4 elasticsearch elasticsearch 4096 Aug 9 2023 elasticsearch-5.6.16-backup
-rw-r--r-- 1 root root 1073741824 Aug 9 2023 emerg_delete.dummy
drwx------ 3 postgres postgres 4096 Aug 9 2023 postgresql
-rwxrwx--- 1 root root 55 Aug 9 2023 README
Thanks!
Ed