Cristian Balan

Cristian Balan

Live

On a fresh Elasticsearch server, any changes to the "Elasticsearch details" page (via Ploi GUI), e.g. updating "Network host", it also adds "discovery.seed_hosts: null" to the /etc/elasticsearch/elasticsearch.yml config file which breaks Elasticsearch (it won't restart).

The reason is failing is not immediately clear but I have eventually found this in the syslog which gave me the clue: Exception in thread "main" ElasticsearchParseException[null-valued setting found for key [discovery.seed_hosts] found at line number [7], column number [23]]

Manually removing the discovery.seed_hosts: null from the elasticsearch.yml file fixes the problem but essentially that gets added back whenever we update the config from the GUI.

N.B. By default, the "Discovery seed hosts" filed in the GUI is empty.

Update: Also, discovery.type: single-node is needed to be able to start Elasticsearch. I suppose that is there the first time the server is installed but then removed when using the Ploi GUI.

Cristian Balan

Cristian Balan

Item author
· · Edited
·

Also, discovery.type: single-node is needed to be able to start Elasticsearch. I suppose that is there the first time the server is installed but then removed when using the Ploi GUI.

1 total vote
Cristian Balan
Cristian Balan

Cristian Balan

Item author
·
·

Came acrross this again today.

I have then created a fresh ES server just to see the default /etc/elasticsearch/elasticsearch.yml config which is:

node.name: es-test
cluster.name: es-test
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200

Updating the "Network host" from the UI amends the config to (notice how it changes the config order too):

node.name: es-test
cluster.name: es-test
http.port: 9200
network.host: 0.0.0.0
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.seed_hosts: null

To get Elasticsearch service restart properly I had to change the config from CLI to:

node.name: es-test
cluster.name: es-test
http.port: 9200
network.host: 0.0.0.0
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.type: single-node
no votes yet
Dennis

Dennis

·
·

It should now default to the single-node which should be completely fine. Can you test? ☺️

no votes yet
Cristian Balan

Cristian Balan

Item author
·
·

Hi @Dennis, thanks for attempting to sort this out.

Now instead of adding discovery.seed_hosts: null to the config, is adding discovery.seed_hosts: single-node which is not exactly right even if it "works" (i.e. allows service to restart) - in my example it was discovery.type: single-node (not "seed_hosts").

I'm afraid this is a bit more complicated.

By default, Elasticsearch is in "development mode" with the network.host: 127.0.0.1 binding to the localhost as per https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#network.host.

As I wanted to access the cluster remotely I've changed the network.host to 0.0.0.0 and now it becomes a production cluster.

However, as per https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#_discovery_configuration_is_required_in_production, production clusters

require at least one of the following settings to be specified in the elasticsearch.yml configuration file:

  • discovery.seed_hosts
  • discovery.seed_providers
  • cluster.initial_master_nodes
  • discovery.zen.ping.unicast.hosts
  • discovery.zen.hosts_provider

This explains why it starts with discovery.seed_hosts: single-node (the old forced discovery.seed_hosts: null failed with Exception in thread "main" ElasticsearchParseException[null-valued setting found for key).

Anyway, according to https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#initial_master_nodes,

When you start an Elasticsearch cluster for the first time, a cluster bootstrapping step determines the set of master-eligible nodes whose votes are counted in the first election. In development mode, with no discovery settings configured, this step is performed automatically by the nodes themselves. Because auto-bootstrapping is inherently unsafe, when starting a new cluster in production mode, you must explicitly list the master-eligible nodes whose votes should be counted in the very first election. You set this list using the cluster.initial_master_nodes setting.

So the most obvious thing when setting network.host for remote connections is to set cluster.initial_master_nodes and you do have an option for that in the UI. Alternatively, if happy with a single node, as per https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html#single-node-discovery we can have discovery.type: single-node instead.

Ok, now I'm happy to use cluster.initial_master_nodes as per your UI ("Initial master nodes") but it doesn't work. In fact it doesn't do anything and when saving it just adds the second field ("Discovery seed hosts") even if is empty (before was discovery.seed_hosts: null now is discovery.seed_hosts: single-node).

So, what needs to happen is for empty fields to be ignored (or deleted if previously configured) and those fields we fill in added to the configuration.

In an ideal world, if the "Network host" field is changed to something other than 127.0.0.1, the "Initial master nodes" should become mandatory. :)

no votes yet
Dennis

Dennis

· · Edited
·

Understood! When a field is left empty, the config option won't appear in the config file itself.

1 total vote
Cristian Balan
Cristian Balan

Cristian Balan

Item author
· · Edited
·

Hi @Dennis,

The "Discovery seed hosts" form field issue was fixed and if left empty it no longer adds the "null" value to the config, which is fine.

However, the "Initial master nodes" (cluster.initial_master_nodes) field doesn't do anything. We actually need that to get the service to restart after changing the "Network host".

no votes yet
Dennis

Dennis

·
·

Oh really? What does the initial_master_nodes do? And currently it doesn't append to the config?

no votes yet
Cristian Balan

Cristian Balan

Item author
·
·

Hi @Dennis, I have posted about initial_master_nodes in my comment above with a link to the documentation and some comments.

Yes, sadly when we fill in the field, is not being added to the config upon save like everything else does.

no votes yet
Cristian Balan

Cristian Balan

Item author
·
·

I have raised a separate bug at https://roadmap.ploi.io/projects/7-bugs/items/898-gui-elasticsearch-initial-master-nodes-not-doing-anything to deal with cluster.initial_master_nodes.

no votes yet
You may use @ to mention someone.

Elasticsearch - discovery.seed_hosts: null added to elasticsearch.yml

1 total vote
Cristian Balan
  • Dennis moved item to board Live

    2 months ago
  • Dennis moved item to board In progress

    2 months ago
  • Dennis moved item to board Live

    3 months ago
  • Dennis moved item to board In progress

    3 months ago
  • Dennis moved item to board Planned

    3 months ago
  • Cristian Balan moved item to project Bugs

    7 months ago
  • Cristian Balan created the item

    7 months ago