Skip to content

Step-by-Step Guide: Configuring a Self-Managed Elastic Cluster on Linux – Single and multi node

Elasticsearch, renowned for its search engine capabilities, boasts powerful security features for data protection. This comprehensive guide walks you through the process of setting up a Multi-Node ElasticSearch cluster with custom configurations on Linux. Starting from scratch, we cover user creation, stack limits, memory map configurations, and Elasticsearch deployment. The guide concludes with verification using CURL.

Elasticsearch, renowned for its power and scalability as a search engine, offers robust security features to safeguard your data. We are set to establish a Multi Node ES cluster with tailored configurations on linux machines. This blog post serves as a step-by-step guide, walking you through the process of configuring an Elastic cluster with everything from scratch using the settings outlined below.

  • First, we will generate a user, choosing any name as an example (e.g., Cloud/Elastic).
  • Next, we will define an open limit for the stack.
  • Following that, we’ll establish memory map limits for the Elastic stack.
  • Once memory limits are in place, we’ll proceed to deploy Elasticsearch.
  • Go through the basic Elasticsearch configuration settings for example setting up cluster name, network configuration, TLS, SSL configuration, etc.
  • With the setup complete, we’ll launch Elasticsearch as a daemon.
  • To verify, we’ll use CURL to confirm the status and configurations of the Elasticsearch nodes.

SSH to Linux Machine and Create the elastic user:

Create elastic user 
sudo useradd elastic

When you run sudo useradd elastic, you are instructing the system to create a new user named “elastic” with default settings. After running this command, a new user account is created, and you can use it to log in and perform various tasks on the system.

It’s worth noting that creating a user with useradd typically only creates the user account without setting a password or creating a home directory. You might need additional commands, such as passwd to set a password for the new user and usermod to add a home directory. The specific steps may vary based on your system configuration and requirements.

Open the limits.conf file as root:
sudo vi /etc/security/limits.conf

Add the following line near the bottom:
elastic – nofile 65536

Elasticsearch uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,535 or higher.

Read more here :

Open the sysctl.conf file as root:
sudo vi /etc/sysctl.conf

Add the following line at the bottom:

Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure vm.max_map_count via sysctl to be at least 262144.

Read more here :,to%20be%20at%20least%20262144%20.

Load the new sysctl values:
sudo sysctl -p

Become the elastic user:
sudo su – elastic

Download the binaries for Elasticsearch 8.10.3 in the elastic user’s a home directory:

curl -O

Note : In case your machine has no internet connection then you can winscp to transfer the tar file to the machine.

Winscp download link :

Unpack the archive:
tar -xzvf elasticsearch-8.10.3-linux-x86_64.tar.gz

Remove the archive:
rm elasticsearch-8.10.3-linux-x86_64.tar.gz

Rename the unpacked directory:
mv elasticsearch-8.10.3 elasticsearch

Now We will Configure each node’s elasticsearch.yml file here as per specifications.
Log in to each node and become the elastic user:

  • sudo su – elastic
  • Open the elasticsearch.yml file:
  • vi $ES_HOME/elasticsearch.yml

Elasticsearch Configuration File Overview

The supplied YAML file encompasses crucial configurations tailored for a production Elasticsearch cluster. Let’s delve into key sections:

Cluster Configuration Your_Cluster_Name

Sets a descriptive name for your cluster. All nodes must share the same cluster name to join the same cluster.

Node Configuration node-01

Specifies a descriptive name for your node. The default node name is the machine’s hostname upon Elasticsearch startup.

Path Configuration /elastic_data_directory/ElasticData

path.logs: /elastic_log_directory/ElasticLogs

Defines data and log directory paths. Your Elastic data and logs will be stored in these paths.

Network Configuration X.X.X.5 

http.port: 9200

Configures network settings, making Elasticsearch accessible on the specified IP and port.

Discovery Configuration – Multi Node Cluster

discovery.seed_hosts: [“X.X.X.5”, “X.X.X.6”, “X.X.X.7”]

cluster.initial_master_nodes: [“node-01”, “node-02”, “node-03”]

Establishes the discovery process with seed hosts and initial master nodes. For a single-node cluster, use:

discovery.seed_hosts: [“X.X.X.5”]

cluster.initial_master_nodes: [“node-01”]

Security Configuration true true certificate required elastic-certificates.p12 elastic-certificates.p12

Enables X-Pack Security and configures SSL for transport.

Configuring TLS between nodes is the basic security setup to prevent unauthorised nodes from accessing your cluster. 

Refer this documentation to create these certificates and read more about Security configuration: Set up basic security for the Elastic Stack | Elasticsearch Guide [8.11]

SSL HTTP Configuration true certificate certificate.pfx

When you enable TLS on the HTTP layer it provides an additional layer of security to ensure that all communications to and from your cluster are encrypted.

For more information on creating certificates, refer to the official documentation.

Set up basic security for the Elastic Stack plus secured HTTPS traffic | Elasticsearch Guide [8.11]

Now we will configure the heap for each node per instructions.

Log in to each master node and become the elastic user:

sudo su – elastic

Open the jvm.options file:

vi /$ES_HOME/elasticsearch/config/jvm.options

Change the following lines:



Log in to each data node and become the elastic user:

sudo su – elastic

Open the jvm.options file:

vi /$ES_HOME/elasticsearch/config/jvm.options

Change the following lines:



Note : By default, Elasticsearch automatically sets the JVM heap size based on a node’s roles and total memory. Using the default sizing is recommended for most production environments.

To override the default heap size, set the minimum and maximum heap size settings, Xms and Xmx. The minimum and maximum values must be the same.

The heap size should be based on the available RAM:

Set Xms and Xmx to no more than 50% of your total memory. Elasticsearch requires memory for purposes other than the JVM heap. For example, Elasticsearch uses off-heap buffers for efficient network communication and relies on the operating system’s filesystem cache for efficient access to files. The JVM itself also requires some memory. It’s normal for Elasticsearch to use more memory than the limit configured with the Xmx setting.

Note : When running in a container, such as Docker, total memory is defined as the amount of memory visible to the container, not the total system memory on the host.

Read more about this here: Advanced configuration | Elasticsearch Guide [8.11] | Elastic

Lets Start Elasticsearch as a daemon on each node.

Log in to each node and become the elastic user:

sudo su – elastic

Switch to the elasticsearch directory:

cd /$ES_HOME/elasticsearch

Start Elasticsearch :


When starting Elasticsearch for the first time, security features are enabled and configured by default. The following security configuration occurs automatically:

Authentication and authorization are enabled, and a password is generated for the elastic built-in superuser.

Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates.

An enrollment token is generated for Kibana, which is valid for 30 minutes.

The password for the elastic user and the enrollment token for Kibana are output to your terminal.

We recommend storing the elastic password as an environment variable in your shell.

Check the startup process by tailing logs:


By default, Elasticsearch logs are output in a plain text format. If you want to modify the Elasticsearch log format to JSON, you can achieve this by adjusting the logging configuration. Here’s how you can do it:

Step 1: Locate the Log Configuration File

Find the file or log4j2.yml file in your Elasticsearch configuration directory. The specific location may vary depending on your installation method and operating system.

For example, the path might be: /$ES_HOME/elasticsearch/ 

Step 2: Modify the Log Configuration

Edit the or log4j2.yml file.

Here is an example configuration to output logs in JSON format along with rolling logs. Choose the appropriate format based on whether your configuration file is

######## Server JSON ############################

appender.rolling.type = RollingFile # Configure the RollingFile appender = rolling

appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_server.json 

appender.rolling.layout.type = ECSJsonLayout 

appender.rolling.layout.dataset = elasticsearch.server 

appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz 

appender.rolling.policies.type = Policies

appender.rolling.policies.time.type = TimeBasedTriggeringPolicy 

appender.rolling.policies.time.interval = 1 

appender.rolling.policies.time.modulate = true 

appender.rolling.policies.size.type = SizeBasedTriggeringPolicy 

appender.rolling.policies.size.size = 256MB 

appender.rolling.strategy.type = DefaultRolloverStrategy

appender.rolling.strategy.fileIndex = nomax

appender.rolling.strategy.action.type = Delete 

appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}

appender.rolling.strategy.action.condition.type = IfFileName 

appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-* 

appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize 

appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB 


######## Server –  old style pattern ###########

appender.rolling_old.type = RollingFile = rolling_old

appender.rolling_old.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_server.log 

appender.rolling_old.layout.type = PatternLayout

appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

appender.rolling_old.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.old_log.gz

Note : The configuration for old style pattern appenders. These logs will be saved in *.log files and if archived will be in * .log.gz files. Note that these should be considered deprecated and will be removed in the future.

Read more here :

Check if elasticsearch is running : 

curl –cacert $ES_HOME/config/certs/certificate.pfx

 -u elastic:$ELASTIC_PASSWORD https://localhost:9200

You should response like below :-

Sample response: cluster_name, uuid, version.number based on configuration and installed version.


This comprehensive guide covers setting up an Elastic cluster from scratch on Linux, ensuring security and optimal configurations. Stay tuned for upcoming blogs covering similar setups for Kibana and Enterprise Search on Linux. Your feedback is valuable, so feel free to share your suggestions in the comments.

Stay tuned for upcoming blogs:

  • Configuring a Self-Managed Elastic Cluster with SSL: A Step-by-Step Guide for Kibana on linux.
  • Configuring a Self-Managed Elastic Cluster with SSL: A Step-by-Step Guide for Enterprise Search on linux.