Do you know: how organisations can save on storage and costs using ILM?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how organisations can save on storage and costs using ILM.

Background

Index Lifecycle Management (ILM) is a strategy used to manage the lifecycle of an index from creation to deletion. By defining policies that dictate how data moves through different phases (hot, warm, cold and delete), organisations can optimise storage costs and ensure efficient access to data, based on its age and relevance.

Main benefits of ILM

• Cost savings: store less frequently accessed data on cheaper storage solutions.
• Performance optimisation: keep high-priority data on faster, more expensive storage for quick access.
• Automated data management: reduce manual intervention with automated data tier transitions.

Solution

Step 1

Begin by analysing the data lifecycle and query patterns:

• Hot Data: frequently accessed and newly ingested data.
• Warm Data: less frequently accessed but still needed for analysis.
• Cold Data: rarely accessed data that can be stored more cost-effectively.
• Delete: data that is no longer needed.

Step 2

To implement ILM in the system, start by creating the ILM policy. Define your ILM policy in your data management system. Here’s an example of how to define an ILM policy in Elasticsearch:

PUT _ilm/policy/<Custom-Name>-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Step 3

Apply ILM policy to indices

PUT /<Custom-Name>*/_settings
{
  "index": {
    "lifecycle": {
      "name": "<Custom-Name>-policy",
      "rollover_alias": "<Custom-Name>-alias"
    }
  }
}

Step 4

Regularly monitor the performance and storage usage to ensure the ILM policy meets your goals. Use metrics and logs to identify any issues or areas for improvement. Adjust the policy as needed based on data access patterns and storage costs.

Step 5

For even further cost savings, consider using searchable snapshots for cold data. This allows for less frequently accessed data to be stored in more cost-effective storage solutions like object storage, while still being searchable when needed.

Step 6

Create a snapshot repository

PUT _snapshot/my_repository
{
  "type": "fs",
  "settings": {
    "location": "/mount/backups"
  }
}

Update your ILM policy

PUT _ilm/policy/<Custom-Name>-policy
{
  "policy": {
    "phases": {
      "cold": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_repository"
          }
        }
      }
    }
  }
}

Final ILM policy

PUT _ilm/policy/<Custom-Name>-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "cold"
            }
          },
          "searchable_snapshot": {
            "snapshot_repository": "my_repository"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Conclusion

Implementing an ILM strategy allows organisations to manage huge amounts of data efficiently, significantly reducing storage costs while maintaining performance. By classifying data into hot, warm, and cold phases and using searchable snapshots, one can optimise storage, ensuring quick access to recent data and cost-effective storage for older data. Regular monitoring and adjustments of ILM policies will ensure continued efficiency and cost savings.