Log Management With ELK Stack: Complete Guide
On this page
Log Management With ELK Stack: Complete Guide
Logs are the raw truth of what your systems are doing. But scattered across dozens of servers and containers, they are nearly useless. The ELK Stack solves this by centralizing, parsing, indexing, and visualizing logs so you can search millions of events in seconds. This guide walks through what ELK is, how to set it up, and how to run it reliably in production.
What Is the ELK Stack?
ELK is an acronym for three open-source projects maintained by Elastic:
- Elasticsearch — a distributed search and analytics engine that stores and indexes your log data.
- Logstash — a server-side data processing pipeline that ingests, transforms, and enriches logs before shipping them to Elasticsearch.
- Kibana — a web UI for searching, visualizing, and building dashboards on top of your indexed data.
In recent years the stack expanded to include Beats — lightweight shippers that run on your hosts and forward data. Because Beats became so central, the collection is now often called the Elastic Stack. The most common Beat is Filebeat, which tails log files and ships them onward.
A typical modern pipeline looks like this:
Application logs → Filebeat → Logstash → Elasticsearch → Kibana
For simpler setups you can skip Logstash entirely and let Filebeat write directly to Elasticsearch using ingest pipelines.
Why Centralized Log Management Matters
Before ELK, debugging a distributed system meant SSHing into machines and grepping through files. That approach breaks down fast:
- Correlation is impossible. A single request may touch ten services. Without central logs you cannot follow it end to end.
- Logs disappear. Containers are ephemeral; when a pod dies, its logs die with it unless they were shipped elsewhere.
- Search is slow.
grepacross terabytes takes minutes. Elasticsearch returns results in milliseconds. - No alerting. Centralized logs let you trigger alerts on error spikes, failed logins, or latency thresholds.
Centralized logging is a foundational pillar of observability, alongside metrics and traces.
Core Architecture Explained
Understanding each component's role helps you design a stack that scales.
Elasticsearch
Elasticsearch stores documents in indices, which are split into shards distributed across nodes in a cluster. Each shard is a self-contained Lucene index. Replicas provide redundancy and increase read throughput. Time-series log data is usually organized with data streams and managed through Index Lifecycle Management (ILM), which automatically rolls over, shrinks, and deletes old indices.
Logstash
Logstash pipelines have three stages defined in a config file:
- Inputs — where data comes from (Beats, Kafka, syslog, files).
- Filters — how data is parsed and transformed (grok, mutate, date, geoip).
- Outputs — where data goes (Elasticsearch, S3, another queue).
Kibana
Kibana is your window into the data. You use it to run queries in the Discover view, build visualizations, assemble dashboards, and manage the stack itself.
Beats
Beats are purpose-built, low-footprint agents. Filebeat handles logs, Metricbeat handles metrics, Packetbeat handles network data, and Winlogbeat handles Windows event logs.
Setting Up ELK With Docker
The fastest way to get a working stack for learning or development is Docker Compose. Below is a minimal single-node example.
version: "3.8"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.14.0
environment:
- discovery.type=single-node
- xpack.security.enabled=true
- ELASTIC_PASSWORD=changeme
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
ports:
- "9200:9200"
volumes:
- esdata:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:8.14.0
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- "5601:5601"
depends_on:
- elasticsearch
logstash:
image: docker.elastic.co/logstash/logstash:8.14.0
volumes:
- ./logstash/pipeline:/usr/share/logstash/pipeline
ports:
- "5044:5044"
depends_on:
- elasticsearch
volumes:
esdata:
Bring it up with docker compose up -d, then visit http://localhost:5601 for Kibana. Set the ES_JAVA_OPTS heap to no more than 50% of available RAM, and never above ~31GB.
Building a Logstash Pipeline
A pipeline config that receives from Filebeat, parses an Nginx access log, and ships to Elasticsearch might look like this:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}
geoip {
source => "clientip"
}
mutate {
convert => { "response" => "integer" }
convert => { "bytes" => "integer" }
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
user => "elastic"
password => "changeme"
index => "nginx-access-%{+YYYY.MM.dd}"
}
}
The grok filter is the workhorse of log parsing. It uses named regular-expression patterns to turn an unstructured line into structured fields. Test your grok patterns in Kibana's Grok Debugger before deploying — malformed patterns are the most common cause of _grokparsefailure tags.
Parsing and Structuring Logs
Structured logs are far easier to work with than free-form text. Wherever possible, emit JSON logs from your applications. Filebeat and Logstash can parse JSON natively, eliminating fragile grok patterns entirely.
Adopt the Elastic Common Schema (ECS) — a standardized set of field names like host.name, source.ip, and event.action. When every service uses the same field names, cross-service dashboards and correlation become trivial. Map your custom fields to ECS equivalents in your pipeline.
Visualizing With Kibana
Once data flows in, create a data view (formerly index pattern) matching your indices, e.g. nginx-access-*. Then:
- Use Discover to search raw events with KQL, e.g.
response >= 500 and url.path : "/checkout". - Build visualizations — line charts for request rates, heat maps for latency, data tables for top error messages.
- Assemble dashboards that combine visualizations into a single operational view.
- Configure alerts in the Stack Management section to notify Slack, email, or PagerDuty when thresholds are crossed.
Production Best Practices
Running ELK at scale requires deliberate choices.
Add a Buffer
Put a message queue like Kafka or Redis between your shippers and Logstash. When ingestion spikes or Elasticsearch slows down, the queue absorbs the backpressure instead of dropping logs.
Manage Index Lifecycle
Configure ILM policies to roll over indices at a size or age threshold, move older data to cheaper storage tiers (hot → warm → cold → frozen), and delete data past its retention window. This keeps clusters fast and storage costs predictable.
Secure the Stack
Enable authentication and TLS (they are on by default in recent versions). Use role-based access control so developers only see the indices they need. Never expose Elasticsearch's port 9200 directly to the internet.
Size Your Cluster
Use dedicated master-eligible nodes for cluster stability, and separate hot/warm data nodes for time-series workloads. Monitor JVM heap pressure, disk watermarks, and shard counts — thousands of tiny shards degrade performance.
Watch Your Own Stack
Use the Stack Monitoring feature or a separate monitoring cluster to observe ELK itself. An unmonitored logging platform is a single point of failure for your entire observability strategy.
Common Pitfalls to Avoid
- Mapping explosions — dynamically indexing thousands of unique field names bloats the cluster. Use explicit mappings and disable dynamic mapping where appropriate.
- Oversharding — a common cause of poor performance. Aim for shard sizes between 10GB and 50GB.
- Ignoring retention — logs grow without bound. Set retention policies from day one.
- Parsing everything with grok — prefer structured JSON logging to reduce CPU cost and brittleness.
Frequently Asked Questions
Is the ELK Stack free?
The core components are available under a free tier and open-source licenses. Advanced features like machine learning, advanced security, and alerting live in paid subscription tiers, though a generous Basic tier is free. OpenSearch is a fully open-source fork if licensing is a concern.
Do I always need Logstash?
No. For simple use cases, Filebeat can ship directly to Elasticsearch and use ingest pipelines for lightweight parsing. Add Logstash when you need complex transformations, multiple outputs, or buffering.
How much hardware does ELK need?
It depends on ingest volume and retention. A small setup handling a few GB per day runs comfortably on a single node with 8–16GB RAM. High-volume production clusters ingesting terabytes daily require multiple dedicated nodes with fast SSDs and ample heap.
What is the difference between ELK and the Elastic Stack?
They refer to the same ecosystem. "ELK" is the original acronym (Elasticsearch, Logstash, Kibana). "Elastic Stack" is the current branding that also includes Beats and other components.
How is ELK different from Grafana Loki or Splunk?
Splunk is a powerful commercial platform with a matching price tag. Grafana Loki is a lighter, cheaper alternative that indexes only labels rather than full log content. ELK sits in between — full-text indexing and rich search, self-hostable, with strong analytics.
How long should I retain logs?
Balance compliance requirements against cost. A common pattern is to keep a window of hot, fully-searchable data and automatically move older data to cheaper frozen or archived tiers, then delete it once it has passed your retention threshold.
Conclusion
The ELK Stack turns chaotic, scattered logs into a searchable, visual, alertable system of record. Start small with a Docker deployment, adopt structured JSON logging and ECS early, and layer in buffering, lifecycle management, and security as you move to production. Done well, ELK becomes the nervous system of your observability practice — the first place you look when something breaks and the fastest way to understand what your systems are really doing.