You may notice that these articles took on architectural decisions and never specific questions like, “How many Logstash instances do I need if I have 10k events/second and my regular expression looks like…”. Instead of trying to address all these in a single article, I’ll break down the flow and describe each in its own post:ĮLK: Performance of the Logstash Indexing layerĮLK: Federated Search with a Tribal node Capacity Planning Where is the clustering that would allow Elasticsearch or the Logstash indexing layer to be highly available or scalable? How will we horizontally scale the Logstash Indexing layer which will be executing CPU-intensive regular expression operations to extract fields? By what mechanism will log files be sent to Logstash? Will we lose log events if one of the Logstash nodes goes down?
Immediately, questions should come to mind. From there, Kibana allows end users to analyze the data using a web application. Let’s start with the simplest logical deployment for ELK, which is a data source on the left, flowing through a Logstash indexing layer that filters and extracts the data, an ultimately sends it to Elasticsearch for indexing/searching. The ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability.īecause so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, message queues, indexes on distinct physical drives, etc… So in this article I want to explore the obvious extension points, and encourage the reader to treat this as a starting point in their own design and deployment.
0 kommentar(er)
