Short on time? Click here to jump to our Elastic Stack Use Case and Business Example!
The Challenge
Many organizations strive to find affordable methods to manage, maintain, and audit their IT logs. From firewalls, to servers, to applications, and more, the variety of log sources and types is nearly endless. Equally endless are the software solutions available for receiving and handling these logs. The log management market is filled with a bountiful supply of solutions, including both free and paid products. Unfortunately for most companies, the solutions that can offer them the desired feature sets are often quite pricey due to the pricing structures designed and utilized across much of the log management space. This holds especially true for organizations who have need of SIEM (security information and event management) solutions and functionality in order to meet compliance guidelines and mandates. This not only introduces additional cost, but it also begins to drastically increase the complexity of the log management solution as the data must now be normalized and processed, a task that is both complex and resource intense in most situations. This often leads to companies giving up early or stopping shy of implementing the solutions they truly need in favor of a solution that is simple, manageable, and affordable.
The Future is Here
However, the world of log management is beginning to rapidly shift as providers begin looking for new methodologies to achieve the necessary goals and mandates that organizations have while providing a simpler to use solution. One such solution that is rising to the top of the pack is Elastic Stack. This stack of products is comprised of a data receiver/retriever (Beats/Logstash), an extremely fast data indexing and search engine (ElasticSearch), and a highly extendable data visualization engine (Kibana). In addition to the core Elastic Stack, there are a variety of data plugins and agents available to both push data into the Elastic Stack as well as to pull data from a variety of log sources including APIs and databases. The best part, is that while Elastic (the company) does charge for some of their more advanced plugins (X-Pack Collection), the core Elastic Stack and its data plugins are completely free.
This sounds great, but how does it help you? How does it help your organization? Due to both the dedicated team at Elastic and community volunteers and supporters, Elastic Stack comes with pre-built parsers and data normalizers for many modern log and data formats. From Cisco networking gear, to netflow, to Windows event logs, to pure JSON sources and logs, to single line standard logs and multiline logs, to many more, Elastic Stack can parse most of them out of the box or with minor modifications. Find out your formats missing, not a problem, it is extremely easy to add new formats and parsers with numerous websites available online which allow you to pretest your patterns completely before you put them into use. Even better is the dynamic indexing system used by Elastic Stack; as new fields are found in event items coming into an index, it will auto add those fields to the index so that they can be queried at a later time. It is also extremely easy to normalize your date/time fields so that Elastic Stack can time graph all of your events. Likewise, fields being imported can be specified to be string/Boolean/integer/or many more options, thus allowing for greater and more exact levels of indexing as well as more comprehensive visualization options within Kibana.
Taking Advantage and Gaining a Business Edge
Ok, this is sounding better, but how can I take advantage of this data and what is the real benefit my organization can gain from such a solution? Without any money spent on software, your organization can immediately begin to visualize events within your organization whether that be from log sources, data contained with databases, or data retrieved for APIs or other sources. Additionally, paid plugins from Elastic and new coming technology, such as Watcher, Prelert, Graph, and more, allow you to quickly view data relationships and data trends, receive alerts and take responsive actions, and much more. Further, with Prelert joining the Elastic family, your company can take advantage of machine learning within the Elastic Stack. True behavioral analytics and data trend analysis on the fly.
Elastic Stack Use Case and Business Example
Looking at an example, consider your organization sells widgets (because what theoretical company in a business example doesn’t…). Every Tuesday just after 11am, your e-commerce web servers come under extremely heavy load. The first couple times it happens you fear you may be experiencing a Denial-of-Service (DoS) attack. After a few times, you realize the timing is to coincidental for it to be a DoS attack and you start to wonder if someone is running an intensive scheduled job. After several dozen emails, phone calls, and a couple boring meetings, you are no closer to knowing the truth of the matter and you know it is only a matter of time before it happens again. What if your company had an Elastic Stack…? How could it help…?
When the event occurs the first time the data flows into Elastic Stack in the form of logs from your e-commerce servers, you are able to quickly search, view, and visualize the data in Kibana, from top hosts, to geo IP locations, to pages accessed, etcetera. You immediately are able to determine that the traffic is perfectly normal traffic, albeit a lot of it, and it is neither malicious, or an expected job running against the system. Further, you actually knew about the “event” before it becomes a major issue because the Elastic alerting component was able to detect the large increase in log traffic from those hosts and the increase in memory and CPU utilization as was discovered by the Elastic Stack system metrics plugin. Correlating that data, the Elastic Stack sent you, the administrator, an email advising you to look into the situation. Now that you have determined the event is benign, you still must discover the root cause of such high load to determine if you need to prepare for it again. Luckily, you are utilizing the various social media and email monitoring plugins available for Logstash. Because of this, you are quickly able to correlate that directly before the traffic storm, a large email and twitter campaign went out which generated significant hype for your company’s widgets. You present the data to your boss, informing him of the root cause of the degraded performance of the e-commerce servers. You inform him that if your company continues to successfully market their widgets, more e-commerce servers will be needed to handle the load. He thanks you and asks you to provide hard data for the purchase justification for new hardware. After all, just because a lot of people visited the site doesn’t mean purchases were made, and management certainly wouldn’t approve additional resources without a favorable ROI. Thankfully, your Elastic Stack also uses a JDBC connector to view data in your organizational databases. This being the case, you are able to easily overlay your e-commerce traffic logs with the purchase orders from the database – all of which are readily and immediately available in beautiful Kibana graphs that can be immediately shared with your management team.
Your management team is extremely pleased with the data you have provided. They ask you to monitor the situation over the next few weeks as the campaigns go out and inform them if the trends continue. By the third and fourth weeks, it has become a trend, not only seen by yourself, by the trend has been observed by Prelert which now warns you ahead of time that a period of degraded performance and high traffic is approaching on Tuesday mornings. The data you have gathered and acted on is invaluable and will help your organization make important decisions in the days ahead.
Applying the Solution to Multiple Problems
Now, while this was a simple IT and business example, it is easy to consider how this might extend into the areas of data analysis for cyber security, service health monitoring, and much more. For those companies that have complex mandate requirements, an Elastic Stack may not be sufficient for all of your needs; however, it is a great tool to have on the front line analyzing and rapidly normalizing traffic and then passing off to an official SIEM solution just the subset of data needed and required for mandate compliance. This allows for more optimal SIEM performance. It also helps make a SIEM solution a more cost affordable purchase since you are limiting the data set which will be sent to the SIEM. Is your company into big data? Not a problem. Utilizing Elastic Stack as your frontline data collection and correlation service present data directly to Hadoop or other big data engines. Equally, Elastic Stack can then be used to retrieve datasets from Hadoop back into the stack for further visibility and analysis.
Looking Ahead
Ultimately, the world of data will continue to rapidly change. Log sources will change, data sources will change, the very data itself will not look the same 5 years from now as it does today. However, that doesn’t mean your organization needs to remain in a data rut. Strive to find a log and data correlation and analysis tool that works for you and that your company will use. It could be the difference in whether or not your company is successfully growing 1, 2, 3, 5, or 10 years from now. Elastic Stack may not be the right fit for your organization, but we strongly encourage you to find a tool that is. Don’t necessarily seek a cookie-cutter, one-size-fits-all solution. One of Elastic Stacks biggest strengths is its ability to integrate with other utilities and software solutions both on the input and the output. Seek out the tool that fits with your organization, your data, and ultimately your people. The only thing worse than not having a necessary solution is having a necessary solution that no one wants to use.
Don’t Know? Ask. Help is Available!
If you have questions or would like advice, please reach out. There are entire online communities dedicated to Elastic Stack and many other log management solutions. Additionally, our architects and engineers are also available if you would like assistance determining which product or products are the right fit for your company. Start the journey towards greater data insight into your business today.
Request Information
[contact-form-7 id=”2544″ title=”Blog Post Contact Form”]
