Elasticsearch and IoT: Match Made in Heaven?

With a mushroom-like growth in the implementation of IoT practical use cases, there has been much thought about how to manage and use the vast amounts of data produced by the IoT devices. Among many options available in the market Elastic Stack is fast becoming the preferred choice among the available options. This article will introduce you to what Elasticsearch can do and how it helps with IoT analytics.

What is Elastic Stack?

Elastic Stack is a group of open source products from Elastic, designed to help users collect data from any type of source, and then search, analyse, and visualise that data in real-time. It consists of different components including Logstash, Elasticsearch and Kibana.

How to use Elastic Stack for IoT Data Analytics

Elastic is a Powerful Analytics Engine

Most IoT applications require real-time aggregations. Logstash is capable of both time-based and hierarchical aggregations. Multiple readings from devices (for instance temperature and humidity) can be aggregated into a single event, which helps users detect changes in a larger environment and facilitate forecasting using a variety of factors.

Here is an IoT dataset collected from meters in a smart home. After the process of aggregation, each event contains both weather information and energy data from household appliances like dishwasher, fridge, etc.

Kibana Lens

Kibana includes a powerful Data Visualizer which you can use to learn more about your data. In particular, if your data is stored in Elasticsearch and contains a time field, you can use the Data Visualizer to identify possible field candidates for anomaly detection.

It is said that the best way to understand data is to visualize it. Kibana Lens is an easy-to-use, intuitive UI that simplifies the process of data visualization through a drag-and-drop experience. This is particularly useful with ever-changing customer demands as IoT data volume continues to grow. With dashboards, IoT streaming data can be converted from one or multiple index patterns into panels. This can bring focus to the data which is important and can help in forming a story to explain the available data. It is possible to display this data in charts, tables, maps and compare panels to identify patterns and connections with subsets of the data. It is possible to copy and close panels, download and share dashboard data. Dashboards can be shared as Embedded code, Permalinks, PDR and PNG report formats.

Below is an example showing how Kibana Lens was used to visualise the total energy consumption over time, broken down against appliances.

It is also possible to identify the relationship between the total energy consumption and weather conditions like temperature and humidity.

Machine Learning

With the latest Elastic machine learning feature, you can detect anomalies in your dataset. As data sets increase in size and complexity, the human effort required to inspect dashboards or maintain rules for spotting infrastructure problems, cyber-attacks, or business issues becomes impractical. Elastic Machine Learning features (such as anomaly detection and outlier detection) make it significantly easier to identify suspicious activities with minimal human intervention.

In this example Elastic Machine Learning was used to detect patterns in the energy consumption data of a smart home kitchen. By plotting the total consumption value against time the model can recognise when there is an unexpected change in the consumption and mark it as an anomaly.

The next screenshot illustrates the forecasting capabilities, by forecasting for the next 1 hour based on identified influencers such as weather conditions and appliances data.

Alert and Action

Most IoT applications require real-time alerting. With no points for guessing Elasticsearch and Kibana again come into picture with excellent native alerting capability. Alerts work by running checks on a schedule to detect conditions. When a condition is met, an alert instance is tracked and followed by one or more actions. Actions typically involve interaction with Kibana services or third-party integrations. Connectors allow actions to talk to these services and integrations. Reading data coming in from IoT devices, any event that indicates abnormal device status can trigger alerts via email or even on slack.

For example, to be notified when the machine learning job in the previous step detects an unexpected change in energy consumption in the smart home kitchen, an alert can be configured to be triggered when an anomaly is found.

It is also easy to choose how notifications are sent from a range of options.

Auto Scaling

IoT data can grow at a fast pace over time, Elasticseach’s auto scaling feature enables an operator to configure tiers of nodes that self-monitor whether or not they need to scale based on an operator-defined policy. Then, via the autoscaling API, an Elasticsearch cluster can report whether or not it needs additional resources to meet the policy.

Supported by a powerful server-side data processing tool

Logstash is an open-source data pipeline that can pull and blend data from diverse sources in real-time. Logstash is most suitable because it can source IoT logs and events from various sources. There are different inputs with different technologies, locations, and services supported. By utilizing these inputs, data can be imported and manipulated as per specific requirements and can later be sent to other systems for storage and processing. In IoT ecosystems the discussion is almost always focused on the multiple sources of data and thus important to make sure they are correctly tagged.

Just like Logstash inputs, Logstash outputs come with a number of outputs that facilitate pushing events to various locations and technologies. It is very much possible to store events in CSV or convert them into messages or send them to various services like IRC (Internet Relay Chat). The number of combinations of IoT inputs and outputs in Logstash makes it a versatile event transformer.

A filter plugin performs intermediary processing on an IoT event. Filters can be applied conditionally depending on the characteristics of the event.

Same tool for Operational Monitoring

There are many IoT applications in the market which can be directly integrated with Elasticsearch. Kibana provides the ability to monitor the entire elastic stack out of the box. Application-related alerts can be configured to make sure issues are addressed before the application/system goes down.

Once your IoT data is ingested into Elastic stack you can answer some of the most critical questions that challenge your daily IoT ecosystem operations like:

How secure is my IoT environment?

How my IoT ecosystem might be exposed to a potential cyber security attack?

What is the operational efficiency of my IoT devices? When they are required to be replaced?
What optimization opportunities are there in my IoT setup?

Why Elastic when we have multiple IoT platforms available in the market?

This brings us to the very important question as to why choose elastic when we have quite a few established IoT platforms available in the market. While choosing the IoT platform some of the most critical features that need to be considered are

Device Management – the capability to identify and correct individual devices and send updates remotely.
Integration Capabilities – Integration with 3rd party applications via APIs, SDKs and gateways.
Analytics – Having powerful tool to produce complex analysis of real-time data, automatic/machine learning and artificial intelligence.
Visualization Features – Capability to provide graphics that allow users to understand processed data.
Processing and Management of actions – Automated actions based on sensors’ incoming data.

As mentioned above, elastic ticks all these boxes. In addition to that it is effortless to change and add new features in your IoT platform with elastic, whereas with conventional IoT platforms you’d have to go through time and money consuming change request processes even for minor releases.

IoT platforms should be adaptable and resistant to change. Otherwise with the ever-changing customer needs and technological improvements you might run into issues and be forced to use technology that is no longer sufficient for the demands of the time.

Lastly, a major factor why elastic should be preferred is its ease of use. The whole point of IoT is to make your life and business processes easier. It shouldn’t add an extra layer of difficulty and complexity to your systems. Elastic based IoT platforms are straightforward and easy to integrate with existing processes.

Conclusion

The best thing about Elastic stack is that all features mentioned above come within a single stack. It means that in no time you will be able to set up your IoT analytics platform.

It is important to mention here that improper elastic implementation for IoT use cases can be extremely costly from both financial and management points of view. ELK can be expensive for IoT when not designed properly. We have seen examples where organizations have spent more than $25,000/month on a setup of less than 20,000 IoT devices. This is where you need Elastic expertise. Check our website to see how our elastic certified engineers can design, develop and implement your IoT analytics and security platform in the most cost-effective manner.