Why ElasticSearch?
- Its open source
- Its hugely scaleable
- Ideal for time series data
It is part of the elasticsearch stack which can provide functionality for the following:
- Graphs (Kibana)
- Analytics (Kibana)
- Alarms
What is Covered in This article
We are going to set up a single elasticsearch node on a Linux Ubuntu 16.04 server and use it to collect data published on a Mosquitto MQTT server. (It assumes you already have your MQTT server up and running.)
For full information and documentation, the IoT open source stack project is now called Zibawa and has a project page of its own -where you will find source code, documentation and case studies.
Installing ElasticSearch
Create a new directory myElasticSearch
mkdir myElasticSearch cd myElasticSearch
Download the Elasticsearch tar :
curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.1/elasticsearch-2.4.1.tar.gz
Then extract it as follows :
tar -xvf elasticsearch-2.4.1.tar.gz
It will then create a bunch of files and folders in your current directory. We then go into the bin directory as follows:
cd elasticsearch-2.4.1/bin
And now we are ready to start our node and single cluster:
./elasticsearch
To store data we can use the command
curl -XPOST 'localhost:9200/customer/external?pretty' -d ' { "name": "Jane Doe" }'
To read the same data we can use
curl -XGET 'localhost:9200/customer/external/1?pretty'
If you can see the data you created, then elasticSearch is up and running!
Install the Python Client for elasticsearch
pip install elasticsearch
Install the PAHO mqtt client on the server
pip install paho-mqtt
Create a Python MQTT client script to store the MQTT data in elastic search
Use the script mqttToElasticSearch.py which uses both the MQTT Paho and ElasticSearch python libraries. You will need to modify the lines at the top depending upon the port and IP address of your MQTT installation.
You can download the file from
https://github.com/mattfield11/mqtt-elasticSearch
Or if you have GIT installed use:
git clone https://github.com/mattfield11/mqtt-elasticSearch.git
The script should be installed into a directory on the same server as you have ElasticSearch running.
Run the Python MQTT client we just downloaded
python mqttToElasticSearch.py
To view the data we just created on elasticsearch
curl 'localhost:9200/my-index/_search?q=*&pretty'
We are now storing our MQTT data in elasticsearch!
In the next few days I will publish how to view MQTT data in Kibana where we will make graphs, and analyse the MQTT data.
Further Information
Zibawa – Open source from device to Dashboard. Project, applications, documentation and source code.
ElasticSearch
Running as a service on Linux I didnt use this, but probably should have!
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service.html#using-systemd
ElasticSearch Python Client
https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html
very detailed one, good
did you publish the MQTT kibana article ?
Yes Mathew, I also did the kibana article. You may also be interested to know that We have now published our code on github. You can find full documentation at https://zibawa.com. The code should still be compatible with Kibana and elasticsearch, but now we are using InfluxDB and Grafana, because of the greater security options available for multi user environments.
Hi,
I wrote the article about using Elastic for storing IoT data. Maybe it can also help.
You can find it here : https://www.researchgate.net/publication/319243453_Building_an_IoT_Data_Hub_with_Elasticsearch_Logstash_and_Kibana
Nice article for Python fans. If you are not so deep in programming, be aware that there are some log shippers like Logagent (a Logstash alternative) support MQTT input and Elasticsearch output plugins – so you could collect MQTT messages without any programming and run the Logagent service on Linux. See documentation: https://sematext.com/docs/logagent/input-plugin-mqtt-client/
Logagent could also collect log files (e.g. sensor data log file) or SQL data and ship it via MQTT to any MQTT broker. The tool is scriptable with JavaScript to transform data structures or filter data by a custom logic.
For the dashboards you could use Sematext Cloud (free plan available to store data with Elasticsearch compatible API), Kibana or any other tool working with Elasticsearch API like Grafana.
Hi, I’m looking for some inspiration how to store data from my mosquitto in the ES and I found this article. Did you consider going by the way of full Elastic Stack and store data with the logstash and own Beat instead of this python script which talks to ES directly? Are they any Proc/Cons? Thank you.
* Pros/Cons
Hi, sorry for my late reply. Yes, logstash is a great way to go. I recently setup a mqtt broker using rabbitMQ mqtt plugin, and then used logstash to pull events from the broker. That obviously requires you to install and configure a bit of software, but will give you a scaleable production ready solution. This article was really just written for fun in my early days with IoT. Logstash is definitely the better way to go.
Hi
i didn’t understand the last. i need to know how to see data from mosquitto in elasticSearch