Tag Archives: mqtt

MQTT and Kibana – Open source Graphs and Analysis for IoT

Following my previous article on how to interface MQTT with ElasticSearch, here, I am belatedly following up with an article on how you can use Kibana to graph the data.

Pre-Requisites.

You should have run through my tutorial on MQTT with Elastic Search, so that you actually have some data to look at.

Installing Kibana

To avoid compatibility issues, you should ensure that you are working with a version of Kibana compatible with your elastic search installation.  The easiest way to ensure this is by updating both.   I won’t repeat the instructions which are available on the Kibana web site.

Kibana and ElasticSearch are two parts of a single product offer, so there is very little difficulty in getting them to work together.

MQTT data format for Kibana

Kibana is ideal for working with time series data. The only tricky thing that I found using Kibana was to get it to interpret time data as a time, rather than string, or numeric.   For this you need to create a Mapping for your elastic search index, which in other words tells elasticsearch that the data you are sending is to be stored and interpreted as a time rather than string or integer.

mappingJson={"mappings": {
  "json": {
    "properties": {
      "timestamp": {
        "type": "date"
        },
     

      "dataFloat":{
      "type": "float"
      }
 


}
}
}
}

The above mapping will tell elasticsearch to expect data with three elements, a timestamp and a float value called dataFloat.  Most importantly this will cause elasticsearch to try to interpret the timestamp field as a time rather than storing it as a string.

Analysing MQTT Data with Kibana

Once you have got elasticsearch to interpret your data as a timestamp, you are able to take advantage of all of the functionality of Kibana that comes out of the box, this includes counts, averages values, derivatives and many others.

Security

The set up we described is great for prototyping in a closed environment, but as we have been developing the project, we found ourselves hampered by the lack of security features on Kibana.  It is possible to provide basic login functionality using NGINX, but we could not find an easy way to provide restricted access to data according to account (this is a paid feature in elastic/Kibana).   For this reason we have started to use Grafana with InfluxDB as an alternative.

Zibawa Open Source Project

Zibawa is a project which brings together a number of open source tools to produce a secure IoT system , fully open source from device to dashboard.  The project includes device manager, device and user security management (LDAP), queue management and monitoring (RabbitMQ), Big data storage and api (InfluxDB) and Dashboards (Grafana).

 

More information

https://www.elastic.co/guide/index.html

Zibawa Open IoT project source code

 

 

 

 

 

 

opensource

Introducing the Open Source IoT stack for Industrie 4.0

The open source IoT stack is a set of open source software which can be used to develop and scale IoT in a business environment.  It is particularly focused towards manufacturing organizations.

Why Open Source?

Continue reading “Introducing the Open Source IoT stack for Industrie 4.0” »

bigdata

Storing IoT data using open source. MQTT and ElasticSearch – Tutorial

Why ElasticSearch?

  • Its open source
  • Its hugely scaleable
  • Ideal for time series data

It is part of the elasticsearch stack which can provide functionality for the following:

  • Graphs (Kibana)
  • Analytics (Kibana)
  • Alarms

What is Covered in This article

We are going to set up a single elasticsearch node  on a Linux Ubuntu 16.04 server and use it to collect data published on a Mosquitto MQTT server.  (It assumes you already have your MQTT server up and running.)

For full information and documentation, the IoT open source stack project is now called Zibawa and has a project page of its own -where you will find source code, documentation and case studies.

Installing ElasticSearch

Create a new directory myElasticSearch

mkdir myElasticSearch
cd myElasticSearch

Download the Elasticsearch tar :

curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.1/elasticsearch-2.4.1.tar.gz

Then extract it as follows :

tar -xvf elasticsearch-2.4.1.tar.gz

It will then create a bunch of files and folders in your current directory. We then go into the bin directory as follows:

cd elasticsearch-2.4.1/bin

And now we are ready to start our node and single cluster:

./elasticsearch

To store data we can use the command

curl -XPOST 'localhost:9200/customer/external?pretty' -d '
{
"name": "Jane Doe"
}'

To read the same data we can use

curl -XGET 'localhost:9200/customer/external/1?pretty'

If you can see the data you created, then elasticSearch is up and running!

Install the Python Client for elasticsearch

pip install elasticsearch

Install the PAHO mqtt client on the server

pip install paho-mqtt

Create a Python MQTT client script to store the MQTT data in elastic search

Use the script mqttToElasticSearch.py which uses both the MQTT Paho and ElasticSearch python libraries.  You will need to modify the lines at the top depending upon the port and IP address of your MQTT installation.

You can download the file from

https://github.com/mattfield11/mqtt-elasticSearch

Or if you have GIT installed use:

git clone https://github.com/mattfield11/mqtt-elasticSearch.git

The script should be installed into a directory on the same server as you have ElasticSearch running.

Run the Python MQTT client we just downloaded

python mqttToElasticSearch.py

To view the data we just created on elasticsearch

curl 'localhost:9200/my-index/_search?q=*&pretty'

We are now storing our MQTT data in elasticsearch!
In the next few days I will publish how to view MQTT data in Kibana where we will make graphs, and analyse the MQTT data.

Further Information

 

Zibawa – Open source from device to Dashboard.  Project, applications, documentation and source code.

https://zibawa.com

ElasticSearch

https://www.elastic.co/

Running as a service on Linux I didnt use this, but probably should have!

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service.html#using-systemd

 

ElasticSearch Python Client

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html