Tag: IoT Processing

Categories
News Tech

A Brief IoT Processing Tutorial – Apache Nifi 1.7 and MiniFi 0.5

In this tutorial, you will learn the basics of using IoT development by using the DeepLearning protocol. The process involves IoT data reading, Big Data processing and data ingestion in order to track simple events, data processing, flow, conversion and storage. Apache Nifi 1.7 and MiniFi 0.5 are mandatory for data ingestion.

Prerequisites

  1. A Raspberry Pi, to which you add a USB camera, and install Python, Sensor libraries, Java 8, Minifi 0.5, Apache MXNet and the latest patches.
  2. The MiniFi will push files such as JSON and JPEG to the Apache NiFi server. You can also choose to push the data to a Nifi/Kafka cluster instead, as long as they use a HDF 3.2 environment.
  3. The Apache NiFi cluster will push the data to Dockerized API, Hive and HDFS which work in a HDP 3.0 or similar third party IPs.
  4. Nifi and Kafka can be used in conjunction with Schema Registry in order to compile tabular data and JSON GPS data.

I personally prefer to stream data into Apache ORC files and use them in order to compile external data tables

I use command lines in order to create two distinct tables and storage locations. The tables are generated automatically and they are stored at the designated location as ORC locations.

The general command is: ‘’CREATE EXTERNAL TABLE IF NOT EXISTS (insert name)’’

In order to facilitate the processing task, I also create a Hive 3 acid table:

  • create table rainbow acid (add the line between brackets)
  • After it is done I load the data
  • Insert in x acid (where x is the name of the file)
  • Select * from x

I then use the most recent version of Hive in order to sort out and separate data. I hope this tutorial was useful for you and that it will help you to get started in IoT. Cheers!

Exit mobile version