TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer)

Test Case Title

TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer)

Goal

See all IOT device changes from a unique location to search them via Developer Dashboard (already shown in previous TC)

Have in a unique location IOT device changes and ETL changes, index them, browse them and search them via the Developer Dashboard.

Modify the data driven filtering for saving / indexing data

See results on Developer Dashboard.

Prerequisites

Using a PC or Mobile with a web browser. Access to the back office since this process for changing the flow if needed. The implementation is performed into a data driven VM based on NIFI that can be scaled up.

The following functionalities are available only for specific Snap4city users with specific privileges.

Expected successful result

See the resulting referral data a real-time data on the developer dashboard

Steps

 

 

The ETL data is stored in tabular form on Apache Phoenix cluster while IOT streams are retrieved by querying the different IOT broker URLS . In addition, the SOLR index is defined as following json structure

{"measureType":"",

        "src":"<ETL|IOT>",

        "latlon":"",

        "unit":"",

        "deviceName":"",

        "className":"",

        "subMeasureType":"",

        "kind":"",

        "dateTime":"",

        "serviceUri":"",

        "id":"",

        "value":",

        "sensorID":""

        }

Because one IOT device data observation correspond to multi measures (for example “Sensore via Bolognese” measures road temperature, air humidity, etc .. ), a process of denormalization (i.e. having single row for each measure) is needed to permits faceted search on the corresponding SOLR index.

Apache NIFI Framework is used to process the data gathered from Apache Phoenix tables by SQL queries and IOT brokers, denormalize and then update the SOLR index dynamically.

In the figure below is depicted a proof-of-concept of the process of data-retrieve, enrich and indexing. The process is composed of three parts: ETL management, IOT management and store and index management.

 

For each table stored in Apache Phoenix produced by an ETL process is identified a NIFI process Group (e.g.: weatherSensorObsrvation, smartbenchobservation etc … See Figure below).

Each Nifi Process Group is composed by a part of data retrieving and a part of data manipulation permitting to capture each  measure per measure per type e.g. road temperature, pressure.

An example of a filter is drawn the figure below.

The interesting data are extracted and a custom json is constructed with valid SOLR data type and then committed to SOLR index and Hbase table.

Click on https://devdash.snap4city.org/

See the resulting Developer dashboard:

TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer) | Snap4City

Error

×

Error message

  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Unable to send e-mail. Contact the site administrator if the problem persists.
  • Notice: Trying to get property of non-object in user_is_logged_in() (line 1576 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_is_logged_in() (line 1576 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in og_field_access() (line 578 of /var/www/html/drupal/sites/default/modules/og/og.module).
  • Notice: Trying to get property of non-object in user_access() (line 806 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 818 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 819 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 825 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 828 of /var/www/html/drupal/modules/user/user.module).
  • Warning: Cannot modify header information - headers already sent by (output started at /var/www/html/drupal/includes/common.inc:2791) in drupal_send_headers() (line 1499 of /var/www/html/drupal/includes/bootstrap.inc).
  • Error: __clone method called on non-object in og_user_access_entity() (line 2249 of /var/www/html/drupal/sites/default/modules/og/og.module).
The website encountered an unexpected error. Please try again later.