MicroServices Suite for Smart City Applications

 

Keywords: Smart City, IOT Applications, MicroServices, Node-RED, Dashboard.  

 

1.Introduction

 

The concept of Smart City is becoming increasingly important and pervasive, making it a major area for research, business and policy makers [Gupta, 2019]. Smart Cities are currently required to provide flexible architectures, in order to satisfy new functional and non-functional requirements that arise and constantly evolve in many different contexts and environments. There is also the necessity of scaling up to handle the increasingly number of users, devices, and services. However, the current common approach is still to develop ad-hoc applications for specific domains, which leads to the production of non-interoperable services [Esposte, 2017]. Smart Cities are becoming every day more focused on activities related to IOT and IOE (Internet of Things/Internet of Everything). This has provoked a switch of technology and approach in field data collection. In the past, most of the city generated data  have been collected from domain specific vertical applications, by sending data to dedicated concentrators (for example, the collection of data from traffic lights, traffic flow sensors, parking, etc.). Each and every vertical application was using its own solution for data collection either via radio, or any other means. On the other hand, the microservice architecture paradigm has arisen from wide-scale industries requirements of building and maintaining large-scale distributed systems [Krylovskiy 2015], supporting an adequate availability, scalability, modularity, as well as flexibility [Fernandez 2019]. The new push of IOT is stimulating cities to adopt the same gateways to collect data of multiple services. This means that the infrastructure of the sensor network has to be properly designed and planned by the city operator. LoraWan can be an option [Gomez 2012], while in many cities 5G solutions are coming and would replace former technologies in short time, at least within the large cities [Li 2018].

    However, the arrival of a more aggregated IOT streams via 5G is not a problem for smart city data aggregators serving the Smart City Control Room [Bellini 2018]. Presently, they collect data from several heterogeneous sources, and to have a reduction in the number of data concentrators to be aggregated would be a benefit not a problem. On the other hand, the IOT/IOE paradigm is also strongly impacting on the infrastructure management [Bonomi 1012], [Li2 2018], since the systematic adoption of IOT has also led to the adoption of PUSH approach (event driven protocols) and it has brought forth the possibility of acting on actuators and not only sensing the city, not to mention the possibility of creating Event Driven Applications. They are also called IOT Applications which have to be capable to process messages and produce reactions in real time [Su 2011]. For many aspects Smart City application are one of the most complex cyber physical systems, CPS, due to their complexity in terms of data, data analytics, and interfaces with the real world, physical and digital on the user interface.

     In the state of the art, several different definitions and architectures have been proposed for IOT frameworks [Sethi 2017]. Most of the IOT platforms adopt a code-based development for the design of IOT Apps. Some of them provide development tools as: Eclipse, or JavaScript. Others provide simple tools as IFTTT [IFTTT], relying on rule-based scripts [Badii, 2013]. Other platforms adopt solutions similar to ETL (Extract Transform Load) processes [ETL Penthao] capable of working for event driven processes, or just surrogating it with high rate polling. One of the most promising tools for creating IOT App is Node-RED [NodeRED], based on Node.JS, which has been proposed by JS Foundation  as fully open source. The Node-RED approach is a mix of visual composition of nodes/blocks to compose the so-called flows that are concurrently executed by a Node.JS engine. It is quite diffused, being also directly provided into official releases of IOT devices as the Raspberry Pi family. It is provided with a minimum set of functionalities (the building blocks/nodes), while other blocks can be easily added loading them from a large library made available by the JS Foundation. The classical nodes provided in the standard version can be classified as: input, output, function, social, storage, analysis, advanced, and dashboard; such nodes are not sufficient for creating Smart City IOT Applications. Actually, for the use in the context of Smart City it was not powerful enough to cope with the basic requirements of such domain, even though its use in the field is quite widespread. In the next section, many other solutions have been considered and compared, in order to select the starting point of our research and innovation action, i.e. the Snap4City platform funded by European Commission, as described below.

 

1.1 Related Work

Several Smart City use cases have been presented so far in literature, transforming the collection of huge quantities of data related to urban environments into useful content for various stakeholders (citizens, tourists, local government and companies) [Lim 2018]. Among them, some of the most notable use cases concerns the cities of Santander [Díaz-Díaz 2017], Manchester, Barcelona [Agbali 2019], Singapore [Cavada 2019], Seoul, San Francisco [Lee 2014], Rio de Janeiro [Kitchin 2014] and Florence. Most presented solutions however still focus on specific domains, aiming at resolving particular problems, with little software reuse [Santana 2017].

        In order to manage the high variety of devices and applications, including IOT, mobile, web and service-oriented framework, the Microservice architecture is being increasingly adopted in recent IOT based solutions for Smart Cities. Actually, Microservices architecture focus on the development of simple and loosely coupled services [Zimmermann 2017], enhancing scalability and availability, facilitating maintenance and fast isolated testing. These aspects aim at simplifying the complexity of traditional service-oriented architecture (SOA) [Dragoni 2017].

    The programming of IOT Applications is performed in several different manners by different tools [Yasumoto, 2016]. For example, in AWS IoT ecosystem a complex stack of Amazon’s services is proposed, which enables developers to define any business logic by programming the flow of IoT Devices’ data towards their visualization in proprietary dashboard. The IoT Devices can activate AWS functions written in: Java, Python and C#. These functions can implement business logic or trigger actions in the IOT world or Cloud. Data rendering is flexible and can be used for data inspection and reporting. Azure IoT provides a development environment to build business logics. The integration with mobile applications is performed by using the Xamarin platform in C# and .NET. In addition, it is possible to write application also in Java, Node.js and Python. For the creation of dashboards, developers can use MS Business intelligence tool. In Google IoT platform, several programming languages can be exploited to program the data flows from device to dashboards, like Java, Node.js, Python, Go, Ruby, PHP, C#. In such cases, data flows are implemented by using programming languages. Not any of them provide a visual editor for IoT Application flow definition to be used even without any software programming skills. In addition, these solutions often require the mandatory use of cloud services, and they are not suitable for any autonomous deploy on-premise scenario. As a limit case, they may permit to have some modules configured on premise IOT Edge with reduced capabilities connected with the cloud solution.

       In [Ray, 2017], a survey on Visual Programming Languages (VPL) for IOT has been proposed. In the survey 13 different VPLs have been considered. The analysis has been mainly focused on comparing them on the basis of the programming environment, licensing, project repository, and supported platforms. Some of them are Open Source platforms, while others are proprietary. Among the Open Source platforms (Node-RED, NETLab, Ardublock, Scratch, Modkit, miniBloq, NooDL), only some of them can be programmed using a Web interface and can be also executed on some dockers or virtual machines on cloud. In [Contreras Masse, 2019],  the usage of haptic interfaces in the context of IOT Applications for smart manufacturing has been reviewed. The analysis has produced a classification and the identification of the main elements needed to the exploitation of haptic devices without entering the problems of the microservices and VPL for programming IOT Applications.  

In the assessment of Open Source VPL for IOT, several important aspects should be taken into account, such as:

• performance in terms of number of messages processed per second, per minute;
• diffusion of the tool, namely the size of the communities supporting the tool, both as developers and users developing applications and additional features;
• openness in terms of flexibility to create new blocks to extend the basic functionalities, for example, exploiting external services, developing data analytics, etc. For example, in the domain of smart city or industry 4.0, you could add visual modules representing the IOT Device elements, as well as Data Analytics;
• Cloud/docker support for their use on cloud and/or support for any execution on embedded IOT, thus executability on several operating systems. Some of these VPLs produce code only for specific embedded devices. On cloud the automatic resource provisioning and the elastic scaling of resources are needed to manage large amount of IOT applications;
• level of expressivity: an IOT VPL can be oriented on defining functional or data driven/flow aspects. The first are typically less expressive, since such constructs are related to the programming language: section, assignment, condition, iteration, etc. Data Flow models are more expressive and may be easily extended with new blocks/functions and this lack of expressivity may be solved if the right blocks are identified. Therefore, this can involve as a result to produce simple data flows even for complex problems. In fact, the complexity of the graphs produced for solving problems in the applicative domain can be measured in terms of number of blocks/modules and connections used to solve such problems;
• usability, including visual editing of data flow via web browser or through specific client tools, etc. The fundamental features, such as: inputs, outputs, functions definition, social gathering, storage load/save, data analytic and dashboard (user interface), may be not enough for creating complex Smart City IOT Applications with low effort. On the other hand, one may have a well-designed VPL, and the effort may be vanished due to the lack of expressive and domain suitable MicroServices;
• managing resources: the possibility of developing IOT applications (as well as portions of them) as data analytics and share them in a marketplace. In the context of VPL for IOT, the development of Data Analytics is quite complex, since processes may have a lot of dependencies and the programmers’ skill may imply the knowledge of the data storage and models, in addition to the knowledge of algorithms for data analysing. This approach may require the involvement of experts of sensors, algorithms, infrastructure, programming, knowledge modelling, domain etc.

 

Some solutions that can be suitably used to address such aspects and overcome the problems described above are, for example: Node-RED (it can be executed on multiple operating systems, it has a visual programming via Web interface and the source code can be loaded on several different embedded devices, etc., [NodeRED]), NETLab is an open source development environment for embedded system [NETLAB]; Ardublock is an Open source block programming for Arduino, mainly functional approach and limited to specific devices [ARDUBLOCK]; Scratch is a visual tool of MIT, adopted for code generation for IOT, very low level programming model, not event driven, functional; Modkit is strongly oriented to Embedded IOT devices; miniBloq is a block functional programming at  very low level; NooDL allows the fast prototyping of the user interface and it is less suitable for data analytics; VISUINO is an open source tool to  develop code for embedded IOT, but providing limited capabilities to be used on cloud; Kura is an Open Source Java visual tool for IOT; FLOGO is an Open Source tool for data flow in Go, natively works on AWS Lambda, [FLOGO] or Wyliodrin a development environment for IOT edge embedded devices, deploy and update.

    Most of the above solutions are strongly focused on producing code for embedded devices with functional model and not data driven/flow, such as: Scratch, Modkit, miniBloq, etc. Therefore, they are not suitable for creating IOT Applications that can be put in execution on IOT Edge, as well as on cloud docker containers, addressing the complexity of Data Driven applications and flows; exploiting Data Analytic. Solutions such as AWS, IOT Azure, and IOT Google are more oriented on traditional programming. The VPL tools based on data driven, open to extensions, supported by large communities, that can be put in execution on both edges, could be only a few like: Node-RED and maybe the emerging FLOGO. In [Roukounaki, 2015], Node-RED, flow editor has been used for creating smart city IOT applications for projects such as VITAL. The suite of nodes implemented is not focused on functional aspects of IOT Applications, so that the resulting flows are still too much focused on technical aspects and not user centred. In fact, as depicted in [Roukounaki, 2015], even the creation of simple activities as the identification of traffic flow sensors results into the development of complex workflows with a high number of nodes/blocks. Specialized workflow formalization models have been also proposed, such as in [Chattopadhyay, 2015]. In [Dey, 2015], a Model Driven Development (MDD) process has been proposed consisting of (i) a tool for creating semantic algorithms, and (ii) a workflow generator on the basis of the first results. The user interface is a subset of Node-RED environment.

       There is a few IOT development environments where algorithm repositories are provided for their exchange in the community, such as in [Pal, 2014], [Aldasoro, 2011]. This feature should be mandatory in cloud based IOT development environments such as: Axeda [AXEDA], BlueMix [IBM], ThingWorx [THINGWOX]. On this last aspect, there are two levels in IOT valorisation environments, such as:

• IOT platforms marketplaces, such as BeeSmart City, Fi-WARE, EOSC, etc., which are typically constrained to a class of solutions or open to all. Their target is the promotion and marketing of IOT or Cloud solutions.
• IOT Applications and solutions resources sharing and marketplaces, such as: Node-RED library, which are typically focussed on specific IOT kind of development processes, and on promoting the interchange.

 

In this paper, the issues related to Snap4City IOT development environment and framework are addressed. The experience described herein refers to the design and implementation of Snap4City platform, https://www.snap4city.org [Badii 2018], which is based on Km4City [Badii 2017]. Snap4City is the solution produced in response to a research challenge launched by Select4Cities PCP (Pre Commercial Procurement) H2020 research and development project of the European Commission (https://www.select4cities.eu ). Select4Cities has identified a large number of functional (mainly Smart City IoT) and non-functional requirements (open source, scalability, security, GDPR compliance, working on cloud and on premise, etc.) which are fully described in their web site and aimed at creating the best solution for the modern Smart Cities supporting IoT/IoE, in hands of public administrations and Living Labs. Most of the identified requirements have been taken from the large association of Living Lab ENOLL (European Network of Living Lab association, https://enoll.org), and upon consultation of smart cities at level of European Commission as EIP-SCC (European Innovation Partnership on Smart Cities and Communities, https://eu-smartcities.eu).

Snap4City has responded to the research challenge and could prove to satisfy all the Select4Cities requirements. Snap4City allows the creation and management of users communities, which collaboratively: (i) may create IOT Solutions, (ii) exploit open and private data with IOT/IOE respecting GDPR, and (iii) create/use processes and IoT Applications that could run on Edge, Mobile and cloud, with the capability of interacting one another and with users via messages, Dashboards and Applications. A specific attention has been given to the creation of an integrated development environment for IOT App (cloud and edge), based on VPL, with dashboards, and supporting data analytics and resource sharing, thus minimizing any required technical expertise as programmers. To this end, a large number of visual elements have been identified and developed as MicroServices for Smart City IOT. The starting point has been Node-RED, of which we could identify many breaches to cover the Smart City domain. In addition, the developed platform has a full support to give user assistance during the development life cycle, from data gathering to dashboard production, as well as in IOT App development and sharing of results with other developers and in the community as self-assessment, together with any security support as described in [Badii 2019].

It should be remarked that this paper is presenting aspects that have not been presented in our past articles in the past, and in particular: the suite of MicroServices, their motivation and requirements for smart city IOT solutions, a number of  integrated examples, the Life Cycle of development, and the validation of the approach.

1.2 Structure of the paper

   This paper is structured as follows. In Section 2, requirements for a microservice based programming solution for smart city applications are analysed and reported. Section 3 presents the Snap4City Architecture with a stress on data flow, IOT and MicroService aspects. In Section 4, the Snap4City MicroServices Library is presented and motivated. Section 5 introduces some applications exploiting the MicroServices in the context of emergency management, personal mobility, and crowd of people monitoring with mobile PAX Counters. In section 6, the Snap4City development life cycle has been formalized. Section 7 presents a validation of the solution performed with city officials in developing IOT Applications. Conclusions are reported in Section 8. 

 

2. Requirements for MicroService based Smart City Applications

As above described, the aim of the research reported in this paper has been to design and implement a visual programming environment where city operators may develop Smart City Applications with IOT by means of visual constructs. In this section, the identified requirements of the challenge are listed.

      As a first step, by analysing the state of the art we realized that a VPL for developing IOT Applications has to provide generic requirements, so as to support:

• data communications sending/receiving messages/ data, providing both push and pull modalities. This means that the language has to be capable to perform requests using many different protocols among the available ones [Dizdarevic 2019], such as pull protocols to get data (e.g.: Rest Call, Web services, FTP, HTTP/HTTPS, etc.) and push protocols to receive data via data driven subscriptions (WS, MQTT, NGSI, COAP, AMQP, etc.). In the context of IOT solution, this feature is quite common. For example, in Node-RED library, you can find a large collection of nodes covering tens of protocols, while large platforms as AWS and IOT Azure are typically limited and address just few protocols. In some cases, such limitations concern the supported security and authentication schemas, since in most cases, username and password (or authentication keys) have to be included directly into the flow in clear text, thus making not all the approaches on IOT security satisfactory [Badii 2019]. For example, in Node-RED each single IOT Device can be connected to a flow using specific credentials directly included in the code. On the other hand, a generic service for IOT Discovering and Registration is missing.
• data save and retrieval to/from some bigdata storage. For example, the capability to store data and exploit them to implement algorithms for predictions,  decision making processes,  value trends overviews, etc.
• data transformation and processing via the algorithm formalization and the possibility to start, pause, recall their execution periodically or sporadically, according to the arrival of some event or other firing conditions.
• calls of external services in order to enable the instrument of delegation to external solutions, when it comes to the computing and processing, for instance the computation of some complex Data Analytic algorithms, or the exploitation of external services such as the Social Media Analysis via REST Call. For this purpose, the classic Node-RED provides access to Watson IBM for machine learning and artificial intelligence reasoning. Desirable applications could be, for example, computing sentiment analysis, performing a clustering classification and recognition, computing some predictions on the basis of historical data of some City IOT values.
• the presentation of data via some Dashboards that would be the user interface of the application. Data visualization should be expressive enough, including values in real time, time trends, histograms, pie charts, etc.
• the use of the Dashboard as a user actuator interface, enabling the user to act: (i) on some internal variables, (ii) to send messages/ commands to IOT Devices and flows.

 

The VPL IOT platforms should present a number of non-functional requirements such as demonstrating capabilities of: robustness (in terms of availability and fault tolerance), scalability (to be capable to serve from small  to very large businesses with corresponding volumes of processes per second), security (authentication, authorization, secure connection, etc.), full privacy respect (compliance with data privacy according to the GDPR, General Data Protection Regulation  European guidelines), interoperability (e.g., communicating with any kind of protocols, devices, services), and openness in terms of open source and in terms of possibility of adding new modules and functionalities, etc. On the other hand, most IOT development environments are not web based development platforms, and are focussed only on generating code for specific embedded as Arduino, see the above mentioned: VISUINO, Ardublock, ModKIT, miniBloq, etc.

 

     In addition, by analysing the requirements identified by Select4Cities, ENOLL, EIP, etc., we realized that a large number of smart city IOT specific requirements have to be satisfied by a VPL to enable the easy development of IOT Applications in the Smart City context, at the service of City Operators and advanced City Users (among citizens, stakeholders, third party developers of SME, researchers on data science, etc.). For example, the VPL should provide specific features to address functionalities to develop applications in domains such as: mobility, environment, parking, culture, health, tourism, energy, and general city services, with no need of knowing technical details about the provided services or service supplier identity. Actually, in almost all cities, different services on mobility, environment, tourism, energy, education, etc., are provided by different operators. Moreover, some of these services are also related one another; for instance, the number of free parking lots and the events in the area, the number of weather forecasts and the number of people in city gardens, the traffic flow and the environmental data regarding NOX, etc. That is, when it is supposed to rain, the number of traffic car increase, and thus the number of free parking slots along certain hours of traffic decreases.

      On such basis, a number of specific functionalities have been identified to develop smart city applications via VPLs, which should be available in terms of MicroServices and building blocks, in order to:

• Access to Smart City data entities on the basis of their functional semantic descriptors. For example, status or prediction of “parking square Carlo IV” and not status of variable “Status” of device “sensor45”, etc.
• Discover and exploit city entities’ data regardless of their technical details, such as their push/pull protocols and gathering model, messages format, provider, etc., while focusing the search on functional aspects and ignoring all the details about the query syntax used in the specific storage (e.g.: SPARQL, SQL, ElasticSearch). For example, to provide answers to the following questions: “let me access to all temperature values in this area, path or close to a given GPS point”; or: “give me the sensors measuring PM10 (or any other environmental variables) close to my position or along a path”, regardless of the actual connection of sensor devices which would be connected to the platform by using different IOT brokers, different protocols, etc.
• Create Graphic User Interface for web and mobiles, with visual representation of complex data in dashboards, allowing also to input data into IOT Applications and the whole system. Node-RED provides a number of basic nodes to create GUI for IOT App (Dashboard nodes), whereas the offered solution presents strong limitations as to its usage in the context of smart city. They are not: presenting maps (with pins, paths, areas, etc.), heatmaps, trajectories, origin destination matrices, comparing trends, Kiviats, multi-pie charts, etc. Furthermore, they are not protected by authentication and secure connection.  In smart city domains, advanced data types need to be managed and shown, for instance: maps, paths on maps, heatmaps, collective trajectories, area shapes on maps, areas, ortho-maps, origin destination matrices, etc. Most smart city data types are related to sequences of GPS points.
• Access to Data Analytics and their production. The real need in the context of Smart City is to grant data analysts with the possibility of creating some data analytic processes (it may be in R, Tensor Flow, Python, Java, etc.) and using them into the data flows as MicroServices, with no need for a programmer to step in, nor for an administrator to take action every time. This approach may create a strong flexibility into the IOT Applications. For example, classical Data Analysis algorithms for smart city could include solutions to:
  
  • Compute the routing from point A to point B according to certain criteria, for example: using car, public transportation, bike or pedestrian; shortest or fastest; avoiding specific zones due to road works or other kinds of emergencies. This implies the production of a sequence of points connected by street segments.
  • Collect and compute metrics on social media data from Twitter (such as: counting Tweets, retweets, as well as extracting Natural Language Processing features [Nesi 2015], i.e. verbs, adjectives, citations, hashtags, and performing sentiment analysis on them). For example, computing metrics related to Tweets collected on the basis of a set of keys, Twitter usernames, hashtags, etc.
• Register new IOT Devices, regardless of their protocol, which are provided via some external or internal IOT Broker. This feature is strongly relevant to the registration of personal devices. For example, glucose meter, security devices for home monitoring, etc. The data collected from those devices have to be kept strictly private, and according to the GDPR guidelines [Badii 2019], [Valtolina 2019].
• Save and retrieve personal data, for example data coming from private devices, as those mentioned in the previous points, and thus according to GDPR.
• Load and publish Data Sets, aiming at ingesting open data and/or creating new open data sets, share them with other cities and networks. A data set is typically a file in some open format containing data description. For example, the position of benches in the city, the position and description of all the restaurants in the city, etc.
• Provide different kinds of events within the city. They can be (i) notifications of consolidated events (such as road accidents, fire brigade and/or police calls), (ii) entertainment and sport events, or (iii) potential emergency calls, not yet consolidated events which may be produced by police cops or citizens in the street just to communicate facts (for example, potholes in the street, a balcony with some problems), etc. They need to be profiled with a number of metadata, and in case of emergency events they should be compliant with CAP standard (Common Alerting Protocol) [CAP], in order to provide tools and instruments to enhance the city resilience capabilities, and be interoperable among several operators: fire brigade, local and national police, civil protection, etc.

 

3. Snap4City Architecture

In this section, the general architecture of Snap4City is presented to contextualize the successive sections where VPL aspects are discussed. The Snap4City platform has a Lambda architecture, as reported in Figure 1. Data are received in push and/or pull, and almost every data can be considered IOT data sources. They may come from IOT Brokers, social media, web server, web sockets, streams, etc.

 

Figure 1: Snap4City Functional Architecture, cloud view where the IOT Applications are only on cloud.

 

    In Snap4City, most data provided from External Services or operators are collected in pull via REST Calls by using scheduled processes written in ETL, through the Penthao Kettle tool. The ETL processes are executed (periodically or sporadically/on demand) on a cluster of Virtual Machines by a scalable and distributed scheduler called DISCES [Badii 2017]. The collected data are regularised/reconciled according to the Km4City Knowledge Base [Badii 2017], and then pushed into the Big Data Cluster storage. The reconciliation process allows the reconnection of data to city entities already in place; for example, by connecting POI (Points of Interests) to civic numbers, traffic flow to street segments, etc. The aim is creating a fully connected knowledge base for the city, where semantic queries can be performed by exploiting physical and conceptual relationships.

    On the other hand, data arriving in Push (data driven) are typically real time data streams, managed by a number of IOT Brokers which an Apache NIFI distributed cluster is subscribed to. The NIFI process is in charge of performing a regularization/reconciliation task on data, according to the Km4City Knowledge Base and then it pushes the resulting data into the Big Data Cluster storage, while indexing and thus creating the Data Shadow. The above described parallel solution tends to be normalized in a single approach based on NIFI [NIFI] when the platform is small. Otherwise, when the platform has large data volumes, a distributed data warehouse for data ingestion based on ETL is more effective. The Big Data cluster storage includes a HDFS Hbase, a Phoenix storage, an RDF store for semantic reasoning based on Km4City (which is operatively implemented on Virtuoso store), and an ElasticSearch Index. The whole solution presents textual indexes on all fields, semantic indexes, and elastic aggregations to perform advanced “faceted” filtering queries. The indexing and query supports are exploited by Snap4City Smart City APIs [Badii 2017]. When small installations are performed, the HDFS cluster can be avoided.

     The operative IOT processes (IOT Applications) can be executed on cloud or on premise. In Figure 1, the cloud case (private or public) is depicted. When IOT Applications are executed on IOT Edge that are located on the field, they may directly communicate to the IOT Applications or Dashboards on cloud or by means of IOT Brokers (on which all the other ca be subscribed). On such grounds, without the risk of losing generality, in Snap4City, the IOT Applications for Smart Cities are obtained as:

 

IOT App = Node-RED + Snap4City MicroServices

 

    The IOT App exploits basic nodes of Node-RED Node.JS plus Snap4City MicroServices that are suitable for smart city data transformation and processing.

The Node-RED platform is based on two components: (1) a web-based visual editor to design flows, and (2) a runtime environment that may execute flows. In Snap4City, the Node-RED editor has been improved to:

• communicate with the so-called Resource Manager,
• save and load IOT Applications and manage SSO (Single Sign On) with Snap4City,
• manage a large set of MicroServices, namely the Snap4City Libraries of Nodes that are accessible from Node-RED official library.

Moreover, the runtime engine of Node-RED has been also improved to (i) manage the security according to SSO and Snap4City model, (ii) execute it on Docker on the elastic management solution of Snap4City.

On the other hand, the changes performed on the Node-RED have been released open and are functional only for use on cloud for large scale. While, Snap4City library can be used on the installation of IOT Edge with standard Node-RED tools, without any restriction.

 

The Snap4City solution has formalized and implemented a large set of MicroServices satisfying the above discussed requirements. The MicroServices provide an easy and formalized access to all the Smart City services which are available in cloud from the platform (including the ones to control a part of the platform itself). They are made available into the Node-RED Node.JS environment to create IOT Applications as VPL. Among the MicroServices, the IOT applications need also to access such services allowing exploiting Data Analytics, Visual Analytics and Dashboards. The latter two aspects can be employed to create the Graphic User Interface (GUI) of the IOT Applications. These tools, orchestrated by the IOT Application flows, may automatically inform, announce, act and produce alerts and warnings on IOT Devices, networks, user interface, external services, etc., and provide support to close the loop towards the user acting/reacting on the GUI and/or Devices, including notifications.

     In the deployments of Smart City IOT as a Service, SCIaaS, such as Snap4City, and in large smart city applications, a relevant number of IOT Applications may need to be deployed on the basis of the on-line requests made by users/organizations. Snap4City has been recently accepted by the EOSC (European Open Science Cloud) marketplace of the European Commission and therefore the IOT Applications need to be managed and allocated on demand, on the basis of the users requesting them, as well as to be put in execution on cloud. To this end, an elastic (vertical and horizontal) infrastructure has been created to manage in scalable manner the IOT Applications in containers [Bellini 2019], and mechanisms to guarantee end-to-end security [Badii 2019]. In Figure 2, the user interface allows the user to manage its own IOT Applications, irrespective of whether they are IOT App on cloud or on IOT Edges/field, whether they are child processes/containers for Data Analytics or WebScraping, etc. Please note that different kinds of IOT Applications are represented by different icons.

 

 Figure 2: Snap4City: the IOT Applications manager as seen by a user.

 

      The Snap4City MicroServices abstract low-level details to the programmer using a visual environment, hiding the complexity of sophisticate algorithms and tools. This is useful and suitable, for example, to provide routing, spatio-temporal search and discovery, for data analytic, dashboarding, networking among IOT devices, IOT data abstraction, etc. The Snap4City MicroServices are distributed into two official libraries of Node-RED nodes by the JS Foundation portal (https://flows.nodered.org/?term=snap4city). The two libraries are dedicated to final users (basic), and to developers (advanced). The version dedicated to Users provides outputs of Node-RED nodes easy to be exploited for non-skilled user on JSON. In fact, most of the output produces single variables and not complex JSON structured messages. On the other hand, the version for Developers (to be installed on top of the basic version for final users), presents a number of nodes/blocks that can accept complex input JSON messages to create strongly dynamic IOT Applications.

    Both Libraries of Snap4City Nodes can be directly installed in any Node-RED tool of any operating system: Linux, Raspberry pi, Windows, etc. In addition, we have also developed an Android App executing Node.JS/Node-RED and our libraries to provide use of them on IOT Edge, also exploiting mobile device sensors on the above-mentioned operating systems. In [Badii 2019b], we have demonstrated how Snap4City approach may work for mobility and transport applications, where safety critical communications and solutions have to be set up, involving IOT networks with IOT Devices and IOT Edge, IOT Apps and Dashboards.

 

     In this paper, a deep view of the analysis and design of Snap4City MicroServices for smart city applications is presented. The analysis has been also supported by the evidence about how the MicroServices can be successfully used for IOT Applications implementation, so that they can be used by City Operators and Final Users.

 

4. Snap4City Library of MicroServices of Smart City

In order to satisfy the smart city requirements reported and discussed in Section 2, in Snap4City a collection of more than 150 MicroServices has been developed as Nodes for Node-RED programming environment. The Node-RED philosophy of visual programming allows the creation of event driven data flow applications where the exchanged messages are in JSON format. On the other hand, also periodic processes can be developed by scheduling one or more internal timers. This means that users can develop IOT Applications as Node-RED flows, exploiting both Push and Pull data protocols, with the same visual programming environment. In the context of smart cities, both protocols are needed, while IOT Applications have to be capable to create flows exploiting a large number of features which are typically not available in Node-RED open library, neither by installing a number of libraries from different providers. Moreover, the Snap4City MicroServices are at a level which can grant an easy development of IOT Applications for smart city  even for non-expert users, as assessed in Section 6.

 

The most relevant families of nodes/MicroServices for smart city are listed below and they perform different kinds of activities, useful in the IOT App construction.

 

-- Access to Smart City Entities, that have different data models, and thus, different MicroServices may be required. Some Entities may have simple sets of Metadata for examples the ones describing the POI, e.g.: title, description, web, email, GPS location, images, opening time, etc.; others may have complex information, and/or also specific data types and structures, for example:

• Status of the first aid: number of people under observation for each colour in the triage, waiting time, etc.;
• Bus stops: all the bus lines including geometry, their planned time schedule, the real time delays, etc.
• Sensors with their values, measure units, types, healthiness criteria, etc.;
• Weather forecast associated with an area/region, which consists of a large set of values: temperature, humidity, pressure, wind, etc., for many different time slots in advance;
• Shape of cycling paths, gardens, parks, difficulties, restrictions, etc.;
• Parking areas, with the number of free spaces, predictions, typical daily trend of free spaces, costs, etc.;
• Events of: (i) entertainment, with their description, photo, start date, end date, etc.; (ii) from police officers on the street; (iii) Events of emergency, such as Civil protection early warning in CAP standard, etc.

In order to simplify this complexity, MicroServices like “Service Info” and/or “Service Info Dev” are provided for Final Users and Developers, respectively. In the IOT Application, a search/Discovery has to be performed as described in the next paragraph, otherwise the developer needs to know the so called ServiceURI, which is the unique identifier of all the city entities in the Km4City Smart City Knowledge Base. In the Snap4City development environment, the ServiceURIs can be recovered directly from the graphic configuration of the MicroService or searched by using the ServiceMap visual tool which provides a GUI for query. This means that IOT App programming is 100% visual, even if a single service is used for accessing a single element.

 

Figure 3: ServiceMap for ServiceURI identification.

-- Search/Discovery of City Entities and their relationships. The search of city data has to allow users to discover data/device values by a semantic search, using a composition of the available query types described in the following:

• Semantic classification. In Snap4City, all the POI and Services are classified with more than 20 classes (mobility, energy, banking, environment, cultural activities etc.), including a total of more than 520 subclasses (see for example, the menu of the ServiceMap reported in Figure 3);
• Geo spatial references: close to a point, max distance from a given point, along a path, into an area/polyline;
• Textual keywords substrings: for example, on the basis of the title and descriptions of city entities;
• Value Types: for example, all sensors measuring temperature, all bus paths, etc.;
• Historical time slot: for example, data values for the last 7 days;
• Prediction time slot: for example, data regarding predicted parking slots for the next 15 minute, 30 minutes, 1 hour;
• etc.

The results of this kind of searches can be a single element with its description as well as a JSON containing a list of entities. In this latter case, the list can be split in single messages using Node-RED Split node. Once the list of ServiceURIs is accessible, their detailed description can be obtained by using the above-mentioned “Service Info” node. The search facility for Final Users is provided via the Node interface, simply performed by employing a user interface for setting parameters. On the contrary, as to the nodes for Developers, search parameters can be also prepared and sent to the search Node in JSON. In both cases, the user does not need to know any query language (e.g., SQL, MySQL, or others), neither if the data are coming from a complex set of queries on SPARQL for RDF store, on Elasticsearch and Phoenix/HBASE. Therefore, the developer of IOT App does not need to know all the tiny details about the large variety of adopted storages. Snap4City provides more than 70 different nodes for searching different Smart City entities, providing results in different data types: POIs, time trends, values, events, schedule of busses, bus lines, recommendations, addresses, routes, etc. This approach simplifies a lot the creation of Smart City Applications.

 

--- Discovering and Exploiting IOT Entities (sensors and actuators) should not be different than discovering any smart city entity. In Snap4City IOT Devices and data values can be accessed, searched and discovered by the above presented MicroServices/nodes. On the other hand, when the user/developer would like to create a data driven IOT Application exploiting specific IOT Devices, some specific MicroServices can be used so as to discover the desired devices regardless of the IOT Broker, IOT protocol or IOT Device. To this end, the IOT Directory MicroService exploits the services of the Km4City Knowledge Base for searching, managing and discovering all the available IOT sensors/actuators [Badii, 2017]. In Snap4City, developers can register on IOT Directory IOT Devices and IOT Brokers supporting a large number of different protocols and authentication models [Valtolina 2019], [Bellini 2019]. The IOT Directory automatically registers new Devices into the Knowledge Base, and each new IOT element receives a ServiceURI, thus becoming a City Entity and POI [Valtolina 2019]. A number of IOT Brokers are provided by Snap4City, while others are managed by third parties. For example, to provide answer to queries such as: “give me all temperature sensors close to my house” (regardless of the data providers, protocol, source, etc.).

 

-- Creation of advanced Graphic User Interface including graphics widgets which are: Dashboards, Virtual Devices to act on the IOT Applications and IOT Devices, advanced tools, etc. This means to give developers the possibility to design the IOT App user interface of the IOT App. The user interface has to show and also to get interactions from users to create messages for the IOT Network including the IOT App that may implement the logic of the user interface. The user interface is built by composing a number of graphic widgets in a connected Dashboard for presenting and collecting data. In Snap4City, the IOT Apps may be connected with Dashboards by means of widgets for:

• rendering data: single content, time trend, bars, histogram, map, pie, semaphore, dynamic signals, buttons, knobs, clock, bars, etc.
• collecting data as a switch, a knob or a keypad, which are interactive elements on Dashboards and are represented into IOT Applications as Input Nodes for the flow, able to provoke events into the IOT App according to the data driven approach.
• showing MicroApplications and External Services in a generic iframe widget.

In addition, other Dashboard Widgets may not have a counterpart into the IOT App and can be directly added to the Dashboard by using the Dashboard Builder editor [Bellini 2018]; which can:

• include Visual Analytics tools such as: External Services, MicroApplications, object Tracking, Origin Destination tools, heatmaps, traffic flows, maps of any kind, etc.
• visualize data from other sources: IOT Brokers/ Devices, data stores, API, etc.
• include Virtual IOT Devices that can send data directly to IOT Brokers, as any IOT Device but coming from the user interface.

This also means that an IOT Application may be connected to multiple Dashboards, and a Dashboard may be connected with multiple IOT Devices and IOT Apps [Bellini 2018].

 

-- Data Analytic, which would mean to provide MicroServices/nodes exposing a number of data analytic services to be exploited into the IOT Applications. In Snap4City, it is possible to: (a) exploit computing of data analytic processes already in place, (b) develop new data analytic processes, (c) calling External Services as rest call. In fact, Snap4City suite provides MicroServices implementing:

1. Data Analytic for computing: routing, auto-ARIMA predictions, anomaly detection, descriptive statistics, machine learning predictions, heatmaps, providing social media data from Twitter Vigilance [Cenni 2017], etc.
2. A generic Data Analytic MicroService where Data Analytic algorithms developed in R-Studio can be put in execution, according to specific guidelines.
3. A REST call invoking External Services. For example, to get access to Twitter Vigilance API, The Weather Channel API, The Things Network API, etc.

 

Figure 4: Developing Data Analytic MicroServices via R Studio to exploit them from IOT Apps in Node-RED.

-- Save and Retrieve Personal Data that could be time series for motion tracking, values of personal devices, clicks on mobiles Applications, POI, shapes, KPI (key performance index), Keys to access IOT Devices services, etc. The possibility to save and retrieve data from a safe storage (with the possibility of assigning delegations according to GDPR) enables a large variety of smart scenarios for the final users and operators. For example, the production of IOT applications saving personal data from personal health devices (e.g., monitoring glucose), from home status, from the location of mobile phones and thus belonging to a specific person. In addition, the Snap4City platform automatically collects all the personal data gathered from mobile Apps accessed on the Snap4City login (provided that the user authorizes their collection with a signed consent, according to GDPR).

 

-- Save, retrieve, and publish Data Sets, as those managed in the open data portals. Most public administrations are used to publish their data sets in open data, and also share their data via federated networks, see for example the harvesting mechanisms of CKAN [CKAN]. Having the possibility of creating data sets from the IOT Applications flow means that city operators can automatize their ingestion/update processes and their production/publication.

 

In the Snap4City suite of MicroServices, there is also a number of tools for managing DISCES backoffice scheduler of processes, as well as for saving LOGS regarding the data access and flows. The former are accessible only by administrators, while the latter can be useful for both developers and administrators. Moreover, a number of so-called GEO-utilities have been identified and implemented in Snap4City such as MicroServices/nodes to:

• Calculate the distance from two GPS locations.
• Verify, once a GPS point and a closed shape/area are given, if the point is inside the area or not, so as for instance to verify the match with administrative areas, to understand whether your dog is in the garden or the monitored bear in the forest, etc.
• Get the most probable value of a variable represented in a Heatmap from any GPS point irrespective of  the presence or absence of sensor in that point. This feature is very useful to decide among different routes: taking the quietest one, taking the less polluted one, taking the less crowded by the traffic, taking the busiest road to meeting people, etc.
• Get the closest civic number and street from a GPS point or vice-versa. This is very useful for geo reversing.
• Get the closest road map segment (typically called node in Open Street Map speaking) from a GPS point or vice-versa. It can be useful for routing and computing precise distances. This is very useful for geo reversing.

This means that the IOT Applications developers do not need to solve the so-called direct or inverse georeferencing problems, since they are provided by default.

 

A number of Node-RED nodes useful in smart city contexts are also available: they have not been implemented by us, as they are globally provided by third parties such as: interaction with Facebook social media, SMS sending, email sending, Telegram, etc.

 

5. Example of Application

In this section, three examples are presented with the aim of depicting the capabilities and the expressiveness of both the IOT Apps and the Snap4City development environment. They are IOT Applications for: (i) emergency management, (ii) routing on the basis of environmental data, (iii) dynamic management of PAXCounters IOT devices for monitoring people flow at museum and events.

5.1 Alerting about critical events involving people in a specific area

In this section, we are presenting a scenario where a public operator (Road Operator) on the field, like a policeman or a public transport driver, notifies to a control room operator (City Operator) a critical event in the city. The notification includes the reporting in real time of the event position, the number of involved people and the seriousness of the event.

The City Operator would like to: monitor events vs services in the city and receive critical event notifications from Road Operators and assess contextual conditions and services status. The control room receives the notifications and can explore the status of the services in the city, evaluate the gravity of the situation and, therefore, take the correct decisions to cope with the event. Especially if many events may arrive from the same area, a critical condition may be early detected and may be an Early Warning propagated to other cops. The Road Operator would like to monitor the status of traffic, parking, environmental variables, speed limits, services and send critical event notifications via coded descriptions. Figure 5 better explains this scenario.

 

Figure 5: Connection among City Operator dashboard in the control room and the one used by Road Operators.

Figure 6 provides an overview of the City Operator’s Dashboard named “The City Operator” can see the City Map (1) and select one or more services on the selector on the left (2) to see them on map. By a click on the elements forming part of the list on the left (2), the services belonging to the selected category are displayed (or hidden) from the map. Multiple categories can be selected at the same time to have a broader overview of what is the situation like in the area where the Road Operator is. By clicking a POI on the map (1), a pop up is shown representing the real time data of the selected POI (if available). The data shown in the pop up can be also represented as real time values in a dedicated widget (3), as well as time trend values in another specific widget (4).

Figure 6: City Operator Dashboard.

When the Road Operator sends the message related to a critical event, the City Operator receives the notification in the widget  (5) that lists all the notifications received. The minimap (6) is automatically centred on the coordinates sent by the Road Operator.

The creation of the City Operator dashboard has been performed by using a wizard guiding the user in the choice of the variables he wants to monitor and insert into the dashboard, as well as the widgets where to display the values produced by such variables [Bellini 2018]. Please note that Widget (5) is a data driven visualization from data arriving from IOT Orion Broker in NGSI.

Please note that the two Dashboards shown in Figure 5 have been created in two different manners, starting from the:

• Dashboard Wizard which permits, through simple guided steps, to select the data sources, then decide which widgets to be used to visualize them , finally arrive to the dashboard creation and of the selected widgets in few clicks.
• IOT Application editor (Node-RED), selecting Dashboard MicroServices, and connecting real time data, to see them automatically appearing in the dashboard as Widgets.  

Please note that both approaches are valid and can be mixed up.

When the Dashboard Wizard is used, as in Figure 6, the user is requested to choose one of the available templates, and then the structure which the new dashboard has to be based on. If the user chooses a particular template, it is also possible to add more widgets in a second moment manually, or by calling again the wizard within the newly created dashboard. In the Wizard, the user can choose data sources to use inside the dashboard. The types of data sources are various: Complex Event, IOTApp, External Service, Heatmap, MicroApplication, My Personal Data, MyData, KPI, MyKPI, POI, MyPOI,  Sensor, Actuator, Special Widget, WFS. Multiple data sources can be selected to be displayed inside the same dashboard. The data generated by the selected sources can be very different from one another, and the user must choose how to represent them within the dashboard. In the Wizard, for each selected data the user may choose among the possible widgets which can represent and display that specific data type. Therefore, the possible graphic widgets are dynamically shown. To make the choice easier for the user, when he has selected a certain data source, widgets are automatically filtered, and vice versa: the user could also firstly select the widget model for the dashboard, and then the wizard will filter the data sources showing only those that may be visualized on the selected widget type.

Figure 6: Dashboard Wizard.

Once the new dashboard is created, if the user wants to add more widgets, the dashboard can be edited through the wizard again and again. The adopted approach allows to easily create interactive dashboards without using any particular programming skills, but relying only on the help of the wizard. The creation of the Road Operator's dashboard has been performed by using the IOT Application editor, which is the second approach listed above. The Dashboard Wizard is not used to create this dashboard, actually it has been created in automatic mode from a Node-Red stream. To allow the IOT App flows to interact with the Dashboards, we have to create in the flow Nodes/Blocks representing Dashboard Widgets. Widgets in the flows can be INPUT widgets (providing data from the Dashboard to the IOT App) as well as OUTPUT widgets (graphically representing some data on the Dashboard). These elements can be regarded as Virtual IOT devices as well, respectively Virtual Sensors and Virtual Actuators. 

The Snap4City Dashboard nodes allow the creation of 5 types of widgets on the dashboards, to represent the different metrics chosen by the user/developer/city operator, among a range of many available ones; these widgets are:  Gauge Chart, Speedometer, Time Trend, Single Content and Web Content. The first 3 types accept numerical values as input: in the first two types, the last sent value of the represented metric is displayed, and in the third type the metric history (as a time graph of all the metric values sent over time) is also shown. The Single Content can receive data in any text format and can also display HTML. The WebContent allows you to send a url and to display the resource that is located at that url within the widget.

The nodes creating Actuators on the Dashboards are of 5 different types: Impulse Button, Switch, Numeric Keyboard, Knob and Geolocation. The Impulse Button sends an "On" signal when pressed and an "Off" signal when released. The Switch allows you to set the button to "On" until its status is changed. The Numeric Keyboard allows you to send any numeric data from the dashboard to the stream showing the last value sent in the widget. The Knob allows you to dynamically change the sent value, depending on how the knob is turned. The Geolocation sends the position retrieved from the browser where the dashboard is shown.

In Figure 7, the relationships between the User Interface in the hands of the Road Operator and the corresponding IOT Application are shown. The Node-RED flow is divided into three parts that are described below.  In Figure 7b, GPS Position: the buttons receive the GPS Position that has been set as “Helsinki”, “Florence”, “Antwerp“, and “My position” in this example. By clicking the GPS position button, the map on the top sets the position accordingly. The flow is making the request to the Smart City API, and the map is shown in the right place with the list of services and PINs on map. In Figure 7c, People Number: The numeric keypad Widget allows the Road Operator to insert the number of people involved in the critical event. The user enters the number and press the “Confirm” button. The confirmed number is shown in the “Last confirmed” box on the top-right of the numpad. In the IOT App, the number is stored in a temporary memory.

Figure 7: On the Road Operator’s user interface and IOT Application managing the logic. Overview and first details.

In Figure 8, Alert Colour: the coloured buttons allow to set the colour code of the emergency. If the user has set correctly the GPS position and has inserted the number of people involved, by pressing the colour code he can also send the message to the City Operator, by posting it on an IOT Virtual Device of the Dashboard.   

 

Figure 8: On the Road Operator’s user interface, last part.

In detail, when the Road Operator presses the colour button indicating the priority of the emergency, a JSON is created containing information about all the previously described aspects such as: the number of people, the coordinates and the colour of the emergency. This JSON is sent as a value of an entity created on a Fi-Ware Orion Broker through a node which is enhanced with the ability to add authentication keys, if compared to the base node provided in Node-Red. This ensures that writing such an entity can occur only if the IoT App, which is sending the data, belongs to the entity owner.

The entity registered on the IOT Orion Broker can be also chosen in the wizard, which allows you to create the widget that you see in th City Operator's dashboard at the top right (5). This widget adapts the displayed value dynamically when a new JSON arrives, thus changing the state of the entity with no need on the City Operator’s side to update anything on the dashboard in order to comply with the Event Driven messages. The flexibility of the dashboards allows to create mixed dashboards containing widgets created with the wizard and widgets automatically created by the Node-Red streams. If the dashboard is created automatically by the creation of widgets from a NodeRed stream, you can open the wizard and add additional widgets through the different modalities explained above. The messages exchanged between the Node-Red streams and the IOT Orion Broker use the NGSI protocol, which is the same protocol used when values are received from the widgets on the dashboards linked to the real IOT Devices of the IOT Orion Broker. The exchange of messages happening among the nodes dedicated to the dashboards (whether they are actuators or viewers), is made via Web Socket Secure protected by an Access Token depending on the owner of the same IOT Application.

The additional nodes created to enhance the functionality of Node-Red in Smart City can also be installed on local users’ devices (such as Raspberry etc..). In some cases  it is necessary to request and obtain an account on the Snap4city platform before exploiting such enhanced functionality. Without this account you cannot get the Access Token ensuring the security of communications among nodes and other components of the Snap4city platform [Badii 2019].

5.2 Check which route is less polluted

The above examples have shown how it is possible to use Snap4City MicroServices for the development of systems useful for the security and control of emergencies in the public sector. The suite of microservices used in the above exercises have been mainly composed of dashboard elements. IOT App with dashboard can also be used by citizens within private applications.  In the example developed in this section, MicroServices retrieve information from the Smart City storage and info to create a dashboard which provides the user with pieces of information on which is the less polluted path at a precise moment to go jogging. If predictive data are available, it can work on predictions.

 

In Figure 9, the dashboard presents two possible paths to go jogging; the values of O3, PM 2.5 and PM 10 of the two paths are displayed in the centre.

 

Figure 9: Personal Dashboard to decide the less polluted path to go jogging.

The flow realizing the logic behind the dashboard is shown in Figure 10. This dashboard has been created automatically, when the Node-Red flow has been deployed the first time. In the flow, the outermost blue nodes on the right correspond to the two maps with the routes, the six central blue nodes on the right are related to the Gauge Charts reporting the values of the pollutants, and the only blue node on the left corresponds to the "Update" button that allows to recalculate the data when pressed. These nodes automatically create the widgets on the dashboard, if the dashboard exists; otherwise, they also create the dashboard, without any required intervention by the user, who has to edit only the name of the dashboard and of the Widgets, when configuring the nodes.

 

 

Figure 10: IOT Application according to the logic of Dashboard of Figure 9.

The orange nodes on the left, called "PARCO DELLE CASCINE" and "PARCO VIA FONTALLERTA", allow the user to calculate the shortest path between two geo-points. With a simple visual configuration, as shown in Figure 11, you can choose a starting point, an arrival point, the time you want to begin the trip and the transport means: the node will produce in output a JSON containing all the information about the calculated route. As you can see, the complexity of the route calculation is transparent to the user who has the result of his request in a simple and intuitive way.

In this Node-Red flow, unlike the previous example, a minimum of programming ability is required: the JSON that is returned from the previous nodes also contains the complete path calculated by the system, and this must be divided into its main points exploiting the function nodes made available by Node-Red (where the user can interact with the variables by writing code in JavaScript). 

 

Figure 11: Setting for routing of IOT App in Figure 10, and of Dashboard in Figure 9.

 

The other six orange nodes on the left, internal to those just described and which have the names of the pollutants, allow you to get punctually the value of that substance in a given geographical point.

In this case too, the complexity lying behind this value is hidden to the user, who manages to have, in just few steps, that value in his hands, thus being able to use it as he pleases. The complexity behind the whole described scenario, which is transparent to the user, implies the retrieval of all the sensors data available in the area under consideration, the calculation of a heatmap (furthermore, interpolating values in the points where sensors are not located), its storage on a geo-server, and finally the recovery of timely data available from the same geo-server with the creation of APIs dedicated to this particular service. The calculation of the heatmap must be done periodically and, in turn, this periodicity has been done with a special Node-Red flow executing (on the pre-determined frequencies or scheduled time intervals) the R script in charge of calculating the heatmap. Please note that the sensors are typically located in some specific points, which may be very distant from the computed routing. A specific service in the smart city back office is dedicated to compute the heatmap and to provide a MicroServices which allows to get the value of pollutant, as well as of temperature, humidity, etc. in any point of the area.

In the flow depicted in Figure 10, the values of the polluting elements are calculated for the main points of the routes returned by the first two nodes, and the average is made so as to have an indicator of pollution for that route. The user who has created a dashboard made in this way can press the update button, before going for jogging and he will know where to go running to breathe cleaner air. The automated update can be programmed as well.

 

5.3 Controlling Personal Mobile PAX Counter

In the first example, we have seen how to create two interoperable dashboards through an IoT App, in order to ensure support for emergencies within a city. We have also demonstrated how this allows a smart interaction with Road Operators, who are located directly on the emergency scene, as well as with City Operators who are located inside the City Control Room. In the second example, we have shown how to create a dashboard retrieving data from city sensors to get real-time information on the state of air quality. We have also stressed how this can be useful for decision support, for instance when making choices based on pieces of information offered by the system and able to improve the user’s health by suggesting the most virtuous behaviour. In both cases, everything was done through MicroServices and the user did not need any additional device to obtain the required info. All the temporary data for the processes has not been permanently stored.

 

This example depicts the interaction with IOT Devices counting people by using Wi-Fi and Bluetooth sniffing in its vicinity (according to GDPR [GDPR]). In Figure 12, a mobile PAX Counter is shown, based on ESP32, sending messages to Snap4City via LoraWAN. Other simpler versions can be located on fixed places, and may also exploit WiFi to send the measures obtained also from different, additional sensors. In the current example, they have been used to monitor a Museum in Antwerp and many other locations. A total of 22 devices have been installed.  The measured values are sent to LoraWan operator The Things Network, which does not provide a system for data analysis, except for actually only offering a way to receive and manage devices.

 

 

Figure 12: A mobile PAX counter based on ESP32.

Museum Case

In the literature, there are several technologies for tracking movements into the museum and understanding the inside flows [Zezzatti et al., 2019]. On the other hand, their costs are very high and for most applications it is not needed to have the internal traces. In the presented IOT Application and Museum case with free entrance, it has been of great interest the monitoring the flows of entering and exiting of visitors, collecting also the history of such data and not only the real-time values with low cost solutions. In fact, the monitoring via PAX Counter of 10 doors would cost less than 3000 euro. To this end, data have to be saved permanently into time series data store, for example into MyKPI, MyPOI storage via MicroServices. The Snap4City approach allows the IOT Developers to create and retrieve data entities trends directly in Dashboard and for Data Analytics, as data Shadow. According to this solution, an IOT App has been created for PaxCounter to receive in Event Driven mode the new data coming from The Things Network (as IOT Gateway), according to the Publish Subscribe model of MQTT; then, such data are saved as MyKPI via MicroServices. Consequently, Dashboard Wizard automatically sees the data entity on the MyKPI and allows us to create automatically Widgets showing their trends, real-time values, etc. Actually, the Snap4City back end provides a suite of scheduled scripts specifically designed to populate automatically and programmatically the Dashboard Wizard with data coming from all the different data sources (Sensors, IOT App, External Services, Heatmaps, Personal Data, KPI, MyKPI, POI, MyPOI etc.). Figure 13 shows the Dashboard presenting the time trends of the entrances and exits of the museum and all its derived data. Such derived values of the initial real-time data can be computed in real-time, saved on personal storage and shown on dashboard via the IOT Application exploiting Snap4City MicroServices. At the same time, while sums are being calculated, the difference between people entering the museum and those leaving is also performed, in order to obtain the number of people inside the museum in another entity.

 

Figure 13: Monitoring MAS Museum people flows.

Through the Dashboard widget and IOT App, it is possible to send commands to the Pax Counter, via the IOT Gateway, to get the status of the device, as well as to change settings. A command that can be useful when having a Pax Counter installed in the museum can be changing the mode of counting people from cyclic to cumulative. In the cyclic mode, every 8 minutes the count is reset and the anonymous data of the devices counted by the Pax Counter are deleted: therefore, in the following 8 minutes, if a person stands near the Pax Counter,  that person is counted again. In cumulative mode, the count is not reset but is increased with new people who arriving near the Pax Counter; therefore, duplicate counts of the same person are avoided until the device is in cumulative mode.

 

Mobile PaxCounter for monitoring events

The mobile PAXCounter presented in Figure 12 has been used for monitoring sport and entertainment events. Each new measured count is associated with GPS coordinates and sent to the platform, thus resulting in trajectories with associated values as depicted in Figure 14. This is done exploiting the Snap4City Tracker Widget.

 

Figure 14: Tracking PAX Counters, measures over time and space.

In this case as already seen above, it could be very important to change modality, allowing to switch from Cyclic to Cumulative counting. To this end, in order to program the change of modality, an IOT Application with a dedicated Dashboard user interface has been realized, as reported in Figure 15.

 

Figure 15: Management of the Mobile PAXCounter with the possibility of programming change of modality.

In this case, the IOT Application is more complex since it requires to (i) collect data; (ii) program the IOT devices; (iii) interact with the user to collect the data for programming purposes; (iv) monitor the status of the IOT device to verify if the setting has been successfully applied; (v) report the status of the IOT device to the user. All these functionalities have been implemented in a single IOT App, as reported in Figure 16.

 

Figure 16: IOT App controlling mobile PAXCounter

Thus, also a custom widget has been developed to allow the definition of both begin and end instants of the programmed period of cumulative mode. The activation of the programmed interval is performed by pressing the "Activate" button that saves the times within the IOT App. The IOT App performs change of modality at the due time, since the IOT Devices are not programmable. In addition, 3 Single Content widgets are instantiated to display the activated period of cumulative mode, the status of IOT App for setting data to the device, and the state of the IOT Device. In Snap4City, Custom widgets can be developed by creating them in SVG and implementing a specific process to connect them to Snap4City MyKPI.

     Once the custom widget has been created, the IOT App is created realizing the control logic covering all the functionalities listed above by exploiting the MicroServices for: dashboard widget, personal storage exploitation, IOT Gateway connection, etc..

 

From the point of view of the life cycle, the above presented IOT Applications can be saved into the Resources Manager, thus allowing other users to download and install in their own resource area, and use them with their own devices.

 

6. Development Life Cycle of Snap4City Applications

The design and development of IOT Cyber Physical System/Solutions, CPS, is much more complex than regular software developments approaches for SW life cycle. An aspect that they share is the adoption of strongly dynamic microcycles. In [Rahman, 2018], a review of life cycles for IOT has been presented. The life cycle proposed is a simplification with respect to the actual needs, since it does not take into account: data analytics, personal/private data, complex dashboards. In most cases, when IOT development life cycles are discussed the focus goes on devices development, finalization, test, deploy, connection, etc., while as in [Dias, 2018] the complexity of the life cycle is much wider.

    The Life Cycle for the IOT Applications/Solutions in complex domains as Smart City involves a number of specialized tools such as those for Maps, Dashboards, etc., that need to be well placed in the life cycle. Therefore, in this section, the development life cycle for Snap4City applications is presented, see Figure 17. In the represented life cycle only the software aspects are address.

 

 

Figure 17: Snap4City Development Life Cycle for Cyber Physical Solutions, where we distinguish among phases, formal documents, tools and processes, MicroService/Nodes and Widgets of the Dashboards. 

    In the context of Snap4City IOT Smart City development, the life cycle should start with the Analysis and Design to produce requirements and specification of the application. This information is collected as details regarding:

• Data analysis have to be identified on the basis of the scenarios goal. And in particular: real time inputs and outputs that could be used; Historical data that could be exploited for showing them or for Data Analytics; data to be scraped from Web pages according to the terms of use; private information of the users or entities involved according to GDPR guidelines. This process of data analysis also may imply the Discovery of Data on the available knowledge base via Service Map or Data Inspector. If the needed data are not available specific ingestion process will be developed.  
• Functional transformations of data and commands, which can be needed on the basis of the data format accessible. This activity could also create business logic of the IOT Application taking into account historical data, inputs, MyKPI, scraped data and Data Analytics to produce outputs on other devices and/or dashboards.
• Analytical identified which are the data analytic to be developed for the application, if needed. For example, for a Smart Parking a free parking slot prediction will be needed and this is evident since the analysis. The Data Analytic will have to be developed from scratch or customized exploiting some ready to use MicroServices.
• Visual Interface. In most cases, the graphic interface is part of the problem. A good visual representation may do the difference from a tedious view a nice user centric view. This part includes the design of the Visual Interface in terms of Widget and Dashboard. The single elements may be presenting data coming from the data store/shadow, may be directly produced from/to the IOT Application data flow in real time, or may need to be developed as Custom Widgets.

 

Once the Analysis and Design has been completed, a global view of the Smart Application has been sketched but need to be implemented. Thus, the different aspects and phases may be addressed by different tools and for large factory also by different teams addressing:

• Data Ingestion and Modeling. In this case, different tools can be used.
  
  • IOT Directory for registering new IOT Devices, IOT Brokers, etc. In some cases, the definition of the IOT Device Model is needed to pass at the IOT Device Registration in Bulk.
  • DataGate/CKAN: for the ingestion of Open Data with geo information as POI.
  • ETL development, a process of ingestion of some file or data that has to be taken in PULL for example, from WS, FTP, HTTP, etc. They could be even developed into the IOT Application tool exploiting MicroServices.
  • External Services: the direct identification of external API that could be exploited into the IOT Application.
  • MyKPI, the modeling and registration of the KPI and data needed to have permanent and personal data into the IOT Application.
• Development of Data Analytic. This process is typically addressed by Data Analysts skilled on data science, accessing to historical and real time data and creating specific algorithms such as: prediction, anomaly detection, interpolation, clustering, etc., exploiting libraries and tools of operative research, statistic, machine learning; using languages and development environment provided by Snap4City for online development such as RStudio, Python, Java, etc. The Data Analytic may produce:
  
  • MicroServices that can be reused in a range of IOT Applications. They produce data on Storage to be visualized on Dashboards by  regular widget such as Heatmaps.
  • Special Tools that may produce view as External Services on Widgets such as Origin Destination Matrices. Also these Special Tools  may produce data into the Big Data Storage,
• Data Flow Development (that is the development of the IOT Applications. The data flow development is typically focused on producing outputs on storage,  MyKPI and/or presenting resulting data on some Widget Dashboard. This IOT Application development exploits the Snap4City library of MicroServices/Nodes plus those that may be newly created for the specific solution. The development of some IOT Applications has been presented in the previous section. The IOT Application implements the business logic of the solution and can be deployed (on cloud/container or on IOT Edge). The Deploy can be performed in a sandbox for testing. When the IOT Applications includes the usage of Dashboard Widgets they are created on Dashboard at the moment of the Deploy of the IOT App.
• User Interface Development (that consists in the development of the Dashboard(s)). The Snap4City Dashboards can be composed by several kind of Widgets mainly classified as
  
  • IOT App Widget directly produced or that have a counterpart representation into the IOT Application (also called virtual sensors, virtual actuators). These widgets are used for managing event driven graphic elements as inputs and outputs of the IOT Application such as: buttons, messages, switch, gauge, data view, trend, keypad, semaphore status, etc.
  • Custom Widgets which can be produced by using SVG tools and may implement synoptic as specific interaction tools with animations, etc.
  • Regular Widgets taken from the large collection of data viewer of Snap4City Dashboard Builder.

 

The final result consists of a number of: data ingestion processes, data analytic processes and MicroServices, IOT Applications, and Dashboards connected each other. The above presented Life Cycle in Figure 17 distinguishes among phases, formal documents, tools and processes, MicroService/Nodes and Widgets of the Dashboards. 

 

The process of the Development, deploy, testing, validation, acceptance testing, is typically iterative, to arrive at the final deploy in production with an acceptance testing. Then, after that the solution passes to maintenance, and has to be continuously monitored to detect eventual problems, that may spring from: missing data, change of data format, exceptions in the data analytic or in the data flow, etc. All these sources of fault should be managed with a correct design and implementation with error management and fault detection. The best solutions, try to produce a resulting services any way, for example in the lack of data for making predictions via machine learning provide the average values, etc.

 

7. Assessment and Experimental Data

In literature, several Smart City experimentation testbeds have been presented, expressing the common requirement of involving citizens, ICT operators and local administrations. This aspect implies the organization and deployment of experimentation facilities such as crowdsensing campaigns, living labs and tools to monitor and enhance the citizens engagement [Choque 2019]. In order to assess the effectiveness of Snap4City IOT Application Development Environment, a training course has been organized with Smart City operators, ICT city officials, City Council Operators (they have never used Node-RED before). During the course, we assigned a number of exercises (https://www.snap4city.org/drupal/node/501) and recorded all the activities. Before playing with exercises, a specific but short training has been performed on Node-RED programming and Snap4City MicroServices. Then, exercises have been presented, and a fixed time slot has been assigned to learners to produce the IOT App. In both cases, the IOT also included the development of a small Dashboard. At the end of the exercise, we closed the sections, asked the users to provide some answers through a written questionnaire, while the IOT App and dashboards have been saved on the platform closed for assessment. Thus, the assessment has taken into account both results, as described below:

1. The answer in paper about the exercise, where users in the audience should mark the needed Node-RED smart city blocks required to solve the problem; the form included the list of NODES/MicroServices that can be used to compose the IOT App.
2. The developed IOT Application into the platform, accessible for our review, with the possibility of testing to verify whether it was 100% complete or not.

Please note that, the assessment has considered the fact that results would not be unique, in the sense that each exercise could be solved in different manners.

The aim was to answer to the questions: “As to the training received from us, was it enough to allow users to develop an IOT App to solve the problem?”, “how much faster is the solution with respect to other solutions?”, “how has it been perceived?”, “do they have already learnt the programming model?”, etc. The audience included 24 skilled users, with less than 20% ICT developers. The following exercises have been proposed:

1. Create an IOT App that reads a value from a Snap4City service (for example the parking lot seen in the previous demo, identified by the serviceUri: http://www.disit.org/km4city/resource/CarParkPieracciniMeyer) and realize a flow which sends different messages (e.g.: “Almost Full Parking slots” or “Free Parking”) on the dashboard, on the basis of a certain threshold.. Apart from these functionalities, the creation of a service which sends to you an email J ! would be considered as a plus. (time required: 15 minutes)
2. Create an IOT Application / flow that: (i) reads a value from a list of Snap4city services (for example the car parks in the Florence City Area, as seen in previous demo), (ii) calculates the average of Free Parking Lots, (iii) sends the value on a dashboard with the four possible nodes seen in the demo. (time required : 20 minutes)

The assessment has been carefully performed, since each exercise can be solved with multiple solutions. To assess the exercise on the paper questionnaire, we have only verified if test-takers have selected the right microservices. A score of 100% has been given if they selected all the right answers, according to the possible solutions, while a fraction of 100% has been assigned proportionally if some answers have been missed. In the assessment of the final IOT App we have verified if the App satisfies the requirements, and a vote has been given according to percentage of such satisfactory outcome. In some cases, we have positively considered a plus when the developer has added more features (for example the email sending) (see Table 1).

	Exercise 1	Exercise 2
Number of users	24	24
Exercise Duration	15 min	20 min
Results on paper	21	21
average score	71,42%	83,92%
Variance	302,8	470,9
Resulted IOT Apps	20	16
average score	85,75%	83,42%
Variance	692,8	449,0
75% of requirements	80,00%	81,25%

Table 1:  Results of the IOT App Development Assessment.

 

As we can remark from the results summarized in Table 1, 21 users have performed Exercise 1 on paper. The average score on paper assessment has been 71,42% with variance of 302,8. This means that the mean value of scores assigned on the basis of the paper assessment has been 71,42% over 100%, which corresponds to the percentage of identified Node-RED blocks needed to solve the problem. In addition, 20 of those 21 actually implemented an IOT App. Only one test-taker has presented the result on paper and he has not been interested in performing the exercise on real tool.  Test-takers obtained an average score on real results in terms of requirements coverage equal to 85,75% and Variance of 692,8. In average, the developed applications covered 85,75% of the requirements. In addition, 80,00% of users realised the IOT App with at least 75% of requirements. That is, in synthesis, a good result confirming that the training has been enough to let them create the IOT App autonomously. The test-takers who solved the exercise in the IOT App editor have actually performed better in the VPL tool, rather than on paper. This also means that they could identify the blocks (the MicroServices), but it stressed also that the development environment has provided a concrete support in correctly identifying the nodes/MicroServices, by means of the over functionality that provide help description when the mouse passes on the icons.

The 24 participants have been also provided with additional questionnaires to assess their appreciation and effectiveness in a more general sense. In this case, we have used a Likert scale at 5 values:

• 91% have demonstrated to be satisfied of the training day, and among them 53,40% have been very satisfied

• 85% have ranked the IOT solution and Data Analysis very good and among them 42,17% have ranked excellent
• 100% have stated that solution would be useful for their work, 78,13% have declared that it would be very useful in their daily work
• 95,28% have found the creation of IOT App somewhat easy, and among them more than 56% easy and very easy
• 93% have described the functions for IOT Apps (MicroServices) creation as complete and satisfactory, and among them 31,25% have evaluated it as very complete
• 50% of users have stated that the IOT App development has been 5 times faster if compared to what they know as possible with other tools at the state of the art.

 

Therefore, answers to our initial questions have been obtained. A short training has been sufficient to learn how to use the Snap4City IOT App development environment, and it is typically considered faster than any other tools (typically they have used ETL, programming languages, and rule bases solutions). The implementation of the applications we presented by using other solutions or even Node-RED standard version is somehow unfeasible due to the complexity some of our MicroServices hide, for example, the routing, the picker for the heatmap, the discovery, the dashboard, the IOT registration and discovery, etc.

 

7. Conclusions

Smart Cities are approaching the IOT world. Most of the early smart city solutions of first generation have been based on ETL processes mainly supporting pull protocols and data silos. The last generation of smart cities are massively using the IOT solutions, and thus are moving towards the use of push protocols / event driven. Therefore, the concept of Event Driven IOT Applications is becoming more widespread and viable with respect to the ETL or to rule-based solutions. In this paper, these aspects have been reviewed, highlighting the requirements for smart city IOT Applications, as well as the ones specific for the MicroServices implementation for IOT Applications in Smart City frameworks. In this paper, we have also presented Snap4City library of MicroServices for Node-RED, that has enabled the creation of a wide range of innovative IOT Applications for Smart City involving Dashboards, IOT Devices, personal storage GDPR compliant, geographical routing and geo-utilities, data analytics, etc. The solution proposed also addressed the formalization of the development life cycle that we have adopted and proposed to develop Cyber Physical System in Snap4City. The proposed solution has been validated against a large number of IOT Applications for the cities of Firenze, Antwerp and Helsinki. In addition, three applications have been reported in the paper and described, putting in evidence the relationships among MicroServices, Dashboard and data. Besides, the assessment of the solution has been performed and reported. The assessment has shown that the suite of MicroServices significantly could make the development time shorter, and it has been strongly appreciated by expert users. The research has been developed in the context of Select4Cities PCP of the European Commission, as Snap4City platform. 

 

8. Acknowledgment

The authors would like to thank to Snap4City consortium and the Select4Cities PCP Consortium, the European Commission H2020 research and innovation program (grant agreement No 688196) as well as all the companies and partners involved in other trials in the same context. Snap4City and Km4City are open technologies and research frameworks of DISIT Lab.

 

9. References

[Agbali 2019] M. Agbali, C. Trillo, I. A. Ibrahim, Y. Arayici, and T. Fernando, “Are Smart Innovation Ecosystems Really Seeking to Meet Citizens’ Needs? Insights from the Stakeholders’ Vision on Smart City Strategy Implementation”, Smart Cities, Vol. 2(2), pp. 307-327; DOI: 10.3390/smartcities2020019, 2019.

[Aldasoro,2011] Reyes-Aldasoro, C.C., Griffiths, M.K., Savas, D., Tozer, G.M.: “Caiman: an online algorithm repository for cancer image analysis”. Comput. Methods Prog. Biomed. 103(2), 97–103 (2011)

[Ardublock] http://blog.ardublock.com/en/

[AXEDA] PTC Axeda IoT Platform. http://www.ptc.com/axeda/product/iot-platform.

[Badii 2013] Badii, Atta, et al. "CityScripts: Unifying Web, IoT and Smart City Services in a Smart Citizen Workspace." JoWUA 4.3 (2013): 58-78.

[Badii 2017] C. Badii, P. Bellini, D. Cenni, A. Difino, P. Nesi, M. Paolucci, “Analysis and Assessment of a Knowledge Based Smart City Architecture Providing Service APIs”, Future Generation Computer Systems, Elsevier, 2017, http://dx.doi.org/10.1016/j.future.2017.05.001.

[Badii 2018] C. Badii, E. G. Belay, P. Bellini, D. Cenni, M. Marazzini, M. Mesiti, P. Nesi, G. Pantaleo, M. Paolucci, S. Valtolina, M. Soderi, I. Zaza, "Snap4City: A Scalable IOT/IOE Platform for Developing Smart City Applications", Int. Conf. IEEE Smart City Innovation, Cina 2018, IEEE Press.

[Badii 2019] C. Badii, P. Bellini, A. Difino, P. Nesi, "Privacy and security aspects on a Smart City IoT Platform", 16th IEEE Int. Conf. on Advanced and Trusted Computing, ATC 2019, Leicester, UK

[Badii 2019b] C. Badii, P. Bellini, A. Difino, P. Nesi, "Sii-Mobility: an IOT/IOE architecture to enhance smart city services of mobility and transportation", Sensors, MDPI    https://doi.org/10.3390/s19010001  https://www.mdpi.com/1424-8220/19/1/1/pdf , 2019

[Bellini 2018] P. Bellini, D. Cenni, M. Marazzini, N. Mitolo, P. Nesi, M. Paolucci, "Smart City Control Room Dashboards: Big Data Infrastructure, from data to decision support", Journal of Visual Languages and Computing, 10.18293/VLSS2018-030

[Bellini 2019] P. Bellini, F. Bugli, P. Nesi, G. Pantaleo, M. Paolucci, I. Zaza, "Data Flow Management and Visual Analytic for Big Data Smart City/IOT",  19th IEEE Int. Conf. on Scalable Computing and Communication, SCALCOM 2019, Leicester, UK

[Bonomi 2012] Bonomi, F., Milito, R., Zhu, J., & Addepalli, S. (2012). Fog computing and its role in the internet of things. In Proceedings of the first edition of the MCC workshop on Mobile cloud computing (pp. 13-16). ACM.

[CAP] https://www.fema.gov/common-alerting-protocol

[Cavada 2019] M. Cavada, M. R.Tight, C. D. F.Rogers, “A smart city case study of Singapore - Is Singapore truly smart?”, Smart City Emergence, Cases From Around the World (Elsevier), pp. 295-314, 2019.

[Cenni 2017] D. Cenni, P. Nesi, G. Pantaleo and I. Zaza, “Twitter Vigilance: a Multi-User platform for Cross-Domain Twitter Data Analytics, NLP and Sentiment Analysis”, in Proc. of the IEEE int. Conf. on Smart City and Innovation, San Francisco, 2017.

[Chattopadhyay, 2015] Chattopadhyay, Tanushyam, et al. "Automated Workflow Formation for IoT Analytics: A Case Study." International Internet of Things Summit. Springer, Cham, 2015.

[Choque 2019] J.  Choque, L. Diez, A. Medela, and L. Muñoz, “Experimentation Management in the Co-Created Smart-City: Incentivization and Citizen Engagement”, Sensors, Vol. 19(2): 411. DOI: 10.3390/s19020411, 2019.

[CKAN] https://ckan.org/

[Contreras Masse, 2019] Contreras Masse, Roberto. "Application of IoT with haptics interface in the smart manufacturing industry." Instituto de Ingeniería y Tecnología (2019).

[Dey, 2015] Dey, Sounak, et al. "A semantic algorithm repository and workflow designer tool: signal processing use case." International Internet of Things Summit. Springer, Cham, 2015.

[Dias, 2018] Dias, João Pedro, and Hugo Sereno Ferreira. "State of the Software Development Life-Cycle for the Internet-of-Things." arXiv preprint arXiv:1811.04159 (2018).

[Díaz-Díaz 2017] R. Díaz-Díaz, L. Muñoz, and D. Pérez-González, “Business model analysis of public services operating in the smart city ecosystem: The case of SmartSantander”. Future Generation Computer Systems, Vol. 76, pp. 198–214, 2017.

[Dizdarevic 2019] Dizdarevic, J., Carpio, F., Jukan, A., & Masip-Bruin, X,  “A Survey of Communication Protocols for Internet of Things and Related Challenges of Fog and Cloud Computing Integration”, ACM Computing Surveys, Vol. 51, No. 6, Article 116, (2019).

[Dragoni 2017] N. Dragoni, S. Giallorenzo, A. L. Lafuente, M. Mazzara, F. Montesi, R. Mustafin, and L. Safina, “Microservices: yesterday, today, and tomorrow”, In book: Present and Ulterior Software Engineering (Springer), DOI: 10.1007/978-3-319-67425-4_12, 2017.

[Esposte, 2017] A. Esposte, F. Kon, F. Costa, and N. Lago, “InterSCity: A Scalable Microservice-based Open Source Platform for Smart Cities”, 6th International Conference on Smart Cities and Green ICT Systems (SMARTGREENS 2017), pp. 35-46, DOI: 10.5220/0006306200350046, Porto, Portugal (2017).

[ETL Penthao] https://help.pentaho.com/Documentation/7.1/0D0/Pentaho_Data_Integration

[Fernandez 2019] J. M. Fernandez, I. Vidal, and F. Valera, “Enabling the Orchestration of IoT Slices through Edge and Cloud Microservice Platforms”, Sensors, MDPI, Vol. 19(13), E2980; DOI:10.3390/s19132980, 2019.

[FLOGO] https://www.flogo.io/

[GDPR] General Data Protection Regulation of the European Commission, https://eugdpr.org/

[Gomez 2012] Gomez, C., Oller, J., & Paradells, J. (2012). “Overview and evaluation of Bluetooth low energy: An emerging low-power wireless technology”. Sensors, 12(9), 11734-11753.

[Gupta, 2019] P. Gupta, S. Chauhan, M. P. Jaiswal, “Classification of Smart City Research - a Descriptive Literature Review and Future Research Agenda”, Information Systems Frontiers 21(1), DOI: 10.1007/s10796-019-09911-3 (2019).

[IBM] IBM Bluemix. www.ibm.com/software/bluemix/welcome/solutions2.html.

[IFTTT] https://ifttt.com/

[Kitchin 2014] R. Kitchin, “The real-time city? Big data and smart urbanism”. GeoJournal, Vol. 79(1), pp. 1–14, 2014.

[Krylovskiy 2015] A. Krylovskiy, M. Jahn, and E. Patti, “Designing a Smart City Internet of Things Platform with Microservice Architecture”, in Proc. of the 3rd International Conference on Future Internet of Things and Cloud (FiCloud 2015), pp. 25-30, Rome, Italy, 2015.

[Lee 2014] J. H. Lee, M. G. Hancock, M. C. Hu, “Towards an effective framework for building smart cities: Lessons from Seoul and San Francisco”, Technological Forecasting and Social Changes, Vol. 89, pp. 80–99, 2014.

[Li 2018] Li, S., Da Xu, L., & Zhao, S. (2018). 5G internet of things: A survey. Journal of Industrial Information Integration, 10, 1-9.

[Li2 2018] Li, H., Ota, K., & Dong, M. (2018). “Learning IoT in edge: Deep learning for the Internet of Things with edge computing”. IEEE Network, 32(1), 96-101.

[Lim 2018] C. Lim, K.-J. Kim, P. P. Maglio, “Smart cities with big data: Reference models, challenges, and considerations”, Cities, Vol. 82, pp. 86–99, 2018.

[Nesi 2015] P. Nesi, G. Pantaleo, G. Sanesi, “A distributed framework for NLP-based keyword and keyphrase extraction from web pages and documents”, 21st International Conference on Distributed Multimedia Systems, DMS 2015, pp. 155-161, DOI: 10.18293/DMS2015-02, Canada, 31 August - 2 September 2015.

[NETLAB] http://www.netlabtoolkit.org/flash/reference/iot-devices/

[NIFI] https://nifi.apache.org/

[NodeRED] A visual tool for wiring the Internet-of-Things. https://nodered.org/

[Pal, 2014] Pal, A., Mukherjee, A., Balamuralidhar, P.: Model driven development for internet of things: towards easing the concerns of application developers. In: International Conference on IoT as a Service, IoT360 Summit, Rome (2014)

[Rahman, 2018] Rahman, Leila Fatmasari, Tanir Ozcelebi, and Johan Lukkien. "Understanding IoT systems: a life cycle approach." Procedia computer science 130 (2018): 1057-1062

[Ray, 2017] Ray, Partha Pratim. "A survey on visual programming languages in internet of things." Scientific Programming 2017 (2017).

[Roukounaki, 2015] Roukounaki, Aikaterini, et al. "Visual development environment for semantically interoperable smart cities applications." International Internet of Things Summit. Springer, Cham, 2015.

[Santana 2017] [Santana et al. 2017] Santana, A. P. Chaves, M. A. Gerosa, F. Kon, and D. S. Milojicic. “Software Platforms for Smart Cities: Concepts, Requirements, Challenges, and a Unified Reference Architecture”. In: ACM Computing Surveys, Vol. 50(6), pp. 78:1–78:37, 2017.

[Sethi 2017] Sethi, P., and Sarangi, S.R., “Internet of Things: Architectures, Protocols, and Applications”, Journal of Electrical and Computer Engineering (2017).

[Su 2011] Su, K., Li, J., & Fu, H. (2011). Smart city and the applications. In Electronics, Communications and Control (ICECC), 2011 International Conference on (pp. 1028-1031). IEEE.

[ThingWorx] ThingWorx IoT Platform. http://www.thingworx.com

[Valtolina 2019] Valtolina, S., Hachem, F., Barricelli, B. R., Belay, E. G., Bonfitto, S., & Mesiti, M. (2019, July). Facilitating the Development of IoT Applications in Smart City Platforms. In International Symposium on End User Development (pp. 83-99). Springer, Cham.

[Yasumoto, 2016] Yasumoto, Keiichi, Hirozumi Yamaguchi, and Hiroshi Shigeno. "Survey of real-time processing technologies of iot data streams." Journal of Information Processing 24.2 (2016): 195-202.

[Zezzatti et al., 2019] Zezzatti, Alberto Ochoa, Roberto Contreras-Massé, and José Mejía. "Innovative Data Visualization of Collisions in a Human Stampede Occurred in a Religious Event using Multiagent Systems." 2019 23rd International Conference in Information Visualization–Part II. IEEE, 2019.

[Zimmermann 2017] O. Zimmermann, “Microservices tenets: Agile approach to service development and deployment”, Computer Science - Research and Development, Vol. 32(3-4), pp. 301–310, 2017.