2024 HOW TO: Deploy/Install your Snap4City Solution on private or public Clouds, VM with Containers
-
First version of Snap4City installation based on Kubernetes (03/01/2022): https://github.com/disit/snap4city-kubernetes
-
Platform Overview: PDF FILE https://www.snap4city.org/download/video/Snap4City-PlatformOverview.pdf
-
june 2022 - APPLIANCE single VM will almost all Snap4City MAIN Version 1.6: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityMAIN-Dashboard-1-6.rar
-
Snap4City installations have to use the following logo linked to Https://www.snap4city.org at the bottom right of each dashboard and in the main page of the service.
-
Snap4City & Snap4Industry Registered Instances Installations
(02-02-2021) Notes on Installing DataCity-Large 2020: HOW TO: Deploy & Installation of Snap4City DataCity-Large 2020
(10 September 2020) Snap4City DataCity-Small (aligned to version 1.5 of VM plus some new features) has been released as DOCKER Compose, see in the following.
(15 August 2020) Snap4CityMAIN version 1.5 has been released with included: synoptics, more widgets, real time buttons, IOT App updated, plus other VMs have been updated for composing a number of configurations: IOTOBSF, IOTDSES, KBSSM. See in the following.
Snap4City platform can be freely downloaded from this portal, and thus you can produce a number of services with these tools in the view of Smart City IOT as a Service, SCIaaS. Therefore, if you are interested in using Snap4City.org for trial from this portal you do not need to perform any installation on your computers.
On the contrary, this page is dedicated to who would like to perform a local installation on premise or cloud. In fact, in this page you can find the instructions for the installation and a number of Snap4City configurations. Snap4City is 100% open source, license free solution, and the Appliances that you can download from this page are license free. You can download and install Snap4City tools on premise or on cloud as you like, by starting from the following Appliances and/or from the GITHUB/DISIT sources (Source Code and licenses), and docker compose provided. In the following figures, the deploy of Snap4City solution from small business to large scalable cases are presented. The configurations are listed and coded with a letter to simplify the identification of the configuration. While other mixed solutions and intermedium configurations are also possible. In addition, some of the tools we are mentioning/providing in the following configurations can be substituted with your legacy tools. For example your CKAN, your GIS, ArcGIS server, etc., etc. For shortening your deploy and/or installation activities we have created a number of Virtual Machines, VM, Appliances and Docker containers/compose. The VMs can be put in execution on VMplayer or VMware workstation or Vcenter Vsphere VMware infrastructure or you can convert them for any kind of Cloud infrastructures, while Containers on Docker. The configurations are presented below from the smaller and simpler solutions to the most capable and wide deploy for large scale smart city with living lab support. For each of them an incremental description and user manual for installation. The same tools can also be downloaded from GITHUB/DISIT.
Configuration Kind |
Min #VM / Docker |
IOT Broker int (any# and kind) |
IOT Broker Ext (any# and kind) |
MAP WMS web |
KBSM semantic engine reasoner |
IOT APP, MicroServices #Containers |
Storage & IOT Data Shadow |
SSO, Roles LDAP |
IOT Security full stack end2end |
Dash + Wizard, widgets, Data Inspector |
GIS input WFS WMS |
GIS output WFS WMS |
Heatmaps WMS |
Mob. App & Micro Apps. |
ASCAPI Smart City API, Federated |
Living Lab |
Optional services supported by the configuration |
A: Alone/MAIN |
1 |
|
X |
X |
|
1 mf |
Small |
X |
X |
X |
X |
|
|
|
(x) |
ETL, DG, CM |
|
B: Basic |
2 |
X |
X |
X |
|
1 mf |
Small |
X |
X |
X |
X |
|
|
|
(x) |
ETL, DG, CM |
|
C: CityStart |
3-4 |
X |
X |
X |
X |
1 mf |
Small |
X |
X |
X |
X |
X |
X |
X |
X |
ETL, DG, R, CM, Eng |
|
D: DataCity |
5-6 |
X |
X |
X |
X |
70 |
Medium |
X |
X |
X |
X |
X |
X |
X |
X |
ETL, DG, R, CM, Eng, LL |
|
E: ExtensiveCity |
6-8+ |
X |
X |
X |
X |
70 |
Scalable |
X |
X |
X |
X |
X |
X |
X |
X |
ETL, DG, R, CM, Eng, LL |
|
F: FullPlatform |
12-14+ |
X |
X |
X |
X |
Scalable |
Scalable |
X |
X |
X |
X |
X |
X |
X |
X |
ETL, DG, R, RS, CM, Eng, LL |
|
F: FullPlatform + LivingLab |
13-15+ |
X |
X |
X |
X |
Scalable |
Scalable |
X |
X |
X |
X |
X |
X |
X |
X |
X |
ETL, DG, R, RS, CM, Eng |
Legenda of the above table for Optional Services
- DG: DataGate CKAN, open data portal, harvesting and federation
- R: Routing for mobile App and for What-IF analysis
- ETL: ETL SDK for developing ETL process for data ingestion and transformation in addition to IOT App already in
- RS: R Studio Server, for developing Data Analytics (other languages can be used as well) with different kind of servers
- Eng: Engager for engaging users on mobile App according to their context and behavior
- CM: Chat Manager, for private chat room management attached to the dashboards
- LL: Living Lab, for supporting the set up and management of the living lab, see how to install DRUPAL for living lab below
- etc.
A: Alone/MAIN) For Small Size Business, for example an industry 4.0 application you can need to install only one VM: Snap4CityMAIN, ContainerCompose "Alone".
it may be used to create dashboards exploting data coming from external third party IOT brokers, from external data stores / databases of several kinds (ODBC, JDBC, Mongo, RDF, etc.), etc. Please note that the solution released as VM allows you to test also Smart IOT Applications, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.
Therefore, 1 (one) VM, one Container Compose with multiple Containers and you can start using.
Snap4CityMAIN (also called dashboard) includes the Snap4City Dashboard Builder and a full starting Snap4City environment for small size businesses as well as for large solutions. Snap4CityMAIN includes tools configured for supporting features in a Snap4City local environment on your premise, so that with limitations only on configuration aspects to make the VM consistent and capable to be executed and provide Snap4City services in your network. it can be the seeding of a larger and scalable solution on your premise. Snap4CityMAIN can be replicated to create a cluster in balance. The database can be shared from the frist Snap4CityMAIN while it cam be also extracted and put in some network storage.
What you can do with ALONE configuration you can:
- create a small smart city business, (or industry 4.0 solution) acquiring data by using this VM only, for example to get data with Node-RED and presenting them on Dashboards;
- create and manage a number of users into your on premise Snap4City platform, according to GDPR;
- directly connect your Dashboard Builder environment to get data from any your database in your network;
- register External Services for showing them into your dashboards;
- register MIcro Applications;
- Communicate with Dashboard by using Web Socket streams.
- define you myKPI and POI and manager them on dashboards, they can be time series, and time series with GPS locations;
- connect the SNAP4CITYMAIN with (a) the IOTOBSF: for ingesting data from the IOT Brokers in push, (b) ETL for ingesting data from batch processes in PULL; This means that with two VM you can create you Industry 4.0 and Smart City solutions. The only limit is the volume of data, the number of IOT Applications, and the lack of massive IOT device management.
- add more IOT or other protocols using the IOT Applications as described into https://www.snap4city.org/65
- add Data Analytics as described in: https://www.snap4city.org/501
- please note that:
- you can start with Snap4City MAIN anche step by step creating more complete and high performance solutions without the need to restart since any solution include also the MAIN
- MAIN does not include any knowledge base map management. You can use your GIS or direcly the maps from open sources. for the KB you need to instal KBSSM
- etc
Alone/MAIN Configuration: Snap4CityMAIN Virtual Machine
Snap4CityMAIN VM can be found on:
- SNAP4CITY MAIN version 1.6 (June 2022): https://www.snap4city.org/download/video/vm/Appliance-Snap4CityMAIN-Dashboard-1-6.rar
- SNAP4CITY MAIN version 1.5: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityMAIN-Dashboard-1.5.rar
- former version 1.4: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityMAIN-Dashboard-1.4.rar
The suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe
Documentation for installation can be found on HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City and the VM appliance Snap4CityMAIN includes tools as:
- Dashboard Builder for creating dashboards of several kinds, and distributing them on a large number of users, also in connection with IOT Applications, including Dashboard Wizards and Native Widgets. A large number of widgets updated and/or added in version 1.5 and most of them are dynamic: time series, bar series, kiviat, multiseries, single content, buttons, tables, event driven with synoptics, etc., most of them allow you to browse on time, etc.)
- if you decide to go for the direct source code installation see: https://github.com/disit/dashboard-builder/wiki
- Data Inspector: for exploring the data accessible on the solution. A large set of new features have been added on Ver 1.5, for managing the collection of data, links to processes, links to IOT Apps, licensing models, owners, etc.)
- Dashboard engine, for collecting data from several data sources with a Pull approach and save them on local database of Dashboard Builder. The Dashboard engine is compatible with several protocols.
- Authentication tool, Single Sign On, SSO: for registering users and managing keys (keycloak based), LDAP support for managing roles and organizations.
- Notificator for programming notifications to be sent when some event occurs on Dashboard metrics.
- IOT Application, IOT App: one Node-RED Docker for implementing IOT Application flows, a number of them, also supporting Snap4City MicroServices, mutual authentications, etc., working with data in Push and Pull and several protocols. The library of Snap4City MicroServices presents some limitations since the Knowledge Base as Servicemap is missing in the configuration. With Version 1.5, stronlgy update of nodes for node-red for managin dynamic dashboard widgets, synoptics directly, barseries, timeseries, etc., in addition the compatibility with a number of new protocols
- MySQL database, local instance for configuration and data management. Which can use also Elastic Search even if not configured yet.
- External Service manager: to collect external services, legacy servers and tools and exploit them on Dashboards. Please remind that External Services as web pages can be also embedded into Dashboard widgets.
- MenuManager also called Main Snap4City: to manager different tools connected on a Snap4City environment, also external services, legacy servers, etc. On version 1.5, added the possibility of having menu attached to the single organization and single dashboards.
- OwnershipsManager: to manage users' ownerships and delegations to access at different kinds of data and resources, delegation to access, revokations, etc. Added in Ver. 1.5 the possibility of controlling ownership and delegation via IOT Applications.
- MyKPI: for collecting and managing MyKPI, personal MyPOI, etc. It includes the Personal Data Manager, based on GDPR. (updated on ver 1.5)
- WS Server: WebSocket Server for secure communications among IOT Applications and Dashboard, Synoptics, widgets, etc. (strongly updated on ver. 1.5, there is also Kafka for managing event driven data, plus WS server)
- MicroApplications: a set of MicroApplications which can be tuned on the content accessible on your area. The MicroApplications may have sense when the Main Snap4City is configured together with KBSSM providing knowledge base of city entities on the maps. Please remind that MicroApplications can be also embedded into Dashboard widgets.
- IOT device management with IOT Directory: IOT Directory: is a tool for managing a large number of IOT Brokers of different kinds, and supporting different IOT protocols. The IOT Directory needs at least a ServiceMap and to perform IOT Discovery. The IOT Directory allows at the users to add IOT Devices, IOT devices in bulk, to manage delegation on devices and IOT Devices values, etc.
- including the process for ingesting IOT Devices from external IOT Brokers (that are IOT Brokers not managed by IOT Directory, so that they are managed by third party for adding IOT Devices).
- on ver 1.5, the management of IOT Brokers, IOT Devices, sensors, actuators, etc. has been much more simplified and automated by increasing its flexibility in managing internal and external Brokers
- Resource Manager: to publish, exchange and access to a large set of resources developed for other smart cities and Snap4City solutions; The Resource Manager includes market place for IOT Applications and flows, ETL processes, Data Analytic processes, etc. It is a very powerful tool for information exchange and for the publication of smart applications. The Resource Manager also allows the user to create MicroServices for the IOT Applications, when they are based on rest call of external services. Added on ver 1.5 the Data Dictionary, to define nature, subnature, value types, value units.
- Synoptic support since version 1.5: a set of tools for creating synoptics and custom Widgets for the dashboards using MyKPI, sensors, costants, variables on WS.
- User Stats since version 1.6 (coming soon): a set of metrics and statistics on resources consumed by each user for self assessment;
- we postponed other fetures to the next versions. On the other hand, a number of features that you see on Snap4City.org can be added on the main taking the source code from GITHUB and make from yourself the configuration needed.
Alone/MAIN Configuration: Snap4CityMAIN Docker Containers
The Docker version for this "Alone Configuration" and is available
- version 1.4 of docker that will be aligned to MAIN 1.5: is coming..... see DataCity-Small docker compose for Version 1.5 below in this web page
- version 1.3 https://github.com/disit/snap4city-docker/tree/master/Alone
- version 1.2 from: https://github.com/disit/snap4city-docker/tree/master/Alone
The configuration in terms of user names and password is the same of the VM described above: HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City, On the other hand, the Container Compose includes the details of the ports used that should not conflict with the host or server in which you deploy it, so t hat you can change in the file in case of conflict. In addition, you have to insert into the hosts file of your operating system: <your-host-ip> dashboard keycloak wsserver iotapp personaldata
This will allow all processes to refer to compoments: dashboard, keycloak, wsserver, iotapp, personaldata, as localhost by using their names and internal mapping. The same if you have other computers in the same network. Please note that, if you are going to make the solution public you have to provide a domain name, etc., and change the configuration according to your setup.
The cerate group of contaiers are put in execution by the Container Compose (version 1.3) and in particular each of them are accessible from Docker HUB and include:
- https://hub.docker.com/r/disitlab/dashboard-builder
- Dashboard Builder for creating dashboards of several kinds, and distributing them on a large number of users, also in connection with IOT Applications, including Dashboard Wizards and Native Widgets.
- if you decide to go for the direct source code installation see: https://github.com/disit/dashboard-builder/wiki
- Data Inspector: for exploring the data accessible on the solution.
- Dashboard engine, for collecting data from several data sources with a Pull approach and save them on local database of Dashboard Builder. The Dashboard engine is compatible with several protocols.
- Notificator for programming notifications to be sent when some event occurs on Dashboard metrics.
- External Service manager: to collect external services, legacy servers and tools and exploit them on Dashboards. Please remind that External Services as web pages can be also embedded into Dashboard widgets.
- MenuManager also called Main Snap4City: to manager different tools connected on a Snap4City environment, also external services, legacy servers, etc.
- OwnershipsManager: to manage users' ownerships and delegations to access at different kinds of data and resources, delegation to access, revokations, etc.
- MicroApplications: a set of MicroApplications which can be tuned on the content accessible on your area. The MicroApplications may have sense when the Main Snap4City is configured together with KBSSM providing knowledge base of city entities on the maps. Please remind that MicroApplications can be also embedded into Dashboard widgets.
- IOT device management with IOT Directory: IOT Directory: is a tool for managing a large number of IOT Brokers of different kinds, and supporting different IOT protocols. The IOT Directory needs at least a ServiceMap and to perform IOT Discovery. The IOT Directory allows at the users to add IOT Devices, IOT devices in bulk, to manage delegation on devices and IOT Devices values, etc.
- except for the process for ingesting IOT Devices from external IOT Brokers (that are IOT Brokers not managed by IOT Directory, so that they are managed by third party for adding IOT Devices). that would be available soon (coming soon):
- Resource Manager since version 1.4 (coming soon): to publish, exchange and access to a large set of resources developed for other smart cities and Snap4City solutions; The Resource Manager includes market place for IOT Applications and flows, ETL processes, Data Analytic processes, etc. It is a very powerful tool for information exchange and for the publication of smart applications. The Resource Manager also allows the user to create MicroServices for the IOT Applications, when they are based on rest call of external services.
- User Stats since version 1.4 (coming soon): a set of metrics and statistics on resources consumed by each user for self assessment;
- Synoptic support since version 1.4 (coming soon): a set of tools for creating synoptics and custom Widget for the dashboards.
- Dashboard Builder for creating dashboards of several kinds, and distributing them on a large number of users, also in connection with IOT Applications, including Dashboard Wizards and Native Widgets.
- https://hub.docker.com/r/disitlab/personaldata
- https://hub.docker.com/r/disitlab/snap4demo-keycloak, https://hub.docker.com/r/disitlab/snap4demo-openldap
- Authentication tool, Single Sign On, SSO: for registering users and managing keys (keycloak based), LDAP support for managing roles and organizations.
- https://hub.docker.com/r/disitlab/snap4city-nodered-adv, https://hub.docker.com/r/disitlab/snap4city-nodered-basic
- IOT Application, IOT App1: one Node-RED Docker for implementing IOT Application flows, a number of them, also supporting Snap4City MicroServices, mutual authentications, etc., working with data in Push and Pull and several protocols. The library of Snap4City MicroServices presents some limitations since the Knowledge Base as Servicemap is missing in the configuration.
- A standard Container for MySQL
- MySQL database, local instance for configuration and data management.
- https://hub.docker.com/r/disitlab/websocketserver
- WS Server: WebSocket Server for secure communication among IOT Applications and Dashboard, Synoptics, widgets, etc.
B: Basic) Addressing a relevant number of IOT devices/ IOT brokers in your smart city or industry 4.0, a few smart Applications with multiple flows
If you have a relevant number of IOT Devices you need to have several IOT brokers. They can be registered with their devices into the Snap4City IOT Directory in the Snap4CityMAIN. Moreover, you may need to implement and thus provide some of those IOT Brokers. You can do that by using one or more IOTOBSF, which includes an IOT Orion Broker supporting NGSI and other protocols see :https://www.snap4city.org/467 for Mutual Authentication. In the context of IOT for Industry applications, you may need to have some intelligence and IOT Applications located on the field, as well as on premise. For example in factory in the case of Industry 4.0, on the farm in the case of smart farm, etc. This smart tools can be IOT Applications working on IOT Edge. This configuration alloow have them connected to the Smart4CityMAINVM, thus sending data to the solution at MyKPI, via IOT brokers, etc. see below the list of the IOT Edge solutions on Snap4City with Security support. You may have IOT brokers and/or IOT Edge, both solutions or only one, depending on your data and preferences. Please note that the solution allows you to test also Smart IOT Application, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.
Therefore, 1 (one) VM + N IOT broker VM and you can start using.
- IOTOBSF includes (i) the standard IOT Orion Broker FiWare, plus (ii) the Snap4City Proxy Filter Security for enforcing mutual authentication and secure connection from IOT Devices and IOT Broker, from IOT Broker and Node-RED IOT Applications of Snap4City. You CAN:
- A) Download and put in execution IOTOBSF VM
- HOW TO: configure/install IOTOBSF VM appliance of Snap4City
- VM version 1.2 to be used with MAIN version 1.5: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityIOTOBSF-1.2.rar
- configured to use eth0 thus simplifing the intallaton on your local cloud.
- the suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe
- VM version 1.1 with MAIN version 1.4: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityIOTOBSF-1.1.rar
- B) Download and put in execution IOTOBSF Docker
- C) create from scratch:
- VM: download and install the official Fi-Ware VM or Docker for IOT Orion Broker that you can find at: https://fiware-orion.readthedocs.io/en/master/ (see also https://fiware-orion.readthedocs.io/en/master/admin/extra/ha/index.html).
- DOCKER: get the official Docker container with Orion Broker from https://hub.docker.com/u/fiware/
- If you need secure connection with Snap4City with mutual authentication you have to install the secure filter into VM or DOCKER that you can find on: https://github.com/disit/snap4city/tree/master/OrionBrokerFilter
- In any cases read: HOW TO: configure/install IOTOBSF VM/Container appliance of Snap4City
- A) Download and put in execution IOTOBSF VM
- IOT Edge you may need to have some intelligence, processes and Applications located on the field, as well as on premise. For example in factory in the case of Industry 4.0, on the farm in the case of smart farm, on the city center for mobility, on some park, etc. etc. This partially authonomous smart tools can be implemented by IOT Application working on IOT Edge, they are IOT Application = Node-RED + Snap4City library of MicroServices Nodes, and can be installed as:
- Linux Ubuntu Appliance VM as IOT Edge: https://www.snap4city.org/298 Snap4City IOT Edge Linux on VM Ubuntu: IOT Application, to be configured and updated at the first installation.
- You Can install them from Docker Hub: https://hub.docker.com/r/disitlab/snap4city-nodered-adv, https://hub.docker.com/r/disitlab/snap4city-nodered-basic
- IOT Application, IOT App1: one Node-RED Docker for implementing IOT Application flows, a number of them, also supporting Snap4City MicroServices, mutual authentications, etc., working with data in Push and Pull and several protocols. The library of Snap4City MicroServices presents some limitations when the Knowledge Base as Servicemap is missing in the configuration.
- You Can install them from Docker Hub: https://hub.docker.com/r/disitlab/snap4city-nodered-adv, https://hub.docker.com/r/disitlab/snap4city-nodered-basic
- Android as IOT Edge https://www.snap4city.org/278 and
- Raspberry Pi as IOT Edge: https://www.snap4city.org/77 You can install Snap4City library in any Node-RED from the official Node-RED library
- Linux Ubuntu Appliance VM as IOT Edge: https://www.snap4city.org/298 Snap4City IOT Edge Linux on VM Ubuntu: IOT Application, to be configured and updated at the first installation.
- IOT Applications = Node-RED + Snap4City library of MicroServices, this means that you can start from any installation of Node-RED and add Snap4City by simply installing Snap4City Library from the official Library of Node-RED, that in Node-RED means to add new Nodes into the Palette as https://nodered.org/docs/user-guide/editor/palette/manager : access for the microservice collection from the JS foundation with full documentation inside
- BASIC: Final User: https://flows.nodered.org/?term=snap4city&num_pages=1
- ADVANCED: DEVELOPER (you have to install both libraries: basic and advanced): https://flows.nodered.org/?term=snap4city&num_pages=1
- Docker Containers of IOT Applications: https://hub.docker.com/r/disitlab/snap4city-nodered-adv, https://hub.docker.com/r/disitlab/snap4city-nodered-basic
- IOT Application, IOT App1: one Node-RED Docker for implementing IOT Application flows, a number of them, also supporting Snap4City MicroServices, mutual authentications, etc., working with data in Push and Pull and several protocols. The library of Snap4City MicroServices presents some limitations since the Knowledge Base as Servicemap is missing in the configuration.
C: CityStart) Addressing a relevant number of GeoLocated Points of Interest, PIN on maps, and eventually heatmaps, a Small Smart City with a few smart applications on cloud and many on edge
If you need to manage a relevant number of city entities located on maps, we suggest you the following minimal configuration which includes KBSSM modeling a knowledge base for street graphs which can be feed with OSM graph or with your GIS as well (also included into it). You can exploit maps from GIS solutions (see https://www.snap4city.org/368) or you can exploit the Knowledge Base approach which is more powerful in terms of semantic relationships among city entities. To this end you need to have KBSSM appliance into your platform. If the workload on Smart City API become very high the KBSSM can be cloned to create a cluster and a balancer. The first KBSSM may keep the storage and share with partners. Please note that the solution allows you to test also Smart IOT Application, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.
Therefore, at the minimum, 2 (two) VM and you can start using. If you have also Geoserver 3 VM. if you have a huge amount of mobile apps exploiting the Smart City data or you have to provide dashboard towards a large number of users, you may need to create a balanced workload, etc.
- KBSSM: includes ServiceMap, ServiceMap3D, SuperServiceMap, Knowledge Base on Virtuoso RDF store, SmartCity API, from OSM to ServiceMap Tool, LOG (for windows may be provided in separate VM)/Flint tools to work on RDF Stores, thus you have a semantic index. As a first step you need to create the Knowledge Base into the RDF store Virtuoso of your territory; this is possible by using the OSM to ServiceMap tool included, it ia matter of few minutes. Once created the ServiceMap (knowledge base) model from the OSM Smart City API for Mobile and Web Applications will be ready; ServiceMap and its API allow you to perform semantic queries, delivering services towards the Mobile Apps, etc. One or more Knowledge Bases can be added. A single Knowledge Base can manage multiple territories/cities/regions. A set of Knowledge bases can be joined to work together by using the so called SuperServiceMap. You can use your ServiceMap with the official SuperServiceMap of Https://www.snap4city.org The addition of the ServiceMap also implies the possibility of using a large amount of MicroServices (nodes) from the IOT Applications, since they exploit the Smart City APIs. The SuperServiceMap and thus the Smart City API also expose API in WFS for GIS integration. The addition of the ServiceMap and thus of the knowledge Base also enable the mechanisms of IOT Discovering into the IOT Directory included into the Snap4CityMAIN.
- VM version 1.2 to be used with Main version 1.5: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityKBSSMVM-1.2.rar
- configured to use eth0 thus simplifing the intallaton on your local cloud.
- Km4City semantic model and knowledge base have been updated
- Advanced smart City API have been updated, including API for interfacing the KB with the IOT Directory and IOT tools.
- VM version 1.1 with MAIN version 1.4: .https://www.snap4city.org/download/video/vm/Appliance-Snap4CityKBSSMVM-1.1.rar
- Including: ServiceMap, Smart City API, SuperServiceMap, Super Smart City API for KB federation, OSM to ServiceMap
- This version 1.0 is working fine with all kind of data, while adding IOT data is supposed to work in DataCity Configuration with at least IOTDSES VM version 1.0
- Docker containers:
- HOW TO: configure/install KBSSM VM appliance of Snap4City
- ServiceMap documentation: https://github.com/disit/servicemap/wiki/ServiceMap-installation
- ServiceMap configuration: https://github.com/disit/servicemap/wiki/ServiceMap-Configuration
- The suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe
- VM version 1.2 to be used with Main version 1.5: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityKBSSMVM-1.2.rar
- GISGeoServer: which is optional and Includes the heatmap manager, a tool and API at which you can send heatmap data and you can get back Heatmaps on Dashboards via WMS protocols. The same GeoServer can also provide Shapes and other GIS Data in WFS/WMS format as back ground images, patterns, maps, etc. These API for the Heatmap are part of the Smart City API.
- HOW TO: configure/install GISGeoServer VM appliance of Snap4City
- VM version 1.0 with MAIN version 1.3 and 1.4: https://www.snap4city.org/download/video/vm/Appliance-Snap4CityGISGeoserver-1.0.rar
- The suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe
Ds: DataCity-Small) A small size Smart City with a fixed number of smart applications on cloud and 2 IOT brokers, relevant volume of data entering into the cloud
To satisfy small size Smart City, probably you need to have a limited number of smart applications on cloud, for example 4-5 IOT App/containers, and 1-2 IOT brokers, that collected IOT Data from whcih you prefer to the Data Shadow in place. You can start with the following configuration with restricted clustering capabilities that may growth with your city needs as well, reaching the Configuration E as described in the following and without radical changes but progressively. You may continue to use IOT Edge even if they have not been represented in the picture. The 2020 version includes full support for synoptics, data dictionary, resource management, new library of microservices aligned to July 2020 (the update can be easily performed from the IOT App in node-RED), IOT edge any kind, etc..
For this configuration on VM you have to install VM MAIN and the other components.
KBSSM: see above for this component
IOTDSES: including a non clustered solution for data shadown based on NIFI + Elastic Search. IOT Data Shadow: the IOT data shadow allows to perform a copy of any message arriving from/to IOT Devices via the IOT Brokers registered on the IOT Directory. This feature consists of a cluster of VMs, and it is necessary when large volume of IOT Data comes into the system. The solution is based on a cluster NIFI, Elastic Search and Kibana. The solution is scalable and reliable, and fault tolerant. Elastic Search is the storage in which the IOT data are stored. Once the data is collected, the historical values are made available for the Dashboards, and IOT Applications via MicroServices of the Smart City API and thus of the Knowledge Base. To start you can have one single IOTDSES while the solution can scale up cloning and clustering the IOTDSES . Each NIFI server is managing on a modest VM more than 120.000 msg per minute.
- VM version 1.1 to be used with Main version 1.5: https://www.snap4city.org/download/video/vm/Appliance-Snap4cityIOTDSES-1.1.rar
- configured to use eth0 thus simplifing the intallaton on your local cloud.
- NIFI process has been updated to work with new Smart City API and new Km4City model, to collect data and reconciliate data towards Elastic search and Kafka.
- VM Version 1.0 with MAIN version 1.3 and 1.4: https://www.snap4city.org/download/video/vm/Appliance-Snap4cityIOTDSES-1.0.rar
- including: NIFI, SQUID and Elastic Search in a non scalable configuration, suitable for working with at least KBSSM version 1.0 and IOTBSF version 1.0
- The suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe
- The Docker Container version is included in the following docker compose. Container regarding Elastic Search are those official.
- HOW TO: configure/install IOTDSES VM appliance of Snap4City
- resources suggested: a minimum of 8Gbytes RAM, 8 CPU cores at 2 Ghz, and 500 Gbytes of storage
The Docker compose version of September 2020 (aligned to ver. 1.5 of VM, a little more advanced) of the whole configuration is: https://github.com/disit/snap4city-docker/tree/master/DataCity-Small
The configuration contains: MAIN, IOT Directory, two IOT Applications (July version), Notificator, one IOT Broker (IOTOBSF), ServiceMap (KBSSM), Advanced Smart City API with SuperServiceMap, Elastic Search, Kibana, Dashboard Builder, Synoptics, Data Inspector new version, Resource Manager new wversion, MyKPI, MyPOI, Knowledge Base, (for OSM to KB conversion use VM), auditing tools, management tools, external services, micrapplications, etc. Starting resources suggested: a minimum of 16-32 Gbytes RAM, 8 CPU cores at 2 Ghz, and 500 Gbytes of storage (it may work/start even with 6 Gbytes but some of the process could be very limited)
What you can do:
- manage the city geo data by ingesting data from OSM to ServiceMap (via separate VM), with our tools, exploiting your GIS tools
- ingesting data driven data from IOT Devices via IOT Orion Broker and related adaptors, it can manage a number of brokers, internal and external.
- ingesting pull data by developing flows into IOT Applications (each IOT App may have multiple flows), exploiting Snap4City MicroServices for the ingestion. The IOT App may go to get the data and pass them into the IOT broker, thus resulting the data automatically indexed, stored and accessible for all the features
- storing data into the data shadow based on elastic search and indexing them into the knowledge base automatically
- the ingested data are automatically made available for the Data Inspector and for the Dashboard wizard and builder
- create your dashboards exploiting maps, time trends, single content, buttons, bar series, multiseries, etc. IOT devices as sensors and actuators, etc. etc. they are data drive, may change dynamically data from IOT App directoy on Dashboards, and a number of new widgets have been added
- Synoptic: production of synoptic templates in SVG (open source editor separated), istantiation of a synoptic connecting them to MyKPI, sensors, contants, etc.
- adding any other external services, IOT Devices, IOT brokers, etc., extended IOT device model with constant attributes, and ingestion in bulk
- developing smart IOT Applications for data transformation in the back of your data flow or in the back of the dashboards for business intelligence
- adding more IOT Applications by changing the Docker Compose or allocating them in other VM or hosts
- registering local users and managing their profiles, with the user manger allowing local registration
- data dictionary has been added to define: nature, subnature, value types and value units, and their relationships
- etc.
D: DataCity) A medium/large size Smart City with a number of smart applications on cloud and a number of IOT brokers, relevant volume of data entering into the cloud
To satisfy a medium size Smart City, probably you need to have a number of smart applications on cloud, for example even 70 IOT App/containers, and a number of IOT brokers, that collected IOT Data from whcih you prefer to the Data Shadow in place. You can start with the following configuration with restricted clustering capabilities that may growth with your city needs as well, reaching the Configuration E as described in the following and without radical changes but progressively. Please note that even a large Smart City usually did not need more than 70 processes. For example, In Helsinki and Antwerp we used no more than 15 processes each. In Florence and whole Tuscany we have about 200 processes, covering the whole region with several relevant cities. So that you can pass at Large Scale Smart Cities maintaining the single MCLSCount and a cluster of IOTDSES for the data shadow of big data: billions and billions of data according to the number of VM you are allocating, not problem on scaling. So that Configuration D with elastic search cluster is called E: ExtensiveCity.
You may continue to use IOT Edge even if they have not been represented in the picture.
At the minimum without any optional, you need to install 4 VM + N IOT broker VM and you can start using.
- IOTDSES: see above for this component
When you have VMs as: Snap4CityMAIN, KBSSM, IOTBSF, GEOServer and a number of IOTOBSF you can create a smart city as DataCity version minus (that means without a cluster of IOT Applications), that can be statically instantiated on some VM with Docker any way. Thus you have a non scalable solution for smart city that you can size according to your needs.
- MCLSCont (coming soon): including Container Cluster Manager first VM: Mesos, Marathon, APIs, DISCES-EM. it consits of a set of tools for managing in elastic manner a large number of containers dockers. which are going to include IOT Applications, ETL processes, Web Scraping Processes, Data Analytics in R Studio, etc. This feature can be added including a number of VM for creating a Marathon/Mesos cluster. This VM includes the DISCES-EM scheduler for elastic management of containers on VSphere infrastructure, plus other minor tools. If you do not need to have scalability on a large number of Containers (IOT Applications, Data Analytic, Web Scraping, etc.) for them you can avoid to install a cluster and may use only one of them: TC10.14- Container Cluster Monitoring including: Marathon and Mesos each VM may manage up to 70 Containers for IOT Applications, but depend on the memory size, similarly 30 Data Analytics, 50 Web Scraping, etc. You can create three of MCLSCont for creating a Master Cluster based on Zookeeper, the additional two MCLSCont have to provide a different configuration with respect to the main MCLSCont. The Data Analytics may include, predictions, heatmap productions, OD matrices estimation, traffic flow reconstruction, GRAL predictions, traffic total flow computing, anomaly detections, etc.
- NCLSCont (coming soon): includes a simpler version of MCLSCont . it is just a node for the Cluster of Containers. You can duplicate this VM as you like to create the cluster and the cluster has to be connected to the Masters as described above based on MCLSCont .
- HOW TO: configure/install MCLSCont VM and NCLSCont VM appliances of Snap4City
Please note that a number of other VMs/Container have been listed in the figure as optionals, such as: CKAN Data Gate, Routing, etc:
- ETLSDK Server: https://www.snap4city.org/download/video/Snap4City-ETL-VM.rar, this ETLSDK Server is a Virtual Machine dedicated to developers of ETL processes. This appliance or eventually a cluster of this VM may be optional and may be needed only if the Smart City has already adopted in the past ETL based data ingestion process. It is strongly similar to the ETL Development environment that Snap4City provide online to its developers. It includes: ETL development environment based on a standard installation of Penthao Kettle, plus Snap4City DISCES for periodic scheduling of data ingestion processes (or at least to learn how to schedule processes in a single or multiple nodes). We suggest you to read documentation about this on: https://www.snap4city.org/24 The Solution is scalable in the sense that this VM can be cloned to create a cluster with the distributed scheduler DISCES to create a Data Cluster. see slides: https://www.snap4city.org/download/video/barc19/ingest/
- ETLNode (please request): is identical to the ETLSDK version above, and includes Phoenix (snap4city drivers), DISCES, HBASE for creating a Big data cluster HDFS storage, on which the connection to the KBSSM is supported.
- HOW TO: configure/install ETLNode VM appliance of Snap4City
- DataSetManager DataGate is additional (please request): the Data Set Manager is an instance of CKAN customized to work with SSO, LDAP released as open source also. It also includes DataGate module for accelerating the data ingestion of data sets. If you have already a CKAN on your premise you can simply install the DataGate module.
- Routing Server (please request): is an additional VM/Containerwith the OSM gathering and routing algorithms. In Snap4City there are two Routers. The one used for the Mobile App and one used for the What-IF Analysis. Both of them are Open Source.
- ChatManager (please request): is an additional VM/Container with the Chat Manager based on Rocket Chat Room system with some configuration only!! all the adptations are already included into the Smat4CityMain. It is directly connected with the Snap4CityMAIN for dashboard system.
- RStudioServer (please request): is an additional VM/Containerwith R Studio, it is a standard deploy of the free version of R Studio Server. This VM/Container or physical server may have separate user management, and assignment of VM for R-studio Server in the main menu of the Snap4City. R Studio Server can be located on Host with NVIDIA Tensor Flow, as well as on other parallel machines. There is a commercial version of R Studio on which it is possible to have SSO with the rest of the platform., that is not commercialized by Snap4City but from an R Studio third party provider.
- Engager Server VM (please request): is an additional VM/Container with Engager tools which are based on DROOLS, for assessing user behavior condition from the data coming from mobile apps (typically on MyKPI), and reacting to the changes of status by providing suggestions, recommendation, engagements, questionnaires, soudages, etc. They can be planned and tuned on the basis of the user profile and behavior.
- advanced Data analytics (please request) that may have a cost such as: traffic reconstruction, environmental data predictions, parking predictions, wifi people flow predictions, offer vs demand of mobility, etc.
- Python server (please request): a VM/Container for developers for creating Python based processes which are deployed as Containers into the platform and made accessible as MicroServices for IOT Application, or direcly scheduled by DISCES.
- Java server (please request): a Virtual machine for developers for creating Java based processes which are deployed as Containers into the platform and made accessible as MicroServices for IOT Application, or direcly scheduled by DISCES.
- your legacy tools can be connected as well.
See DCL: DataCity-Large Structure and Configuration
E: ExtensiveCity) A Large size Smart City with smart applications on cloud and a number of IOT brokers, large volume of data streams entering in the cloud
So that you can pass from configuration D to E managing a Large Scale Smart Cities maintaining the single MCLSCount and a cluster of IOTDSES for the data shadow of big data: billions and billions of data according to the number of VM/Container you are allocating, not problem on scaling. So that Configuration D with elastic search cluster is called E: ExtensiveCity. You may continue to use IOT Edge even if they have not been represented in the picture.
F: FullPlatform) For Huge scale Smart City with a Huge number of smart applications and processes on cloud (thousands), any number of IOT brokers
A large scale deploy for several millions of messages per day entering into the cloud can be obtained by using the same components as described in the following figure. This solution add to the E configuration described above a cluster for managing containers including: IOT Applications, Data Analytic, ETL processes, Web Scraping Processes, R Studio processes, Java, etc. created with MCLSCount and NCLSCount. The resulting configuration is for very huge scale smart City and also capable to manage a large Living Lab Community of developers with their personal IOT App, processes of several different kinds. You may continue to use IOT Edge even if they have not been represented in the picture.
F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as Full Platform plus Living lab Support
In this case, the schema is practically identical of the F:FullPlatform but with the addition of:
- LivingLabPortal (please request): a VM/Container including Drupal with all modules intalled for managing the CRM of the Living Lab, registration and user management, multiple organizations, multiple groups, articles and blogs, comments on alll products, automatic alignement of LDAP users with those of Drupal, Single Sign On, SSO, among the users of Snap4City and those of Drupal, monitoring of user activity, GDPR support, SOLR index of queries, reports, indexing, etc.
For the DRUPAL modules to be installed to set up the Living Lab support we suggest:
- install drupal 7
- take and install our DRUPAL 7 Snap4City Custom modules from: https://github.com/disit/snap4city/tree/master/Drupal-Modules :
- Simple LDAP (7.x-1.4) (enable Role and User submodules)
- OpenID Connect (7.x-1.0-beta8+2-dev)
- OpenIDConnectPatch
- Organic Groups (7.x-2.10)
- OrganicGroupSimpleLdap
- ProfileSnap4City
- many other DRUPAL standard modules can be installed for: editing content, indexing and search, etc. etc.
- Setup drupal:
- add Content type "Organization" with machine name "disitorg" and set it as Group in the Organic groups section
- add Content type "Organization group" with machne name "disitgroup" and set it as Group and Group content" ...
Strongly suggested to install also optional or similar legacy services such as:
- Data Analytics in R Studio, Tensor Flow (NVIDIA)
- Hbase / Phoenix Big Data Store
- Data Gate CKAN for Open Data
- Chat Manager on Dashboard
- ETLSDK servers
- User Engagement Server
- Routing Solutions
What you can exploit from Https://www.snap4city.org community a set of On Line Services:
- Help Desk and contacts;
- updated documentation, test cases, videos, slides, tutorials, Swagger of API, and a community for discussing on them;
- Virtual Assistant to support the developers in getting the needed documentation;(please request);
- Development tools as: LOG (linked open graph) also called Knowledge Base Graphs, and Knowledge Base Queries in SPARQL;
- Several Knowledge Bases of cities and regions: Toscana and all its cities and provinces, Helsinki, Antwerp, Sardegna and many cities, Garda, etc.;
- an environment in which the new tools (even your if you propose) are tested and incubated for all;
- access to DISIT.org Https://www.disit.org
- access to Km4City.org https://www.km4city.org
- ....
If you have special requests please ask to Snap4City via Contact form https://www.snap4city.org/3
Map of features per VM/Container see also (Source Code and licenses)
Features included into the VM Appliance: (you can update from Github/disit) |
SNAP4CITYMAIN | IOTOBSF | KBSSM | MCLSCont NCLSCont |
IOTDSES | GISGeoserver | ||
|
VM |
VM |
VM, img, Src |
VM |
VM | VM | VM | VM |
X |
|
|
|
|||||
X |
|
|
|
|||||
KPI Support, POI support |
X |
|
|
|
||||
SSO: Authentication, Authorization, Roles |
X |
|
|
|
||||
X |
|
|
|
|||||
IOT Application = Node-RED + Snap4City MicroServices |
X |
|
|
|||||
X |
|
|
|
|||||
X |
|
|
|
|||||
Menu Profiler |
X |
|
|
|
||||
X |
|
|
|
|||||
ETL editor, EDL Development Environment |
|
X |
|
|
||||
|
X |
|
|
|||||
IOT Broker: Orion Broker + Security Filter |
|
|
|
(X) |
||||
IOT Device Management: IOT Directory (includes IOT brokers management) | X | |||||||
Resource Manager: auditing, market place | X | |||||||
Synoptics and custom widgets | X (since version 1.5) | |||||||
User Statistics and self assessment | X (since version 1.6) | |||||||
Micro Applications | X | |||||||
ServiceMap, ServiceMap3D, SuperServiceMap, Knowledge Base, Virtuoso, SmartCity API, from OSM to SM Knowledge base, semantic index: servicemap (includes Smart City API, Swagger), WFS API for GIS integration | X | |||||||
IOT Application Manager: Mesos, Marathon, APIs, DISCES-EM, IOT Application = Node-RED + Snap4City MicroServices; Web Scraping, DataAnalytics, ETL processes, Java processes, etc. |
(X: only 1 IOT App) | X | ||||||
IOT Data Shadow: NIFI + Elastic Search (it can be used as a starting point for clustered NIFI and ES creating large storage) | X | |||||||
GIS, GeoServer, heatmap server, heatmap manager, WFS and WMS support | (X: WFS) | X |
Main links for the main components. All source code is on GITHUB/DISIT https://github.com/disit see licenses for all: https://www.snap4city.org/7
If you need specific VM/Appliance for the following components please ask to snap4city@disit.org providing a description of your purpose.
Architecture |
|
Knowledge base, semantic index: servicemap (includes Smart City API, Swagger) |
|
IOT Device Management: IOT Directory (includes IOT brokers management) |
|
IOT Data Shadow: NIFI Cluster + Elastic Search (large storage) |
|
Resource Manager: auditing, market place |
|
IOT Application Manager: Mesos, Marathon, DISCES-EM, APIs |
|
Data Analytics: RStudio + Plumber |
|
WEB scraping: Portia in connection with IOT Application Manager |
|
Monitoring tools: depending on the modules adopted |
It is a distributed feature |
Chat room on Dashboard: Rocket |
https://www.snap4city.org/515 |
Web App: the Series “in a Snap” App, MicroApplications |
|
ETL Data ingestion and storage: HBASE, Phoenix, DISCES (huge storage) |
|
Engager, user engagement |
Comments
new features are needed ?
roottooladmin1 - Sun, 09/01/2019 - 10:38Please provide us your comments and suggestion to go further in providing you this kind of solutions ready to use!!! Snap4City Team.
Context broker with nifi subscription problem
aliferisi - Tue, 08/24/2021 - 13:29Hello,
I'm currently having an issue while trying to create an entity through CLI. I was expecting broker to pass the message (entity) to nifi for ingestion but it didn't.
Installation:
I have deployed IOTOBSF and IOTDSES (nifi, elasticsearch and kibana) on the same VM.
I have subscribed nifi to broker, and i created an entity. However i am getting this warning in the docker logs for orion:
orion_1 | time=2021-08-24T07:05:00.836Z | lvl=WARN | corr=9918d6ae-04a9-11ec-a28b-0242ac120003; cbnotif=1 | trans=1629725321-127-00000000009 | from=172.18.0.1 | srv=<none> | subsrv=/ | comp=Orion | op=AlarmManager.cpp[361]:notificationError | msg=Raising alarm NotificationError nifi:1030/ingestngsi: notification failure for sender-thread: Couldn't resolve host name
I don't understand why is that a problem since my hostfile contains this line: <my_host_ip> iotobsf iotdses-address nifi kibana elastcisearch
After, i replaced hostname with my_host_ip for the subscription and i got the following warning:
orion_1 | time=2021-08-24T08:50:31.304Z | lvl=WARN | corr=567a8324-04b8-11ec-8961-0242ac120003; cbnotif=1 | trans=1629725321-127-00000000052 | from=172.18.0.1 | srv=<none> | subsrv=/ | comp=Orion | op=AlarmManager.cpp[361]:notificationError | msg=Raising alarm NotificationError <my_host_ip>:1030/ingestngsi: notification failure for sender-thread: Couldn't connect to server
I also connected nifi container to the pre-existed iotobsf_default network so the containers could communicate with each other. Or was this unnecessary, since ports are exposed?
Any help will be highly appreciated.
Thanks in advance.
we are anwering directly on
roottooladmin1 - Fri, 08/27/2021 - 18:49we are answering directly on email to you on this topic,
paolo
"I have deployed IOTOBSF and
roottooladmin1 - Mon, 08/30/2021 - 11:00"I have deployed IOTOBSF and IOTDSES (nifi, elasticsearch and kibana) on the same VM."
They are single VM so that it is not clear how you have composed into the same VM?