CLICK on the above thumbnail to get a booklet on data analytics of Snap4City, Snap4Industry cases
For the development of data analytics, the data scientist and developers can use Python and/or RStudio from the online platform and on their premises. Python and RStudio platforms may exploit any kind of libraries such as Keras, Pandas, and hardware accelerator as NVIDIA to use Tensor Flow, etc. The developers can access the KB and Big Data store respecting the privacy and the data licensing by using authenticated Smart City APIs. The access has to permit the reading of historical and real-time data, and saving the resulting data provided by the algorithms, for example, heatmap-related predictions, the assessment of data quality, and labels of detected anomalies. Data scientist’ work may be finished once they develop the algorithm they should be aware of. On the other hand, the same algorithm (e.g., for computing heatmaps, parking prediction), should allow being:
• Used on different services of the same kind located in different places and based on several parameters (e.g., target precision and list of data sources). This means that data analytics itself has to be designed with the needed flexibility and generality;
• put in execution from IoT App by passing a set of parameters and collecting the results on the Data Storage or as a result of the invocation. The executions can be periodic or event-driven — e.g., the arrival of a request or the arrival of the new set of data values;
• controlled for collecting eventual errors and mistakes, in debug and at run time for logging. This may be for informing the developer and/or the administrator of eventual mistakes and problems by sending notifications; and
• dynamically allocated on the cloud in one or multiple instances to plan a massive computation of the same data analytic process on several data sets and services at the same time.
In Python and/or RStudio cases, the script code has to include a library for creating a REST Call, namely: Plumber for RStudio and Flask for Python. In this manner, each process presents a specific API, which is accessible from an IoT Application as a MicroService, that is, a node of the above-mentioned Node-RED visual programming tool for data flow. Data scientists can develop and debug/test the data analytic processes on the Snap4City cloud environment since it is the best way to access at the Smart City API with the needed permissions. The source code can be shared among developers with the tool “Resource Manager”, which also allows the developers to perform queries and retrieve source code made available by other developers.
Read more on: https://www.snap4city.org/download/video/course/da/
Snap4City Analytics, the data analytics at your disposal on Snap4City
The data analytics in Snap4City are focussed on providing support for Decision Makers to improve quality of life, match the Sustainable Development Goals, specific KPIs; and assess the conditions for 15 Min City Indexes.
The following examples and those reported in the training course can give you an idea of the capability of the platform. We recommend that you browse the training course: https://www.snap4city.org/download/video/course/da/
A selection of our Data Analytics are:
- Mobility and Transport
- What if analysis: routing, traffic flow, demand vs offer, pollutant, etc. (Simulation + ML)
- Traffic flow reconstruction from sensors and other sources (simulation + ML)
- Predictions for: traffic flow, smart parking, smart bike sharing, people flows, etc. (ML, DL)
- Public Transportation: Ingestion and modelling of GTFS and Transmodel
- Analysis of transport/mobility supply and demand mobility based on public transportation and multiple data sources (Simulation)
- Assessing quality of public transportation (analysis)
- Accidents heatmaps, anomaly detection (analysis, ML)
- Tracking fleets, and people, via devices: OBU, OBD2, mobile apps, etc.
- Routing and multimodal routing (multi-stop travel planning), constrained routing, dynamic routing
- Computing Origin-Destination Matrices from different kinds of data (analysis)
- Computing typical trajectories based on tracks (analysis, ML)
- Computing Messages for Connected drive
- Slow and Fast Mobility 15 Minute City Indexes (analysis, ML)
- Computing and comparing traffic flow on devices and at the city border (analysis)
- Typical time trends for traffic flow and IoT Time series. (analysis, ML)
- Impact of COVID-19 on mobility and transport
- City Users and Social
- People detection and classification: persona, carts, bikes, etc. (ML, DL)
- People counting and tracking (via thermal cameras, ML, DL)
- People counting via head counting (via thermal cameras, ML, DL)
- People flows prediction and reconstruction, (ML, DL)
- Wi-Fi data, mobile apps data, Mobile Data, etc.
- Computing User engagement and suggestions for sustainable mobility (Rule Based, ML)
- User’s behaviour analysis,
- Origin-Destination matrices, hot places, schedule, Recency and frequency, permanence, typical trajectory, etc.
- People flow analysis from PAX Counters and heterogenous data sources
- Social media analysis on a specific channel, specific keywords: see Twitter Vigilance,
- Reputation, service assessment: MultiLingual NLP and Sentiment Analysis, SA
- Tweet proneness, retweet-ability of tweets, impact guessing
- Audience predictions on TV channels and physical events, locations
- SDG, 15 Minute City Index , etc. (modelling and computability)
- Environment and Weather
- Predictions of pollution as: NOX based on traffic flow, PM10, etc., for the next 48 hours, or longer term.
- Long-term predictions of European Commission KPIs on
- NO2 average value over the year, PM10, …….
- Prediction of landslides, 24 hours in advance
- Computation of CO2 based on traffic flows
- each road for each time slot of the day
- Prediction of MicroClimate conditions for the diffusion of
- NO2, PM10, PM2.5, etc.
- Heatmaps production, dense data interpolation for
- Weather conditions: temperature, humidity, wind, DEW
- Pollutants and Aerosol: NO, NO2, CO2, PM10, PM2.5, etc.
- Impact of COVID-19 on Environmental aspects
- Management and Strategies
- What-if analysis, dynamic routing, origin-destination matrices production from a large range of sources
- Early warning computation
- Estimation of KPI and local indexes for: quality of life (15MinCityIndex)
- Production Optimization
- Planning and Monitoring renovation works via objective KPIs
- Managing Maintenance and teams
- Predictive Maintenance and costs predictions: chemical plant, vehicles, boats
- Resilience and Risks Analysis
- Resilience analysis according to European Guidelines on Resilience of critical infrastructure, and transport systems
- Risk analysis: natural and nonnatural disaster
- Time Series
- Time Series Anomaly detection
- Data quality assessment and control
- short and long-term prediction
- Interpolation of Data on the regular grid for calibrated heatmaps
- Semantic Reasoning
- Ontology Modelling and integration, expert system construction
- Knowledge modelling and reasoning on RDF stores: spatial, temporal, relational
- Virtual Assistant construction
- Matrices, Images, Maps, and 3D Digital Models
- Conversion of Satellite data images into regular ground images
- Extraction information from Orthomaps, LIDAR, etc., regarding city structures
- 3D Digital Twin of Cities and Objects: pattern extraction, 3D model reconstruction
- Etc.
They are developed by using a large range of statistics, operatig research, ML, AI, XAI techniques such as:
- RF, XGBoost, BRNN, RNN, SVR, MLP,
- DNN, LSTM, CNN-LSTM, Autoencoders, …, YOLO, etc.
- Clustering: K-means, K-Medoid, etc.,
- Simulated Annealing, Genetic Search, Taboo Search, etc.
- XAI: Shap, variations
------------------------------
- First step to reduce the fossil emission is to monitor the present emission, identify the most critical points and factors. The emissions from fossil combustion are mainly CO2 and NO1, NO2. And they are due to vehicles, house heating and industries… but where they are now precisely in the city and lands
- Getting pollution status from satellite data
- "Exploiting Satellite Data in the Context of Smart City Applications", The 5th IEEE International Conference on Smart City Innovations - Track 1: Theory, Modeling and Methodologies, 2021. http://ieeesmartworld.org/sci/
- https://www.snap4city.org/drupal/sites/default/files/files/Copernicus-final.pdf
- Measuring CO2
- "Estimating CO2 Emissions from IoT Traffic Flow Sensors and Reconstruction", Sensors, MDPI, 2022.
- https://www.mdpi.com/1424-8220/22/9/3382/
- Long terms prediction of NO2
- Long Term Prediction of NO2 KPI of European Commission reference values , PDF
- "Long Term Predictions of NO2 Average Values via Deep Learning", Proc. of the 2021 International Conference on Computational Science and Its Applications. Published on LNCS Springer. https://iccsa.org/
- https://www.snap4city.org/download/video/long_term_predictions_of_no2.pdf
- 24-48 hours prediction of NOX
- https://www.snap4city.org/drupal/node/530
- “Real-Time Automatic Air Pollution Services from IOT Data Network”, proc. Of IEEE Symposium on Computers and Communications (ISCC), MOCS track, 10th Workshop on Management of Cloud and Smart City System, 2020 July 7th, Rennes, France. https://conferences.imt-atlantique.fr/iscc2020/
- Https://www.snap4city.org/download/video/NOXprediction 2020.pdf
- Getting pollution status from satellite data
- As first step the present traffic and housing conditions have to be assed, then the progresses can be kept under control. The assessment can be performed by computing indicators. For example the 15 Min City index for serviceability at 15 minute distance by walking in the city and with different parameters in peripheral and rural areas, which includes an index for housing and slow and fast mobility. Relevant aspects are: (i) to assess the capability of the public transport to satisfy the demand, (ii) predicting traffic flow and reconstruction, (iii) making the what if analysis to be capable to assess how the traffic conditions would change due to changes in the road network, for example for restructuring, etc.
- 15 Min City Index:
- 15MinCityIndex: understanding city areas by means of 13 different aspects, PDF
- "Computing 15MinCityIndexes on the basis of Open Data and Services", Proc. of the 2021 International Conference on Computational Science and Its Applications. Published on LNCS Springer. https://iccsa.org/
- https://www.snap4city.org/drupal/sites/default/files/files/computing15minCityIndex_ICCSA_v0-3.pdf
- Planning public transport according to the analysis of demand wrt offer match.
- Data Analytic: Analyzing Public Transportation Offer wrt Mobility Demand, DORAM Tool
- Analyzing demand with respect to offer of mobility, Applied Science, MDPI, 2022.
- https://www.mdpi.com/2076-3417/12/18/8982
- Traffic Flow Analysis, prediction and reconstruction:
- “Traffic Flow Reconstruction by Solving Indeterminacy on Traffic Distribution at Junctions”, Future Generation Computer Systems, Elsevier, 2020, 2021. https://authors.elsevier.com/sd/article/S0167-739X(20)30835-9
- Https://www.snap4city.org/download/video/TrafficFlowReconstruction2020.pdf
- “Real-Time System for Short- and Log-Term Prediction of Vehicle Flow”, IEEE Smart Data Service as Proc of IEEE Services, 2020.
- Https://www.snap4city.org/download/video/TrafficPredicitons2020.pdf
- Dashboard info: Mobility and Environment What-IF Analysis, Florence
- What-IF analysis and Dynamic Routing Support
- 15 Min City Index:
--- former version---------
This page reports what Snap4City has done for Helsinki and Antwerp in terms of Data Analytic.
List of all scenarious: https://www.snap4city.org/4
Please note that Snap4City has also other capabilities and tools on this matter so that see also:
- traffic flow reconstruction from sensors and other sources: https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MTc5NQ==
- parking predictions: https://www.snap4city.org/drupal/node/200 https://ieeexplore.ieee.org/abstract/document/8430514/
- wi-fi people flow prediction and reconstruction https://www.sciencedirect.com/science/article/pii/S1045926X17300083
- what-if analysis, dynamic routing https://www.snap4city.org/drupal/node/521
- origin destination matrices production from a large range of sources: https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MTc3NA==
- analysis of the demand vs offer of mobility according to public transportation and multiple data sources https://www.snap4city.org/drupal/node/554
- resilience and risk analysis: https://www.snap4city.org/drupal/node/520
- early warning computation: https://www.snap4city.org/drupal/node/520
- accidents heatmaps
- traffic flow predictions
- NOX pollution prediction on the basis of traffic flow, 48 hours see: https://www.snap4city.org/drupal/node/500
- pollution prediction at 48 hours, every hour
- user engagement for sustainable mobility: https://www.snap4city.org/drupal/node/486
- user behaviour analysis: https://www.snap4city.org/drupal/node/486
- data reconstruction calibration
- tracking fleets, people, devices: https://www.snap4city.org/drupal/node/456
- OBD2 support: https://www.snap4city.org/drupal/node/514
- People flow analysis from PAX Counters: https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MjE4MQ==
- social media analysis on specific channel, specific keywords: see Twitter Vigilance, for NLP and Sentiment Analysis, SA
- data quality assessment, prediction, anomaly detection
- maintenance prediction and costs predictions
- ReTweet proneness, retweettability of tweets: https://link.springer.com/article/10.1007/s11042-018-5865-0
- Audience prediction to TV channels and phisical events: https://link.springer.com/article/10.1007/s11042-017-4880-x
- etc.
All the algorithms have worked continuously providing services to the users and also information to US: What they requested, What they visited, POI; where they passes: trajectories and OD matrix, Language, Routing, Clicking on services and icons, Engaging: responding, Searching.
Description of the main categories of Data Analysis provided.
- Discovery is provided by the Knowledge Base and it is attached to POI. This implies to create a geoindex.
- Discovery of public transport and multimodal routing are based on OSM, KB, and GTFS data
- The browsing of data on Public Transportation implies the analysis and processing of data collected as GTFS, in most cases, they are incomplete or they lack of cross linking that are reconstructed at the moment of ingestion that may happen once per day or once per season according to the updates released by the transportation companies. So that the Snap4City process has to continuously check the presence of updates on these data.
- Full test search implies a continuous full text indexing of all data entering into the platform
- Routing is computed on demand available from Mobile, ServiceMap and MicroApplications:
- https://www.km4city.org/webapp-super/?operation=privatetransport/pathfinder&coordinates=51.222744;4.405380&lang=eng#b
- https://www.km4city.org/webapp-super/?operation=privatetransport/pathfinder&coordinates=60.170437;24.938215&lang=eng#b
- Recently routing is also available on What-IF analysis are dynamic real time routing on dashboard according to eventual barriers defined into the scenarious.
- EAQI: European Air Quality Index, as well as many other Air Quality Indexes are computed on the basis of data obtained in real time for the different cities Their computation algorithms are described in https://www.snap4city.org/drupal/node/413
- Heatmaps data are computed in real time and periodically on the basis of sensors data via interpolation taking into account sensors data and positions. Those of Enfuser (PM10, PM2.5, AQI) are collected in advance as predictions.
- GRAL PM10 predictions are computed by using a complex differential algorithm model taking into account: city graphs, routing possible, 3D structure of the city building, traffic flow density reconstructed or obtained statistically, population of vehicles, forecast for wind, etc.
- Origin Destination and Trajectories are computed periodically, being an analysis. They depend on the data coming from Snap4City Mobile App and so now are feed by demo data.
- Weather data, and forecast are from sensors data and Open Weather service.
- Engagement monitoring of the Antwerp in a Snap mobile App as depicted in dashboard:
- Engagement monitoring of the Helsinki in a Snap mobile App as depicted in dashboard:
- Hot areas and places Click Density are computed on the basis of the data collected from mobile applications and/or PAX counters. See for example Click Density on:
- Trajectories from Mobile PAX Counters that need to be revised and regularized:
- Suggestions are provided taking into account the GPS position and the knowledge of the City
- Self assessment: this feature is based on an algorithm that assess all the activities performed by the users (according to the authorization received) to assess how much of the Snap4City has been covered in terms of feature by the activity. The values are reported to the users in terms of Level reached, and trends for the detailed data we collect and compute for. https://www.snap4city.org/userstats/ssoLogin.php?redirect=https://www.snap4city.org/userstats/user.php?title=Daily+User+Stats+%26nbsp%3B&time=day
- Twitter data are provided from Twitter Vigilance service of DISIT lab: