2 min read Data Science

COVID-19 Dashboard

What parts of the world are susceptible to Corona outbreak? We used Big Data and Data Engineering in this project to find out.
COVID-19 Dashboard
A dashboard showing potential Corona hotspots based on population data.

We are in the midst of a global pandemic. At the time this project started, the Corona virus was still just a headline for most - but in the meantime it reached and impacted all of our lives. Fighting such a pandemic happens in many ways on multiple scales. We are interested in how this can be done on the societal level: using data. In this project, me and my teammates built a pipeline capable of processing a large dataset and created a visualization of the areas most vulnerable to Corona which includes reported cases in real-time.

Architecture

To quickly summarize the application: a backend downloads- and processes Corona data and population data. A clustering algorithm is applied to determine the most 'vulnerable' areas to Corona outbreak. Then, all resulting insights are stored in a MongoDB database and exposes through an API. A frontend then integrates with Mapbox to show the data visually, on a map. Because we were working with large amounts of data, some sophisticated technologies were required to properly process the data:

  • Apache Kafka
  • Apache Spark
  • Apache Zeppelin to author PySpark scripts

Brought together, this can be put in a diagram as follows:

Batch architecture
The Corona dashboard batch processing architecture.

All components are hosted on Google Cloud Platform (GCP). To also demonstrate merging both batch- and stream data in a single dashboard, also another architecture was built. This time, we took in tweets that are concerned about Corona through the keyword 'Corona' and used a Map-Reduce technique to compute the total amount of Corona-tweets sent by each country in the world. This was then again, stored in a MongoDB database and exposed as an API. See the stream processing architecture below:

Stream architecture
The Corona dashboard stream processing architecture.

To conclude both architectures and how they come together in a single application, see the following diagram:

Full architecture
Both the batch- and stream processing pipelines visualized in a single diagram.

To see the front-end in action, see the live dashboard.

Full report

For further reading, check out the GitHub repository:

disease-spread

Published by:

Jeroen Overschie

You might also like...

Mar
26
RAG on GCP: production-ready GenAI on Google Cloud Platform (on Xebia.com ⧉)

RAG on GCP: production-ready GenAI on Google Cloud Platform (on Xebia.com ⧉)

GCP has an increasing set of managed services that can help you build production-ready Retrieval Augmented Generation (RAG) applications. How can you best leverage them to build RAG applications? Let’s explore together. Read along!
6 min read
Dec
28
Dataset enrichment using LLM’s ✨ (on Xebia.com ⧉)

Dataset enrichment using LLM’s ✨ (on Xebia.com ⧉)

Large Language Models (LLM's) have proven to be useful for numerous tasks. LLM's can summarise text, generate text, classify text or translate text. What LLM's can also be used for is converting unstructured data to structured data.
Dec
08
Scaling up: bringing your Azure DevOps CI/CD setup to the next level 🚀 (on Xebia.com ⧉)

Scaling up: bringing your Azure DevOps CI/CD setup to the next level 🚀 (on Xebia.com ⧉)

Azure DevOps pipelines are a great way to automate your CI/CD process. But what if you have many projects? In this blog post, we will show you how you can scale up your Azure DevOps CI/CD setup for reusability and easy maintenance.
9 min read
Nov
21
How to create a Devcontainer for your Python project 🐳 (on Xebia.com ⧉)

How to create a Devcontainer for your Python project 🐳 (on Xebia.com ⧉)

Instead of giving other developers a setup document, let’s make sure we also create formal instructions so we can automatically set up the development environment. Devcontainers let us do exactly this.
10 min read
Apr
18
From Linear Regression to Neural Networks

From Linear Regression to Neural Networks

How are linear regression, logistic regression and neural networks related? What is overfitting and how do we fight it? In this post, we find answers to these questions in an interactive way by working with a real-world dataset on penguins.
20 min read