Real-Time Data Processing with Confluent Cloud

This is a complete end-to-end demo showcasing the power of the Confluent Cloud Data Streaming Platform (DSP). It focuses on a third-party reseller that offers products from major vendors like Amazon and Walmart.

The demo illustrates how Confluent Cloud can be used to filter, enrich, and transform online orders in real time. Additionally, it demonstrates seamless integration with existing systems, specifically Amazon OpenSearch.

General Requirements

• Confluent Cloud API Keys - Cloud resource management API Keys with Organisation Admin permissions
• Terraform (v1.9.5+) - The demo resources is automatically created using Terraform.
• Git CLI - Git CLI to clone the repo

Installing CLI tools on MAC

Install git and terraform by running:

brew install git terraform

Installing CLI tools on Windows

Install git and terraform by running:

winget install --id Git.Git -e
winget install --id Hashicorp.Terraform -e

Setup

Mac Setup

First, clone the repo and change directory to demo-infrastructure

git clone https://github.com/AhmedsZamzam/getting-started-flink-cc.git
cd getting-started-flink-cc/demo-infrastructure

Open and edit the terraform.tfvars (variables) file, use the following command in your terminal:

open -a TextEdit terraform.tfvars

Replace the placeholders with your own API keys and name. For example, if your confluent_cloud_api_key is ABCDEFG, your confluent_cloud_api_secret is 123456789ABDCEFGHLJKLMNOP=7-23, and your name is Kevin, your file should look like this:

confluent_cloud_api_key = "ABCDEFG"
confluent_cloud_api_secret = "123456789ABDCEFGHLJKLMNOP=7-23"
prefix = "kevin"

In demo-infrastructure directory run the following commands in terminal to set up the whole demo environment

terraform init
terraform apply --auto-approve

Windows Setup

First, clone the repo and change directory to demo-infrastructure

git clone https://github.com/AhmedsZamzam/getting-started-flink-cc.git
cd getting-started-flink-cc\demo-infrastructure

Open and edit the terraform.tfvars (variables) file, use the following command in CMD:

start terraform.tfvars

Choose Notepad.

Replace the placeholders with your own API keys and name. For example, if your confluent_cloud_api_key is ABCDEFG, your confluent_cloud_api_secret is 123456789ABDCEFGHLJKLMNOP=7-23, and your name is Kevin, your file should look like this:

confluent_cloud_api_key = "ABCDEFG"
confluent_cloud_api_secret = "123456789ABDCEFGHLJKLMNOP=7-23"
prefix = "kevin"

In demo-infrastructure folder run the following commands in CMD to set up the whole demo environment

terraform init
terraform apply --auto-approve

Labs

NOTE: GO TO LAB1 ON THE ACTUAL WORKSHOP DAY

Next Lab: Lab 1: Getting Started with Flink

Tear down

NOTE: TEARDOWN AFTER YOU COMPLETED THE WORKSHOP

In demo-infrastructure run the following commands to destroy the whole demo environment

terraform destroy --auto-approve