#104 Loading data to Azure Event Hubs using Databricks

Azure Event Hubs

Before heading to technical specifics, let's revise all Azure-Event-Hubs-related terms and concepts:

1. Azure Event Hubs: It is a fully managed real-time data ingestion service. Azure Event Hubs is a popular choice for event-driven and big data applications due to its scalability, reliability, and integration with various Azure services.

2. Azure Event Hubs components:

Event Hubs components (https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about)

• Event Hub (or Event Streams): Actual stream / event pipeline where the events are ingested (equivalent to Kafka topic).
• Event Producers: They are the sources that send data (events) to Event Hubs.
• Event Hubs Namespace: A container that holds 1+ Event Hubs.
• Event Consumers: Applications or services that read and process events from Event Hubs.
• Checkpointing: Ensures consumers can resume from the correct position if interrupted.
• Capture: Can automatically save raw event data into Blob Storage or Data Lake.
• Partition: Event Hubs organizes sequences of events sent to an event hub into one or more partitions.