Lecture 4

The Spark Ecosystem & Architecture

Instructor: Manish Kumar | Connecting Code to Hardware Level

1. Introduction & The Goal

In this lecture, the primary goal is to understand what the Spark Ecosystem looks like and, more importantly, to visualize exactly how it connects from the high-level code we write all the way down to the physical hardware level where data is actually processed.

Instructor's Promise: "Stay until the end of the video, and the entire responsibility of making you understand how this holistic picture connects is mine."

2. The Spark Architecture Stack

Spark is composed of several distinct layers. We can divide the ecosystem into High-Level APIs (Libraries) at the top, and the Low-Level API (Spark Core) at the bottom. Let's visualize how they sit on top of one another.

Visual: The Complete Spark Ecosystem Stack

HIGH-LEVEL APIs (LIBRARIES)
Spark SQL / DataFrames
Spark Streaming
MLlib (Machine Learning)
GraphX
LOW-LEVEL API
Spark Core (RDDs)

The foundation. All high-level code compiles down to here.

EXECUTION
Spark Compute Engine
RESOURCE MANAGEMENT
Cluster Manager

YARN, Mesos, or Standalone

HARDWARE
💻
💻
💻

Physical / Virtual Nodes

A. High-Level APIs to Low-Level API Translation

In practical scenarios, we write code using DataFrames or Spark SQL (the High-Level APIs). These libraries sit and run on top of the Spark Core.

Internally, whatever logic or transformation you write using DataFrames is automatically compiled and transformed down into the Spark Core. In the Spark Core, your code is converted into RDDs (Resilient Distributed Datasets).

B. Spark Engine to Cluster Manager Interaction

After the code becomes an RDD, it enters the Spark Compute Engine.

Crucial Distinction: The Spark Engine's only job is to process. It does not manage the hardware, and it does not know the details of the cluster.

The 100GB Request Example:

The Spark Engine realizes it needs resources to execute the RDDs. It goes to the Cluster Manager and says, "I need 100 GB of RAM."

  • The Cluster Manager has all the details of the actual hardware.
  • It checks if 100 GB is currently available across the nodes.
  • If Yes: It responds and allocates the resources so the process can run.
  • If No: It puts the application request into a FIFO Queue (First In, First Out). The job that arrived first will get the memory as soon as it becomes available, and the others will wait their turn.

C. Popular Cluster Managers

The Cluster Manager is an independent component that Spark connects to. The instructor listed the most popular ones used in the industry:

YARN

Yet Another Resource Negotiator (Most common, from Hadoop ecosystem)

Mesos

Another popular resource manager framework.

Standalone

Spark's very own, built-in cluster manager.

Summary of the Holistic Picture

You write high-level DataFrame code → Spark transforms it internally to RDDs in the Spark Core → The Spark Engine takes the execution plan and asks the Cluster Manager for resources → The Cluster Manager checks hardware availability and allocates the RAM/CPU → The code runs on physical hardware nodes.