Lecture 10

DAG & Lazy Evaluation (Deep Dive)

Instructor: Manish Kumar | Understanding the "Why" Behind Spark's Speed

1. Introduction & Prerequisite

Instructor's Prerequisite Warning:

The instructor explicitly states that this video is heavily dependent on the previous lecture ("Transformations and Actions"). If you do not understand Transformations and Actions, this lecture will be difficult to grasp. He recommended checking the I-button link to review those concepts first.

What is a Spark Application?

The instructor starts by writing a code snippet on his laptop screen to serve as a foundation. He explains that whenever you write a complete block of code in Spark—consisting of multiple Transformations and Actions—that entire collection of jobs is collectively known as a Spark Application.

"हमारा जो कोड बना होता है वो पूरा ट्रांसफॉर्मेशन एंड एक्शन से बने होते हैं" (Our entire code is built purely from Transformations and Actions).

2. Decoding DAG (Directed Acyclic Graph)

Before showing how Spark builds it, the instructor breaks down the full form of DAG, word by word, so students genuinely understand the term.

Directed
Directional Flow: The flow of data is always moving forward. It moves from Node A → Node B → Node C. It never goes backward.
Acyclic
No Cycles / No Loops: If data goes from A → B → C, node C will never loop back to point A. The process is linear and ends at a certain logical endpoint.
Graph
A visual representation consisting of Vertices (Nodes) and Edges (Lines connecting the nodes).
In Spark: The Vertices are your RDDs/Transformations, and the Edges represent the flow of data between them.

Visual: Cyclic Graph vs. DAG

Cyclic Graph (Not Spark)
A
B
C

Contains a loop (C goes back to A).

Directed Acyclic Graph (Spark)
Read
Filter
Write

Linear flow. No loops. Ends at an Action.

How Spark creates it: When you write a code snippet like df = spark.read... followed by df.filter() and df.select(), Spark is literally drawing this DAG structure internally in memory.

3. Lazy Evaluation & "The Why"

What is it? When you write Transformations, Spark just adds them to the DAG. It does absolutely no computation at that moment. It sits lazily. It only starts calculating and executing the DAG when a final Action (like write(), show(), count()) is called.

Interview Focus: WHY do we need Lazy Evaluation?

If an interviewer asks you why Spark is lazy, you must explain it using an optimization scenario. To set the context, the instructor compared Spark to Pandas.

"पांडास में क्या होता है ईगर इवैल्यूएशन होता है..." (In Pandas, execution is 'Eager'. It runs code line-by-line instantly. Spark does not do this to save time).

The 10 Lakh (1M) vs 1 Lakh (100k) Example

CORE CONCEPT

The Scenario: You have Table A with 10 Lakh (1 Million) records and Table B with 1 Lakh (100,000) records. You write code that tells Spark to:

  1. Join Table A and Table B to create Table C.
  2. Filter Table C for Age > 18. (Assume only 10,000 people are > 18).
  3. Write the final 10,000 records to disk (Action).
If Eager (Line-by-Line)

If executed instantly, the engine would blindly join all 1 Million records with 100k records first.

Result: Massive network shuffle. Huge CPU and RAM usage.

Then, it applies the filter, throwing away 99% of that huge result down to 10,000 rows.

Time taken: ~10 Hours
If Lazy (Spark's Way)

Spark waits for the write() action. The Catalyst Optimizer looks at the whole DAG and realizes:

"Why join the whole tables? I will push the filter UP!"

Spark filters Table A and Table B first, reducing them to just a few thousand records locally, and then joins them.

Time taken: ~10 Seconds

Lazy evaluation allows the Catalyst Optimizer to read the entire plan and logically rearrange operations (like pushing filters before joins) to drastically optimize execution time and cost.

4. Visualizing Optimization in the Spark UI (Sparky)

The instructor mentions that the Spark UI is often affectionately called "Sparky" by developers. If an interviewer asks "What is Sparky?", they are referring to the Spark Web UI.

The Spark UI Tabs

The UI contains several tabs: Job, Stage, Environment, and Executor. The instructor notes that we will study these deeply in the future, but for now, we just need to see how the UI proves our optimization theory.

Instructor's UI Screenshot Proof

Manish showed an actual screenshot of a DAG generated inside the Spark UI. He pointed out the specific blocks the UI drew to prove that Spark rearranged the code:

  • FileScan: This block represents Spark reading the raw file.
  • Filter: Right after scanning, Spark immediately applied the filter (even if you wrote the filter later in your code).
  • Project: This block represents the select() operation, picking only the required columns.

Lecture Conclusion

"मैं चाहता था की आपको जितना लो लेवल पे जाके समझ में ए जाएगा उसके बाद आपको हर एक डाउट जो होगा ना वो बिल्कुल बचेगा ही नहीं"

(I could have explained this in a few lines, but I wanted to go down to the low level. Now, if someone asks you why we do lazy evaluation, you will be able to give such a detailed answer that the interviewer will be happy, and you will have zero doubts remaining.) - Manish Kumar