1. Introduction & Prerequisite
Instructor's Prerequisite Warning:
The instructor explicitly states that this video is heavily dependent on the previous lecture ("Transformations and Actions"). If you do not understand Transformations and Actions, this lecture will be difficult to grasp. He recommended checking the I-button link to review those concepts first.
What is a Spark Application?
The instructor starts by writing a code snippet on his laptop screen to serve as a foundation. He
explains that whenever you write a complete block of code in Spark—consisting of multiple
Transformations and Actions—that entire collection of jobs is collectively known as a Spark
Application.
"हमारा जो कोड बना होता है वो पूरा ट्रांसफॉर्मेशन एंड एक्शन से बने होते हैं" (Our entire code is
built purely from Transformations and Actions).
2. Decoding DAG (Directed Acyclic Graph)
Before showing how Spark builds it, the instructor breaks down the full form of DAG, word by word, so students genuinely understand the term.
In Spark: The Vertices are your RDDs/Transformations, and the Edges represent the flow of data between them.
Visual: Cyclic Graph vs. DAG
Cyclic Graph (Not Spark)
Contains a loop (C goes back to A).
Directed Acyclic Graph (Spark)
Linear flow. No loops. Ends at an Action.
How Spark creates it: When you write a code snippet like
df = spark.read... followed by df.filter() and df.select(), Spark
is literally drawing this DAG structure internally in memory.
3. Lazy Evaluation & "The Why"
What is it? When you write Transformations, Spark just adds them to the DAG. It does
absolutely no computation at that moment. It sits lazily. It only starts calculating and
executing the DAG when a final Action (like write(), show(),
count()) is called.
Interview Focus: WHY do we need Lazy Evaluation?
If an interviewer asks you why Spark is lazy, you must explain it using an optimization scenario. To set the context, the instructor compared Spark to Pandas.
"पांडास में क्या होता है ईगर इवैल्यूएशन होता है..." (In Pandas, execution is 'Eager'. It runs code line-by-line instantly. Spark does not do this to save time).
The 10 Lakh (1M) vs 1 Lakh (100k) Example
CORE CONCEPTThe Scenario: You have Table A with
10 Lakh (1 Million) records and Table B with 1 Lakh
(100,000) records. You write code that tells Spark to:
- Join Table A and Table B to create Table C.
- Filter Table C for
Age > 18. (Assume only 10,000 people are > 18). - Write the final 10,000 records to disk (Action).
If Eager (Line-by-Line)
If executed instantly, the engine would blindly join all 1 Million records with 100k records first.
Result: Massive network shuffle. Huge CPU and RAM usage.
Then, it applies the filter, throwing away 99% of that huge result down to 10,000 rows.
If Lazy (Spark's Way)
Spark waits for the write() action. The
Catalyst Optimizer looks at the whole DAG and realizes:
"Why join the whole tables? I will push the filter UP!"
Spark filters Table A and Table B first, reducing them to just a few thousand records locally, and then joins them.
Lazy evaluation allows the Catalyst Optimizer to read the entire plan and logically rearrange operations (like pushing filters before joins) to drastically optimize execution time and cost.
4. Visualizing Optimization in the Spark UI (Sparky)
The instructor mentions that the Spark UI is often affectionately called "Sparky" by developers. If an interviewer asks "What is Sparky?", they are referring to the Spark Web UI.
The Spark UI Tabs
The UI contains several tabs: Job, Stage, Environment, and Executor. The instructor notes that we will study these deeply in the future, but for now, we just need to see how the UI proves our optimization theory.
Instructor's UI Screenshot Proof
Manish showed an actual screenshot of a DAG generated inside the Spark UI. He pointed out the specific blocks the UI drew to prove that Spark rearranged the code:
FileScan: This block represents Spark reading the raw file.Filter: Right after scanning, Spark immediately applied the filter (even if you wrote the filter later in your code).Project: This block represents theselect()operation, picking only the required columns.
Lecture Conclusion
"मैं चाहता था की आपको जितना लो लेवल पे जाके समझ में ए जाएगा उसके बाद आपको हर एक डाउट जो होगा ना वो
बिल्कुल बचेगा ही नहीं"
(I could have explained this in a few lines, but I wanted to go down to the low level. Now, if someone
asks you why we do lazy evaluation, you will be able to give such a detailed answer that the interviewer
will be happy, and you will have zero doubts remaining.) - Manish Kumar