Lecture 8

Transformations & Actions

Instructor: Manish Kumar | Narrow vs Wide Dependencies & Lazy Evaluation

🔥 Instructor's "Bitter Truth" (कड़वा सच)

Before starting the technical topic, the instructor gave a serious reality check based on student behavior from the previous videos.

  • The Intentional Trap: In the previous video (handling corrupted records), the instructor intentionally did not provide the links to the CSV file or the Spark Documentation in the description.
  • The Observation: People commented "Great course, I am following," but not a single person asked for the missing data or documentation. This proved to the instructor that no one is actually practicing the code on their own machines.
  • The Bitter Truth: "सबको पैसे ज्यादा चाहिए, लेकिन मेहनत कोई नहीं करना चाहता" (Everyone wants high packages and money, but no one wants to do the hard work).
  • Final Warning: Watching these videos while lying down will not get you a job. You must implement the code. If you don't practice, learning Spark is useless.

1. Potential Interview Questions

This topic forms the absolute core of Spark processing. Expect these questions:

Q1: What is a Transformation and what is an Action in Spark?
Q2: What is Lazy Evaluation?
99% Asked Q3: What is the difference between Narrow and Wide Dependency Transformations?
Q4: Why do we need Wide Dependencies / How does data flow internally?

2. Transformations & Lazy Evaluation

Transformations are instructions given to Spark to modify the data (e.g., filter, select, withColumn, drop). They always return a new DataFrame/RDD.

Lazy Evaluation (आलसी रवैया)

When you write a chain of transformations in Spark (e.g., read data → filter it → drop a column → select a column), Spark does absolutely nothing. It does not execute the code immediately. Instead, it acts lazy and simply notes down the instructions, creating a logical plan (DAG). It waits until an Action is called to actually start working.

Types of Transformations

A. Narrow Dependency Transformations

These are operations where the data required to compute the result lives on a single partition (on the same machine). Spark does not need to move data across the network to other machines to figure out the answer.

Examples: filter() select() drop() withColumn()

Performance: Very fast, because there is no network shuffle overhead.

B. Wide Dependency Transformations

These are operations where data required to compute the result resides across multiple partitions (different machines). To complete the task, Spark must move and combine data across the network. This movement of data is called a Shuffle.

Examples: groupBy() join() orderBy() repartition() distinct()
Instructor's Example: Why Shuffle Happens (groupBy)

Imagine we want to calculate the total salary by department (IT vs HR). The raw data is scattered randomly across 3 nodes.

Node 1
IT, HR Data
Node 2
IT, HR Data
Node 3
IT, HR Data
NETWORK SHUFFLE
Partition A
All IT Data
Partition B
All HR Data

To compute the final sum, Spark must pull all 'IT' data from all 3 nodes to one single partition, and all 'HR' data to another. This physical movement of data over the network makes Wide Dependencies very expensive operations.

3. Actions & The OOM Danger

Actions are the triggers. Once an Action is executed, Spark stops being lazy, looks at the logical plan it created from the Transformations, optimizes it, and finally executes the job on the cluster.

show() count() write() collect() take()

⚠️ The Extreme Danger of df.collect()

df.show() - Safe

By default, this only brings the top 20 records back to the Application Driver to print on your screen. It is perfectly safe.

df.collect() - Dangerous

This action commands all Executors to gather ALL of their processed data and send it back to the Application Driver as an array/list.

The 100GB to 4GB Crash Scenario

If you have 20 Executors that process 100 GB of data in total, and you call collect(), they will all try to force their massive data payloads into the Driver Node at once.

💻💻💻 100 GB Data
(From Workers)
↓
4 GB Driver RAM 💥 OUT OF MEMORY

This immediately throws the famous "Driver Out of Memory Exception".

Why Learn This Deep Theory?

"We learn these internals so we can write optimized code. If we know an operation is a Wide Dependency, we can try to optimize or remove it if possible. When you give these internal explanations in an interview—explaining exactly how data shuffles and how OOM happens—the interviewer instantly knows your level of expertise." - Manish Kumar