Lecture 1

Introduction & Overview of Apache Spark

Instructor: Manish Kumar | Fundamentals of Spark

IMPORTANT Instructor's Prerequisites & Warning

Before diving into the technical concepts, Manish Kumar established strict ground rules for this course:

  • No PDFs or PPTs will be provided: Do not ask for PDFs in the comment section. The instructor explicitly refuses to provide them because downloaded PDFs often just sit unopened in laptops or phones.
  • Make Your Own Notes: You must create your own hand-written notes or take screenshots. Writing things down manually is the only way to truly retain the information.
  • Active Learning Required: Simply lying down ("let ke dekhoge") and watching the video like a movie will yield zero results. This is a guarantee from the instructor.
  • Doubt Resolution: If you face doubts while actively practicing and making notes, the instructor will personally help resolve them in the comment section.

*Note: This lecture had to be re-recorded because the initial recording failed.*

What is Apache Spark?

"Apache Spark is a unified computing engine and a set of libraries for parallel data processing on a computer cluster."

1. Unified Engine

It is a single, unified platform that handles various types of workloads (which will be detailed in later lectures).

2. Computing Engine (Crucial Detail)

Spark does NOT have storage. It has absolutely no capacity to store data permanently. Its sole purpose is to compute (process) data. It relies entirely on external storage systems to fetch and save data.

3. Set of Libraries

Spark provides a set of pre-built libraries that developers can use to easily interact with the engine.

4. Computer Cluster

Spark does not run on just one machine. It runs on a "cluster" — which means multiple computers connected together, working simultaneously as a single logical unit to execute tasks.

Understanding Parallel Data Processing

To explain Parallel Data Processing, the instructor used the "Father and Sons" analogy. This perfectly illustrates how a Master-Worker cluster operates.

Visual: The Father-Son Analogy

The Father
(Has 10 heavy tasks to complete)

If the father does all 10 tasks alone, it will take a massive amount of time.

DIVIDES THE 10 TASKS
Son 1
Executes Task Chunk
Son 2
Executes Task Chunk
Son 3
Executes Task Chunk
Son 4
Executes Task Chunk

Key Takeaways from the Analogy:

  • By distributing the 10 tasks among 4 sons, the workload is completed much faster.
  • Fault Tolerance Foreshadowing: The instructor noted: "What if a son encounters a task he doesn't know how to do?" In this case, the parent (Master) will step in to help or reassign it. This hints at how Spark handles task failures, which will be covered in later architecture videos.