Lecture 2

Why Apache Spark? (The Big Data Problem)

Instructor: Manish Kumar | Understanding the origins of Spark

1. The Pre-Spark Era: Traditional Databases

To understand why Apache Spark came into existence, we first need to look at what existed before it and how data was being handled.

The Past: Structured Data

Before the boom of the internet and mobile applications, data was relatively simple. We relied on traditional database servers such as:

Oracle Teradata Exadata MySQL

These databases were built strictly to handle Structured Data. Structured data is data that fits perfectly into a tabular format—meaning it has fixed rows and columns. The instructor likened this to a standard Excel Sheet (e.g., Column 1, Column 2, Column 3 with fixed data types).

2. The Shift & The "Big Data" Problem

With the rise of the internet, smartphones, and mobile applications, data generation exploded. Traditional databases started failing because they couldn't handle the new reality of data. This problem is defined by the 3 Vs of Big Data:

🌊

Volume

Data was no longer just a few Gigabytes. The sheer amount of data grew massively into Terabytes and Petabytes.

⚡

Velocity

The speed at which data was being generated became incredibly fast (e.g., millions of records pouring in per second/hour).

🗂️

Variety

Data was no longer just structured tables. It became Semi-Structured (JSON, XML) and Unstructured (Images, Video, Audio, Logs).

Traditional RDBMS simply could not store, process, or handle this new Volume, Velocity, and Variety of data.

3. Approaches to Scaling (Solving the Problem)

To solve the Big Data problem, the industry had to figure out how to scale hardware to store and process all this information. There were two approaches:

Visual: Vertical vs. Horizontal Scaling

Monolithic Machine
+ 64GB RAM
+ 16GB RAM
4GB RAM Base
Vertical Scaling

(Adding power to ONE machine)

Network Cluster
Node 1
Node 2
Node 3
Node 4
Horizontal Scaling

(Adding MORE commodity machines)

A. Monolithic Approach (Vertical Scaling)

Analogy: Imagine you have a laptop with 4GB RAM. To handle more work, you upgrade it to 8GB, then 16GB, then 64GB.

  • Physical Limits: Eventually, the motherboard cannot physically support more RAM or heat dissipation. You hit a hard limit.
  • Cost: Upgrading to super-tier hardware is exponentially expensive.
  • Single Point of Failure (SPOF): Major Flaw. If this one massive machine crashes (e.g., motherboard burns or power fails), your entire system is completely down.

B. Distributed Approach (Horizontal Scaling)

Instead of making one machine super powerful, you take multiple standard, cheaper machines (commodity hardware) and connect them via a network to work together as a single unit.

  • No Limits: If you need more power, just plug another machine into the network.
  • High Availability (No SPOF): It uses a Master-Worker architecture. If one machine (Worker) fails for any reason, the Master node detects it and simply assigns that specific workload to a different machine. The system never goes down completely.

Apache Spark and Hadoop were built entirely on the Distributed (Horizontal) Approach.

4. The Solutions: Hadoop & Spark

To make the Distributed Approach work (treating 100 separate computers as one single computer), new software frameworks had to be invented.

💡 Instructor's Final Advice

The instructor noted that while Hadoop's processing engine is becoming outdated, its file storage system—HDFS (Hadoop Distributed File System)—is still incredibly relevant and widely used in the industry today. He strongly recommends that students take the time to read up on how HDFS works, as it is a crucial foundation for any Data Engineer.