1. The Pre-Spark Era: Traditional Databases
To understand why Apache Spark came into existence, we first need to look at what existed before it and how data was being handled.
The Past: Structured Data
Before the boom of the internet and mobile applications, data was relatively simple. We relied on traditional database servers such as:
These databases were built strictly to handle Structured Data. Structured data is data that fits perfectly into a tabular format—meaning it has fixed rows and columns. The instructor likened this to a standard Excel Sheet (e.g., Column 1, Column 2, Column 3 with fixed data types).
2. The Shift & The "Big Data" Problem
With the rise of the internet, smartphones, and mobile applications, data generation exploded. Traditional databases started failing because they couldn't handle the new reality of data. This problem is defined by the 3 Vs of Big Data:
Volume
Data was no longer just a few Gigabytes. The sheer amount of data grew massively into Terabytes and Petabytes.
Velocity
The speed at which data was being generated became incredibly fast (e.g., millions of records pouring in per second/hour).
Variety
Data was no longer just structured tables. It became Semi-Structured (JSON, XML) and Unstructured (Images, Video, Audio, Logs).
Traditional RDBMS simply could not store, process, or handle this new Volume, Velocity, and Variety of data.
3. Approaches to Scaling (Solving the Problem)
To solve the Big Data problem, the industry had to figure out how to scale hardware to store and process all this information. There were two approaches:
Visual: Vertical vs. Horizontal Scaling
Vertical Scaling
(Adding power to ONE machine)
Horizontal Scaling
(Adding MORE commodity machines)
A. Monolithic Approach (Vertical Scaling)
Analogy: Imagine you have a laptop with 4GB RAM. To handle more work, you upgrade it to 8GB, then 16GB, then 64GB.
- Physical Limits: Eventually, the motherboard cannot physically support more RAM or heat dissipation. You hit a hard limit.
- Cost: Upgrading to super-tier hardware is exponentially expensive.
- Single Point of Failure (SPOF): Major Flaw. If this one massive machine crashes (e.g., motherboard burns or power fails), your entire system is completely down.
B. Distributed Approach (Horizontal Scaling)
Instead of making one machine super powerful, you take multiple standard, cheaper machines (commodity hardware) and connect them via a network to work together as a single unit.
- No Limits: If you need more power, just plug another machine into the network.
- High Availability (No SPOF): It uses a Master-Worker architecture. If one machine (Worker) fails for any reason, the Master node detects it and simply assigns that specific workload to a different machine. The system never goes down completely.
Apache Spark and Hadoop were built entirely on the Distributed (Horizontal) Approach.
4. The Solutions: Hadoop & Spark
To make the Distributed Approach work (treating 100 separate computers as one single computer), new software frameworks had to be invented.
-
1. Hadoop
Hadoop came first. It introduced a distributed programming model and a distributed file system. It was the first major framework to solve the Big Data problem by making a cluster of machines act as a single unit.
-
2. Apache Spark
Spark was invented later to solve the slowness and limitations of Hadoop's processing model. (The exact differences will be covered in the next lecture).
💡 Instructor's Final Advice
The instructor noted that while Hadoop's processing engine is becoming outdated, its file storage system—HDFS (Hadoop Distributed File System)—is still incredibly relevant and widely used in the industry today. He strongly recommends that students take the time to read up on how HDFS works, as it is a crucial foundation for any Data Engineer.