Lecture 5

Reading Data in Apache Spark

Instructor: Manish Kumar | DataFrameReader API, Options & Read Modes

1. Introduction & Prerequisites

Instructor's Note: The instructor states, "Today is our 3rd lecture of Spark." (Though chronologically labeled as Lecture 5 in the series). The absolute focus of this session is to learn the exact mechanics of reading data into Spark.

Before any data can be read, the instructor explains that two fundamental components must be running in the environment (demonstrated inside a Databricks Notebook):

2. The Core Structure: DataFrameReader API

The instructor breaks down the theoretical structure of the code before writing it. Spark reads data using the DataFrameReader API. Every read operation follows a specific, chained syntax.

Visual: Anatomy of the Read Command

spark.read
← Entry point (Calls DataFrameReader)
.format("csv")
← File type (CSV, JSON, Parquet, etc.)
.option("key", "val")
← Configurations (Headers, Modes, etc.)
.schema(...)
← (Optional) Custom table schema
.load("path")
← The Action: Fetches from DBFS/HDFS path

Python Syntax Tip: The instructor highly recommends using the backslash (\) character in Python. This allows you to break the chained command into multiple, highly readable lines without getting syntax errors from the Python compiler.

CRITICAL Handling Bad Data: Spark Read Modes

When using .option(), one of the most vital concepts discussed by the instructor is handling Corrupted Records. Real-world files are often messy (missing columns, wrong delimiters). Spark provides three specific Read Modes to dictate how the engine reacts when it hits a bad row of data.

1. PERMISSIVE (The Default)

If you do not specify a mode, Spark automatically uses `PERMISSIVE`.

  • Spark will try to read all records.
  • If it encounters a corrupted row, it will insert null into all the respective columns for that row.
  • It creates a special, automatic internal column called _corrupt_record and dumps the entire broken string into that column so you can investigate it later.

2. DROPMALFORMED

Activated via .option("mode", "DROPMALFORMED").

  • Spark will completely ignore the bad data.
  • If a row is corrupted, it is completely dropped from the final DataFrame.
  • Warning: You lose data silently. You won't know how many rows were dropped unless you do a count comparison.

3. FAILFAST

Activated via .option("mode", "FAILFAST").

  • Zero tolerance for bad data.
  • The absolute moment Spark encounters even one single corrupted row, it throws a massive exception/error.
  • The entire Spark job halts and fails immediately.

3. Practical Implementation & Issue Resolution

Step 1: Uploading Data in Databricks

The instructor navigated to the Data Tab → Create Table and uploaded a file named flights_data.csv. Databricks stored it in the DBFS (Databricks File System) at the path: dbfs:/FileStore/tables/flights_data.csv.

Step 2: Viewing the Data

After writing the basic read command, the data is stored in a DataFrame variable df. The instructor highlighted two ways to view it:

Issue 1: The Missing Headers (_c0, _c1)

When running df.show() for the first time, a major issue appeared. Spark did not recognize the headers. Instead, it assigned arbitrary column names: _c0, _c1, and _c2.

Worse, the actual column names (Destination_Country, Origin_Country, count) were pushed down and treated as the very first row of actual data.

The Fix:

We must explicitly inform Spark that the first row is a header using the option method.

.option("header", True)

Issue 2: Incorrect Data Types (Everything is a String)

To verify the column data types, the instructor used df.printSchema().
Instructor Pro-Tip: "जैसे ही मैं तप (Tab) प्रेस करूंगा तो यह ऑटो फाइल कर देगा" (As soon as you type 'print' and press Tab, Databricks will auto-fill 'printSchema' so you don't have to type it manually).

The schema revealed a problem: Destination_Country was a String, Origin_Country was a String, and count was also a String! Since "count" contains numbers (15, 30), it should be an Integer. Because we gave Spark no instructions, its default behavior is to treat every column in a CSV as a String.

The Fix:

We instruct Spark to do a sample read of the data to guess the correct data types automatically by enabling inferSchema.

.option("inferSchema", True)

The Final Perfected Code

# Reading the CSV file with all necessary options
df = spark.read \
    .format("csv") \
    .option("header", True) \
    .option("inferSchema", True) \
    .option("mode", "PERMISSIVE") # Optional: Explicitly defining the read mode \
    .load("dbfs:/FileStore/tables/flights_data.csv")

# Verifying the schema and data
df.printSchema() # The 'count' column will now correctly display as Integer
display(df)

Instructor's Closing Remark: "We have learned how to read data. I am not teaching transformations today. The video has become long because I explain everything in deep detail. We will learn transformations and other things in the next video."