Lecture 11

Reading JSON Files in Spark

Instructor: Manish Kumar | Multiline, Extra Columns & NoSQL Sources

ЁЯУЪ

Context & Recap

In the previous classes, we learned how to parse CSV files, how to identify corrupted records, and how to store them using badRecordsPath. Today, we move on to one of the most widely used data formats in the industry: JSON. Reading JSON is as easy as reading CSV, but there are specific nuances (like multiline formatting and nested structures) that every Data Engineer must know.

ЁЯОп Potential Interview Questions

Before writing any code, the instructor outlined exactly what an interviewer will ask you regarding JSON files:

  1. What is JSON and how do you read it in Spark?
  2. Where do we typically get JSON data from in a real project?
  3. What happens if our defined Schema has 3 keys, but the incoming JSON file has an extra 4th key-value pair?
  4. What is a "Line Delimited" JSON vs a "Multiline" JSON, and how does Spark handle them?
  5. How do you handle a corrupted JSON record (e.g., missing a closing bracket })?
  6. How do you flatten a deeply nested JSON? (Teaser for the next lecture)

1. Where does JSON come from?

As a Data Engineer, you might wonder why we deal with JSON so much. Why isn't everything just a neat CSV table?

APIs & NoSQL Databases

JSON (JavaScript Object Notation) is the standard format for web APIs. Additionally, NoSQL databases like MongoDB heavily rely on JSON/BSON because they support semi-structured data.

"MongoDB basically semi-structured type ke data ko support karta hai... wahan se hume JSON format mein data milta hai."

NoSQL DB / Web API
↓
Spark Reads JSON
↓
Process & Flatten
↓
Write to Structured Table

2. The "Multiline" Challenge

This is the most common error beginners face when reading JSON in Spark. Spark expects JSON to be Line-Delimited by default.

Line-Delimited (Default Spark Behavior)
{"id": 1, "name": "John", "age": 30}
{"id": 2, "name": "Jane", "age": 25}
Works Perfectly. Spark reads one complete JSON object per row.
Multiline (Pretty-Printed)
{
  "id": 1,
  "name": "John",
  "age": 30
}
Fails / Corrupts! Spark reads line 1 ({) and thinks it's an incomplete, broken record.

The Solution:

If your JSON spans multiple lines, you must explicitly tell Spark using an option:

df = spark.read \
    .format("json") \
    .option("multiline", "true") \
    .load("/path/to/file.json")

3. Handling Schema Mismatches & Corruptions

Scenario A Extra Key-Value Pairs

Question: What if your defined Schema expects 3 columns (ID, Name, Age), but the JSON file suddenly sends a 4th column (Salary)?

Answer: Spark handles this gracefully. It will simply ignore the extra column ("Salary") and only pick up the 3 columns defined in your schema. It will not throw an error.

Scenario B Syntactical Corruption

Question: What if the JSON is missing a comma, a quote, or a closing bracket?

Answer: Spark treats it as a Corrupted Record. The handling is exactly the same as we learned in the CSV lecture. Depending on the mode (PERMISSIVE, DROPMALFORMED, FAILFAST), Spark will process it. You can store these using badRecordsPath and view them using the internal _corrupt_record column.

4. Flattening JSON & Instructor's Warning

Oftentimes, JSON data is deeply nested (arrays inside objects inside objects). Currently, if you load it, Spark creates complex StructType or ArrayType columns which are hard to analyze. The process of converting these nested structures into flat, standard rows and columns is called Flattening a JSON.

Instructor's Note: "This is where the actual hard work is. We will learn how to flatten JSONs in the upcoming dedicated lectures, as it is a major part of our daily job."

ЁЯФе Instructor's Strict Advice on Practice

"рдореИрдВ рд╕рд┐рд░реНрдл рдЖрдкрдХреЛ рдбрд╛рдЯрд╛ рд╕реЗрдЯ рджреВрдВрдЧрд╛ рдФрд░ рдПрдХ рд▓рд┐рдВрдХ рджреВрдВрдЧрд╛... рдирд╛ рд╣реА рдЖрдкрдХреЛ рдпреЗ рдиреЛрдЯрдмреБрдХ рдорд╛рдВрдЧрдиреЗ рд╣реИрдВ рдФрд░ рдирд╛ рд╣реА рдЖрдкрдХреЛ рдЬреЛ рд╣рдорд╛рд░рд╛ рдбреЗрдЯрд╛рдмреЗрд╕ рдХрд╛ рдиреЛрдЯрдмреБрдХ рд╣реИ рд╡реЛ рднреА рдирд╣реАрдВ рдорд╛рдВрдЧрдирд╛ рд╣реИред"

Translation & Meaning: The instructor is purposely not providing his Databricks notebook. He will only provide the raw dataset and documentation links. He strictly demands that students write the code from scratch on their own machines to build muscle memory, rather than just copy-pasting his notebook.