Context & Recap
In the previous classes, we learned how to parse CSV files, how to identify corrupted records, and how
to store them using badRecordsPath. Today, we move on to one of the most widely used data
formats in the industry: JSON. Reading JSON is as easy as reading CSV, but there are
specific nuances (like multiline formatting and nested structures) that every Data Engineer must know.
ЁЯОп Potential Interview Questions
Before writing any code, the instructor outlined exactly what an interviewer will ask you regarding JSON files:
- What is JSON and how do you read it in Spark?
- Where do we typically get JSON data from in a real project?
- What happens if our defined Schema has 3 keys, but the incoming JSON file has an extra 4th key-value pair?
- What is a "Line Delimited" JSON vs a "Multiline" JSON, and how does Spark handle them?
- How do you handle a corrupted JSON record (e.g., missing a closing bracket
})? - How do you flatten a deeply nested JSON? (Teaser for the next lecture)
1. Where does JSON come from?
As a Data Engineer, you might wonder why we deal with JSON so much. Why isn't everything just a neat CSV table?
APIs & NoSQL Databases
JSON (JavaScript Object Notation) is the standard format for web APIs. Additionally, NoSQL databases like MongoDB heavily rely on JSON/BSON because they support semi-structured data.
"MongoDB basically semi-structured type ke data ko support karta hai... wahan se hume JSON format mein data milta hai."
2. The "Multiline" Challenge
This is the most common error beginners face when reading JSON in Spark. Spark expects JSON to be Line-Delimited by default.
"id": 1,
"name": "John",
"age": 30
}
{) and thinks it's an
incomplete, broken record.
The Solution:
If your JSON spans multiple lines, you must explicitly tell Spark using an option:
.format("json") \
.option("multiline", "true") \
.load("/path/to/file.json")
3. Handling Schema Mismatches & Corruptions
Scenario A Extra Key-Value Pairs
Question: What if your defined Schema expects 3 columns (ID, Name, Age), but
the JSON file suddenly sends a 4th column (Salary)?
Answer: Spark handles this gracefully. It will simply ignore
the extra column ("Salary") and only pick up the 3 columns defined in your schema. It will
not throw an error.
Scenario B Syntactical Corruption
Question: What if the JSON is missing a comma, a quote, or a closing
bracket?
Answer: Spark treats it as a Corrupted Record. The handling is
exactly the same as we learned in the CSV lecture. Depending on the mode
(PERMISSIVE, DROPMALFORMED, FAILFAST), Spark will process
it. You can store these using badRecordsPath and view them using the internal
_corrupt_record column.
4. Flattening JSON & Instructor's Warning
Oftentimes, JSON data is deeply nested (arrays inside objects inside objects). Currently, if you load
it, Spark creates complex StructType or ArrayType columns which are hard to
analyze. The process of converting these nested structures into flat, standard rows and columns is
called Flattening a JSON.
Instructor's Note: "This is where the actual hard work is. We will learn how to flatten JSONs
in the upcoming dedicated lectures, as it is a major part of our daily job."
ЁЯФе Instructor's Strict Advice on Practice
"рдореИрдВ рд╕рд┐рд░реНрдл рдЖрдкрдХреЛ рдбрд╛рдЯрд╛ рд╕реЗрдЯ рджреВрдВрдЧрд╛ рдФрд░ рдПрдХ рд▓рд┐рдВрдХ рджреВрдВрдЧрд╛... рдирд╛ рд╣реА рдЖрдкрдХреЛ рдпреЗ рдиреЛрдЯрдмреБрдХ рдорд╛рдВрдЧрдиреЗ рд╣реИрдВ рдФрд░ рдирд╛ рд╣реА рдЖрдкрдХреЛ рдЬреЛ рд╣рдорд╛рд░рд╛ рдбреЗрдЯрд╛рдмреЗрд╕ рдХрд╛ рдиреЛрдЯрдмреБрдХ рд╣реИ рд╡реЛ рднреА рдирд╣реАрдВ рдорд╛рдВрдЧрдирд╛ рд╣реИред"
Translation & Meaning: The instructor is purposely not providing his Databricks notebook. He will only provide the raw dataset and documentation links. He strictly demands that students write the code from scratch on their own machines to build muscle memory, rather than just copy-pasting his notebook.