ЁЯОп The 5 Potential Interview Questions
Following the new teaching methodology, the instructor starts by listing the exact questions an interviewer might ask regarding this topic:
- Have you ever handled corrupted records in your project?
- When do you say a record is corrupted? (How do you identify it?)
- What happens when we encounter a corrupted record in different Read Modes?
- How can we print the bad records?
- Where do you store corrupted records?
"рдЕрдЧрд░ рдпреЗ рд╣рдо рдЯреЙрдкрд┐рдХ рдкрдврд╝рддреЗ рд╣реИрдВ рддреЛ рдЪрд▓рд┐рдП рд▓реЗрдЯрдирд╛ рдХрд░рддреЗ рд╣реБрдП рд╡реАрдбрд┐рдпреЛ рдХреЛ рд╕реНрдЯрд╛рд░реНрдЯ рдХрд░рддреЗ рд╣реИрдВ... рдЖрдЬ рдХреЗ рдЗрд╕ рд╡реАрдбрд┐рдпреЛ рдореЗрдВ рд╣рдо рдмрд╛рдд рддреЛ рдХрд░реЗрдВрдЧреЗ рд╣реА рд▓реЗрдХрд┐рди рдЙрд╕реЗ рдкрд╣рд▓реЗ рд╣рдо рд▓реЛрдЧ рдпрд╣ рджреЗрдЦ рд▓реЗрддреЗ рд╣реИрдВ рдХреА рдХреМрди-рдХреМрди рд╕реЗ рдРрд╕реЗ рдкреЛрдЯреЗрдВрд╢рд┐рдпрд▓ рдЗрдВрдЯрд░рд╡реНрдпреВ рдХреНрд╡реЗрд╢реНрдЪрдВрд╕ рд╣реИрдВ..." (If we study this topic, let's start without delay... Before talking about it, let's see the potential interview questions so you know what to expect).
1. Defining and Handling Corrupted Data
What is a Corrupted Record?
A record is considered corrupted when it does not match the expected schema or data format. For
example, if a column expects an Integer but receives a String, or if a
JSON file has a missing closing bracket }, Spark will treat that specific row as a
corrupted record.
Different Read Modes (Quick Recap)
How Spark reacts to these corrupted records depends entirely on the Read Mode you specify:
PERMISSIVE
The default mode. It tries to load everything. If it finds
a bad row, it puts null in the columns and saves the bad string.
DROPMALFORMED
It completely ignores and drops the corrupted row, only loading the perfect records.
FAILFAST
Throws an immediate exception and stops the entire job upon hitting a single bad record.
2. Printing & Storing the Corrupted Records
In a real production environment, you cannot simply drop malformed records and forget about them. The business expects you to track what data failed and why.
Q4 How to Print Bad Records
If you are using the default PERMISSIVE mode, Spark automatically creates an
internal column to hold the broken raw data. You can print this by selecting the exact column
name: _corrupt_record.
Q5 Where Do You Store Corrupted Records?
To store the corrupted records physically on the disk so they can be analyzed later, Spark
provides a specific option called badRecordsPath. We introduce this inside the
.option() method while reading the file.
.format("json") \
.option("badRecordsPath", "/path/to/error_folder/") \
.load("/path/to/raw_data.json")
3. Analyzing the Stored Bad Records
When Spark encounters bad records and you have badRecordsPath enabled, it writes those
records to the specified folder.
What does the output look like?
Spark saves the error files in JSON format. The file contains metadata such as the file size, modification time, and most importantly, the actual error details. If you read this JSON file back into a Spark DataFrame to view it, you will see fields like:
- path (The original file that had the error)
- reason (Why it failed)
- record (The actual corrupted string/row)
Instructor's Pro-Tip: The Truncate Issue
When you run df.show() to view these error logs, you will notice that the "reason" or
"path" columns are very long, and Spark cuts them off with a dot-dot-dot (...). The
information doesn't look correct or fully visible.
How to fix this:
df.show(truncate=False)
By passing truncate=False, Spark will display the full string on the page, allowing you
to see the exact location, the exact reason for the failure, and the exact corrupted record.
Lecture Conclusion
"We have completely answered all the questions... Now if someone asks 'Have you worked on it?', we can confidently say 'Yes we have worked on it.' If anyone asks you any question related to bad records, I think you will be able to answer it. This is always my attempt, to explain everything in full detail." - Manish Kumar