Databricks Certified Associate Developer for Apache Spark 3.5 - Python Sample Questions:

1. 38 of 55.
A data engineer is working with Spark SQL and has a large JSON file stored at /data/input.json.
The file contains records with varying schemas, and the engineer wants to create an external table in Spark SQL that:
Reads directly from /data/input.json.
Infers the schema automatically.
Merges differing schemas.
Which code snippet should the engineer use?

A) CREATE EXTERNAL TABLE users
USING json
OPTIONS (path '/data/input.json', mergeAll 'true');
B) CREATE EXTERNAL TABLE users
USING json
OPTIONS (path '/data/input.json', mergeSchema 'true');
C) CREATE EXTERNAL TABLE users
USING json
OPTIONS (path '/data/input.json', inferSchema 'true');
D) CREATE TABLE users
USING json
OPTIONS (path '/data/input.json');

2. An engineer notices a significant increase in the job execution time during the execution of a Spark job. After some investigation, the engineer decides to check the logs produced by the Executors.
How should the engineer retrieve the Executor logs to diagnose performance issues in the Spark application?

A) Use the command spark-submit with the -verbose flag to print the logs to the console.
B) Locate the executor logs on the Spark master node, typically under the /tmp directory.
C) Fetch the logs by running a Spark job with the spark-sql CLI tool.
D) Use the Spark UI to select the stage and view the executor logs directly from the stages tab.

3. 26 of 55.
A data scientist at an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user.
Before further processing, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns.
The PII columns in df_user are name, email, and birthdate.
Which code snippet can be used to meet this requirement?

A) df_user_non_pii = df_user.select("name", "email", "birthdate")
B) df_user_non_pii = df_user.remove("name", "email", "birthdate")
C) df_user_non_pii = df_user.drop("name", "email", "birthdate")
D) df_user_non_pii = df_user.dropFields("name", "email", "birthdate")

4. A data engineer writes the following code to join two DataFrames df1 and df2:
df1 = spark.read.csv("sales_data.csv") # ~10 GB
df2 = spark.read.csv("product_data.csv") # ~8 MB
result = df1.join(df2, df1.product_id == df2.product_id)

Which join strategy will Spark use?

A) Shuffle join, because AQE is not enabled, and Spark uses a static query plan
B) Shuffle join because no broadcast hints were provided
C) Broadcast join, as df2 is smaller than the default broadcast threshold
D) Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently

5. A data scientist has identified that some records in the user profile table contain null values in any of the fields, and such records should be removed from the dataset before processing. The schema includes fields like user_id, username, date_of_birth, created_ts, etc.
The schema of the user profile table looks like this:

Which block of Spark code can be used to achieve this requirement?
Options:

A) filtered_df = users_raw_df.na.drop(how='all')
B) filtered_df = users_raw_df.na.drop(how='any')
C) filtered_df = users_raw_df.na.drop(how='all', thresh=None)
D) filtered_df = users_raw_df.na.drop(thresh=0)

Solutions: