Databricks Certified-Data-Engineer-Professional Value Pack (Frequently Bought Together)

A data engineer is working on a Databricks notebook that requires several third-party Python libraries. Some of these are available on PyPI, while others are custom-developed and stored as local.wheel (.whl) and source (.tar.gz) files in an S3 bucket. The goal is to ensure all dependencies are installed and correctly available across multiple jobs running on any automated cluster in a Unity Catalog-enabled workspace. The engineer needs to install the required dependencies in a way that ensures a consistent environment setup across interactive notebooks and jobs and complies with workspace security policies (no internet access). Which approach should the engineer use to install and manage these dependencies while also ensuring reproducibility and compliance?

• A. Create a Python wheel file for the entire project, upload it to the Databricks Workspace Files or Volumes, and install it using a Cluster Library or pip install in a requirements.txt declared within a Databricks Asset Bundle.
• B. Use an init script on the cluster to install all dependencies using pip, referencing the local file system.
• C. Install all dependencies manually in the driver node of an interactive cluster, then export the environment and reimport on job clusters using %conda.
• D. Use %pip install in every notebook and job to install packages directly from PyPl and custom S3 paths.

Which method can be used to determine the total wall-clock time it took to execute a query?

• A. In the Spark UI, take the job duration of the longest-running job associated with that query.
• B. In the Spark UI, take the sum of all task durations that ran across all stages for all jobs associated with that query.
• C. Open the Query Profiler associated with that query and use the Total wall-clock duration metric.
• D. Open the Query Profiler associated with that query and use the Aggregated task time metric.

A junior data engineer has manually configured a series of jobs using the Databricks Jobs UI.
Upon reviewing their work, the engineer realizes that they are listed as the "Owner" for each job.
They attempt to transfer "Owner" privileges to the "DevOps" group, but cannot successfully accomplish this task.
Which statement explains what is preventing this privilege transfer?

• A. A user can only transfer job ownership to a group if they are also a member of that group.
• B. The creator of a Databricks job will always have "Owner" privileges; this configuration cannot be changed.
• C. Databricks jobs must have exactly one owner; "Owner" privileges cannot be assigned to a group.
• D. Other than the default "admins" group, only individual users can be granted privileges on jobs.
• E. Only workspace administrators can grant "Owner" privileges to a group.

A data engineer is building a streaming data pipeline to ingest JSON files from cloud storage into a Delta Lake table. The pipeline must process files incrementally, handle schema evolution automatically, ensure exactly-once processing, and minimize manual infrastructure management.
How should the data engineer fulfill these requirements?

• A. Use traditional Spark Structured Streaming with Auto Loader, manually configuring checkpoints location and enabling schema inference with "mergeSchema"= "true"
• B. Use Lakeflow Spart Declarative Pipelines with Auto Loader and enabling schema inference with
  "cloudFiles.schemaEvolutionMode"= "addNewColumns"
• C. Use Lakeflow Spark Declarative Pipelines with a static DataFrame read, merge schema with spark.conf.set ("spark.databricks.delta.schema.autoMerge.enabled", "true")
• D. Use Auto Loader in batch mode with a daily job to overwrite the Delta table.

A data pipeline uses Structured Streaming to ingest data from kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka_generated timesamp, key, and value. Three months after the pipeline is deployed the data engineering team has noticed some latency issued during certain times of the day.
A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recoded by Apache Spark) as well the Kafka topic and partition. The team plans to use the additional metadata fields to diagnose the transient processing delays.
Which limitation will the team face while diagnosing this problem?

• A. Updating the table schema requires a default value provided for each file added.
• B. Updating the table schema will invalidate the Delta transaction log metadata.
• C. New fields cannot be added to a production Delta table.
• D. Spark cannot capture the topic partition fields from the kafka source.
• E. New fields will not be computed for historic records.