Data Engineering Associate with Databricks Practice Exam

To execute a single micro-batch to process all available data, which code should be used?

1. trigger(once=True)

2. trigger(once=False)

3. execute(singleBatch=True)

4. process(allData=True)

The correct answer is: trigger(once=True)

The choice of implementing trigger(once=True) is appropriate for executing a single micro-batch to process all available data in Spark Structured Streaming. When set to "once," the trigger specifies that the stream should process all available data that has been ingested up to that point, completing the execution of a single micro-batch. This is especially useful when you want to handle a certain batch of data without entering into a continuous streaming process. It allows for a finite and defined execution, giving users control over how much data is processed at a time. The result is that all the data available in the input source is processed in that single execution. In contrast, other options do not effectively meet the requirement for processing all available data in a single micro-batch scenario. For instance, trigger(once=False) indicates a continuous processing mode, which is designed for ongoing data ingestion rather than a single batch. The execute(singleBatch=True) and process(allData=True) options do not align with the standard API for configuring triggers in Spark Structured Streaming and would result in syntax errors or incorrect configurations.