Data Engineering Associate with Databricks Practice Exam

What are examples of narrow transformations in Spark?

1. GroupBy, join, aggregate

2. Filter, contains, map

3. Distinct, union, sort

4. Shuffle, repartition, coalesce

The correct answer is: Filter, contains, map

Narrow transformations in Spark are operations that do not require data to be shuffled across partitions. This means that each partition output from a narrow transformation depends only on a partition from its parent RDD. The operations that are classified as narrow transformations typically involve the transformation of data in a straightforward manner, as each input partition is mapped to a specific output partition without the need to exchange data between partitions. In this context, filtering, checking if a collection contains an element, and mapping a function over elements are all operations that can be performed independently on each partition. Hence, they can be efficiently executed without redistributing the data across the entire cluster. For instance, a filter operation applies a Boolean function to every element in the data, returning only those that satisfy the condition while keeping the partition intact. Similarly, the map function applies a transformation to each element, only needing access to the data contained within the individual partition. The other options include operations that involve shuffling or other transformations that may lead to data reassignment across partitions. GroupBy, join, and aggregate require combining data from different partitions, making them shuffle operations. Distinct and union can also lead to shuffling since they need to ensure all unique items are captured or combined across partitions. Shuffle, repart