Databricks-Machine-Learning-Associate Exam Dumps : Databricks Certified Machine Learning Associate Exam

To filter rows in a Spark DataFrame based on a condition, you use thefiltermethod along with a column condition. The correct syntax in PySpark to accomplish this task isspark_df.filter(col("price") > 0), which filters the DataFrame to include only those rows where the value in the "price" column is greater than 0. Thecolfunction is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different types of data manipulation frameworks like pandas.References:

• PySpark DataFrame API documentation (Filtering DataFrames).

To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use theapplyInPandasfunction. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance.

prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) )

In this code:

• groupby("device_id"): Groups the DataFrame by the "device_id" column.
• applyInPandas(apply_model, schema=apply_return_schema): Applies theapply_modelfunction to each group and specifies the schema of the return DataFrame.

References:

• PySpark Pandas UDFs Documentation

In the context of Spark MLlib, an estimator refers to an algorithm which can be "fit" on a DataFrame to produce a model (referred to as a Transformer), which can then be used to transform one DataFrame into another, typically adding predictions or model scores. This is a fundamental concept in machine learning pipelines in Spark, where the workflow includes fitting estimators to data to produce transformers.

References

• Spark MLlib Documentation:https://spark.apache.org/docs/latest/ml-pipeline.html#estimators