New Release Databricks-Machine-Learning-Professional ML Data Scientist Questions

To drop a column from a Delta table, you need to enable column mapping mode for the table and use the ALTER TABLE statement with the DROP COLUMN clause. Alternatively, you can use the spark.read.format(“delta”).table(path) method to read the Delta table as a DataFrame and then use the drop method to drop the column. This will return a new DataFrame with the column removed. You can then overwrite the originalDelta table with the new DataFrame using the write.format(“delta”).mode(“overwrite”) method. The other options are incorrect because:

• Option A: spark.read.format(“delta”).load(path) will not read the Delta table as a table, but as a directory of files. This will not preserve the schema and metadata of the table, and will not allow you to overwrite the table with the new DataFrame.
• Option C: Delta tables can be modified using the Delta Lake APIs or SQL commands. Delta Lake supports schema evolution and column mapping to enable dropping columns from Delta tables.
• Option D: spark.read.table(path) will not work for Delta tables, as it expects a table name, not a path. To read a Delta table by its path, you need to use the spark.read.format(“delta”).load(path) or spark.read.format(“delta”).table(path) methods.
• Option E: spark.sql(“SELECT * EXCEPT star_rating FROM path”) will not drop the column from the Delta table, but only return a new DataFrame with the column excluded. This will not affect the original Delta table, and will not allow you to overwrite the table with the new DataFrame. References: Drop columns, Delta Lake Quickstart, Delta Lake column mapping

Batch deployment is the most common paradigm for machine learning projects, as it accounts for 80-90% of deployments1. Batch deployment leverages databases and object storage to process a large batch of input data at once, rather than processing each input data point individually in real time. Batch deployment is well suited for large data volume scenarios that need efficient processing but response time isn’t critical. For example, you might use batch deployment to generate recommendations for customers based on their purchase history, or to detect anomalies in sensor data collected over a period of time. Batch deployment is less expensive and more efficient than real-time deployment, as the underlying compute runs only during the batch job and can be scaled up or down as needed. Batch deployment can be implemented using Databricks Machine Learning endpoints, which provide a unified interface to invoke and manage model deployments across compute types2. References:

• Databricks Machine Learning Professional Exam Guide, Section 3: Model Deployment, p. 4
• Databricks Machine Learning Documentation, Deploying ML Models, Batch Inference, p. 1
• Machine Learning in Production - Databricks, The Four Deployment Paradigms, p. 9
• Deploy machine learning models in production environments, Deployment methods, p. 1

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows users to express streaming computations the same way as batch computations on static data, using DataFrame and Dataset APIs. Structured Streaming can handle both unbounded and bounded data sources, and can process data in micro-batches or continuously. Structured Streaming can be used to provide continuous processing of data for machine learning applications, such as data ingestion, preprocessing, feature engineering, and model inference. StructuredStreaming can also integrate with MLflow and Delta Lake to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance123 References:

• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks
• [Continuous Machine Learning with Structured Streaming and MLflow - Databricks]

To restore the original scikit-learn model object, the correct method is mlflow.sklearn.load_model, which takes the model URI as an argument and returns a scikit-learn model. This method can load models with the sklearn flavor, which preserves the original attributes and methods of the model. The mlflow.load_model method is a generic method that can load models with any flavor, but it returns a pyfunc model, which does not have the feature_importances_ attribute. The client.list_artifacts method is used to list theartifacts associated with a run, not to load a model. The client.pyfunc.load_model method is similar to mlflow.load_model, but it takes a local file path instead of a model URI. The MLflow Experiments UI can show the logged parameters, metrics, and artifacts of a run, but it cannot show the model object itself. References:

• MLflow Models
• MLflow Python API