In this self-paced course, we will take a deeper dive into the process of custom recipe building to enhance Driverless AI. We will build three recipes using Visual Studio Code text editor. Each recipe will then be uploaded and tested using Driverless AI.

Recipes:

Note: Aquarium's Driverless AI Test Drive lab has a license key built-in, so you don't need to request one to use it. Each Driverless AI Test Drive instance will be available to you for two hours, after which it will terminate. No work will be saved. If you need more time to further explore Driverless AI, you can always launch another Test Drive instance or reach out to our sales team via the contact us form.

In the Get Started and with Open Source Custom Recipes self-paced course we covered the following:

Note: If you have not done so, complete the Get Started with Open Source Custom Recipes, the material covered will be needed for the successful completion of this self-paced course.

To recap, H2O Driverless AI is an artificial intelligence (AI) platform for automatic machine learning. Driverless AI automates some of the most tedious and challenging data science and machine learning tasks such as feature engineering, algorithm selection, model validation, model tuning, model selection, model explanation, model documentation, and model deployment. It aims to achieve the highest predictive accuracy, comparable to expert data scientists, but in a much shorter time thanks to end-to-end automation.

Driverless AI, version 1.7.0 and newer, allows Domain Scientists to combine their subject matter expertise with the broadness of Driverless by giving Data Scientists the option to upload their own transformers, scorers, and custom datasets. Driverless AI's Bring Your Own Recipe (BYOR) lets you apply your domain expertise and optimize your specific Driverless AI model(s) with just a few clicks. Driverless AI treats customer recipes as first-class citizens in the automatic machine learning workflow.

Driverless AI's Automatic Machine Learning Workflow

Driverless AI's Automatic Machine Learning workflow is represented in the image below:

dai-byor-how-it-works

The workflow is as follows; first, we start with tabular data in X and Y format, where X are the predictors and Y the value we want to predict. You can bring in the data from various connectors such as:

See Deeper Dive and Resources at the end of this task for more information about Enabling Data Connectors.

Once the data has been loaded to Driverless AI, Driverless AI performs Automatic Visualizations of the data and outputs the available graphs for the data used. The graphs allow you to have a better understanding of your data.

The data is then sent through Driverless AI's Automatic Model Optimization. The Automatic Model Optimization is a generic algorithm that learns over time what is working and what is not to make the best model for your data. This process includes model recipes, advanced feature engineering, algorithms (such as Xgboost, TensorFlow, LightGBM), and model tuning.

After the model has been finalized, Driverless AI then auto-generates model documentation that explains everything that happened in the experiment. This model documentation also describes how the model that was generated/created makes decisions. Additionally, the Machine Learning Interpretability of the models is generated to explain modeling results in a human-readable format. Once experiments have been completed, Driverless AI automatically generates both Python and Java scoring pipelines so that the model is ready to go for production.

BYOR

Bring Your Own Recipe (BYOR) is part of the Automatic Model Optimization process. It is here that Data scientists, through their subject matter expertise and domain knowledge that they get to augment the Automatic Model Optimization by creating and uploading their own transformations, scorers, and algorithms. Driverless AI allows the uploaded scorers, algorithms, and transformations to compete with the existing Driverless AI recipes and allows the best recipe to be used.

dai-byor-how-it-works-w-recipes

Recipes

Custom recipes are Python code snippets that can be uploaded into Driverless AI at runtime, like plugins. No need to restart Driverless AI. You can provide custom recipes for transformers, models, and scorers. During the training of a supervised machine learning modeling pipeline (aka experiment), Driverless AI can then use these code snippets as building blocks, in combination with all built-in code pieces (or instead of). By providing your own custom recipes, you can gain control over the optimization choices that Driverless AI makes to best solve your machine learning problems.

Python API

Driverless AI custom recipes allow for full customization of the entire ML pipeline through the scikit learn Python API. The Python API allows for custom feature engineering, custom loss functions, and custom ML Algorithms. This API is based off on how scikit learn works.

When building our custom recipes:

For custom feature engineering or a transformer, you will have two main parts:

Fit_transform - takes the X and Y data and changes the X variables(pulls out the year from a date, does arithmetic in multiple columns, target encoding). It can also add new data such as the zip code with the zip code package and bring in population or cities. Custom statistical Transformation Embeddings for numbers, categories, text, date/time, time-series, image audio, zip, latitude/longitude, ICD.

Transform - the transform gets called when you run the model and get predictions. The transform will be inside the scoring pipeline. When used in production, the transform will be present and be used in the validation and test sets. The transform does not have access to Y, and it alters the data based on what happened on the fit_transform.

For custom optimization functions or scorers, you can bring loss or gain functions. We can look further into precision and recall of a model through variations of the F metric. Driverless AI comes with F1, F2, and F0.5 scorers, where F1 is the harmonic mean of precision and recall, and the F2 score gives more weight to recall than precision. If you wanted to give precision higher weight, you could incorporate an F4 or F6 function as recipes for scorers.

Other things that can be done using scorers:

For custom ML Algorithms, there are two functions that are needed:

Best Practices for Recipes

Recipes are meant to be built by people you trust, and each recipe should be code-reviewed before going to production. If you decide to make your custom recipes, you can keep them internal or shared them with the Driverless AI team by making a pull request to the Open Source Driverless AI Recipes GitHub Repo. This repo was built and is currently maintained by H2O.ai's Kaggle Grand Masters. All custom recipes will be put through various acceptance tests that include missing values, various examples for binary, and regression to see if your recipe can handle all different types.

Take a few minutes to review the recommended best practices for building recipes in terms of:

The Writing Recipes Process

1. First, write and test your idea on sample data before wrapping it as a recipe.
2. Download the Driverless AI Recipes Repository for easy access to examples.
3. Use the Recipe Templates to ensure you have all the required components.

We will be building three simple recipes, a transformer, a scorer, and a model in the following three tasks. It is assumed that you have downloaded the Driverless AI Recipes Repository rel-1.9.1 and that you have access to the examples and recipe templates.

Note: The recipes you will find in the "Driverless AI Recipes Repository rel 1.9.1" may be slightly different from those referenced in this self-paced course. If you decide to build the recipes using the code from this self-paced course or the code found on the repository, you should still be able to run the various experiments.

Deeper Dive and Resources

A transformer (or feature) recipe is a collection of programmatic steps, the same steps that a data scientist would write a code to build a column transformation. The recipe makes it possible to engineer the transformer in training and production. The transformer recipe and recipes, in general, provides a data scientist the power to enhance the strengths of DriverlessAI with custom recipes. These custom recipes would bring in nuanced knowledge about certain domains - i.e., financial crimes, cybersecurity, anomaly detection, etc. It also provides the ability to extend DriverlessAI to solve custom solutions for time-series[1].

Where can Driverless AI Transformers be Used?

Driverless AI has transformer recipes for the following categories:

See the Deeper Dive and Resources at the end of this task to learn more about the Driverless AI Transformers GitHub Repo and more.

Custom Transformer Recipe

The custom transformer that we will build is the Summation of multiple Columns. Driverless AI comes with mathematical interactions between two columns. Mathematical interactions such as addition, subtraction, multiplication, and division. What if you wanted to do a mathematical interaction of 3 or more columns?

This transformer recipe will add three or more numeric columns and give the sum. For instance, it would take the values of X1, X2, and X3 to add them and provide the Sum, which might be predictive in our model.

ID

X1

X2

X3

SUM

1

10

5

3

18

2

1

2

3

6

3

0

9

0

9

4

1.3

7

2

10.3

Essentials to building a Transformer

These are the main steps in building our transformer:

a. Extending the Transformer Base Class
b. Specifying whether our recipe is enabled
c. Specifying whether to do acceptance tests
d. Select which columns can be used
e. Transform the training data
f. Transform the validation or testing data
g. When to use the transformer

If you want to see the overall Sum Transformer code for some guidance, look at the end of this task to verify your code is aligned correctly.

Extending the Transformer Base Class

Let's start by creating a transformer recipe class in Python called SumTransformer that inherits from the CustomTransformer Base Class. This transformer will be responsible for computing the Summation of multiple Columns.

1. Open your text editor and create a new file.
2. Save the new file as sum.py
 3. Copy and paste the following code into your .py file.

The python code for Extending the Transformer Base Class is as follows:

"""Adds together 3 or more numeric features"""				
from h2oaicore.transformer_utils import CustomTransformer 
import datatable as dt
import numpy as np

class SumTransformer(CustomTransformer):

The SumTransformer class inherits from the CustomTransformer Base Class. Before creating the SumTransformer class, we import the necessary modules:

There are two types of Base Classes for a transformer the genetic custom transformer and the more specialized custom TimeSeries Transformer.

Specifying Whether our Recipe is Enabled

We will let Driverless AI know our recipe is enabled.

4. Copy and paste the following code below the Extending the Transformer Base Class section of code in your .py file.

The python code for Specifying Whether our Recipe is Enabled is as follows:

@staticmethod
def is_enabled():
    return True

The is_enabled method returns that our recipe is enabled. If this method returns False, our recipe is disabled, and the recipe will be completely ignored.

Specifying Whether to do Acceptance Tests

We will let Driverless AI know to do acceptance tests during the upload of our recipe.

5. Copy and paste the following code below the Specifying Whether our Recipe is Enabled section of code in your .py file.

The python code for Specifying Whether to do Acceptance Tests is as follows:

@staticmethod
def do_acceptance_test():
    return True

The do_acceptance_test method returns that the acceptance tests should be performed during our recipe's upload. Acceptance tests perform a number of sanity checks on small data and attempt to provide helpful instructions for fixing any potential issues. If our recipe required specific data or did not work on random data, this method should return False, so acceptance tests are not performed.

Select Which Columns to Use

We will let Driverless AI know which columns can be used with this transformer.

6. Copy and paste the following code below the Specifying Whether to do Acceptance Tests section of code in your .py file.

The python code for Selecting Which Columns to Use is as follows:

@staticmethod
def get_default_properties():
	  return dict(col_type="numeric", min_cols=3, max_cols="all", relative_importance=1)

The get_default_properties method returns a dictionary that states this transformer accepts numeric int/float column(s) with a minimum number of 3 columns accepted to a maximum number of all columns accepted as input and has a relative importance of 1 in terms of priority in the generic algorithm.

The table below shows the type of original column(s) that a transformer can accept:

Column Type (str)

Description

"all"

all column types

"any"

any column types

"numeric"

numeric int/float column

"categorical"

string/int/float column considered a categorical for feature engineering

"numcat"

allow both numeric or categorical

"datetime"

string or int column with raw datetime such as %Y/%m/%d %H:%M:%S or %Y%m%d%H%M

"date"

string or int column with raw date such as %Y/%m/%d or %Y%m%d

"text"

string column containing text (and hence not treated as categorical)

"time_column"

the time column specified at the start of the experiment (unmodified)

After selecting the column type, we will select the minimum and the maximum number of columns. Since Driverless AI comes with a transformer that can sum two columns, we will set the minimum number of columns to 3 and the maximum number of columns to all for our transformer. This means that when Driverless AI runs our transformer, it will always choose between 3 and all columns.

The relative importance will be set to 1 for the most part; however, if you want your custom transformer to have higher importance, you can always increase its value. Setting the relative importance to 1 will let Driverless AI know that your custom transformer should have a higher priority in the generic algorithm through relative_importance.

Transforming the Training Data

We will fit the transformer on the training data and return a transformed frame with new features that have the same number of rows and any number of columns greater than or equal to 1.

7. Copy and paste the following code below the Select Which Columns to Use section of code in your .py file.

The python code for Transforming Training Data is as follows:

def fit_transform(self, X: dt.Frame, y: np.array = None):
    return self.transform(X)

The fit_transform method fits the transformer on the training data X, which is a datatable of n rows and m columns between min_cols and max_cols specified in get_default_properties method, defaults the target column y parameter to None for API compatibility and returns a transformed datatable frame with new features. This transformed frame will be passed onto the predictors.

Note:

The fit_transform method is always called before the transform method is called. The output can be different based on whether the fit_transform method is called on the entire frame or a subset of rows. The output must be in the same order as the input data.

Transforming the Validation or Testing Data

We are going to transform the validation or testing data on a row-by-row basis.

8. Copy and paste the following code below the Transforming the Training Data section of code in your .py file.

The python code for Transforming Testing Data is the following method:

def transform(self, X: dt.Frame):
	return X[:, dt.sum([dt.f[x] for x in range(X.ncols)])]

The transform method selects all columns using datatable f-expression single-column selector in each row. It performs the summation on each column in that row. It performs this computation until the summation of all columns on each row has finished and returns the same number of rows as the original frame, but with one column in each row equalling the summation. The transformed frame will be passed onto the predictors.

Note: In a lot of cases, the fit_transform and transform function will be doing the exact same thing, they will not be using the y-value. If the "y" is needed, then the code for both functions might differ (ie. time series).

When to Use the Transformer

The last part of the code that will be added will determine when this transformer should be used.

9. Copy and paste the following code below the class SumTransformer(CustomTransformer): section of code in your .py file.

The python code for When to Use the Transformer is as follows:

_regression = True
_binary = True
_multiclass = True
_numeric_output = True
_is_reproducible = True
_included_model_classes = None  # List[str]
_excluded_model_classes = None  # List[str]
_testing_can_skip_failure = False  # ensure tested as if shouldn't fail

When writing transformers, we need to ask ourselves the following types of questions:

All the Custom Transformer Recipe Code

Your text editor should look similar to the page below:

sum-transformer

In case you want to copy and paste all the code to test it out:

"""Adds together 3 or more numeric features"""
# Extending the Transformer Base Class
from h2oaicore.transformer_utils import CustomTransformer
import datatable as dt
import numpy as np


class SumTransformer(CustomTransformer):
    # When to Use the Transformer
    _regression = True
    _binary = True
    _multiclass = True
    _numeric_output = True
    _is_reproducible = True
    _included_model_classes = None  # List[str]
    _excluded_model_classes = None  # List[str]
    _testing_can_skip_failure = False  # ensure tested as if shouldn't fail

    # Specifying Whether our Recipe is Enabled
    @staticmethod
    def is_enabled():
        return True

    # Specifying Whether to do Acceptance Tests
    @staticmethod
    def do_acceptance_test():
        return True

    # Select Which Columns to Use 
    @staticmethod
    def get_default_properties():
        return dict(col_type="numeric", min_cols=3, max_cols="all", relative_importance=1)

    # Transforming the Training Data
    def fit_transform(self, X: dt.Frame, y: np.array = None):
        return self.transform(X)

    # Transforming the Validation or Testing Data
    def transform(self, X: dt.Frame):
        return X[:, dt.sum([dt.f[x] for x in range(X.ncols)])]

Challenge

The final step in building the custom transformer recipe is to upload the custom recipe to Driverless AI and check that it passes the acceptance test. If your recipe is not passing the Driverless AI's acceptance test, see Task 5: Troubleshooting.

Take the new custom SumTransformer and test it in a dataset of your choice.

Note: The dataset needs to have more than three quantitative columns that can be added together

If you have questions on how to upload the transformer recipe to Driverless AI, see Get Started with Open Source Custom Recipes - Task 3: Recipe: Transformer.

References

[1] How to write a Transformer Recipe for Driverless AI by Ashrith Barthur

Deeper Dive and Resources

A scorer recipe helps evaluate the performance of your model. There are many methods to evaluate performance, and Driverless AI has many scorers available by default. However, if you want to test a different scorer for your particular model, then BYOR is an excellent way of testing a particular scorer and then compare the model results through Driverless AI's Project Workspace feature.

Note: Driverless AI will compare the scorer you uploaded with the existing scorers and select the best scorer fit for your dataset. If your scorer was not selected as the default scorer by Driverless AI in your experiment and you still would like to see how your dataset would perform with your scorer recipe, you can manually select your scorer in the Experiments Page under Scorers.

Where can scorers be used?

Driverless AI has Scorer recipes for the following categories:

Custom Scorer Recipe

The custom scorer that we will build in this section is a "False Discovery Rate" scorer. This scorer works in binary classification problems where a model will predict what two categories(classes) the elements of a given set belong to. The results of the model can be classified as True or False. However, there will be elements classified as True even though they are False. Those elements are called Type I Errors (reverse precision) or False Positives. Our False Discovery Rate scorer will use the False Positive Rate equation to obtain the percentage of False Positives out of all the elements that were classified as Positive:

False Positive Rate (FPR) = False Positives/(False Positives + True Negatives) = FP/(FP + TN)

The False Positive Rate is the algorithm where all the elements that were incorrectly classified (False Positives) will be divided by the sum of all the elements that were correctly classified (True Positives) and incorrectly classified (False Positives).

If you would like to review or learn more about binary classification in Driverless AI, view our Machine Learning Experiment Scoring and Analysis - Financial Focus.

Essentials to Building a Scorer

a. Extending the Scorer Base Class
b. Specifying whether our recipe is enabled
c. Implementing the Scoring Metric
d. Optimizing a Scorer

If you want to see the overall False Discovery Rate Scorer code for some guidance, look at the end of this task to verify your code is aligned correctly.

Extending the Scorer Base Class

Let's start by creating a scorer recipe class in Python called MyFDRScorer that inherits from the CustomScorer base class. This scorer will be responsible for computing the False Discovery Rate.

1. Open your text editor and create a new file.
2. Save the new file as false_discovery_rate.py.
3. Copy and paste the code above into your .py file.

The python code for Extending the Scorer Base Class is a follows:

"""Weighted False Discovery Rate: `FP / (FP + TP)` at threshold for optimal F1 Score."""
import typing
import numpy as np
from h2oaicore.metrics import CustomScorer, prep_actual_predicted
from sklearn.preprocessing import LabelEncoder, label_binarize
from sklearn.metrics import precision_score
import h2o4gpu.util.metrics as daicx

class MyFDRScorer(CustomScorer):

The MyFDRScorer class inherits from the CustomScorer Base Class. Unlike transformers, there is only a single base class for scorers, and it's called CustomScorer. Prior to creating the MyFDRScorer class, we import the necessary modules:

Specifying Whether our Recipe is Enabled

We will let Driverless AI know our recipe is enabled.

5. Copy and paste the following code below the Extending the Scorer Base Class section of code in your .py file.

The python code for Specifying Whether our Recipe is Enabled is as follows:

@staticmethod
def is_enabled():
    return False  # Already part of Driverless AI 1.9.0+

The is_enabled method returns when our recipe is enabled. If this method returns False, our recipe is disabled and the recipe will be completely ignored.

Implementing Scoring Metric

We are going to compute a score from the actual and predicted values by implementing the scoring function.

6. Copy and paste the following code below the Specifying Whether our Recipe is Enabled section of code in your .py file.

The code for Implementing Scoring Metric is below:

@staticmethod
def _metric(tp, fp, tn, fn):
    if (fp + tp) != 0:
        return fp / (fp + tp)
    else:
        return 1

def protected_metric(self, tp, fp, tn, fn):
    try:
        return self.__class__._metric(tp, fp, tn, fn)
    except ZeroDivisionError:
        return 0 if self.__class__._maximize else 1  # return worst score if ill-defined

def score(self,
          actual: np.array,
          predicted: np.array,
          sample_weight: typing.Optional[np.array] = None,
          labels: typing.Optional[np.array] = None,
          **kwargs) -> float:

    if sample_weight is not None:
        sample_weight = sample_weight.ravel()
    enc_actual, enc_predicted, labels = prep_actual_predicted(actual, predicted, labels)
    cm_weights = sample_weight if sample_weight is not None else None

    # multiclass
    if enc_predicted.shape[1] > 1:
        enc_predicted = enc_predicted.ravel()
        enc_actual = label_binarize(enc_actual, labels).ravel()
        cm_weights = np.repeat(cm_weights, predicted.shape[1]).ravel() if cm_weights is not None else None
        assert enc_predicted.shape == enc_actual.shape
        assert cm_weights is None or enc_predicted.shape == cm_weights.shape

    cms = daicx.confusion_matrices(enc_actual.ravel(), enc_predicted.ravel(), sample_weight=cm_weights)
    cms = cms.loc[
        cms[[self.__class__._threshold_optimizer]].idxmax()]  # get row(s) for optimal metric defined above
    cms['metric'] = cms[['tp', 'fp', 'tn', 'fn']].apply(lambda x: self.protected_metric(*x), axis=1, raw=True)
    return cms['metric'].mean()  # in case of ties

The score method has four input arguments:

  1. actual: an array of actual values from the target column.
    • If the raw data is text, then the array will be an array of text.
  2. predicted: an array of predicted numeric values.
    • For regression problems, then the predicted value will be a value that is appropriate for your feature space.
    • For binary classification problems, the predicted value would be a numeric column with values between 0 and 1. The predicted value represents how likely it is for an object to belong to one class.
    • For multi-classification problems, the predicted value would be multiple numeric values that represent the probabilities for every class.
  3. sample_weight: allows for some rows to be given higher importance than other rows.
    • The sample_weight column is usually assigned by the user when setting up the Driverless AI experiment.
  4. label: a list of class labels that help with labeling data for classification problems .

The score method starts by overwriting the sample_weight multidimensional array with a contiguous flattened 1D array if the sample_weight is not "none." Then prep_actual_predicted method encodes the predicted data, and behind the scenes, it uses the LabelEncoder to encode the actual values from the target column and the labeled data. The LabelEncoder helps label the data so that the values are 0 or 1 instead of between 0 and 1. The False Discovery Rate needs values of 0 or 1; therefore, all the actual and predicted values will be labeled 0 or 1. The cm_weights are assigned the sample_weight if the sample_weight is not "none," else the cm_weights are assigned none.

Then there is a check to account for if we are dealing with a multiclass classification problem. But since we are dealing with binary classification, we will skip explaining what happens if the check on enc_predicted array's column number is greater than one is true.

Next h2o4gpu's confusion matrices will be called cms = daicx.confusion_matrices(enc_actual.ravel(), enc_predicted.ravel(), sample_weight=cm_weights) to compute the confusion matrices for ROC analysis for all possible prediction thresholds. We will then get the rows for the optimal metric f1, which you will see in the next section is the optimizer defined by variable _threshold_optimizer = f1. Then we will use cms to get the true negatives(tn), false positives(fp), false negative(fn) and true positives(tp) from those rows.

Afterward, the prediction thresholds for tp, fp, tn, and fn are passed from the cms confusion matrices to the protected_metric method inside the lambda function, which is then passed to the _metric method where the False Positive Rate of fp/(fp + tp) is computed as long as the sum of fp + tp does not equal 0. If that sum does equal zero (meaning division by zero), then the _metric method returns a 1.Thus, the _metric method returns the False Positive Rate to the protected_metric method, which returns it to the lambda function, which assigns it to cms['metric'].

Finally, this score method will return the mean float score via return cms['metric'].mean(). This is the number we are trying to optimize, which can be a high or low value.

Optimizing the Scorer

Let's define for Driverless AI what makes a good scorer and when to use it.

7. Copy and paste the following code below the class MyFDRScorer(CustomScorer): section of code in your .py file

The python code for optimizing the scorer

_binary = True
_multiclass = True
_maximize = False
_threshold_optimizer = "f1"
_is_for_user = False  # don't let the user pick since can be trivially optimized (even when using F1-optimal thresh)

Here are some questions we are going answer based on our optimization variable values, so Driverless AI will know what is appropriate for this scorer.

What is the problem type this scorer applies to?

We set _binary to True so that this scorer applies to binary classification problem types. Also, _multiclass is set to True, so this scorer can also handle multiclass classification problem types. We will be using scorer to work with binary classification problems.

Is a higher or smaller score value better?

The perfect binary classification model would have no false positives; for this reason, _maximize is set to False since we want the scorer to get as close as possible to zero. When we set this to false, we will tell Driverless AI to stop working after finding a model that had a score of zero. If Driverless AI was supposed to create 200 models, and on the 100th model, the score is zero, there is no need for Driverless AI to keep working to searching for more perfect models. This optimization helps save the CPU.

What is threshold optimizer for binary classification?

We set _threshold_optimizer to f1 scorer metric. This means that our False Discovery Rate scorer uses f1 optimal metric to get rows from the confusion matrices.

Is the scorer for the user?

The _is_for_user variable is set to False, so users won't have to pick since the scorer can be optimized even when using F1 optimal threshold.

All the Custom Scorer Recipe Code

Your text editor should look similar to the page below:

false-discovery-rate-scorer

In case you want to copy and paste all the code to test it out:

"""Weighted False Discovery Rate: `FP / (FP + TP)` at threshold for optimal F1 Score."""
# Extending the Scorer Base Class
import typing
import numpy as np
from h2oaicore.metrics import CustomScorer, prep_actual_predicted
from sklearn.preprocessing import LabelEncoder, label_binarize
from sklearn.metrics import precision_score
import h2o4gpu.util.metrics as daicx


class MyFDRScorer(CustomScorer):
    # Optimizing the Scorer
    _binary = True
    _multiclass = True
    _maximize = False
    _threshold_optimizer = "f1"
    _is_for_user = False  # don't let the user pick since can be trivially optimized (even when using F1-optimal thresh)

    # Specifying Whether our Recipe is Enabled
    @staticmethod
    def is_enabled():
        return False  # Already part of Driverless AI 1.9.0+

    # Implementing Scoring Metric with helper methods _metric and protected_metric
    @staticmethod
    def _metric(tp, fp, tn, fn):
        if (fp + tp) != 0:
            return fp / (fp + tp)
        else:
            return 1

    def protected_metric(self, tp, fp, tn, fn):
        try:
            return self.__class__._metric(tp, fp, tn, fn)
        except ZeroDivisionError:
            return 0 if self.__class__._maximize else 1  # return worst score if ill-defined

    def score(self,
              actual: np.array,
              predicted: np.array,
              sample_weight: typing.Optional[np.array] = None,
              labels: typing.Optional[np.array] = None,
              **kwargs) -> float:

        if sample_weight is not None:
            sample_weight = sample_weight.ravel()
        enc_actual, enc_predicted, labels = prep_actual_predicted(actual, predicted, labels)
        cm_weights = sample_weight if sample_weight is not None else None

        # multiclass
        if enc_predicted.shape[1] > 1:
            enc_predicted = enc_predicted.ravel()
            enc_actual = label_binarize(enc_actual, labels).ravel()
            cm_weights = np.repeat(cm_weights, predicted.shape[1]).ravel() if cm_weights is not None else None
            assert enc_predicted.shape == enc_actual.shape
            assert cm_weights is None or enc_predicted.shape == cm_weights.shape

        cms = daicx.confusion_matrices(enc_actual.ravel(), enc_predicted.ravel(), sample_weight=cm_weights)
        cms = cms.loc[
            cms[[self.__class__._threshold_optimizer]].idxmax()]  # get row(s) for optimal metric defined above
        cms['metric'] = cms[['tp', 'fp', 'tn', 'fn']].apply(lambda x: self.protected_metric(*x), axis=1, raw=True)
        return cms['metric'].mean()  # in case of ties

Challenge

The final step in building the custom scorer recipe is to upload the custom recipe to Driverless and check that it passes the acceptance test. If your recipe is not passing the Driverless AI's acceptance test, see Task 5: Troubleshooting.

Take the false_discovery_rate scorer and test it in a dataset of your choice.

Note: The dataset needs to be for a binary classification problem

If you have questions on how to upload the scorer recipe to Driverless AI, see "Get Started with Open Source Custom Recipes - Task 4: Recipe: Scorer".

Deeper Dive and Resources

A model recipe is a recipe for a machine learning technique that can be used to build prediction models. Driverless AI has an extensive list of models by default; however, a new model can be loaded and be used to compare the Driverless AI models. Current Driverless models can be enhanced or slightly modified. Just like with the scorer's recipes, you can compare the results of the model recipe you created with the model that Driverless AI selected for your dataset through Driverless AI's Project Workspace feature.

Note: Driverless AI will compare the model you uploaded with the existing models and select the best model fit for your dataset. If the model was not selected as the top model by Driverless AI and you still would like to see how your dataset would perform with your model recipe, you can turn off all default models in Driverless AI and only select your model.

Where can models be used?

Driverless AI has Model recipes for the following categories:

Custom Model Recipe

Models are complex and harder to build; therefore, in this self-paced course, we will build a high-level model to understand the general mechanics of building a model.

The custom model that we will build is sklearn's Extra Trees or extremely randomized trees model from Sklearn. There is Extra Trees for classification ExtraTreeClassifier[1] and regression ExtraTreeRegressor [2].

From scikit learn:

"An extra-trees classifier. This class implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting."[1]

"An extra-trees regressor. This class implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting."[2]

This algorithm might give a slightly different prediction compared to other models such as XGboost, or Random Forest and it might be worth trying to see the results.

Essentials to building a Model

a. Extending the Model Base Class
b. Fit the model
c. Set details on fit model(set model parameters)
d. Get predictions

If you want to see the overall ExtraTrees Model code for some guidance, look at the end of this task to verify your code is aligned correctly.

Extending the Model Base Class

Let's start by creating a model recipe class in Python called ExtraTreesModel that inherits from the CustomModel base class.

1. Open your text editor and create a new file
2. Save the new file as extra_trees.py
3. Copy and paste the following code into your .py file.

The python code for Extending the Model Base Class is as follows:

"""Extremely Randomized Trees (ExtraTrees) model from sklearn"""
import datatable as dt
import numpy as np
from h2oaicore.models import CustomModel
from sklearn.ensemble import ExtraTreesClassifier, ExtraTreesRegressor
from sklearn.preprocessing import LabelEncoder
from h2oaicore.systemutils import physical_cores_count
 
class ExtraTreesModel(CustomModel):

The ExtraTreesModel class inherits from the CustomModel Base Class. Prior to creating the ExtraTreesModel class, we import the necessary modules:

There are four types of Base Classes for a model:

a. CustomModel(Genetic Model)
b. CustomTimeSeriesModel
c. CustomTensorFlowModel
d. CustomTimeSeriesTensorFlowModel

As you can see in the code above, we will focus on the Genetic model (CustomModel). To note, CustomTimeSeriesModel's has everything covered in this task plus an update function because Time Series Model needs to know how to look at past data, especially when scoring takes place, and there is new data that needs to be used. If the model you are working on requires TensorFlow or both TensorFlow and time-series, then the following models are available: CustomTensorFlowModel or CustomTimeSeriesTensorFlowMode. Both models will require their base class to be extended, and there will be additional options for each.

Fit the model

4. Copy and paste the following code below the Extending the Model Base Class section of code in your .py file.

The python code for Fitting the model is as follows:

def fit(self, X, y, sample_weight=None, eval_set=None, sample_weight_eval_set=None, **kwargs):

        orig_cols = list(X.names)
 
        if self.num_classes >= 2:
           lb = LabelEncoder()
           lb.fit(self.labels)
           y = lb.transform(y)
           model = ExtraTreesClassifier(**self.params)
        else:
           model = ExtraTreesRegressor(**self.params)
           
        # Can your model handle missing values??
        # Add your code here to handle missing values 
        """
        """
        model.fit(X, y)

The next part of the custom model is to fit the model. This is where the X and y values will be brought in. If there are rows that have higher importance than others, then a weight column can be added and flagged through the sample_weight. Other items that can be incorporated and flagged are: evaluation sets and evaluation set with sample weights.

The first part of this function is to save the names of all the predictors that came in orig_cols.

An if-else statement is then used to tell the function to work with both classification and regression problems; in other words, the extra tree's model needs to work for every ML problem. The if-else statement states that if there are two or more classes (binary or multiclass). Then you can use the LabelEncoder and call the ExtraTreesClassifier. If there are less than two classes, then the problem is a regression problem, and the ExtraTreesRegressor is called. At this point, an object model has been created, sklearn's extra trees, which is appropriate for this particular custom model.

After any algorithm-specific prep work has been done, the model can be fitted via model.fit(X,y).

Things to note:
 If the model you are working with is a model that can't handle missing values, it needs to be accounted for on your code. Suggestions include:

Driverless AI only accepts custom models that can handle missing values; therefore, for this task, you need to write your own section to handle missing values or see our recommendation at the end of the section.

Here is some code that you could use to replace missing values with a value smaller than all observed values:

        self.min = dict()
        for col in X.names:
            XX = X[:, col]
            self.min[col] = XX.min1()
            if self.min[col] is None or np.isnan(self.min[col]):
                self.min[col] = -1e10
            else:
                self.min[col] -= 1
            XX.replace(None, self.min[col])
            X[:, col] = XX
            assert X[dt.isna(dt.f[col]), col].nrows == 0
        X = X.to_numpy()

Set details on fit model(set model parameters)

5. Copy and paste the following code below the Fitting the model section of code in your .py file.

The code for set model parameters is below:

#Set model parameters
importances = np.array(model.feature_importances_)
 
self.set_model_properties(model=model,
                          features=orig_cols,
                          importances=importances.tolist(),
                          iterations=self.params['n_estimators'])

After the model has been fit, the next part is to set the model properties. There are four model properties that Driverless AI needs to know:

Get Predictions

The final part is to get the predictions.

6. Copy and paste the following code below the #Set model parameters section of code in your .py file.

The python code for get predictions is as follows:

#Get predictions
    def predict(self, X, **kwargs):

        #Can your model handle missing values??
        #Add your code here
        """
        """
        if self.num_classes == 1:
           preds = model.predict(X)
        else:
           preds = model.predict_proba(X)
        return preds

To get the predictions, the predict() function is called. The only thing needed to predict at this point is the model. If the predictions can't handle null values, as is in our case, there needs to be additional code that must be added under #Can your model handle missing values??.

Since the model needs to handle both classification and regression scenarios, the if-then statement in the code is used to identify each case. For classification, when the number of classes is 1 then the model.predict(x) gets called. Then for regression, the function model.predict_proba(x) gets called.

Things to note:

If the model you are working with is a model that can't handle missing values, it needs to be accounted for on your code. Suggestions include:

Here is a sample code for handling the missing values in the prediction section:

        X = dt.Frame(X)
        for col in X.names:
            XX = X[:, col]
            XX.replace(None, self.min[col])
            X[:, col] = XX
        model, _, _, _ = self.get_model_properties()
        X = X.to_numpy()

The Custom Scorer Recipe Code That was Covered So Far

Your text editor should look similar to the page below:

extra-trees-model-1

extra-trees-model-2

In case you want to copy and paste all the code we have covered so far to test it out:

"""Extremely Randomized Trees (ExtraTrees) model from sklearn"""
# Extending the Model Base Class
import datatable as dt
import numpy as np
from h2oaicore.models import CustomModel
from sklearn.ensemble import ExtraTreesClassifier, ExtraTreesRegressor
from sklearn.preprocessing import LabelEncoder
from h2oaicore.systemutils import physical_cores_count


class ExtraTreesModel(CustomModel):
    # Challenge
    """
    #Code for when to use the model, and how to set the parameters, and n_estimators

    _regression = True
    _binary = True
    _multiclass = True
    _display_name = "ExtraTrees"
    _description = "Extra Trees Model based on sklearn"
    _testing_can_skip_failure = False  # ensure tested as if shouldn't fail

    def set_default_params(self, accuracy=None, time_tolerance=None,
                           interpretability=None, **kwargs):
        # Fill up parameters we care about
        self.params = dict(random_state=kwargs.get("random_state", 1234),
                           n_estimators=min(kwargs.get("n_estimators", 100), 1000),
                           criterion="gini" if self.num_classes >= 2 else "mse",
                           n_jobs=self.params_base.get('n_jobs', max(1, physical_cores_count)))

    # Code for how to modify model parameters

    def mutate_params(self, accuracy=10, **kwargs):
        if accuracy > 8:
            estimators_list = [100, 200, 300, 500, 1000, 2000]
        elif accuracy >= 5:
            estimators_list = [50, 100, 200, 300, 400, 500]
        else:
            estimators_list = [10, 50, 100, 150, 200, 250, 300]
        # Modify certain parameters for tuning
        self.params["n_estimators"] = int(np.random.choice(estimators_list))
        self.params["criterion"] = np.random.choice(["gini", "entropy"]) if self.num_classes >= 2 \
            else np.random.choice(["mse", "mae"])

    """

    # Fit the model
    def fit(self, X, y, sample_weight=None, eval_set=None, sample_weight_eval_set=None, **kwargs):
        orig_cols = list(X.names)
        if self.num_classes >= 2:
            lb = LabelEncoder()
            lb.fit(self.labels)
            y = lb.transform(y)
            model = ExtraTreesClassifier(**self.params)
        else:
            model = ExtraTreesRegressor(**self.params)

        # Can your model handle missing values??
        # Add your code here
        """
        self.min = dict()
        for col in X.names:
            XX = X[:, col]
            self.min[col] = XX.min1()
            if self.min[col] is None or np.isnan(self.min[col]):
                self.min[col] = -1e10
            else:
                self.min[col] -= 1
            XX.replace(None, self.min[col])
            X[:, col] = XX
            assert X[dt.isna(dt.f[col]), col].nrows == 0
        X = X.to_numpy()
        """
        model.fit(X, y)

        # Set model parameters
        importances = np.array(model.feature_importances_)
        self.set_model_properties(model=model,
                                  features=orig_cols,
                                  importances=importances.tolist(),
                                  iterations=self.params['n_estimators'])

    # Get Predictions
    def predict(self, X, **kwargs):
        #Can your model handle missing values??
        #Add your code here
        """
        X = dt.Frame(X)
        for col in X.names:
            XX = X[:, col]
            XX.replace(None, self.min[col])
            X[:, col] = XX
        model, _, _, _ = self.get_model_properties()
        X = X.to_numpy()
        """
        
        if self.num_classes == 1:
            preds = model.predict(X)
        else:
            preds = model.predict_proba(X)
        return preds

Challenge

The final step in building the custom model recipe is to complete the parts that were left uncompleted. You will need to write a code to determine when to use the model. First, you need to specify if the model should work with a classification or regression case. You can also set a name and description for the model. Second, you need to determine the "n_estimators" and, lastly, set up test code for testing for null values or handling missing values, like the examples that were given in each section. Once those are set, you can upload the custom recipe to Driverless and check that it passes the acceptance test. If your recipe is not passing the Driverless AI's acceptance test, see Task 5: Troubleshooting.

Here is a sample code that could be used to complete the first task of the challenge:

    _regression = True
    _binary = True
    _multiclass = True
    _display_name = "ExtraTrees"
    _description = "Extra Trees Model based on sklearn"
    _testing_can_skip_failure = False  # ensure tested as if shouldn't fail

    def set_default_params(self, accuracy=None, time_tolerance=None,
                           interpretability=None, **kwargs):
                           
     #Fill up parameters we care about
        self.params = dict(random_state=kwargs.get("random_state", 1234),
                           n_estimators=min(kwargs.get("n_estimators", 100), 1000),
                           criterion="gini" if self.num_classes >= 2 else "mse",
                           n_jobs=self.params_base.get('n_jobs', max(1, physical_cores_count)))

If you need to do parameter tuning after setting your default parameters, here is some code for modifying model parameters. Feel free to try it as is or tune it more:

    def mutate_params(self, accuracy=10, **kwargs):
        if accuracy > 8:
            estimators_list = [100, 200, 300, 500, 1000, 2000]
        elif accuracy >= 5:
            estimators_list = [50, 100, 200, 300, 400, 500]
        else:
            estimators_list = [10, 50, 100, 150, 200, 250, 300]
        # Modify certain parameters for tuning
        self.params["n_estimators"] = int(np.random.choice(estimators_list))
        self.params["criterion"] = np.random.choice(["gini", "entropy"]) if self.num_classes >= 2 \
            else np.random.choice(["mse", "mae"])

Take the extra_tress_model and test it using a dataset of your choice.

If you have questions on how to upload the transformer recipe to Driverless AI, see "Get Started with Open Source Custom Recipes - Task 5: Recipe: Model".

References

[1] sklearn Extra Trees Classifier
[2] sklearn Extra Trees Regressor

Deeper Dive and Resources

When uploading a new recipe to Driverless AI, there are multiple things that can happen:

Recipe did not Make the Cut

One of the most significant advantages when loading a recipe to Driverless AI is that Driverless AI will subject your recipe to its acceptance tests. If your recipe did not pass the acceptance, Driverless AI would let you know right away if your recipe made the cut. If your recipe does not make the cut, you will receive feedback from Driverless AI to improve it.

Other tips:

How can I debug my recipe?

Recipe Made the Cut to Driverless AI and was not Used in the Experiment

You were able to load your recipe to Driverless AI successfully; however, your recipe was not used by Driverless AI; now what? Driverless AI takes best-fit recipes for your dataset, so if you don't see your recipe being used, you can manually select your recipe when setting up your Experiment.

Other tips:

The transformer recipe didn't lead to the highest variable importance for the experiment

That's nothing to worry about. It's unlikely that your features have the strongest signal of all features. Even ‘magic' Kaggle grandmaster features don't usually make a massive difference, but they still beat most of the competition.

Deeper Dive and Resources

Try to build your own recipe. Driverless AI has many datasets that you can use to test your new custom recipe.

H2O custom recipes reside in the H2O Driverless AI Recipes GitHub repo. There are multiple branches of Driverless AI recipes so make sure that you are using the same branch as the Driverless AI version you have.

For this self-paced course, we are using Driverless AI 1.9.1 and we will download the relevant custom recipes from that branch using the following shell commands.

mkdir -p $HOME/recipes/{models/algorithms,scorers/classification/binary,transformers/numeric}
wget https://raw.githubusercontent.com/h2oai/driverlessai-recipes/rel-1.9.1/transformers/numeric/sum.py -P $HOME/recipes/transformers/numeric/
wget https://raw.githubusercontent.com/h2oai/driverlessai-recipes/rel-1.9.1/scorers/classification/binary/false_discovery_rate.py -P $HOME/recipes/scorers/classification/binary/
wget https://raw.githubusercontent.com/h2oai/driverlessai-recipes/rel-1.9.1/models/algorithms/extra_trees.py -P $HOME/recipes/models/algorithms

Feel free to open the custom recipes in your favorite text editor or IDE.