Skip to content

Metrics: Adversarial Similarity

H2O Model Validation offers an array of metrics in the form of histograms, charts, and a table to understand an Adversarial Similarity validation test. Below, each metric is described in turn.

Histogram: IsReference-Probability

The IsReference-Probability histogram displays the AUC probability values obtained during the Adversarial Similarity test.

  • In the Adversarial Similarity test, predictions (AUC probability values) are generated to determine how much a dataset’s row differs from those belonging to the training dataset.

  • The target column is called is_test, and the predictions generated by the adversarial model are called IsReference-Probability.

  • Higher IsReference-Probability values to the histogram's right will indicate a greater difference between the training dataset and the test dataset.

  • A balance of IsReference-Probability values on both sides of the histogram will indicate a 1.0 AUC score (a perfect separation between the two datasets).

  • The Y-axis refers to the relative frequency of an AUC probability (IsReference-Probability) value in one of the IsReference-Probability intervals.

    • Futhermore, the relative frequency (Y-axis) refers to the percent-wise frequencies or number of occurrences of values in the datasets. H2O Model Validation performs this relative or percent-wise calculation to compare numbers between the training and reference datasets. Since H2O Model Validation expects each dataset to have a different size, making a fair comparison on direct frequencies will be harder. Therefore, H2O Model Validation divides the number of occurrences by the length of the dataset.
IsTest probability

Dissimilarity score

The Dissimilarity score displays the Area Under the Curve (AUC) value for the dataset (test dataset) being compared to the training dataset. A higher AUC value indicates a higher dissimilarity.

Disimilarity score

Chart: Feature importance

The Feature importance chart displays the top features generating a high AUC value for the Adversarial Similarity test. In other words, the chart displays the features from top to lowest, driving the separation between the two datasets being compared during the Adversarial Similarity test.

Note

Clicking on the bar of a feature will trigger the display of the following plot: Plot: Feature PDP.

Feature importance

Plot: Feature PDP

The Feature Partial Dependence Plot (PDP) displays the impact the different values of the selected feature have on the predicted values. The selected feature refers to the feature (clicked) on the Feature importance chart.

Feature PDP

The following histogram and graph are available for the selected feature in the Feature Partial Dependence Plot (PDP):

To access either the histogram or graph above, consider the following instructions when viewing the Feature Partial Dependence Plot (PDP):

  1. Click Kebab menu.

    Kebab menu

Histogram: Feature histogram per data

The Feature histogram per data displays the selected feature on the Feature importance chart.

Feature histogram per data

Graph: Feature missing ratios

The Feature missing ratios graph displays any missing ratio values of the selected feature in the train and test dataset (the dataset being compared to the training dataset). The selected feature refers to the feature (clicked) on the Feature importance chart.

Feature missing ratios

Table: Shapley

The Shapley table contains adversarial model results from a Shapley perspective. Each row in the table represents a row of the test dataset (being compared to the training dataset). Rows are ordered from the highest to lowest predictive value (is_test.1). In other words, rows higher in the table represent the rows of the test dataset most dissimilar to the training dataset.

Note

H2O Model Validation in the Shapley table displays the top N highest (dissimilar) or lowest (similar) results (is_test.1 prediction values), where N=40.

Shapley table

The first three columns of the Shapley table refer to the row's ID, the target column, and the predictive value given to the row, while the rest of the columns refer to the train columns of the model.

Name Description
Adv-ID Row ID
Adv-IsReference Target column
Adv-IsReference-Probability Predictive value given to the row

Clicking on one row ID (374) will display a local Shapley chart highlighting the highest and lowest contributing features for the row's prediction value (is_test.1).

Test row local shapley

A global Shapley chart is available for the Shapley table, which highlights the features that, on average, increase the prediction value. For more information, see Chart: Global Shapley.

Show similar

By default, rows in the Shapley table are ordered from highest to lowest predictive value (is_test.1). To switch the default order and view rows from lowest to highest predictive values, consider the following instructions:

  1. When viewing the Shapley table, click Show similar.

Show dissimilar

To revert to the default order of the rows in the Shapley table, where test rows are ordered from highest to lowest predictive value (is_test.1), consider the following instructions:

  1. When viewing the Shapley table, click Show dissimilar.

Download Shapley values

To download the Shapley values of the test dataset, consider the following instructions when viewing the Shapley table:

  1. Click Download Shapley Values.

Note

The .csv file contains adversarial model results from a Shapley perspective for the test dataset (reference dataset).

Chart: Global Shapley

The global Shapley chart displays the top features increasing the average model predictions the most, driving the most dissimilarity between the datasets.

Global shapley