ICE) Explainer Demo

This example demonstrates how to interpret a scikit-learn model using the H2O Eval Studio library and retrieve the data and partial dependence plot.

[15]:

import logging
import os

import pandas
import datatable
import webbrowser

from h2o_sonar import interpret
from h2o_sonar.lib.api import commons, explainers
from h2o_sonar.lib.api.models import ModelApi
from h2o_sonar.explainers.pd_ice_explainer import PdIceExplainer

from sklearn.ensemble import GradientBoostingClassifier

[16]:

# dataset
dataset_path = "../../data/creditcard.csv"
target_col = "default payment next month"
df = pandas.read_csv(dataset_path)
(X, y) = df.drop(target_col, axis=1), df[target_col]

# results
results_location = "../../results"
os.makedirs(results_location, exist_ok=True)

[17]:

# explainer description
interpret.describe_explainer(PdIceExplainer)

[17]:

{'id': 'h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer',
 'name': 'PdIceExplainer',
 'display_name': 'Partial Dependence Plot',
 'description': 'Partial dependence plot (PDP) portrays the average prediction\nbehavior of the model across the domain of an input variable along with +/- 1\nstandard deviation bands. Individual Conditional Expectations plot (ICE) displays\nthe prediction behavior for an individual row of data when an input variable is\ntoggled across its domain.\n\nPD binning:\n\n**Integer** feature:\n\n* bins in **numeric** mode:\n    * bins are integers\n    * (at most) `grid_resolution` integer values in between minimum and maximum\n      of feature values\n    * bin values are created as evenly as possible\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are integers\n    * top `grid_resolution` values from feature values ordered by frequency\n      (int values are converted to strings and most frequent values are used\n      as bins)\n* quantile bins in **numeric** mode:\n    * bins are integers\n    * bin values are created with the aim that there will be the same number of\n      observations per bin\n    * q-quantile used to created ``q`` bins where ``q`` is specified by PD parameter\n* quantile bins in **categorical** mode:\n    * not supported\n\n**Float** feature:\n\n* bins in **numeric** mode:\n    * bins are floats\n    * `grid_resolution` float values in between minimum and maximum of feature\n       values\n    * bin values are created as evenly as possible\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are floats\n    * top `grid_resolution` values from feature values ordered by frequency\n      (float values are converted to strings and most frequent values are used\n      as bins)\n* quantile bins in **numeric** mode:\n    * bins are floats\n    * bin values are created with the aim that there will be the same number of\n      observations per bin\n    * q-quantile used to created ``q`` bins where ``q`` is specified by PD parameter\n* quantile bins in **categorical** mode:\n    * not supported\n\n**String** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * bins are strings\n    * top `grid_resolution` values from feature values ordered by frequency\n* quantile bins:\n    * not supported\n\n**Date/datetime** feature:\n\n* bins in **numeric** mode:\n    * bins are dates\n    * `grid_resolution` date values in between minimum and maximum of feature\n      values\n    * bin values are created as evenly as possible:\n        1. dates are parsed and converted to epoch timestamps i.e integers\n        2. bins are created as in case of numeric integer binning\n        3. integer bins are converted back to original date format\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are dates\n    * top `grid_resolution` values from feature values ordered by frequency\n      (dates are handled as opaque strings and most frequent values are used\n      as bins)\n* quantile bins:\n    * not supported\n\nPD out of range binning:\n\n**Integer** feature:\n\n* OOR bins in **numeric** mode:\n    * OOR bins are integers\n    * (at most) `oor_grid_resolution` integer values are added below minimum and\n      above maximum\n    * bin values are created by adding/substracting rounded standard deviation\n      (of feature values) above and below maximum and minimum `oor_grid_resolution`\n      times\n        * 1 used used if rounded standard deviation would be 0\n    * if feature is of unsigned integer type, then bins below 0\n      are not created\n        * if rounded standard deviation and/or `oor_grid_resolution` is so high\n          that it would cause lower OOR bins to be negative numbers, then standard\n          deviation of size 1 is tried instead\n* OOR bins in **categorical** mode:\n    * same as numeric mode\n\n**Float** feature:\n\n* OOR bins in **numeric** mode:\n    * OOR bins are floats\n    * `oor_grid_resolution` float values are added below minimum and above maximum\n    * bin values are created by adding/substracting standard deviation\n      (of feature values) above and below maximum and minimum `oor_grid_resolution`\n      times\n* OOR bins in **categorical** mode:\n    * same as numeric mode\n\n**String** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * OOR bins are strings\n    * value `UNSEEN` is added as OOR bin\n\n**Date** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * OOR bins are strings\n    * value `UNSEEN` is added as OOR bin\n\n',
 'model_types': ['iid', 'time_series'],
 'can_explain': ['regression', 'binomial', 'multinomial'],
 'explanation_scopes': ['global_scope', 'local_scope'],
 'explanations': [{'explanation_type': 'global-partial-dependence',
   'name': 'PartialDependenceExplanation',
   'category': None,
   'scope': 'global',
   'has_local': None,
   'formats': []},
  {'explanation_type': 'local-individual-conditional-explanation',
   'name': 'IndividualConditionalExplanation',
   'category': None,
   'scope': 'local',
   'has_local': None,
   'formats': []}],
 'parameters': [{'name': 'sample_size',
   'description': 'Sample size for Partial Dependence Plot.',
   'comment': '',
   'type': 'int',
   'val': 25000,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'max_features',
   'description': 'Partial Dependence Plot number of features (to see all features used by model set to -1).',
   'comment': '',
   'type': 'int',
   'val': 10,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'features',
   'description': 'Partial Dependence Plot feature list.',
   'comment': '',
   'type': 'list',
   'val': None,
   'predefined': [],
   'tags': ['SOURCE_DATASET_COLUMN_NAMES'],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'oor_grid_resolution',
   'description': 'Partial Dependence Plot number of out of range bins.',
   'comment': '',
   'type': 'int',
   'val': 0,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'quantile-bin-grid-resolution',
   'description': 'Partial Dependence Plot quantile binning (total quantile points used to create bins).',
   'comment': '',
   'type': 'int',
   'val': 0,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'grid_resolution',
   'description': 'Partial Dependence Plot observations per bin (number of equally spaced points used to create bins).',
   'comment': '',
   'type': 'int',
   'val': 20,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'center',
   'description': 'Center Partial Dependence Plot using ICE centered at 0.',
   'comment': '',
   'type': 'bool',
   'val': False,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'sort_bins',
   'description': 'Ensure bin values sorting.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'histograms',
   'description': 'Enable histograms.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'quantile-bins',
   'description': 'Per-feature quantile binning (Example: if choosing features\n                 F1 and F2, this parameter is \'{"F1": 2,"F2": 5}\'. Note, you can\n                 set all features to use the same quantile binning with the\n                 `Partial Dependence Plot quantile binning` parameter and then\n                 adjust the quantile binning for a subset of PDP features with\n                 this parameter).',
   'comment': '',
   'type': 'str',
   'val': '',
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'numcat_num_chart',
   'description': 'Unique feature values count driven Partial Dependence Plot binning and chart selection.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'numcat_threshold',
   'description': 'Threshold for Partial Dependence Plot binning and chart selection (<=threshold categorical, >threshold numeric).',
   'comment': '',
   'type': 'int',
   'val': 11,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'debug_residuals',
   'description': 'Debug model residuals.',
   'comment': '',
   'type': 'bool',
   'val': False,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''}],
 'keywords': ['run-by-default',
  'can-add-feature',
  'explains-feature-behavior',
  'h2o-sonar']}

Interpretation

[18]:

# scikit-learn model
gradient_booster = GradientBoostingClassifier(learning_rate=0.1)
gradient_booster.fit(X, y)

# explainable model
model = ModelApi().create_model(target_col=target_col, model_src=gradient_booster, used_features=X.columns.to_list())

interpretation = interpret.run_interpretation(
    dataset=df,
    model=model,
    target_col=target_col,
    results_location=results_location,
    log_level=logging.INFO,
    explainers=[
        commons.ExplainerToRun(
            explainer_id=PdIceExplainer.explainer_id(),
            params="",
        )
    ]
)

2023-03-12 23:27:53,358 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 BEGIN calculation
2023-03-12 23:27:53,359 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 loading dataset
2023-03-12 23:27:53,360 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 loaded dataset has 10000 rows and 25 columns
2023-03-12 23:27:53,360 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 getting features list, importanceand metadata
2023-03-12 23:27:53,361 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 all most important model features: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
2023-03-12 23:27:53,362 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 features used by model: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
2023-03-12 23:27:53,362 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885: calculating PD for features ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4']
2023-03-12 23:27:53,363 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 feature metadata: {'id': [], 'categorical': [], 'numeric': [], 'catnum': [], 'date': [], 'time': [], 'datetime': [], 'text': [], 'image': [], 'date-format': [], 'quantile-bin': {}}
2023-03-12 23:27:53,363 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 1 frame strategy: True
2023-03-12 23:27:53,364 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 residual PD/ICE should NOT be calculated, but y has been specified - setting it None
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 10.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 20.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 20.0%

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 20.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 30.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 30.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 30.0%

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 40.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 40.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 50.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 50.0%

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 60.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 60.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 60.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 70.0%

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 70.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 70.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 80.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 80.0%

X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
X does not have valid feature names, but GradientBoostingClassifier was fitted with feature names
2023-03-12 23:27:55,425 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 saving PD to ../../results/h2o-sonar/mli_experiment_a30a4f57-9c78-4c5f-9c8e-85198b523c1c/explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-pd-dai-model.json
2023-03-12 23:27:55,428 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 computation finished & stored to: ../../results/h2o-sonar/mli_experiment_a30a4f57-9c78-4c5f-9c8e-85198b523c1c/explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-pd-dai-model.json
2023-03-12 23:27:55,431 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: ID/True
2023-03-12 23:27:55,441 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: ID/False
2023-03-12 23:27:55,452 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: LIMIT_BAL/True
2023-03-12 23:27:55,453 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: LIMIT_BAL/False
2023-03-12 23:27:55,456 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: SEX/True
2023-03-12 23:27:55,458 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: SEX/False
2023-03-12 23:27:55,462 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: EDUCATION/True
2023-03-12 23:27:55,468 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: EDUCATION/False
2023-03-12 23:27:55,472 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: MARRIAGE/True
2023-03-12 23:27:55,480 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: MARRIAGE/False
2023-03-12 23:27:55,483 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: AGE/True
2023-03-12 23:27:55,485 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: AGE/False
2023-03-12 23:27:55,488 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_0/True
2023-03-12 23:27:55,492 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_0/False
2023-03-12 23:27:55,502 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_2/True
2023-03-12 23:27:55,505 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_2/False
2023-03-12 23:27:55,516 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_3/True
2023-03-12 23:27:55,520 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_3/False
2023-03-12 23:27:55,532 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_4/True
2023-03-12 23:27:55,534 - h2o_sonar.explainers.pd_ice_explainer.PdIceExplainerLogger - INFO - PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 creating histogram: PAY_4/False

h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 90.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 90.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 90.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 90.0%
h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer: progress 100.0%

../_images/notebooks_h2o-sonar-pd-explainer_5_12.png

../_images/notebooks_h2o-sonar-pd-explainer_5_13.png

../_images/notebooks_h2o-sonar-pd-explainer_5_14.png

../_images/notebooks_h2o-sonar-pd-explainer_5_15.png

../_images/notebooks_h2o-sonar-pd-explainer_5_16.png

../_images/notebooks_h2o-sonar-pd-explainer_5_17.png

../_images/notebooks_h2o-sonar-pd-explainer_5_18.png

../_images/notebooks_h2o-sonar-pd-explainer_5_19.png

../_images/notebooks_h2o-sonar-pd-explainer_5_20.png

../_images/notebooks_h2o-sonar-pd-explainer_5_21.png

Explainer Result

[19]:

# retrieve the result
result = interpretation.get_explainer_result(PdIceExplainer.explainer_id())

[20]:

# open interpretation HTML report in web browser
webbrowser.open(interpretation.result.get_html_report_location())

[20]:

True

[21]:

# summary
result.summary()

[21]:

{'id': 'h2o_sonar.explainers.pd_ice_explainer.PdIceExplainer',
 'name': 'PdIceExplainer',
 'display_name': 'Partial Dependence Plot',
 'description': 'Partial dependence plot (PDP) portrays the average prediction\nbehavior of the model across the domain of an input variable along with +/- 1\nstandard deviation bands. Individual Conditional Expectations plot (ICE) displays\nthe prediction behavior for an individual row of data when an input variable is\ntoggled across its domain.\n\nPD binning:\n\n**Integer** feature:\n\n* bins in **numeric** mode:\n    * bins are integers\n    * (at most) `grid_resolution` integer values in between minimum and maximum\n      of feature values\n    * bin values are created as evenly as possible\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are integers\n    * top `grid_resolution` values from feature values ordered by frequency\n      (int values are converted to strings and most frequent values are used\n      as bins)\n* quantile bins in **numeric** mode:\n    * bins are integers\n    * bin values are created with the aim that there will be the same number of\n      observations per bin\n    * q-quantile used to created ``q`` bins where ``q`` is specified by PD parameter\n* quantile bins in **categorical** mode:\n    * not supported\n\n**Float** feature:\n\n* bins in **numeric** mode:\n    * bins are floats\n    * `grid_resolution` float values in between minimum and maximum of feature\n       values\n    * bin values are created as evenly as possible\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are floats\n    * top `grid_resolution` values from feature values ordered by frequency\n      (float values are converted to strings and most frequent values are used\n      as bins)\n* quantile bins in **numeric** mode:\n    * bins are floats\n    * bin values are created with the aim that there will be the same number of\n      observations per bin\n    * q-quantile used to created ``q`` bins where ``q`` is specified by PD parameter\n* quantile bins in **categorical** mode:\n    * not supported\n\n**String** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * bins are strings\n    * top `grid_resolution` values from feature values ordered by frequency\n* quantile bins:\n    * not supported\n\n**Date/datetime** feature:\n\n* bins in **numeric** mode:\n    * bins are dates\n    * `grid_resolution` date values in between minimum and maximum of feature\n      values\n    * bin values are created as evenly as possible:\n        1. dates are parsed and converted to epoch timestamps i.e integers\n        2. bins are created as in case of numeric integer binning\n        3. integer bins are converted back to original date format\n    * minimum and maximum is included in bins\n      (if `grid_resolution` is bigger or equal to 2)\n* bins in **categorical** mode:\n    * bins are dates\n    * top `grid_resolution` values from feature values ordered by frequency\n      (dates are handled as opaque strings and most frequent values are used\n      as bins)\n* quantile bins:\n    * not supported\n\nPD out of range binning:\n\n**Integer** feature:\n\n* OOR bins in **numeric** mode:\n    * OOR bins are integers\n    * (at most) `oor_grid_resolution` integer values are added below minimum and\n      above maximum\n    * bin values are created by adding/substracting rounded standard deviation\n      (of feature values) above and below maximum and minimum `oor_grid_resolution`\n      times\n        * 1 used used if rounded standard deviation would be 0\n    * if feature is of unsigned integer type, then bins below 0\n      are not created\n        * if rounded standard deviation and/or `oor_grid_resolution` is so high\n          that it would cause lower OOR bins to be negative numbers, then standard\n          deviation of size 1 is tried instead\n* OOR bins in **categorical** mode:\n    * same as numeric mode\n\n**Float** feature:\n\n* OOR bins in **numeric** mode:\n    * OOR bins are floats\n    * `oor_grid_resolution` float values are added below minimum and above maximum\n    * bin values are created by adding/substracting standard deviation\n      (of feature values) above and below maximum and minimum `oor_grid_resolution`\n      times\n* OOR bins in **categorical** mode:\n    * same as numeric mode\n\n**String** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * OOR bins are strings\n    * value `UNSEEN` is added as OOR bin\n\n**Date** feature:\n\n* bins in **numeric** mode:\n    * not supported\n* bins in **categorical** mode:\n    * OOR bins are strings\n    * value `UNSEEN` is added as OOR bin\n\n',
 'model_types': ['iid', 'time_series'],
 'can_explain': ['regression', 'binomial', 'multinomial'],
 'explanation_scopes': ['global_scope', 'local_scope'],
 'explanations': [{'explanation_type': 'global-partial-dependence',
   'name': 'Partial Dependence Plot (PDP)',
   'category': 'DAI MODEL',
   'scope': 'global',
   'has_local': 'local-individual-conditional-explanation',
   'formats': ['application/json']},
  {'explanation_type': 'local-individual-conditional-explanation',
   'name': 'Individual Conditional Expectations (ICE)',
   'category': 'DAI MODEL',
   'scope': 'local',
   'has_local': None,
   'formats': ['application/vnd.h2oai.json+datatable.jay']},
  {'explanation_type': 'global-html-fragment',
   'name': 'Partial Dependence Plot (PDP)',
   'category': 'DAI MODEL',
   'scope': 'global',
   'has_local': None,
   'formats': ['text/html']}],
 'parameters': [{'name': 'sample_size',
   'description': 'Sample size for Partial Dependence Plot.',
   'comment': '',
   'type': 'int',
   'val': 25000,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'max_features',
   'description': 'Partial Dependence Plot number of features (to see all features used by model set to -1).',
   'comment': '',
   'type': 'int',
   'val': 10,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'features',
   'description': 'Partial Dependence Plot feature list.',
   'comment': '',
   'type': 'list',
   'val': None,
   'predefined': [],
   'tags': ['SOURCE_DATASET_COLUMN_NAMES'],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'oor_grid_resolution',
   'description': 'Partial Dependence Plot number of out of range bins.',
   'comment': '',
   'type': 'int',
   'val': 0,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'quantile-bin-grid-resolution',
   'description': 'Partial Dependence Plot quantile binning (total quantile points used to create bins).',
   'comment': '',
   'type': 'int',
   'val': 0,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'grid_resolution',
   'description': 'Partial Dependence Plot observations per bin (number of equally spaced points used to create bins).',
   'comment': '',
   'type': 'int',
   'val': 20,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'center',
   'description': 'Center Partial Dependence Plot using ICE centered at 0.',
   'comment': '',
   'type': 'bool',
   'val': False,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'sort_bins',
   'description': 'Ensure bin values sorting.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'histograms',
   'description': 'Enable histograms.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'quantile-bins',
   'description': 'Per-feature quantile binning (Example: if choosing features\n                 F1 and F2, this parameter is \'{"F1": 2,"F2": 5}\'. Note, you can\n                 set all features to use the same quantile binning with the\n                 `Partial Dependence Plot quantile binning` parameter and then\n                 adjust the quantile binning for a subset of PDP features with\n                 this parameter).',
   'comment': '',
   'type': 'str',
   'val': '',
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'numcat_num_chart',
   'description': 'Unique feature values count driven Partial Dependence Plot binning and chart selection.',
   'comment': '',
   'type': 'bool',
   'val': True,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'numcat_threshold',
   'description': 'Threshold for Partial Dependence Plot binning and chart selection (<=threshold categorical, >threshold numeric).',
   'comment': '',
   'type': 'int',
   'val': 11,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''},
  {'name': 'debug_residuals',
   'description': 'Debug model residuals.',
   'comment': '',
   'type': 'bool',
   'val': False,
   'predefined': [],
   'tags': [],
   'min_': 0.0,
   'max_': 0.0,
   'category': ''}],
 'keywords': ['run-by-default',
  'can-add-feature',
  'explains-feature-behavior',
  'h2o-sonar']}

[22]:

# parameters
result.params()

[22]:

{'sample_size': 25000,
 'max_features': 10,
 'features': None,
 'oor_grid_resolution': 0,
 'quantile-bin-grid-resolution': 0,
 'grid_resolution': 20,
 'center': False,
 'sort_bins': True,
 'histograms': True,
 'quantile-bins': '',
 'numcat_num_chart': True,
 'numcat_threshold': 11,
 'debug_residuals': False}

Display PD Data

[23]:

result.data(feature_name="EDUCATION")

[23]:

	bin	pd	sd	oor
	▪▪▪▪	▪▪▪▪▪▪▪▪	▪▪▪▪▪▪▪▪	▪
0	0	0.7639	0.424706	0
1	1	0.7248	0.446637	0
2	2	0.7248	0.446637	0
3	3	0.7248	0.446637	0
4	4	0.7248	0.446637	0
5	5	0.7248	0.446637	0
6	6	0.7248	0.446637	0

[24]:

result.data(feature_name="PAY_3")

[24]:

	bin	pd	sd	oor
	▪▪▪▪	▪▪▪▪▪▪▪▪	▪▪▪▪▪▪▪▪	▪
0	−2	0.7248	0.446637	0
1	−1	0.7248	0.446637	0
2	0	0.7248	0.446637	0
3	1	0.7248	0.446637	0
4	2	0.7248	0.446637	0
5	3	0.7248	0.446637	0
6	4	0.7248	0.446637	0
7	5	0.7248	0.446637	0
8	6	0.7248	0.446637	0
9	7	0.7248	0.446637	0
10	8	0.7248	0.446637	0

Plot PD Data

[25]:

result.plot(feature_name="EDUCATION", override_feature_type=result.format.KEY_CATEGORICAL)

../_images/notebooks_h2o-sonar-pd-explainer_15_0.png

[26]:

result.plot(feature_name="PAY_3")

../_images/notebooks_h2o-sonar-pd-explainer_16_0.png

Save Explainer Log and Data

[27]:

# save the explainer log
result.log(path="./pd-ice-demo.log")

[28]:

!head pd-ice-demo.log

2023-03-12 23:27:53,358 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 BEGIN calculation
2023-03-12 23:27:53,359 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 loading dataset
2023-03-12 23:27:53,360 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 loaded dataset has 10000 rows and 25 columns
2023-03-12 23:27:53,360 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 getting features list, importanceand metadata
2023-03-12 23:27:53,361 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 all most important model features: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
2023-03-12 23:27:53,362 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 features used by model: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
2023-03-12 23:27:53,362 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885: calculating PD for features ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4']
2023-03-12 23:27:53,363 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 feature metadata: {'id': [], 'categorical': [], 'numeric': [], 'catnum': [], 'date': [], 'time': [], 'datetime': [], 'text': [], 'image': [], 'date-format': [], 'quantile-bin': {}}
2023-03-12 23:27:53,363 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 1 frame strategy: True
2023-03-12 23:27:53,364 INFO PD/ICE a30a4f57-9c78-4c5f-9c8e-85198b523c1c/745668cc-bbc8-42be-9b39-47487112a885 residual PD/ICE should NOT be calculated, but y has been specified - setting it None

[29]:

# save the explainer data
result.zip(file_path="./pd-ice-demo-archive.zip")

[30]:

!unzip -l pd-ice-demo-archive.zip

Archive:  pd-ice-demo-archive.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    11434  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/result_descriptor.json
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-8.jay
   160272  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-3.jay
    13658  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-pd-dai-model.json
  1601936  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-1.jay
     2349  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model.json
   320440  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-5.jay
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-9.jay
   560688  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-4.jay
  1601944  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-2.jay
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-10.jay
    80184  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/mli_dataset_y_hat.jay
  1601784  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-6.jay
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/work/h2o_sonar-ice-dai-model-7.jay
      110  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html.meta
    21396  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-2-class-0.png
    12565  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-5-class-0.png
    14256  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-1-class-0.png
    12616  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-8-class-0.png
    19038  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-3-class-0.png
     4027  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/explanation.html
    14358  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-0-class-0.png
    12571  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-7-class-0.png
    13577  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-6-class-0.png
    18249  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-9-class-0.png
    13688  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_html_fragment/text_html/pd-feature-4-class-0.png
      165  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay.meta
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_8_class_0.jay
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_6_class_0.jay
  1601936  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_0_class_0.jay
  1601944  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_1_class_0.jay
  1601784  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_5_class_0.jay
   560688  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_3_class_0.jay
   160272  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_2_class_0.jay
   320440  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_4_class_0.jay
     7096  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/explanation.json
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_9_class_0.jay
    80184  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/y_hat.jay
   881024  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/local_individual_conditional_explanation/application_vnd_h2oai_json_datatable_jay/ice_feature_7_class_0.jay
      151  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json.meta
      924  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_3_class_0.json
     2751  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_1_class_0.json
     1412  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_9_class_0.json
     2660  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_0_class_0.json
      563  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_4_class_0.json
     1414  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_6_class_0.json
     2557  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_5_class_0.json
     1413  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_8_class_0.json
     1413  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_7_class_0.json
     7156  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/explanation.json
      322  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/global_partial_dependence/application_json/pd_feature_2_class_0.json
        2  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/model_problems/problems_and_actions.json
     5746  2023-03-12 23:27   explainer_h2o_sonar_explainers_pd_ice_explainer_PdIceExplainer_745668cc-bbc8-42be-9b39-47487112a885/log/explainer_run_745668cc-bbc8-42be-9b39-47487112a885.log
---------                     -------
 19122325                     53 files

[ ]: