Skip to content

Metrics: Size Dependency

H2O Model Validation offers an array of metrics in the form of a plot and heatmap to understand a Size Dependency test. Below, each metric is described in turn.

Plot: Modeling Scores for Different Training Data Sizes (MSDTDS)

The MSDTDS plot visualizes several scorer values for models (identical models) trained on different training data sizes. Observing the MSDTDS plot can enable you to understand and find an acceptable train dataset size for your model.

Plot details:

  • Y-Axis: The Y-Axis refers to scorer values
  • X-Axis: The X-Axis refers to the size of the training data
  • Yellow line: The yellow line refers to the validation data of the model
  • Red line: The red line refers to the test data of the model
  • Red or yellow dots: The red and yellow dots refer to a child model or the parent model.
    • Child model: A child model refers to a model trained on one of the sub-training samples obtained from the original training data

    • Parent model: The parent model refers to a trained model with the original training data and not with one of the sub-training samples.

      Note

      To learn more about original training data and sub-training samples, see Size Dependency.

Scores for models in the MSDTDS plot are positioned from left to right. The first model (also known as a child experiment) represents the experiment with the smallest sub-training sample obtained from the original train data. The model furthest to the right, known as the parent experiment, represents the original experiment (model) that used the whole original train data. The model right before the parent experiment represents the model with the highest sub-training sample.

Example

Considering the below two graphs (figures) of a model and that a low RMSE score indicates a better model performance, we can conclude that as the model increases its training data size, the RMSE score will decrease while improving the model performance. When comparing the first child experiment with the parent experiment, we can see that by increasing the training data size by 144.0.66k, we will obtain a 0.48 RMSE score. A lower RMSE when compared to the one the first child experiment received.

Go to Instance

Figure 1: Child experiment

Go to Instance

Figure 2: Parent experiment

Heatmap: Feature Importance for Different Training Data Sizes (FIDTDS)

The FIDTDS heatmap visualizes the most important features for different models trained on different sizes of train data during the retraining process of a Driverless AI experiment.

Example

Based on the below heatmap, we can see that throughout the retraining process of a Driverless AI experiment with different sizes of train data, the variable customers was kept as the most important feature, followed by the variable open.

Go to Instance