Skip to main content

DAI Engine Examples

This guide will walk you through the Driverless AI Engine client usage in a few examples.

Advanced users and app developers

See the module documentation for full client reference.

Starting a Driverless AI engine​

import h2o_engine_manager

# Initialize the client for Driverless AI engine.
dai_engine_client = h2o_engine_manager.login().dai_engine_client

# Create the Driverless AI engine using the default configuration.
# This call returns immediately, and does not wait for the engine to be ready.
engine = dai_engine_client.create_engine(
    display_name="My Basic DAI Engine"
)

# This call blocks until the engine is ready.
# It may take a few minutes.
engine.wait()

# Now DAI is ready to be connected to.
# This returns an initialized driverlessai.Client
dai = engine.connect()

# Import a dataset
ds = dai.datasets.create(
    data='s3://h2o-public-test-data/smalldata/iris/iris.csv',
    data_source='s3'
)

# Print column summary for the first three columns
print(ds.column_summaries()[0:3])

Advanced Driverless AI engine​

import h2o_engine_manager

# Initialize the client for Driverless AI engine.
dai_engine_client = h2o_engine_manager.login().dai_engine_client

# Set the name of the workspace that will contain this engine.
workspace_id = "default"
# Set the name of the engine.
engine_id = "my-engine"

# Create the Driverless AI engine.
# This call returns immediately, and does not wait for the engine to be ready.
engine = dai_engine_client.create_engine(
    workspace_id=workspace_id,
    engine_id=engine_id,
    display_name="My Advanced DAI Engine",
    version="1.10.5",
    cpu=4,
    gpu=1,
    memory_bytes="12Gi",
    storage_bytes="64Gi",
    max_idle_duration="3000s",
    max_running_duration="5000s",
    config={"max_runtime_minutes": "120", "feature_engineering_effort": "5"}
)

# This call blocks until the engine is ready.
# It may take a few minutes.
engine.wait()

# Now DAI is ready to be connected to.
# This returns an initialized driverlessai.Client
dai = engine.connect()

Fetching a Driverless AI engine​

This fetches an existing engine from the previous example.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

engine = dai_engine_client.get_engine(
    workspace_id="default",
    engine_id="my-engine"
)

print(engine)

Listing Driverless AI engines​

The list_engine method supports pagination, filtering and sorting. Following example shows basic listing usages.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

# Specify a workspace in which you want to list engines.
workspace_id = "default"

# By default, engines are ordered by their time of creation in descending order
# and limited to 50 items per page.
list1 = dai_engine_client.list_engines(workspace_id=workspace_id)

# To list a second page, pass the next_page_token from the previous page.
# This example increases the page size to 100.
list2 = dai_engine_client.list_engines(
    workspace_id=workspace_id,
    page_size=100,
    page_token=list1.next_page_token
)

# Engine ordering works on any documented field. This example orders engines
# by the most recently started first and the amount of CPU units second.
list3 = dai_engine_client.list_engines(
    workspace_id=workspace_id,
    order_by="resume_time desc, cpu desc"
)

# Engine filtering works on any documented field. This example filters engines
# only running engines with 5 CPU units or more.
list4 = dai_engine_client.list_engines(
    workspace_id=workspace_id,
    filter="state = STATE_RUNNING AND cpu <= 5"
)
info

A convenience function list_all_engines is provided that pages through all engines and returns a complete list. Although it is strongly recommended to use pagination, this function is useful for quick prototyping.

Pausing a Driverless AI engine​

You should pause your engines when you are not using them to save costs. Pausing terminates all running jobs but all other state is persisted.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

engine = dai_engine_client.get_engine(
    workspace_id="default",
    engine_id="my-engine"
)

engine.pause()

# You may wait for the engine to finish pausing.
engine.wait()
info

An engine can be automatically paused after exceeding the idle or uptime limit.

Resuming a Driverless AI engine​

A paused engine can be resumed to continue using it.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

engine = dai_engine_client.get_engine(
    workspace_id="default",
    engine_id="my-engine"
)

# This return immediately and does not wait for the engine to be ready.
engine.resume()

# This call blocks until the engine is ready.
# It may take a few minutes.
engine.wait()

dai = engine.connect()
# ...

Updating a Driverless AI engine​

You can modify an existing engine with the update method. The engine is only updatable when paused.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

# Fetch the engine first.
engine = dai_engine_client.get_engine(
    workspace_id="default",
    engine_id="my-engine"
)

# Update the engine fields directly.
engine.cpu = 2

# Persist changes to the engine.
engine.update()

Deleting a Driverless AI engine​

Engine deletion is a destructive operation that immediately terminates the engine and deletes all data contained within. It cannot be undone.

import h2o_engine_manager

dai_engine_client = h2o_engine_manager.login().dai_engine_client

engine = dai_engine_client.get_engine(
    workspace_id="default",
    engine_id="my-engine"
)

engine.delete()

# You may wait for the engine to finish deleting.
engine.wait()

Asynchronous waiting​

The client provides a way to concurrently wait for an engine using the async/await syntax. While within an async function context, you can take advantage of asyncio wait:

engine = dai_engine_client.create_engine(...)

# This method utilizes asyncio.sleep() to asynchronously poll engine status.
await engine.wait_async()

Eventual consistency​

The API is eventually consistent, meaning a that when reading immediately after a write operations may not reflect the written changes immediately. This client is designed to eliminate the hurdles of eventual consistency if used correctly. Always use the object returned by write operations.

Incorrect - Error not found
dai_engine_client.create_engine(...)
engine = dai_engine_client.get_engine(...)
Correct
engine = dai_engine_client.create_engine(...)

Example notebook​

You can find a Jupyter notebook with examples here.

Feedback