Connector service data sources

This page describes the configuration parameters for each supported data source type. Connector and Worker resources are provisioned by a Workspace Administrator using these parameters; Data Scientists and Data Engineers then open connections and stream data using the provisioned connector and worker references.

Note

The data sources available in your environment depend on which Connector Templates your Platform Administrator has published. Contact your Platform Administrator if a data source type you need is not listed.

The examples on this page use a client created as shown in Connector service Python client.

PostgreSQL

Stream tabular data from a PostgreSQL database using SQL queries or table extraction.

Connector configuration

Parameter	Description
PGHOST	Database host (required)
PGPORT	Database port (default: 5432)
PGDATABASE	Database name (required)
PGUSER	Database user (required)
PGPASSWORD	Database password (use Secure Store instead of inline values)
PGSCHEMA	PostgreSQL schema (default: public)
PGSSLMODE	SSL mode: disable, require, verify-ca, or verify-full (default: disable)

Streaming a query

import os

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"query": "SELECT id, name, amount FROM orders WHERE status = 'completed'"},
) as session:
    session.stream_to_csv("orders.csv")

Supported data types

PostgreSQL types are mapped to the following column types in the stream schema:

PostgreSQL Type	Output Type
boolean	Boolean
smallint	Int16
integer	Int32
bigint	Int64
real	Float32
double precision	Float64
numeric / decimal	Decimal (sent as a string to preserve precision)
text, varchar, char, name	String
bytea	Binary
date	Date
time	Time
timetz	String
timestamp	Timestamp
timestamptz	Timestamp with time zone
uuid	UUID (string value)
json	JSON (string value)
jsonb	JSONB (string value)
interval	Interval (string value)
Arrays of integer, float, boolean, text, or uuid	JSON (JSON-encoded array)
Other arrays	String (PostgreSQL array literal, for example {a,b,c})
NULL	Null

---

Snowflake

Stream tabular data from a Snowflake data warehouse using SQL queries or table extraction.

Connector configuration

Parameter	Description
SNOWFLAKE_ACCOUNT	Snowflake account identifier (required)
SNOWFLAKE_USER	Snowflake user
AUTH_TYPE	Authentication method: password or keypair (required)
SNOWFLAKE_PASSWORD	Snowflake password, for AUTH_TYPE=password (use Secure Store instead of inline values)
SNOWFLAKE_PRIVATE_KEY	PEM-encoded, unencrypted PKCS#8 private key, for AUTH_TYPE=keypair (use Secure Store instead of inline values)
SNOWFLAKE_DATABASE	Database name (required)
SNOWFLAKE_SCHEMA	Schema name (required)
SNOWFLAKE_WAREHOUSE	Compute warehouse to use
SNOWFLAKE_ROLE	Optional role to assume

Note

Snowflake requires PKCS#8 keys. If your key starts with -----BEGIN RSA PRIVATE KEY-----, it is in PKCS#1 format. Convert it with openssl pkcs8 -topk8 -nocrypt -in key.pem -out key-pkcs8.pem.

Streaming a query

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"query": "SELECT * FROM SALES_DATA WHERE REGION = 'EMEA'"},
) as session:
    # Stream to a pandas DataFrame
    df = session.stream_to_pandas()
    print(f"Loaded DataFrame with shape {df.shape}")

# To write Parquet instead, open a new session and call
# session.stream_to_parquet("sales_data.parquet")

Note

The Snowflake user and role must have USAGE privileges on the database, schema, and warehouse specified in the connector configuration, and SELECT on the tables you query. Contact your Workspace Administrator if you receive authorization errors.

---

Amazon S3

Download files from an S3 bucket. Supports both single-object downloads and prefix-based batch downloads.

Connector configuration

Parameter	Description
PROVIDER	Set to s3
AWS_BUCKET	S3 bucket name (required)
AWS_REGION	AWS region (required)
AWS_ACCESS_KEY_ID	AWS access key ID (use Secure Store instead of inline values)
AWS_SECRET_ACCESS_KEY	AWS secret access key (use Secure Store instead of inline values)
AWS_SESSION_TOKEN	Optional STS session token for temporary credentials

If you do not set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the worker uses the default AWS credential chain. For example, it can use an IAM role for service accounts (IRSA) that is attached to the worker.

Downloading a single file

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"paths": {"paths": [{"pattern": "data/exports/report.csv"}]}},
) as session:
    n_files, total_bytes = session.stream_to_files("downloads/")
    print(f"Downloaded {n_files} file(s), {total_bytes} bytes total")

Downloading files by prefix

Use a pattern that ends in / to download all objects under that path. A pattern without a trailing / or * is treated as a single object key.

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"paths": {"paths": [{"pattern": "exports/2026/q1/"}]}},
) as session:
    n_files, total_bytes = session.stream_to_files("downloads/")

Note

Each file is written under the output directory using its full object key. For example, exports/2026/q1/report.csv is written to downloads/exports/2026/q1/report.csv.

Note

The IAM identity used by the connector requires s3:GetObject to download a single object, and s3:GetObject plus s3:ListBucket to download by prefix.

---

MinIO (S3-compatible)

MinIO and other S3-compatible object stores are supported using the same blob worker with a custom endpoint.

Connector configuration

Parameter	Description
PROVIDER	Set to minio
AWS_BUCKET	Bucket name (required)
AWS_ENDPOINT_URL	MinIO endpoint URL (required)
AWS_ACCESS_KEY_ID	Access key ID (required; use Secure Store instead of inline values)
AWS_SECRET_ACCESS_KEY	Secret access key (required; use Secure Store instead of inline values)
AWS_DISABLE_HTTPS	Set to true for plaintext (non-TLS) endpoints

The worker always uses path-style addressing for MinIO.

Downloading files

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"paths": {"paths": [{"pattern": "my-prefix/"}]}},
) as session:
    n_files, total_bytes = session.stream_to_files("downloads/")

---

Google Cloud Storage

Download files from a GCS bucket.

Connector configuration

Parameter	Description
PROVIDER	Set to gcs
GCS_BUCKET	GCS bucket name (required)
GOOGLE_APPLICATION_CREDENTIALS	Optional path to a service account key file inside the worker pod

If you do not set GOOGLE_APPLICATION_CREDENTIALS, the worker uses Application Default Credentials, such as GKE Workload Identity.

Note

GOOGLE_APPLICATION_CREDENTIALS is a file path, not the key itself. In this release, you cannot store the service account key in Secure Store for this data source. Use one of these options instead:

• Workload Identity (recommended): leave GOOGLE_APPLICATION_CREDENTIALS unset. The worker uses the Google service account bound to the worker pod's Kubernetes service account. No key is stored.
• Key file from a Kubernetes Secret: an administrator stores the key in a Kubernetes Secret and mounts it into the worker container through the Worker's pod template. Set GOOGLE_APPLICATION_CREDENTIALS to the mounted file path, for example /var/secrets/gcs-key.json.

---

Azure Blob Storage

Download files from an Azure Blob Storage container.

Connector configuration

Parameter	Description
PROVIDER	Set to azblob
AZURE_STORAGE_ACCOUNT	Storage account name (required)
AZURE_STORAGE_CONTAINER	Container name (required)
AZURE_STORAGE_KEY	Storage account key (use Secure Store instead of inline values)

If you do not set AZURE_STORAGE_KEY, the worker uses Azure default credentials, such as a managed identity.

Note

The data source type for Azure Blob Storage is azure-blob. The PROVIDER value is azblob.

---

Troubleshooting

Symptom	Cause	Resolution
Authorization error on Snowflake	The user or role lacks USAGE on the database, schema, or warehouse, or SELECT on the table	Ask your Workspace Administrator to grant the required Snowflake privileges and update the connector's SNOWFLAKE_ROLE
AUTH_TYPE is required on Snowflake	The connector configuration does not set AUTH_TYPE	Set AUTH_TYPE to password or keypair
AccessDenied on S3 prefix download	The IAM identity lacks s3:ListBucket	Prefix downloads require bucket-level list permission; single-key downloads need only s3:GetObject
AccessDenied on S3 single-key download of a missing object	Without s3:ListBucket, S3 reports a missing key as AccessDenied, not NoSuchKey	Check that the object key is correct
[INTERNAL] internal server error when downloading a single object	The object key does not exist or has a leading slash. The client does not show a "not found" error for a missing object.	Check the key. S3 keys do not start with /: use exports/file.csv, not /exports/file.csv
A prefix download returns 0 files	No object matches the prefix, or the prefix has a leading slash	Check the prefix. A prefix that matches nothing is not an error
Connection fails immediately with credentials error	The password or key in the connector configuration is wrong or expired	Ask your Workspace Administrator to update the connector credentials
No rows returned from a query	The query returned zero results	Verify the query filter conditions and that the target table has data

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O AI Cloud | Docs to cloud-feedback@h2o.ai