H2O Eval Studio Python API

H2O Eval Studio Python API reference.

Python API

• h2o_sonar package
  • Subpackages
    • h2o_sonar.explainers package
      • Submodules
        • h2o_sonar.explainers.adversarial_similarity_explainer module
          • AdversarialSimilarityExplainer
        • h2o_sonar.explainers.backtesting_explainer module
          • BacktestingExplainer
        • h2o_sonar.explainers.calibration_score_explainer module
          • CalibrationScoreExplainer
        • h2o_sonar.explainers.dataset_and_model_insights_explainer module
          • DatasetAndModelInsightsExplainer
        • h2o_sonar.explainers.dia_explainer module
          • DiaArgs
          • DiaExplainer
        • h2o_sonar.explainers.drift_explainer module
          • DriftDetectionExplainer
        • h2o_sonar.explainers.dt_surrogate_explainer module
          • DecisionTreeConstants
          • DecisionTreeSurrogateExplainer
        • h2o_sonar.explainers.fi_kernel_shap_explainer module
          • KernelShapFeatureImportanceExplainer
        • h2o_sonar.explainers.fi_naive_shapley_explainer module
          • NaiveShapleyMojoFeatureImportanceExplainer
        • h2o_sonar.explainers.friedman_h_statistic_explainer module
        • h2o_sonar.explainers.morris_sa_explainer module
          • MorrisSensitivityAnalysisExplainer
        • h2o_sonar.explainers.pd_2_features_explainer module
          • PdFor2FeaturesArgs
          • PdFor2FeaturesExplainer
        • h2o_sonar.explainers.pd_ice_explainer module
          • PdIceArgs
          • PdIceExplainer
        • h2o_sonar.explainers.residual_dt_surrogate_explainer module
          • ResidualDecisionTreeSurrogateExplainer
        • h2o_sonar.explainers.residual_pd_ice_explainer module
          • ResidualPdIceExplainer
        • h2o_sonar.explainers.segment_performance_explainer module
          • SegmentPerformanceExplainer
        • h2o_sonar.explainers.size_dependency_explainer module
          • SizeDependencyExplainer
        • h2o_sonar.explainers.summary_shap_explainer module
          • PdIceArgs
          • PdIceExplainer
        • h2o_sonar.explainers.transformed_fi_shapley_explainer module
          • ShapleyMojoTransformedFeatureImportanceExplainer
      • Module contents
    • h2o_sonar.evaluators package
      • Submodules
        • h2o_sonar.evaluators.abc_byop_evaluator module
          • AbcByopEvaluator
        • h2o_sonar.evaluators.bleu_evaluator module
          • BleuEvaluator
        • h2o_sonar.evaluators.classification_evaluator module
          • ClassificationEvaluator
        • h2o_sonar.evaluators.contact_information_byop_evaluator module
          • ContactInformationByopEvaluator
        • h2o_sonar.evaluators.fairness_bias_evaluator module
          • FairnessBiasEvaluator
        • h2o_sonar.evaluators.gptscore_evaluator module
          • GptScoreEvaluator
        • h2o_sonar.evaluators.gptscore_machine_translation_evaluator module
          • GptScoreMachineTranslationEvaluator
        • h2o_sonar.evaluators.gptscore_question_answering_evaluator module
          • GptScoreQuestionAnsweringEvaluator
        • h2o_sonar.evaluators.gptscore_summary_without_reference_evaluator module
          • GptScoreSummaryWithoutReferenceEvaluator
        • h2o_sonar.evaluators.gptscore_summary_with_reference_evaluator module
          • GptScoreSummaryWithReferenceEvaluator
        • h2o_sonar.evaluators.language_mismatch_byop_evaluator module
          • LanguageMismatchByopEvaluator
        • h2o_sonar.evaluators.parameterizable_byop_evaluator module
          • ParameterizableByopEvaluator
        • h2o_sonar.evaluators.perplexity_evaluator module
          • PerplexityEvaluator
        • h2o_sonar.evaluators.pii_leakage_evaluator module
          • PiiLeakageEvaluator
        • h2o_sonar.evaluators.rag_answer_correctness_evaluator module
          • AnswerCorrectnessEvaluator
        • h2o_sonar.evaluators.rag_answer_relevancy_evaluator module
          • AnswerRelevancyEvaluator
        • h2o_sonar.evaluators.rag_answer_relevancy_no_judge_evaluator module
          • RagAnswerRelevancyNoJudgeEvaluator
        • h2o_sonar.evaluators.rag_answer_similarity_evaluator module
          • AnswerSemanticSimilarityEvaluator
        • h2o_sonar.evaluators.rag_chunk_relevancy_evaluator module
          • ContextChunkRelevancyEvaluator
        • h2o_sonar.evaluators.rag_context_precision_evaluator module
          • ContextPrecisionEvaluator
        • h2o_sonar.evaluators.rag_context_recall_evaluator module
          • ContextRecallEvaluator
        • h2o_sonar.evaluators.rag_context_relevancy_evaluator module
          • ContextRelevancyEvaluator
        • h2o_sonar.evaluators.rag_faithfulness_evaluator module
          • FaithfulnessEvaluator
        • h2o_sonar.evaluators.rag_groundedness_evaluator module
          • RagGroundednessEvaluator
        • h2o_sonar.evaluators.rag_hallucination_evaluator module
          • RagHallucinationEvaluator
        • h2o_sonar.evaluators.rag_ragas_evaluator module
          • RagasEvaluator
        • h2o_sonar.evaluators.rag_tokens_presence_evaluator module
          • ConditionEvaluator
          • RagStrStrEvaluator
          • constraints_to_condition()
        • h2o_sonar.evaluators.rouge_evaluator module
          • RougeEvaluator
        • h2o_sonar.evaluators.sensitive_data_leakage_evaluator module
          • SensitiveDataLeakageEvaluator
        • h2o_sonar.evaluators.sexism_byop_evaluator module
          • SexismByopEvaluator
        • h2o_sonar.evaluators.stereotype_byop_evaluator module
          • StereotypeByopEvaluator
        • h2o_sonar.evaluators.summarization_byop_evaluator module
          • SummarizationByopEvaluator
        • h2o_sonar.evaluators.summarization_evaluator module
          • SummarizationEvaluator
          • load_summac()
          • pairwise_distances_wrapper()
          • segment_calc()
        • h2o_sonar.evaluators.toxicity_evaluator module
          • ToxicityEvaluator
      • Module contents
    • h2o_sonar.lib package
      • Subpackages
        • h2o_sonar.lib.api package
          • Submodules
          • h2o_sonar.lib.api.commons module
          • h2o_sonar.lib.api.datasets module
          • h2o_sonar.lib.api.explainers module
          • h2o_sonar.lib.api.explanations module
          • h2o_sonar.lib.api.formats module
          • h2o_sonar.lib.api.interpretations module
          • h2o_sonar.lib.api.judges module
          • h2o_sonar.lib.api.models module
          • h2o_sonar.lib.api.persistences module
          • h2o_sonar.lib.api.plots module
          • h2o_sonar.lib.api.problems module
          • h2o_sonar.lib.api.results module
          • Module contents
        • h2o_sonar.lib.container package
          • Submodules
          • h2o_sonar.lib.container.explainer_container module
          • Module contents
        • h2o_sonar.lib.integrations package
          • Submodules
          • h2o_sonar.lib.integrations.genai module
          • h2o_sonar.lib.integrations.mv_adapter module
          • h2o_sonar.lib.integrations.ragas_adapter module
          • Module contents
      • Module contents
    • h2o_sonar.methods package
      • Subpackages
        • h2o_sonar.methods.core package
          • Submodules
          • h2o_sonar.methods.core.method module
          • h2o_sonar.methods.core.stats module
          • Module contents
        • h2o_sonar.methods.utils package
          • Submodules
          • h2o_sonar.methods.utils.fairness_utils module
          • h2o_sonar.methods.utils.h2o_utils module
          • h2o_sonar.methods.utils.histogram module
          • Module contents
      • Module contents
    • h2o_sonar.utils package
      • Submodules
        • h2o_sonar.utils.binning module
          • build_qtile_bins()
          • qbin_column()
          • quantile_bin()
        • h2o_sonar.utils.caching module
          • cache_all_models()
          • cache_baai_bge_small_en()
          • cache_baai_bge_small_env15()
          • cache_bert_base_uncased()
          • cache_bge_m3()
          • cache_detoxify_models()
          • cache_eval_studio_models()
          • cache_gptscore_evaluator_model()
          • cache_hkunlp_instructor()
          • cache_lmppl_perplexity_evaluator_model()
          • cache_nltk()
          • cache_nltk_averaged_perceptron_tagger()
          • cache_nltk_punkt()
          • cache_nltk_wordnet()
          • cache_summac_vitc()
          • cache_tiktoken_blobs()
          • cache_vectara_hallucination_model()
        • h2o_sonar.utils.crypto module
          • decrypt()
          • encrypt()
          • resolve_encryption_key()
        • h2o_sonar.utils.io module
          • from_list_explainers_args_json()
          • from_run_interpretation_args_json()
          • load_list_explainers_args_json()
          • load_run_interpretation_args_json()
          • to_list_explainers_args_json()
          • to_run_interpretation_args_json()
        • h2o_sonar.utils.normalization module
          • normalize_importance()
        • h2o_sonar.utils.preprocessing module
          • MultiColumnLabelEncoder
          • categorical_encoder()
        • h2o_sonar.utils.problem_detection module
          • get_feature_importance_problems()
        • h2o_sonar.utils.perturbations module
          • AbcSynAntPerturbator
          • AntonymPerturbator
          • CommaPerturbator
          • ContextualMisinformationPerturbator
          • EncodingPerturbator
          • EncodingPerturbatorBase16
          • KeywordTyposCharacterPerturbator
          • Perturbator
          • PerturbatorDescriptor
          • PerturbatorRegistry
          • QwertyPerturbator
          • RandomCharacterDeletePerturbator
          • RandomCharacterInsertPerturbator
          • RandomCharacterReplacementPerturbator
          • RandomOCRCharacterPerturbator
          • SynonymPerturbator
          • WordSwapPerturbator
          • register_ootb_perturbators()
        • h2o_sonar.utils.sampling module
          • DatasetSampler
          • HeadOfDatasetSampling
          • NoDatasetSampling
          • RandomPandasDatasetSampling
          • StratifiedDatasetSampling
          • downsample_dataset()
        • h2o_sonar.utils.sanitization module
          • DriverlessAiSanitizationMap
          • SanitizationMap
          • sanitize_frame()
          • sanitize_markdown()
          • sanitize_names()
          • sanitize_strings()
        • h2o_sonar.utils.testing module
          • InMemoryLlmHostPromptCache
          • LlmHostPromptCache
          • RagTestCaseConfig
          • RagTestConfig
          • RagTestLab
          • RagTestLabPromptCache
          • RagTestSuiteConfig
          • TestLab
      • Module contents
  • Submodules
    • h2o_sonar.config module
      • ConfigItemType
        • ConfigItemType.CONNECTION
        • ConfigItemType.LICENSE
      • ConfigKeys
        • ConfigKeys.KEY_DESCRIPTION
        • ConfigKeys.KEY_ENCRYPTED
        • ConfigKeys.KEY_NAME
      • ConnectionConfig
        • ConnectionConfig.ENCRYPTED_FIELDS
        • ConnectionConfig.KEY_AUTH_SERVER_URL
        • ConnectionConfig.KEY_CLIENT_ID
        • ConnectionConfig.KEY_ENV_URL
        • ConnectionConfig.KEY_EXTRA_PARAMS
        • ConnectionConfig.KEY_KEY
        • ConnectionConfig.KEY_PASSWORD
        • ConnectionConfig.KEY_REALM_NAME
        • ConnectionConfig.KEY_SERVER_ID
        • ConnectionConfig.KEY_SERVER_URL
        • ConnectionConfig.KEY_TOKEN
        • ConnectionConfig.KEY_TOKEN_USE_TYPE
        • ConnectionConfig.KEY_TYPE
        • ConnectionConfig.KEY_USERNAME
        • ConnectionConfig.from_dict()
        • ConnectionConfig.to_dict()
      • ConnectionConfigType
        • ConnectionConfigType.AMAZON_BEDROCK
        • ConnectionConfigType.AZURE_OPENAI_CHAT
        • ConnectionConfigType.DRIVERLESS_AI
        • ConnectionConfigType.DRIVERLESS_AI_AIEM
        • ConnectionConfigType.DRIVERLESS_AI_STEAM
        • ConnectionConfigType.H2O_3
        • ConnectionConfigType.H2O_GPT
        • ConnectionConfigType.H2O_GPT_E
        • ConnectionConfigType.H2O_LLM_OPS
        • ConnectionConfigType.HF_SPACES
        • ConnectionConfigType.OLLAMA
        • ConnectionConfigType.OPENAI_CHAT
        • ConnectionConfigType.OPENAI_RAG
      • EvaluationJudgeConfig
        • EvaluationJudgeConfig.KEY_COLLECTION_ID
        • EvaluationJudgeConfig.KEY_CONNECTION
        • EvaluationJudgeConfig.KEY_DESCRIPTION
        • EvaluationJudgeConfig.KEY_JUDGE_TYPE
        • EvaluationJudgeConfig.KEY_KEY
        • EvaluationJudgeConfig.KEY_LLM_MODEL_NAME
        • EvaluationJudgeConfig.KEY_NAME
        • EvaluationJudgeConfig.from_dict()
        • EvaluationJudgeConfig.to_dict()
      • EvaluationJudgeType
        • EvaluationJudgeType.azure_openai_llm
        • EvaluationJudgeType.custom
        • EvaluationJudgeType.h2ogpt
        • EvaluationJudgeType.h2ogpte
        • EvaluationJudgeType.h2ogpte_llm
        • EvaluationJudgeType.h2ollmops
        • EvaluationJudgeType.ollama
        • EvaluationJudgeType.openai_llm
        • EvaluationJudgeType.openai_rag
      • H2o3Config
        • H2o3Config.DEFAULT_AUTO_CLEANUP
        • H2o3Config.DEFAULT_AUTO_START
        • H2o3Config.DEFAULT_AUTO_STOP
        • H2o3Config.DEFAULT_HOST
        • H2o3Config.DEFAULT_MAX_MEM_SIZE
        • H2o3Config.DEFAULT_MIN_MEM_SIZE
        • H2o3Config.DEFAULT_PORT
        • H2o3Config.KEYS
        • H2o3Config.KEY_AUTO_CLEANUP
        • H2o3Config.KEY_AUTO_START
        • H2o3Config.KEY_AUTO_STOP
        • H2o3Config.KEY_HOST
        • H2o3Config.KEY_MAX_MEM_SIZE
        • H2o3Config.KEY_MIN_MEM_SIZE
        • H2o3Config.KEY_PORT
      • H2oSonarConfig
        • H2oSonarConfig.CFG_CREATE_HTML_REPRESENTATIONS
        • H2oSonarConfig.CFG_CUSTOM_EXPLAINERS
        • H2oSonarConfig.CFG_DEVICE
        • H2oSonarConfig.CFG_DO_SAMPLE
        • H2oSonarConfig.CFG_ENABLE_DATASET_DOWNLOADING
        • H2oSonarConfig.CFG_FORCE_EVAL_JUDGE
        • H2oSonarConfig.CFG_H2O_AUTO_CLEANUP
        • H2oSonarConfig.CFG_H2O_AUTO_START
        • H2oSonarConfig.CFG_H2O_AUTO_STOP
        • H2oSonarConfig.CFG_H2O_HOST
        • H2oSonarConfig.CFG_H2O_MAX_MEM_SIZE
        • H2oSonarConfig.CFG_H2O_MIN_MEM_SIZE
        • H2oSonarConfig.CFG_H2O_PORT
        • H2oSonarConfig.CFG_HTTP_SSL_CERT_VERIFY
        • H2oSonarConfig.CFG_LOOK_AND_FEEL
        • H2oSonarConfig.CFG_MODEL_CACHE_DIR
        • H2oSonarConfig.CFG_MP_START_METHOD
        • H2oSonarConfig.CFG_NUM_QUANTILES
        • H2oSonarConfig.CFG_PER_EXPLAINER_LOGGER
        • H2oSonarConfig.CFG_SAMPLE_SIZE
        • H2oSonarConfig.ENV_VAR_CFG_PREFIX
        • H2oSonarConfig.KEY_CONNECTIONS
        • H2oSonarConfig.KEY_CREATE_HTML_REPRESENTATIONS
        • H2oSonarConfig.KEY_CUSTOM_EXPLAINERS
        • H2oSonarConfig.KEY_DEVICE
        • H2oSonarConfig.KEY_EVALUATION_JUDGES
        • H2oSonarConfig.KEY_FORCE_EVAL_JUDGE
        • H2oSonarConfig.KEY_HTTP_SSL_CERT_VERIFY
        • H2oSonarConfig.KEY_LICENSES
        • H2oSonarConfig.KEY_LOOK_AND_FEEL
        • H2oSonarConfig.KEY_MODEL_CACHE_DIR
        • H2oSonarConfig.KEY_PER_EXPLAINER_LOGGER
        • H2oSonarConfig.MP_START_METHOD_FORK
        • H2oSonarConfig.MP_START_METHOD_FORKSERVER
        • H2oSonarConfig.MP_START_METHOD_SPAWN
        • H2oSonarConfig.VALUE_AUTO
        • H2oSonarConfig.VALUE_CPU
        • H2oSonarConfig.VALUE_CUDA
        • H2oSonarConfig.VALUE_GPU
        • H2oSonarConfig.VALUE_MPS
        • H2oSonarConfig.VALUE_NPU
        • H2oSonarConfig.VALUE_STR_FALSE
        • H2oSonarConfig.VALUE_STR_TRUE
        • H2oSonarConfig.add_connection()
        • H2oSonarConfig.add_evaluation_judge()
        • H2oSonarConfig.add_license()
        • H2oSonarConfig.copy()
        • H2oSonarConfig.describe_config_item()
        • H2oSonarConfig.describe_config_items()
        • H2oSonarConfig.env_and_override()
        • H2oSonarConfig.from_toml()
        • H2oSonarConfig.get_connection()
        • H2oSonarConfig.get_evaluation_judge()
        • H2oSonarConfig.get_license()
        • H2oSonarConfig.http_ssl_cert_verify
        • H2oSonarConfig.load()
        • H2oSonarConfig.load_and_override()
        • H2oSonarConfig.save()
        • H2oSonarConfig.to_dict()
        • H2oSonarConfig.to_toml()
      • LicenseConfig
        • LicenseConfig.ENCRYPTED_FIELDS
        • LicenseConfig.KEY_KEY
        • LicenseConfig.KEY_LICENSE
        • LicenseConfig.KEY_LICENSE_FILE
        • LicenseConfig.KEY_PRODUCT
        • LicenseConfig.from_dict()
        • LicenseConfig.to_dict()
      • ProductLicenseConfig
        • ProductLicenseConfig.DRIVERLESS_AI
      • TokenUseType
        • TokenUseType.ACCESS_TOKEN
        • TokenUseType.API_KEY
        • TokenUseType.REFRESH_TOKEN
        • TokenUseType.SECRET
    • h2o_sonar.errors module
      • DatasetTooBigError
      • ExplainerCompatibilityError
      • InvalidArgumentError
      • InvalidArgumentValueError
      • InvalidDataError
      • MliError
      • MliJsonDeserializationError
      • MliJsonSerializationError
      • MliNotFoundError
      • MliPredictMethodError
      • MliTypeError
      • MliUnsupportedDataFormatError
      • MliUnsupportedError
      • MliUnsupportedOperationError
      • RenderingError
      • UnknownExplainerError
    • h2o_sonar.h2o_sonar_cli module
      • main()
    • h2o_sonar.interpret module
      • add_config_item()
      • describe_explainer()
      • gc()
      • get_config()
      • list_explainers()
      • list_interpretations()
      • register_explainer()
      • resolve_container()
      • run_interpretation()
      • unregister_explainer()
      • upload_interpretation()
    • h2o_sonar.loggers module
      • SonarFileLogger
        • SonarFileLogger.FORMATTER
        • SonarFileLogger.data()
        • SonarFileLogger.debug()
        • SonarFileLogger.error()
        • SonarFileLogger.info()
        • SonarFileLogger.warning()
      • SonarLogger
        • SonarLogger.FILE_NAME_H2O_SONAR_LOG
        • SonarLogger.data()
        • SonarLogger.debug()
        • SonarLogger.error()
        • SonarLogger.info()
        • SonarLogger.warning()
      • SonarPrintLogger
        • SonarPrintLogger.data()
        • SonarPrintLogger.debug()
        • SonarPrintLogger.error()
        • SonarPrintLogger.info()
        • SonarPrintLogger.warning()
      • debug()
      • error()
      • fatal()
      • get_level()
      • info()
      • log()
      • setLevel()
      • warn()
      • warning()
    • h2o_sonar.version module
  • Module contents

Indices and Tables

• Module Index

• Index