Generative Models

H2O Eval Studio provides evaluation of generative machine learning models.

Evaluating RAGs and LLMs

• Evaluating RAGs and LLMs
  • Test Data Preparation
  • System Under Evaluation Connection
  • Test Data Resolution
  • LLM Evaluation
  • Evaluation Results Analysis

RAG and LLM Hosts

• RAG and LLM Hosts
  • Enterprise h2oGPTe LLM Host
  • H2O GPT LLM Host
  • H2O LLMOps LLM Host
  • Open AI Assistants with File Search (formerly Retrieval) Tool LLM Host
  • Open AI Chat LLM Host
  • Microsoft Azure Open AI Chat LLM Host
  • Open AI Chat API Compatible LLM Host
  • Amazon Bedrock
  • ollama LLM Host

Test Case, Suite, Lab and LLM Dataset

• Test Case, Suite, Lab and LLM Dataset
  • Test Case
  • Test
  • Test Suite
  • Test Lab
  • LLM Dataset

Evaluator Parametrization

• Evaluator Parameters

Evaluators

• Evaluators
  • Answer Correctness Evaluator
  • Answer Semantic Similarity Evaluator
  • Answer Semantic Sentence Similarity Evaluator
  • Context Relevancy Evaluator
  • Context Relevancy (Soft Recall and Precision) Evaluator
  • Groundedness (Semantic Similarity) Evaluator
  • Hallucination Evaluator
  • RAGAS Evaluator
  • Text Matching Evaluator
  • Context Precision Evaluator
  • Fact-Check (Agent-based) Evaluator
  • Faithfulness Evaluator
  • Context Recall Evaluator
  • Answer Relevancy Evaluator
  • Answer Relevancy (Sentence Similarity) Evaluator
  • PII Leakage Evaluator
  • Encoding Guardrail Evaluator
  • Sensitive Data Leakage Evaluator
  • Toxicity Evaluator
  • Fairness Bias Evaluator
  • Contact Information Evaluator
  • Language Mismatch (Judge) Evaluator
  • Looping Detection Evaluator
  • Parameterizable BYOP Evaluator
  • Perplexity Evaluator
  • Sexism (Judge) Evaluator
  • Step Alignment and Completeness Evaluator
  • Stereotypes (Judge) Evaluator
  • Summarization (Completeness and Faithfulness) Evaluator
  • Summarization (Judge) Evaluator
  • Summarization with reference (GPTScore) Evaluator
  • Summarization without reference (GPTScore) Evaluator
  • BLEU Evaluator
  • ROUGE Evaluator
  • Classification Evaluator
  • Machine Translation (GPTScore) Evaluator
  • Question Answering (GPTScore) Evaluator

BYOJ: Bring Your Own Judge

• BYOJ: Bring Your Own Judge
  • Force Custom Judge from H2O Eval Studio Configuration
  • LLM Judge Specified in the Evaluator Parameters
  • Privacy Safe LLM Models

BYOP: Bring Your Own Prompt

• BYOP: Bring Your Own Prompt
  • Prompt Template Design
  • Parametrize BYOP Evaluator
  • Implement Your Own Prompt-Based Evaluator
  • Custom LLM Judge Configuration

Perturbations

• Perturbations
  • Perturbations Step-by-Step
  • Random Character Perturbation
  • Y/Z Perturbation
  • Comma Perturbation
  • Word Swap Perturbation
  • Synonym Perturbation
  • Antonym Perturbation
  • Random Character Insertion Perturbation
  • Random Character Deletion Perturbation
  • Random Character Replacement Perturbation
  • Keyboard Typos Perturbation
  • OCR Error Character Perturbation
  • Contextual Misinformation Perturbation
  • Perturbations API
  • Using Perturbations to Assess Model Robustness