Generative Models

H2O Sonar provides evaluation of generative machine learning models.

Evaluating RAGs and LLMs

• Evaluating RAGs and LLMs
  • Test Data Preparation
  • System Under Evaluation Connection
  • Test Data Resolution
  • LLM Evaluation
  • Evaluation Results Analysis

RAG and LLM Hosts

• RAG and LLM Hosts
  • Enterprise h2oGPTe LLM Host
  • H2O GPT LLM Host
  • H2O LLMOps LLM Host
  • Open AI Assistants with File Search (formerly Retrieval) Tool LLM Host
  • Open AI Chat LLM Host
  • Microsoft Azure Open AI Chat LLM Host
  • Anthropic Claude Chat LLM Host
  • Open AI Chat API Compatible LLM Host
  • Amazon Bedrock
  • ollama LLM Host

Test Case, Suite, Lab and LLM Dataset

• Test Case, Suite, Lab and LLM Dataset
  • Test Case
  • Test
  • Test Suite
  • Test Lab
  • LLM Dataset

Evaluator Parametrization

• Evaluator Parameters

Evaluators

• Evaluators
  • Reproducibility
  • Agent Sanity Check Evaluator
  • Answer Accuracy (Semantic Similarity) Evaluator
  • Answer Correctness Evaluator
  • Answer Semantic Similarity Evaluator
  • Answer Semantic Sentence Similarity Evaluator
  • Context Relevancy Evaluator
  • Context Relevancy (Soft Recall and Precision) Evaluator
  • Groundedness (Semantic Similarity) Evaluator
  • Hallucination Evaluator
  • RAGAS Evaluator
  • Text Matching Evaluator
  • Context Precision Evaluator
  • Fact-Check (Agent-based) Evaluator
  • Faithfulness Evaluator
  • Context Recall Evaluator
  • Context Mean Reciprocal Rank Evaluator
  • Answer Relevancy Evaluator
  • Answer Relevancy (Sentence Similarity) Evaluator
  • PII Leakage Evaluator
  • JSON Schema Evaluator
  • Encoding Guardrail Evaluator
  • Sensitive Data Leakage Evaluator
  • Toxicity Evaluator
  • Fairness Bias Evaluator
  • Contact Information Evaluator
  • Language Mismatch (Judge) Evaluator
  • Looping Detection Evaluator
  • Parameterizable BYOP Evaluator
  • Perplexity Evaluator
  • Questions Drift Evaluator
  • Sexism (Judge) Evaluator
  • Step Alignment and Completeness Evaluator
  • Stereotypes (Judge) Evaluator
  • Summarization (Completeness and Faithfulness) Evaluator
  • Summarization (Judge) Evaluator
  • Summarization with reference (GPTScore) Evaluator
  • Summarization without reference (GPTScore) Evaluator
  • BERTScore Evaluator
  • BLEU Evaluator
  • ROUGE Evaluator
  • Self-Consistency Evaluator
  • Classification Evaluator
  • Machine Translation (GPTScore) Evaluator
  • Question Answering (GPTScore) Evaluator

BYOJ: Bring Your Own Judge

• BYOJ: Bring Your Own Judge
  • Force Custom Judge from H2O Sonar Configuration
  • LLM Judge Specified in the Evaluator Parameters
  • Privacy Safe LLM Models

BYOP: Bring Your Own Prompt

• BYOP: Bring Your Own Prompt
  • Prompt Template Design
  • Parametrize BYOP Evaluator
  • Implement Your Own Prompt-Based Evaluator
  • Custom LLM Judge Configuration

Perturbations

• Perturbations
  • Perturbations Step-by-Step
  • Random Character Perturbator
  • Antonym Perturbator
  • Comma Perturbator
  • Contextual Misinformation Perturbator
  • Copy Perturbator
  • Keyboard Typos Perturbator
  • OCR Error Character Perturbator
  • Random Character Deletion Perturbator
  • Random Character Insertion Perturbator
  • Random Character Replacement Perturbator
  • Synonym Perturbator
  • Word Swap Perturbator
  • Y/Z Perturbator
  • Perturbators API
  • Using Perturbations to Assess Model Robustness