Test Case, Suite, Lab and LLM Dataset ===================================== H2O Sonar can evaluate standalone LLMs (Large Language Models) and LLMs used by RAG (Retrieval-augmented generation) applications using the evaluation data shipped with the product and/or custom evaluation data provided by the user. Evaluation data can be provided in the form of test cases, tests, test suites, and test labs. LLM dataset - which is normalized test lab - is used as the input of the evaluation process. - :ref:`Test Case` - :ref:`Test Suite` - :ref:`Test Lab` - :ref:`LLM Dataset` Terminology: - :ref:`Raw Test Data` - A dataset - provided by the user, exported from a system or generated - which can be used to create the evaluation data. - :ref:`Test Case` - A single test case that is used to evaluate the LLM / RAG - it consists of a prompt, ground truth, conditions, and other parameters that are used for the evaluation. - :ref:`Test` - A set of test cases linked with a collection of documents (corpus) which must be provided in case of RAG evaluation (test cases typically make sense only in the context of particular corpus - the corpus is empty in case of LLM evaluation). - :ref:`Test Suite` - A set of tests which can be used to evaluate the LLM / RAG model. - :ref:`Test Lab` - A test suite completed with the actual data - answers generated by the LLM / RAG models for given prompts, retrieved contexts (in case of RAG), durations, and costs. - :ref:`LLM Dataset` - A normalized test lab that is used as the input of the evaluation process. - :ref:`Report` - A collection of metrics and visualizations that describe the validity of a RAG / LLM model on a given test configuration. Test Case ---------- Test case is a single test case that is used to evaluate the LLM / RAG - it consists of a prompt, ground truth, conditions, and other parameters that are used for the evaluation. Example: .. code-block:: json { "key": "d981bf23-9790-4c5e-905e-2ab56cba7ece", "prompt": "What was the revenue of Brazil?", "categories": [ "question_answering", "perturbed", "perturbed_by:h2o_sonar.utils.perturbations.SynonymPerturbator:PerturbationIntensity.HIGH" ], "relationships": [ { "type": "perturbation_source", "target": "9c3a7df3-67df-4819-babb-20636611f077", "target_type": "test_case" } ], "expected_output": "Brazil revenue was 15,969 million.", "condition": "\"15,969\" AND \"million\"" } Format description: - `key` - Key is optional. In case that it is specified it must be unique within the test suite. - `prompt` - Prompt which is used to evaluate the LLM / RAG model. - `categories` - Optional categories which are used to classify the test cases. - `relationships` - Optional relationships which are used to interlink the test cases. - `relationships/type` - Type of the relationship - for instance test case which was used as the `perturbation_source` in this case. - `relationships/target` - Key of the relationship target - a test case key. - `relationships/target_type` - Target type - for instance `test_case`. - `expected_output` - Ground truth / expected output. - `condition` - Optional condition is used by Text Matching evaluator (and may be used by other evaluators) to evaluate the LLM / RAG model. Python code which can be used to create the **test case**: .. code-block:: python testing.RagTestCaseConfig( prompt="What was the revenue of Brazil?", expected_output="Brazil revenue was 15,969 million.", condition='"15,969" AND "million"' ) Test ---- Test is a set of test cases linked with a collection of documents (corpus) which must be provided in case of RAG evaluation (test cases typically make sense only in the context of particular corpus - the corpus is empty in case of LLM evaluation). Example: .. code-block:: json { "key": "48099214-7b33-443a-a456-eb44519521f1", "documents": [], "test_cases": [ { "key": "9c3a7df3-67df-4819-babb-20636611f077", "prompt": "Respond to the following question with single letter answer...", "categories": [ "question-answering" ], "relationships": [], "condition": "regexp(^A.*)", "expected_output": "A" }, { "key": "6709c0e1-de57-4b5b-b826-2060bf935574", "prompt": "Respond to the next question with one letter answer...", "categories": [ "question-answering", "perturbed", "perturbed_by:h2o_sonar.utils.perturbations.SynonymPerturbator:PerturbationIntensity.HIGH" ], "relationships": [ { "type": "perturbation_source", "target": "9c3a7df3-67df-4819-babb-20636611f077", "target_type": "test_case" } ], "condition": "regexp(^D.*)", "expected_output": "D" } ] } Format description: - `key` - Key is optional. In case that it is specified it must be unique within the test suite. - `documents` - In case of RAG evaluation :ref:`Test` has a set of documents - corpus. - `test_cases` - Test is formed by a set of :ref:`Test Case`s which are used to evaluate the LLM / RAG model in the context of given corpus specified within the test's `documents` field. Python code which can be used to create the **test**: .. code-block:: python # test test = testing.RagTestConfig(documents=[doc_url]) # test cases test_case_1 = testing.RagTestCaseConfig( prompt="What was the revenue of Brazil?", expected_output="Brazil revenue was 15,969 million.", condition='"15,969" AND "million"', config=test, ) ... test_case_n = testing.RagTestCaseConfig( prompt="What was the revenue of Argentina?", expected_output="Argentina revenue was 5,969 million.", condition='"5,969" AND "million"', config=test, ) Test Suite ---------- Test suite is a set of tests which can be used to evaluate the LLM / RAG model. .. code-block:: json { "name": "Test Suite Example", "description": "This is a test suite example.", "tests": [ { "key": "da503caf-a2dc-4c0a-b5f7-618bebd7860d", "documents": [ "https://public-data.s3.amazonaws.com/Coca-Cola-FEMSA-Results-1Q23-vf-2.pdf" ], "test_cases": [ { "key": "d981bf23-9790-4c5e-905e-2ab56cba7ece", "prompt": "What was the revenue of Brazil?", "categories": [ "question_answering" ], "relationships": [], "expected_output": "Brazil revenue was 15,969 million.", "condition": "\"15,969\" AND \"million\"", } ] }, { "key": "be087c6b-ad41-4b2e-a9f1-0ffd5e6743ec", "documents": [ "https://public-data.s3.amazonaws.com/bradesco-2022-integrated-report.pdf" ], "test_cases": [ { "key": "3db6878b-1511-4b58-b42f-985ca9699d1d", "prompt": "Who is the chairman of the board?", "categories": [ "question_answering" ], "relationships": [], "expected_output": "The chairman of the board was Luiz Carlos Trabuco Cappi.", "condition": "\"Luiz Carlos Trabuco Cappi\"", }, { "key": "13011931-70c4-40b9-aacc-3870f521dbaa", "prompt": "What was the number of agreements that include human rights clauses, in 2022?", "expected_output": "The number of agreements that include human rights clauses in 2022 was 22.", "categories": [ "question_answering" ], "relationships": [], "condition": "\"22\"" } ] } ] } Format description: - `/name` - Test suite name. - `/description` - Test suite description. - `/tests` - Test suite is formed by a set of :ref:`Test`s. - `/tests/documents` - In case of RAG evaluation :ref:`Test` has a set of documents - corpus. - `/tests/key` - Key is optional. In case that it is specified it must be unique within the test suite. - `test_cases` - Test suite is formed by a set of :ref:`Test Case`s which are used to evaluate the LLM / RAG model in the context of given corpus. Python code which can be used to create the **test suite**: .. code-block:: python # test suite test_suite = testing.RagTestSuiteConfig() # test test = testing.RagTestConfig(documents=[doc_url_1, doc_url_2]) # test cases test_suite.add_test_case( testing.RagTestCaseConfig( prompt="What was the revenue of Brazil?", expected_output="Brazil revenue was 15,969 million.", condition='"15,969" AND "million"', config=test, ) ) ... test_suite.add_test_case( testing.RagTestCaseConfig( prompt="What was the revenue of Argentina?", expected_output="Argentina revenue was 5,969 million.", condition='"5,969" AND "million"', config=test, ) ) Test Lab -------- Test lab is a test suite completed with the actual data - answers generated by the LLM / RAG models for given prompts, retrieved contexts (in case of RAG), duration, and cost. Example: .. code-block:: json { "name": "TestLab", "description": "Test lab for RAG evaluation.", "raw_dataset": { "inputs": [ { "input": "What is the purpose of the document?", "corpus": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "context": [], "categories": [ "question_answering" ], "relationships": [], "expected_output": "The purpose of this document is to provide comprehensive guid...", "output_condition": "\"guidance\" AND \"model risk management\"", "actual_output": "", "actual_duration": 0.0, "cost": 0.0, "model_key": "0c34cdae-d2a4-4557-a305-d182adb01f6f" }, { "input": "What is model validation?", "corpus": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "context": [], "categories": [ "question_answering" ], "relationships": [], "expected_output": "Model validation is the set of processes and activities inte...", "output_condition": "\"processes\" AND \"activities\"", "actual_output": "", "actual_duration": 0.0, "cost": 0.0, "model_key": "14ac0946-6547-48d1-b19a-41bc90d34849" } ] }, "dataset": { "inputs": [ { "input": "What is the purpose of the document?", "corpus": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "context": [ "Changes in\nregulation have spurred some of the recent developments, parti...", "Analysis of in-\nsample fit and of model performance in holdout samples (d...", "Validation reports should articulate model aspects that were reviewed, hig...", "Page 1\nSR Letter 11-7\nAttachment\nBoard of Governors of the Federal Rese...", "Documentation of model development and validation should be sufficiently\n...", ], "categories": [ "question_answering" ], "relationships": [], "expected_output": "The purpose of this document is to provide comprehensive gu...", "output_condition": "\"guidance\" AND \"model risk management\"", "actual_output": "The purpose of the document is to provide comprehensive guida...", "actual_duration": 8.280992269515991, "cost": 0.0036560000000000013, "model_key": "0c34cdae-d2a4-4557-a305-d182adb01f6f" }, { "input": "What is model validation?", "corpus": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "context": [ "But it is not enough for model developers and users\nto understand and acce...", "While they may grant\nexceptions to typical procedures of model validation ...", "Management should have a clear plan for using the results of sensitivity an...", "Validation reports should articulate model aspects that were reviewed, high...", "Rigorous model validation plays a critical role in\nmodel risk management; ..." ], "categories": [ "question_answering" ], "relationships": [], "expected_output": "Model validation is the set of processes and activities int...", "output_condition": "\"processes\" AND \"activities\"", "actual_output": "Model validation is the process of assessing the quality of a...", "actual_duration": 19.800140142440796, "cost": 0.004117999999999998, "model_key": "14ac0946-6547-48d1-b19a-41bc90d34849" } ] }, "models": [ { "connection": "612db877-cba2-4ab8-bacf-88d84b396450", "model_type": "h2ogpte", "name": "RAG model h2oai/h2ogpt-4096-llama2-70b-chat (docs: ['sr1107a1.pdf'])", "collection_id": "263c58e6-7424-4f9e-aa02-347d7281439b", "collection_name": "RAG collection (docs: ['sr1107a1.pdf'])", "llm_model_name": "h2oai/h2ogpt-4096-llama2-70b-chat", "documents": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "key": "0c34cdae-d2a4-4557-a305-d182adb01f6f" }, { "connection": "612db877-cba2-4ab8-bacf-88d84b396450", "model_type": "h2ogpte", "name": "RAG model h2oai/h2ogpt-4096-llama2-13b-chat (docs: ['sr1107a1.pdf'])", "collection_id": "263c58e6-7424-4f9e-aa02-347d7281439b", "collection_name": "RAG collection (docs: ['sr1107a1.pdf'])", "llm_model_name": "h2oai/h2ogpt-4096-llama2-13b-chat", "documents": [ "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf" ], "key": "14ac0946-6547-48d1-b19a-41bc90d34849" } ], "llm_model_names": [ "h2oai/h2ogpt-4096-llama2-70b-chat", "h2oai/h2ogpt-4096-llama2-13b-chat" ], "docs_cache": { "https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf": "/tmp/pytest-of-me/sr1107a1.pdf" } } Format description: - `/name` - Test lab name. - `/description` - Test lab description. - `/raw_dataset` - Test suite - which does not contain resolved data - normalized as LLM dataset. - `/dataset` - Test suite with resolved data normalized as LLM dataset. - `/models` - List of models which were used to resolve the test suite - generated the actual data for the test cases like answers, duration, and cost. - `/models/connection` - Key of the connection (stored in the H2O Sonar configuration) which was used to connect to the model host. - `/models/model_type` - Type of the model host (e.g. ``h2ogpte`` or ``h2ogpt``). - `/models/name` - Name of the model. - `/models/collection_id` - Corpus collection ID in case of RAG evaluation (not used in case of LLM evaluation). - `/models/collection_name` - Corpus collection name in case of RAG evaluation (not used in case of LLM evaluation). - `/models/llm_model_name` - Name of the LLM model which was evaluated and which generated the actual data. - `/models/documents` - List of documents which were used in the evaluation of a RAG as **corpus** (not used in case of LLM evaluation). - `/models/key` - Key of the model which is used to identify the model in the test suite. It is used in ``/dataset/model_key`` to link the resolved data with the model which created the data. - `/llm_model_names` - List of LLM model (names) which are hosted by ``models`` and which generated the actual data. - `/docs_cache` - Cache of the documents used in the test suite in case of RAG evaluation. Python code which can be used to create the **test lab**: .. code-block:: python # test lab test_lab = testing.RagTestLab.from_llm_test_suite( llm_host_connection=h2o_gpte_connection, llm_test_suite=test_suite, llm_model_type=models.ExplainableModelType.h2ogpte_llm, llm_model_names=llm_model_names, ) # create collections and upload corpus documents (RAG only) test_lab.build() # resolve test lab test_lab.complete_dataset() The **host-specific configuration** can be used to create the test lab. In this case, the host client might be used to obtain the available parameters, which can be modified by the user and then passed to the code that resolves the test lab. In order to be able to compare RAGs/LLMs with different configurations, it is possible to provide multiple configurations for the same model. The configuration is a dictionary that contains the parameters that are used to create the model -see :ref:`RAG and LLM Hosts` for reference. For instance: .. code-block:: python # # LLMs to be evaluated # llm_model_names = ["h2oai/h2ogpt-4096-llama2-70b-chat"] # # custom host configurations # # A # host configuration proto w/ default values custom_config_a = genai.H2oGpteRagClient.config_factory() # modify system prompt custom_config_a["system_prompt"] = ( "You are h2oGPTe, an expert question-answering AI system created by H2O.ai " "that performs like GPT-4 by OpenAI." ) # modify LLM temperature in order to make the model more creative custom_config_a["llm_args"] = {"temperature": 0.9} # B # host configuration proto w/ default values custom_config_b = genai.H2oGpteRagClient.config_factory() # modify system prompt custom_config_b["system_prompt"] = ( "You are flaky, a rookie question-answering AI system that performs poorly." ) # modify LLM temperature to avoid hallucinations custom_config_b["llm_args"] = {"temperature": 0.0} # # LLM configurations: LLM name -> list of host configurations # llm_host_configs = { llm_model_names[0]: [ custom_config_a, custom_config_b, ] } # # test lab # test_lab = testing.RagTestLab.from_llm_test_suite( llm_host_connection=h2o_gpte_connection, llm_test_suite=test_suite, llm_model_type=models.ExplainableModelType.h2ogpte_llm, llm_model_names=llm_model_names, llm_host_configs=llm_host_configs, ) # create collections and upload corpus documents (RAG only) test_lab.build() # resolve test lab test_lab.complete_dataset() LLM Dataset ----------- LLM dataset is a normalized test lab that is used as the input of the evaluation process. LLM dataset can be serialized as JSon, CSV or Pandas frame. Example: .. code-block:: json { "inputs": [ { "input": "Which of the following statements accurately describes the impact of ...", "corpus": [ "https://www.wikipedia.org/" ], "context": [ "Physical Review Letters. 99 (14): 141302. arXiv:0704.1932. Bibcode:2007P...", "The homogeneously distributed mass of the universe would result in a rou...", "Modified Newtonian dynamics (MOND) is a hypothesis that proposes a modif..." ], "categories": [ "question_answering" ], "relationships": [], "expected_output": "MOND is a theory that reduces the discrepancy between the...", "output_condition": "", "actual_output": "MOND is a theory that reduces the discrepancy between the o...", "actual_duration": 0.415, "model_key": "4b60abf6-0c8d-4765-804a-e830a211f152" } ] } Format description: - `/inputs` - List of test cases. - `/inputs/input` - Prompt. - `/inputs/corpus` - List of documents which are used as the corpus in case of this test case. - `/inputs/context` - List of chunks - retrieved contexts. - `/inputs/categories` - List of categories used to classify the test case. - `/inputs/relationships` - List of relationships. - `/inputs/expected_output` - Expected output. - `/inputs/output_condition` - Condition used by Text Matching evaluator (and may be used by other evaluators) to evaluate the LLM / RAG model. - `/inputs/actual_output` - Actual output. - `/inputs/actual_duration` - Duration of the request to RAG / LLM model. - `/inputs/model_key` - Key of the model which generated the actual data - must be valid key of a model when running the evaluation process. Python code which can be used to create the **LLM dataset**: .. code-block:: python # LLM dataset dataset = datasets.LlmDataset() # add row(s) to the dataset dataset.add_input( i="What is the target column of the model?", actual_output="default payment next month", )