List LLMs and their compatible vision model names

Overview

When working with a particular large language model (LLM), a user can identify the vision model that is compatible with it. This is important because integrating the two can enhance the overall performance and capabilities of applications that rely on text and visual information.

By leveraging both models, developers can create more sophisticated applications to interpret, analyze, and generate content based on multimodal inputs.

Example

from h2ogpte import H2OGPTE
from tabulate import tabulate

client = H2OGPTE(
    address="https://h2ogpte.genai.h2o.ai",
    api_key='sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
)

table = [[key, value] for key, value in client.get_llm_and_auto_vision_llm_names().items()]

print(tabulate(table, headers=["LLM", "Vision model"], tablefmt="pretty"))

+------------------------------------------------+------------------------------------------------+
|                      LLM                       |                  Vision model                  |
+------------------------------------------------+------------------------------------------------+
|           h2oai/h2o-danube3-4b-chat            |           mistralai/Pixtral-12B-2409           |
|     meta-llama/Meta-Llama-3.1-70B-Instruct     |           mistralai/Pixtral-12B-2409           |
|  meta-llama/Meta-Llama-3.1-405B-Instruct-FP8   |           mistralai/Pixtral-12B-2409           |
|           Qwen/Qwen2.5-72B-Instruct            |           mistralai/Pixtral-12B-2409           |
|           Qwen/Qwen2-VL-72B-Instruct           |           Qwen/Qwen2-VL-72B-Instruct           |
|           mistralai/Pixtral-12B-2409           |           mistralai/Pixtral-12B-2409           |
|      mistralai/Mixtral-8x7B-Instruct-v0.1      |           mistralai/Pixtral-12B-2409           |
|  meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo  |           mistralai/Pixtral-12B-2409           |
|  meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo   |           mistralai/Pixtral-12B-2409           |
|       upstage/SOLAR-10.7B-Instruct-v1.0        |           mistralai/Pixtral-12B-2409           |
|       mistralai/Mistral-7B-Instruct-v0.3       |           mistralai/Pixtral-12B-2409           |
|             google/gemma-2-27b-it              |           mistralai/Pixtral-12B-2409           |
| meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo | meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo |
| meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo | meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo |
|     meta-llama/Llama-3.2-3B-Instruct-Turbo     |           mistralai/Pixtral-12B-2409           |
|                  mistral-tiny                  |           mistralai/Pixtral-12B-2409           |
|              mistral-small-latest              |           mistralai/Pixtral-12B-2409           |
|                 mistral-medium                 |           mistralai/Pixtral-12B-2409           |
|              mistral-large-latest              |           mistralai/Pixtral-12B-2409           |
|             gemini-1.5-pro-latest              |             gemini-1.5-pro-latest              |
|            gemini-1.5-flash-latest             |            gemini-1.5-flash-latest             |
|            claude-3-haiku-20240307             |            claude-3-haiku-20240307             |
|            claude-3-sonnet-20240229            |            claude-3-sonnet-20240229            |
|           claude-3-5-sonnet-20240620           |           claude-3-5-sonnet-20240620           |
|                     gpt-4o                     |                     gpt-4o                     |
|                  gpt-4o-mini                   |                  gpt-4o-mini                   |
+------------------------------------------------+------------------------------------------------+