Evaluation of InternVL 1.5 Series#

To evaluate the performance of the InternVL 1.5 series across various tasks, follow the instructions for each specific dataset. Ensure that the appropriate number of GPUs is allocated as specified. The following tests will primarily use InternVL-Chat-V1-5 as an example.

1⃣️ We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.

2⃣️ Please note that evaluating the same model using different testing toolkits like InternVL and VLMEvalKit can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.

3⃣️️ Note, the dataset description is generated by GPT-4 and may contain errors.

Model Preparation#

model name	type	param	download	size
InternVL-Chat-V1-5	MLLM	25.5B	🤗 HF link	48.0 GB
Mini-InternVL-Chat-2B-V1-5	MLLM	2.2B	🤗 HF link	4.2 GB
Mini-InternVL-Chat-4B-V1-5	MLLM	4.2B	🤗 HF link	7.8 GB

Before evaluation, download the trained model we provide.

cd pretrained/
# pip install -U huggingface_hub
# Download OpenGVLab/InternVL-Chat-V1-5
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternVL-Chat-V1-5 --local-dir InternVL-Chat-V1-5
# Download OpenGVLab/Mini-InternVL-Chat-2B-V1-5
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/Mini-InternVL-Chat-2B-V1-5 --local-dir Mini-InternVL-Chat-2B-V1-5
# Download OpenGVLab/Mini-InternVL-Chat-4B-V1-5
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/Mini-InternVL-Chat-4B-V1-5 --local-dir Mini-InternVL-Chat-4B-V1-5

The directory structure is:

pretrained
├── Mini-InternVL-Chat-2B-V1-5
├── Mini-InternVL-Chat-4B-V1-5
└── InternVL-Chat-V1-5

Evaluation using InternVL Codebase#

Data Preparation#

Please prepare the evaluation data according to the guidance provided here.

MME#

MME is a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on both perception and cognition abilities across 14 different subtasks, ensuring robust and diverse testing of these models.

Please use the following command to perform the test with 1 GPU:

GPUS=1 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mme --dynamic --max-num 12

The expected test results are:

total score: 1658.3683473389356

   existence  score: 190.0
   count  score: 175.0
   position  score: 171.66666666666669
   color  score: 178.33333333333331
   posters  score: 173.8095238095238
   celebrity  score: 142.05882352941177
   scene  score: 156.5
   landmark  score: 179.5
   artwork  score: 144.0
   OCR  score: 147.5


=========== Cognition ===========
total score: 533.5714285714286

   commonsense_reasoning  score: 133.57142857142858
   numerical_calculation  score: 117.5
   text_translation  score: 185.0
   code_reasoning  score: 97.5

# 1658.3683473389356 + 533.5714285714286 = 2191.939775910364

OKVQA#

OKVQA (Outside Knowledge Visual Question Answering) is a dataset designed for visual question answering tasks that require external knowledge beyond what is visible in the image, featuring over 14,000 questions to evaluate the reasoning abilities of AI models.

Please use the following command to perform the test with 8 GPU:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-okvqa-val --dynamic

The expected test results are:

okvqa_val 0.6203147047165996

TextVQA#

TextVQA is a dataset designed to evaluate visual question answering models by requiring them to read and reason about text present within images, containing 45,336 questions over 28,408 images from the OpenImages dataset.

The TextVQA dataset provides official OCR results, specifically Rosetta OCR tokens. During testing with InstructBLIP and LLaVA 1.5, the OCR results are input to the LLM as a prompt. If you want to input Rosetta OCR tokens, use the following command:

We do not use Rosetta OCR tokens, run this command:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-textvqa-val --dynamic --max-num 24

The expected test results are:

['pretrained/InternVL-Chat-V1-5', 'textvqa_val', 0.8061000000000043]

VizWiz#

The VizWiz VQA dataset is a visual question answering dataset created to help answer visual questions posed by blind individuals. It contains over 31,000 visual questions, where users took a picture using a mobile phone and recorded a spoken question about it. Each question comes with 10 crowdsourced answers. This dataset addresses tasks such as predicting the answer to a visual question and determining whether a visual question can be answered.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-vizwiz-val --dynamic

The expected test results are:

vizwiz_val 0.6353929150266284

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-vizwiz-test --dynamic

For the test set, submit the results to the evaluation server.

ChartQA#

The ChartQA dataset is a comprehensive benchmark for question answering about charts that involves both visual and logical reasoning. It includes a mix of 9.6K human-written questions and 23.1K machine-generated questions derived from chart summaries. This dataset is designed to evaluate models that can understand and analyze charts by answering complex questions that often require multiple logical and arithmetic operations, as well as referencing visual features of the charts.

The ChartQA dataset includes two test sets: chartqa_test_human and chartqa_test_augmented. The final score for model evaluation is calculated as the average of the scores on these two test sets:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-chartqa-test --dynamic --max-num 12

The expected test results are:

['chartqa_test_human', {'relaxed_accuracy': 0.736}]
['chartqa_test_augmented', {'relaxed_accuracy': 0.9408}]
# average score = (73.6 + 94.08) / 2 = 83.8

DocVQA#

The DocVQA dataset consists of 50,000 questions on 12,000+ document images. It is designed for visual question answering tasks where questions are answered using text within the document images. The dataset includes OCR transcriptions and ground truth answers, supporting evaluation of models that interpret and extract information from documents.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-docvqa-val --dynamic --max-num 18

The expected test results are:

Overall ANLS: 0.9049

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-docvqa-test --dynamic --max-num 18

For the test set, submit the results to the evaluation server.

The expected test results are:

Overall ANLS: 0.909

AI2D#

The AI2D dataset contains over 5,000 grade school science diagrams with extensive annotations and 15,000 multiple-choice questions for research on diagram understanding and question answering.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-ai2d-test --dynamic

The expected test results are:

ai2diagram_test {'accuracy': 0.8073186528497409}

InfographicVQA#

The InfographicVQA dataset is a collection of infographics accompanied by natural language questions and answers. This dataset includes a diverse range of infographics sourced from thousands of different websites, ensuring a variety of layouts and designs. It comprises 30,035 questions across 5,485 images, split into training, validation, and test sets.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-infovqa-val --dynamic --max-num 24

The expected test results are:

Overall ANLS: 0.7235

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-infovqa-test --dynamic --max-num 24

For the test set, submit the results to the evaluation server.

The expected test results are:

Overall ANLS: 0.725

GQA#

The GQA dataset is a large-scale visual question answering dataset designed for real-world visual reasoning and compositional question answering. It contains over 22 million questions grounded in real images, each accompanied by detailed scene graphs that describe objects, their attributes, and relationships within the scene. The dataset includes images from the Visual Genome dataset, with questions that require various reasoning skills such as spatial understanding and multi-step inference.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 vqa-gqa-testdev --dynamic

The expected test results are:

Accuracy: 65.76%

ScienceQA#

The ScienceQA dataset is a large-scale benchmark for multimodal science question answering, consisting of 21,208 multiple-choice questions derived from elementary and high school science curricula. This dataset features a diverse range of topics across natural science, social science, and language science. It includes questions with image context (48.7%), text context (48.2%), and both (30.8%).

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 scienceqa --dynamic

The expected test results are:

Acc@1: 0.9404972731779871

POPE#

The POPE (Polling-based Object Probing Evaluation) dataset is designed to evaluate object hallucination in MLLMs. The dataset consists of 3,000 questions related to the captions of 500 images. By treating the MLLMs’ answers to these questions as a binary classification task, the dataset allows researchers to measure accuracy, precision, recall, and F1 scores to determine the extent of hallucination in the models.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 pope --dynamic

The expected test results are:

Category: random, # samples: 2910
TP      FP      TN      FN
1227    21      1389    273
Accuracy: 0.8989690721649485
Precision: 0.9831730769230769
Recall: 0.818
F1 score: 0.8930131004366811
Yes ratio: 0.4288659793814433
0.893, 0.899, 0.983, 0.818, 0.429
====================================
Category: popular, # samples: 3000
TP      FP      TN      FN
1227    48      1452    273
Accuracy: 0.893
Precision: 0.9623529411764706
Recall: 0.818
F1 score: 0.8843243243243243
Yes ratio: 0.425
0.884, 0.893, 0.962, 0.818, 0.425
====================================
Category: adversarial, # samples: 3000
TP      FP      TN      FN
1227    82      1418    273
Accuracy: 0.8816666666666667
Precision: 0.9373567608861727
Recall: 0.818
F1 score: 0.8736205055179779
Yes ratio: 0.43633333333333335
0.874, 0.882, 0.937, 0.818, 0.436
====================================

(89.3 + 88.4 + 87.4) / 3 = 88.3

Tiny LVLM#

The Tiny LVLM-eHub is a streamlined evaluation benchmark designed to assess the multimodal capabilities of MLLMs, including models like Bard. It focuses on six categories of multimodal abilities: visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, and embodied intelligence.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 tiny_lvlm --dynamic

The expected test results are:

Visual_Knowledge_Acquisition: 0.7542857142857143
Object_Hallucination: 0.8933333333333333
Visual_Commonsense: 0.634
Visual_Perception: 0.5775
Visual_Reasoning: 0.7236363636363636
Overall: 3.582755411255411

MMMU#

The MMMU dataset is a comprehensive benchmark designed to evaluate multimodal models on college-level tasks that require domain-specific knowledge and reasoning. It includes 11,500 questions sourced from college exams, quizzes, and textbooks, spanning six disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions cover 30 subjects and feature 30 types of images, such as charts, diagrams, maps, tables, and more.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmmu-val --dynamic

The expected test results are:

{'Overall-Art and Design': {'num': 120, 'acc': 0.608}, 'Art': {'num': 30, 'acc': 0.7}, 
'Art_Theory': {'num': 30, 'acc': 0.8}, 'Design': {'num': 30, 'acc': 0.767},
'Music': {'num': 30, 'acc': 0.167}, 'Overall-Business': {'num': 150, 'acc': 0.413}, 
'Accounting': {'num': 30, 'acc': 0.467}, 'Economics': {'num': 30, 'acc': 0.4}, 
'Finance': {'num': 30, 'acc': 0.4}, 'Manage': {'num': 30, 'acc': 0.4}, 
'Marketing': {'num': 30, 'acc': 0.4}, 'Overall-Science': {'num': 150, 'acc': 0.38}, 
'Biology': {'num': 30, 'acc': 0.6}, 'Chemistry': {'num': 30, 'acc': 0.233}, 
'Geography': {'num': 30, 'acc': 0.4}, 'Math': {'num': 30, 'acc': 0.333}, 
'Physics': {'num': 30, 'acc': 0.333}, 'Overall-Health and Medicine': {'num': 150, 'acc': 0.433}, 
'Basic_Medical_Science': {'num': 30, 'acc': 0.5}, 'Clinical_Medicine': {'num': 30, 'acc': 0.5}, 
'Diagnostics_and_Laboratory_Medicine': {'num': 30, 'acc': 0.333}, 
'Pharmacy': {'num': 30, 'acc': 0.367}, 'Public_Health': {'num': 30, 'acc': 0.467}, 
'Overall-Humanities and Social Science': {'num': 120, 'acc': 0.617}, 
'History': {'num': 30, 'acc': 0.633}, 'Literature': {'num': 30, 'acc': 0.8},
'Sociology': {'num': 30, 'acc': 0.567}, 'Psychology': {'num': 30, 'acc': 0.467},
'Overall-Tech and Engineering': {'num': 210, 'acc': 0.362}, 'Agriculture': {'num': 30, 'acc': 0.567},
'Architecture_and_Engineering': {'num': 30, 'acc': 0.267}, 'Computer_Science': {'num': 30, 'acc': 0.367},
'Electronics': {'num': 30, 'acc': 0.3}, 'Energy_and_Power': {'num': 30, 'acc': 0.333}, 
'Materials': {'num': 30, 'acc': 0.467}, 'Mechanical_Engineering': {'num': 30, 'acc': 0.233}, 
'Overall': {'num': 900, 'acc': 0.452}}

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmmu-test --dynamic

For the test set, submit the results to the evaluation server.

The expected test results are:

# result of the test-en set
A_Overall (test)	0.8217488789237668
# result of the test-cn set
A_Overall (test)	0.8195067264573991

MMVet (GPT-4-0613)#

⚠️ Warning: Here, we use GPT-4-0613 as the judge model, while in VLMEvalKit, GPT-4-Turbo is used as the judge model. Using different versions of GPT-4 can result in significant score variations. Therefore, testing the same model with the two codebases can lead to notable score differences.

The MM-Vet dataset is a comprehensive benchmark designed to evaluate the integrated capabilities of MLLMs. It encompasses six core vision-language (VL) capabilities: recognition, knowledge, optical character recognition (OCR), spatial awareness, language generation, and math. The dataset includes 200 images and 218 questions, each requiring one or more of these capabilities to answer. The evaluation uses an open-ended LLM-based approach, allowing assessment across various answer styles and question types.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmvet --dynamic

Then, submit the results to the evaluation server. The expected test results are:

total
62.7

MMBench#

The MMBench dataset is a comprehensive multi-modality benchmark designed to evaluate the fine-grained abilities of vision-language models. It contains around 3,000 multiple-choice questions covering 20 ability dimensions, structured into a hierarchical taxonomy. These dimensions include perception and reasoning abilities, further broken down into specific skills like coarse and fine-grained perception, attribute reasoning, and logic reasoning.

For the English dev / test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmbench-dev-en --dynamic
GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmbench-test-en --dynamic

Then, submit the results to the evaluation server. The expected test results are:

mmbench-dev-en: 80.8
mmbench-test-en: 82.2

For the Chinese dev / test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmbench-dev-cn --dynamic
GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmbench-test-cn --dynamic

Then, submit the results to the evaluation server. The expected test results are:

mmbench-dev-cn: 80.3
mmbench-test-cn: 82.0

CCBench#

CCBench, a multi-modal benchmark in the domain of Chinese Culture, is designed to evaluate the performance of MLLMs on tasks specifically related to Chinese cultural content.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 ccbench-dev --dynamic

Then, submit the results to the evaluation server. The expected test results are:

A_Overall (dev)	0.7

SEED#

CCBench is a multimodal benchmark specifically designed to evaluate models on tasks related to Chinese culture. It is part of the larger MMBench suite of benchmarks, developed by the OpenCompass Community, and aims to provide fine-grained evaluations across various capabilities of vision-language models. CCBench includes 510 questions in a multiple-choice format, focusing on cultural knowledge and understanding.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 seed --dynamic

The expected test results are:

Acc@1: 0.6999444135630906
length: 17990
Accuracy for each data type:
Data type Scene Understanding: 80.37%
Data type Instance Identity: 80.45%
Data type Instance Location: 78.03%
Data type Instance Attributes: 72.39%
Data type Instances Counting: 69.19%
Data type Spatial Relation: 59.82%
Data type Instance Interaction: 77.32%
Data type Visual Reasoning: 78.85%
Data type Text Understanding: 55.81%
Data type Action Recognition: 54.08%
Data type Action Prediction: 44.82%
Data type Procedure Understanding: 40.18%
Total accuracy: 69.99%
Image accuracy: 75.99%
Video accuracy: 47.27%

MMVP#

The MMVP dataset is designed to benchmark the performance of multimodal large language models (MLLMs) in visual question answering tasks. This dataset focuses on identifying “CLIP-blind pairs,” which are images that appear similar to the CLIP model despite having clear visual differences. The MMVP dataset includes 300 images derived from ImageNet-1k and LAION-Aesthetics, each paired with straightforward questions to evaluate the models’ visual capabilities. It highlights the challenges these systems face, often leading to incorrect responses and hallucinated explanations.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mmvp --dynamic

The expected test results are:

Evaluating MMVP ...
Results saved to results/MMVP_240727104303.jsonl
The accuracy is 0.5933333333333334

LLaVA-Bench (GPT-4-0613)#

⚠️ Warning: Here, we use GPT-4-0613 as the judge model, while in VLMEvalKit, GPT-4-Turbo is used as the judge model. Using different versions of GPT-4 can result in significant score variations. Therefore, testing the same model with the two codebases can lead to notable score differences.

The LLaVA-Bench-in-the-Wild dataset is designed to evaluate the capabilities of MLLMs in handling more complex and diverse visual tasks. It includes a set of 24 images with 60 associated questions, covering a range of indoor and outdoor scenes, memes, paintings, and sketches. Each image is paired with detailed, manually curated descriptions and questions that test the model’s generalizability to novel domains.

export OPENAI_API_KEY='your openai key'
GPUS=1 sh evaluate.sh pretrained/InternVL-Chat-V1-5 llava-bench --dynamic

The expected test results are:

all *93.8* 85.8 80.5
llava_bench_complex [8.643, 8.286] 95.9
llava_bench_complex 95.9 86.4 82.9
llava_bench_conv [8.471, 8.029] 94.8
llava_bench_conv 94.8 84.7 80.3
llava_bench_detail [8.6, 7.633] 88.8
llava_bench_detail 88.8 86.0 76.3

MathVista#

The MathVista dataset is a comprehensive benchmark for evaluating mathematical reasoning within visual contexts. It consists of three newly created datasets—IQTest, FunctionQA, and PaperQA—designed to address logical reasoning on puzzle test figures, algebraic reasoning over functional plots, and scientific reasoning with academic paper figures, respectively.

export OPENAI_API_KEY='your-openai-key'
GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mathvista-testmini --dynamic

The expected test results are:

Correct: 535, Total: 1000, Accuracy: 53.5%
1000
Number of test problems: 1000

Type: [question_type]
[free_form]: 47.17% (217/460)
[multi_choice]: 58.89% (318/540)

Type: [answer_type]
[float]: 0.00% (0/40)
[integer]: 51.67% (216/418)
[text]: 58.89% (318/540)
[list]: 50.00% (1/2)

Type: [language]
[english]: 53.31% (499/936)
[chinese]: 56.45% (35/62)
[persian]: 50.00% (1/2)

RefCOCO Series#

RefCOCO, RefCOCO+, and RefCOCOg are datasets used for tasks involving referring expression comprehension, segmentation, and generation. These datasets are built upon the MSCOCO dataset, and they are essential for evaluating models in natural language processing and computer vision.

GPUS=8 sh evalulate.sh pretrained/InternVL-Chat-V1-5 refcoco --dynamic

The expected test results are:

RefCOCO val, 91.4
RefCOCO testA, 93.7
RefCOCO testB, 87.1
RefCOCO+ val, 87.0
RefCOCO+ testA, 92.3
RefCOCO+ testB, 80.9
RefCOCO‑g val, 88.5
RefCOCO‑g test, 89.3

MVBench#

MVBench is a comprehensive multimodal video understanding benchmark developed to evaluate the temporal comprehension capabilities of MLLMs. It includes 20 challenging video tasks that require temporal understanding and cannot be effectively solved using a single frame. The benchmark uses a novel static-to-dynamic method, transforming static tasks into dynamic ones to systematically generate video tasks that demand a wide range of temporal skills, from perception to cognition.

We evaluate our models on MVBench by extracting 16 frames from each video, and each frame was resized to a 448x448 image.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-5 mvbench --dynamic --max-num 1

The expected test results are:

{"Action Sequence": 63.0, "Action Prediction": 65.0, "Action Antonym": 51.0, "Fine-grained Action": 44.0, 
"Unexpected Action": 74.5, "Object Existence": 48.5, "Object Interaction": 71.5, "Object Shuffle": 36.5, 
"Moving Direction": 37.5, "Action Localization": 36.5, "Scene Transition": 85.5, "Action Count": 36.0, 
"Moving Count": 39.5, "Moving Attribute": 61.5, "State Change": 50.0, "Fine-grained Pose": 58.5, 
"Character Order": 62.0, "Egocentric Navigation": 35.0, "Episodic Reasoning": 44.5, 
"Counterfactual Inference": 42.0, "Avg": 52.125}

Evaluation using VLMEvalKit Codebase#

Data Preparation#

VLMEvalKit will automatically download the data for evaluation, so you do not need to prepare it manually.

MathVista#

torchrun --nproc-per-node=8 run.py --data MathVista_MINI --model InternVL-Chat-V1-5 --verbose

The expected test results are:

"Task&Skill","tot","prefetch","hit","prefetch_rate","acc"
"Overall","1000","545","544","54.50000000000001","54.400000000000006"
"scientific reasoning","122","89","69","72.95081967213115","56.557377049180324"
"textbook question answering","158","102","85","64.55696202531645","53.79746835443038"
"numeric commonsense","144","39","42","27.083333333333332","29.166666666666668"
"arithmetic reasoning","353","145","208","41.07648725212464","58.92351274787535"
"visual question answering","179","92","92","51.39664804469274","51.39664804469274"
"geometry reasoning","239","150","98","62.76150627615063","41.00418410041841"
"algebraic reasoning","281","178","114","63.345195729537366","40.569395017793596"
"geometry problem solving","208","141","81","67.78846153846155","38.94230769230769"
"math word problem","186","68","119","36.55913978494624","63.97849462365591"
"logical reasoning","37","16","6","43.24324324324324","16.216216216216218"
"figure question answering","269","142","167","52.78810408921933","62.0817843866171"
"statistical reasoning","301","140","216","46.51162790697674","71.76079734219269"

HallusionBench#

HallusionBench is a comprehensive benchmark designed to evaluate image-context reasoning in MLLMs, focusing on identifying issues related to language hallucination and visual illusion. The dataset consists of 346 images paired with 1,129 questions crafted by human experts. These questions are divided into two categories: Visual Dependent (VD) and Visual Supplement (VS), allowing the benchmark to assess the nuanced understanding and interpretation of visual data by MLLMs.

torchrun --nproc-per-node=8 run.py --data HallusionBench --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 00:46:23,688 - Evaluation - INFO - Score:
2024-04-29 00:46:23,688 - Evaluation - INFO -           split       aAcc       fAcc       qAcc
     Overall  66.771819  40.173410  40.879121
          VD  63.620981  40.000000  34.296029
          VS  71.944444  40.517241  51.123596
   VD_figure  77.500000  65.853659  53.846154
      VS_map  56.250000  18.181818  18.750000
 VD_illusion  66.666667  41.935484  34.722222
    VS_table  75.892857  46.428571  55.813953
      VD_ocr  78.651685  58.139535  58.139535
      VS_ocr  59.259259  38.461538  22.222222
    VS_chart  81.538462  50.000000  72.368421
   VD_video  51.176471  10.416667  13.043478
    VD_math  56.481481  25.000000  27.777778


result = (66.77 + 40.17 + 40.87) / 3 = 49.3

MMStar#

The MMStar dataset is an advanced multimodal benchmark designed to evaluate the capabilities of MLLMs. It comprises 1,500 carefully selected samples that are balanced and purified to ensure they exhibit visual dependency and minimal data leakage. The dataset evaluates models across six core capabilities and 18 detailed axes, focusing on complex multimodal tasks that require advanced reasoning and understanding of visual content.

torchrun --nproc-per-node=8 run.py --data MMStar --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 18:21:56,491 - Evaluation - INFO -   split   Overall  ...   math  science & technology
0  none  0.572667  ...  0.564                 0.408

OCRBench#

OCRBench is a comprehensive evaluation benchmark designed to assess the OCR capabilities of MLLMs. It includes five components: Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). The benchmark encompasses data from 29 datasets, making it one of the most thorough OCR evaluation tools available. OCRBench aims to reveal both the strengths and weaknesses of MLLMs, particularly in handling multilingual text, handwritten text, non-semantic text, and mathematical expressions. The benchmark includes 1,000 question-answer pairs, all manually verified for precision.

torchrun --nproc-per-node=8 run.py --data OCRBench --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 00:28:29,681 - Evaluation - INFO - Score:
2024-04-29 00:28:29,681 - Evaluation - INFO - Text Recognition:238
2024-04-29 00:28:29,681 - Evaluation - INFO - Scene Text-centric VQA:178
2024-04-29 00:28:29,681 - Evaluation - INFO - Doc-oriented VQA:151
2024-04-29 00:28:29,681 - Evaluation - INFO - Key Information Extraction:153
2024-04-29 00:28:29,681 - Evaluation - INFO - Handwritten Mathematical Expression Recognition:4
2024-04-29 00:28:29,681 - Evaluation - INFO - Final Score:724
2024-04-29 00:28:29,681 - Evaluation - INFO - Final Score Norm:72.4

MMMU#

torchrun --nproc-per-node=8 run.py --data MMMU_DEV_VAL --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 18:20:04,977 - Evaluation - INFO -         split  Overall  ...  Science  Tech & Engineering
0         dev     0.48  ...     0.36            0.428571
1  validation     0.45  ...     0.38            0.371429

RealWorldQA#

The RealWorldQA dataset is a benchmark designed to evaluate the real-world spatial understanding capabilities of multimodal AI models. It consists of over 700 images, each accompanied by a question and a verifiable answer, focusing on various real-world scenarios, including those captured from vehicles. This dataset aims to test how well AI models comprehend physical environments and spatial relations, enhancing their ability to interpret and analyze real-world scenes.

torchrun --nproc-per-node=8 run.py --data RealWorldQA --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 00:35:13,282 - Evaluation - INFO - Score:
2024-04-29 00:35:13,282 - Evaluation - INFO -   split   Overall
0  none  0.660131

MMVet (GPT-4-Turbo)#

torchrun --nproc-per-node=8 run.py --data MMVet --model InternVL-Chat-V1-5 --verbose

The expected test results are:

2024-04-29 18:32:38,748 - Evaluation - INFO - Score:
2024-04-29 18:32:38,748 - Evaluation - INFO -   Category  tot        acc
    rec  187  61.818182
    ocr  108  68.981481
   know   84  46.428571
    gen   80  44.875000
   spat   75  63.600000
   math   26  62.307692
Overall  218  61.513761

Note that because the version of GPT-4 used for scoring differs from the official server, the scores tested by VLMEvalKit will be slightly different.

LLaVA-Bench (GPT-4-Turbo)#

torchrun --nproc-per-node=8 run.py --data LLaVABench --model InternVL-Chat-V1-5 --verbose

The expected test results are:

"split","Relative Score (main)","VLM Score","GPT4 Score"
"overall","82.0","63.7","77.7"
"conv","82.9","74.1","89.4"
"detail","72.0","48.0","66.7"
"complex","86.0","65.7","76.4"

VideoMME#

The Video-MME dataset is a comprehensive benchmark designed to evaluate the capabilities of MLLMs in video analysis. It is the first benchmark specifically tailored for this purpose, focusing on a high-quality assessment of models’ performance in processing sequential visual data.

torchrun --nproc-per-node=8 run.py --data Video-MME --model InternVL-Chat-V1-5 --verbose --nframe 16

The expected test results are:

"overall": {
    "overall": "0.5363",
    "domain": {
        "Knowledge": "0.5679",
        "Film & Television": "0.5833",
        "Sports Competition": "0.4756",
        "Artistic Performance": "0.5444",
        "Life Record": "0.5111",
        "Multilingual": "0.5111"
    },
    ...
}

Citation#

If you find this project useful in your research, please consider citing:

@article{chen2024far,
  title={How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites},
  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
  journal={Science China Information Sciences},
  volume={67},
  number={12},
  pages={220101},
  year={2024},
  publisher={Springer}
}
@inproceedings{chen2024internvl,
  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={24185--24198},
  year={2024}
}

Evaluation of InternVL 1.5 Series

Contents

Evaluation of InternVL 1.5 Series#

Model Preparation#

Evaluation using InternVL Codebase#

Data Preparation#

MME#

OKVQA#

TextVQA#

VizWiz#

ChartQA#

DocVQA#

AI2D#

InfographicVQA#

GQA#

ScienceQA#

POPE#

Tiny LVLM#

MMMU#

MMVet (GPT-4-0613)#

MMBench#

CCBench#

SEED#

MMVP#

LLaVA-Bench (GPT-4-0613)#

MathVista#

RefCOCO Series#

MVBench#

Evaluation using VLMEvalKit Codebase#

Data Preparation#

MathVista#

HallusionBench#

MMStar#

OCRBench#

MMMU#

RealWorldQA#

MMVet (GPT-4-Turbo)#

LLaVA-Bench (GPT-4-Turbo)#

VideoMME#

Citation#