Evaluation of InternVL-Chat-V1-1#

To evaluate the performance of the InternVL-Chat-V1-1 model across various tasks, follow the instructions for each specific dataset. Ensure that the appropriate number of GPUs is allocated as specified.

1⃣️ We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. For certain datasets like MMVet and LLaVA-Bench, different GPT-4 versions used as judges cause significant result discrepancies between two codebases.

2⃣️ Please note that evaluating the same model using different testing toolkits like InternVL and VLMEvalKit can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.

3⃣️️ Note, the dataset description is generated by GPT-4 and may contain errors.

Model Preparation#

model name	type	param	download	size
InternVL-Chat-V1-1	MLLM	19.1B	🤗 HF link	35.0 GB

Download the above model weights and place them in the pretrained/ folder.

cd pretrained/
# pip install -U huggingface_hub
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternVL-Chat-V1-1 --local-dir InternVL-Chat-V1-1

The directory structure is:

pretrained
└── InternVL-Chat-V1-1

Evaluation using InternVL Codebase#

Data Preparation#

Please prepare the evaluation data according to the guidance provided here.

MME#

MME is a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on both perception and cognition abilities across 14 different subtasks, ensuring robust and diverse testing of these models.

Please use the following command to perform the test with 1 GPU:

GPUS=1 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mme

The expected test results are:

=========== Perception ===========
total score: 1664.5088035214085

         existence  score: 185.0
         count  score: 173.33333333333334
         position  score: 163.33333333333334
         color  score: 190.0
         posters  score: 161.22448979591837
         celebrity  score: 149.11764705882354
         scene  score: 153.5
         landmark  score: 167.5
         artwork  score: 144.0
         OCR  score: 177.5


=========== Cognition ===========
total score: 360.7142857142857

         commonsense_reasoning  score: 130.71428571428572
         numerical_calculation  score: 70.0
         text_translation  score: 110.0
         code_reasoning  score: 50.0

OKVQA#

OKVQA (Outside Knowledge Visual Question Answering) is a dataset designed for visual question answering tasks that require external knowledge beyond what is visible in the image, featuring over 14,000 questions to evaluate the reasoning abilities of AI models.

Please use the following command to perform the test with 8 GPU:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-okvqa-val

The expected test results are:

okvqa_val 0.6406262386048285

TextVQA#

TextVQA is a dataset designed to evaluate visual question answering models by requiring them to read and reason about text present within images, containing 45,336 questions over 28,408 images from the OpenImages dataset.

The TextVQA dataset provides official OCR results, specifically Rosetta OCR tokens. During testing with InstructBLIP and LLaVA 1.5, the OCR results are input to the LLM as a prompt. If you want to input Rosetta OCR tokens, use the following command:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-textvqa-val-ocr

The expected test results are:

textvqa_val_ocr 0.686240000000003

If you do not want to input Rosetta OCR tokens, use this command:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-textvqa-val

The expected test results are:

textvqa_val 0.6420000000000028

VizWiz#

The VizWiz VQA dataset is a visual question answering dataset created to help answer visual questions posed by blind individuals. It contains over 31,000 visual questions, where users took a picture using a mobile phone and recorded a spoken question about it. Each question comes with 10 crowdsourced answers. This dataset addresses tasks such as predicting the answer to a visual question and determining whether a visual question can be answered.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-vizwiz-val

The expected test results are:

vizwiz_val 0.5899899435054417

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-vizwiz-test

For the test set, submit the results to the evaluation server.

The expected test results are:

57.3

ChartQA#

The ChartQA dataset is a comprehensive benchmark for question answering about charts that involves both visual and logical reasoning. It includes a mix of 9.6K human-written questions and 23.1K machine-generated questions derived from chart summaries. This dataset is designed to evaluate models that can understand and analyze charts by answering complex questions that often require multiple logical and arithmetic operations, as well as referencing visual features of the charts.

The ChartQA dataset includes two test sets: chartqa_test_human and chartqa_test_augmented. The final score for model evaluation is calculated as the average of the scores on these two test sets:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-chartqa-test

The expected test results are:

['chartqa_test_human', {'relaxed_accuracy': 0.4034}]
['chartqa_test_augmented', {'relaxed_accuracy': 0.795}]
average score = (40.34 + 79.5) / 2 = 59.9

DocVQA#

The DocVQA dataset consists of 50,000 questions on 12,000+ document images. It is designed for visual question answering tasks where questions are answered using text within the document images. The dataset includes OCR transcriptions and ground truth answers, supporting evaluation of models that interpret and extract information from documents.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-docvqa-val

The expected test results are:

Overall ANLS: 0.476

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-docvqa-test

For the test set, submit the results to the evaluation server.

The expected test results are:

Overall ANLS: 0.481

AI2D#

The AI2D dataset contains over 5,000 grade school science diagrams with extensive annotations and 15,000 multiple-choice questions for research on diagram understanding and question answering.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-ai2d-test

The expected test results are:

ai2diagram_test {'accuracy': 0.7240140932642487}

InfographicVQA#

The InfographicVQA dataset is a collection of infographics accompanied by natural language questions and answers. This dataset includes a diverse range of infographics sourced from thousands of different websites, ensuring a variety of layouts and designs. It comprises 30,035 questions across 5,485 images, split into training, validation, and test sets.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-infovqa-val

The expected test results are:

Overall ANLS: 0.3334

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-infovqa-test

For the test set, submit the results to the evaluation server.

The expected test results are:

Overall ANLS: 0.320

GQA#

The GQA dataset is a large-scale visual question answering dataset designed for real-world visual reasoning and compositional question answering. It contains over 22 million questions grounded in real images, each accompanied by detailed scene graphs that describe objects, their attributes, and relationships within the scene. The dataset includes images from the Visual Genome dataset, with questions that require various reasoning skills such as spatial understanding and multi-step inference.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 vqa-gqa-testdev

The expected test results are:

Accuracy: 62.46%

ScienceQA#

The ScienceQA dataset is a large-scale benchmark for multimodal science question answering, consisting of 21,208 multiple-choice questions derived from elementary and high school science curricula. This dataset features a diverse range of topics across natural science, social science, and language science. It includes questions with image context (48.7%), text context (48.2%), and both (30.8%).

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 scienceqa

The expected test results are:

Acc@1: 0.90133019335647

POPE#

The POPE (Polling-based Object Probing Evaluation) dataset is designed to evaluate object hallucination in MLLMs. The dataset consists of 3,000 questions related to the captions of 500 images. By treating the MLLMs’ answers to these questions as a binary classification task, the dataset allows researchers to measure accuracy, precision, recall, and F1 scores to determine the extent of hallucination in the models.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 pope

The expected test results are:

Category: random, # samples: 2910
TP      FP      TN      FN
1196    14      1396    304
Accuracy: 0.8907216494845361
Precision: 0.9884297520661157
Recall: 0.7973333333333333
F1 score: 0.8826568265682657
Yes ratio: 0.41580756013745707
0.883, 0.891, 0.988, 0.797, 0.416
====================================
Category: popular, # samples: 3000
TP      FP      TN      FN
1196    47      1453    304
Accuracy: 0.883
Precision: 0.9621882542236525
Recall: 0.7973333333333333
F1 score: 0.8720379146919431
Yes ratio: 0.41433333333333333
0.872, 0.883, 0.962, 0.797, 0.414
====================================
Category: adversarial, # samples: 3000
TP      FP      TN      FN
1196    89      1411    304
Accuracy: 0.869
Precision: 0.930739299610895
Recall: 0.7973333333333333
F1 score: 0.858886894075404
Yes ratio: 0.42833333333333334
0.859, 0.869, 0.931, 0.797, 0.428
====================================

(88.3 + 87.2 + 85.9) / 3 = 87.1

Tiny LVLM#

The Tiny LVLM-eHub is a streamlined evaluation benchmark designed to assess the multimodal capabilities of MLLMs, including models like Bard. It focuses on six categories of multimodal abilities: visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, and embodied intelligence.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 tiny_lvlm

The expected test results are:

Visual_Knowledge_Acquisition: 0.74
Object_Hallucination: 0.8966666666666666
Visual_Commonsense: 0.6
Visual_Perception: 0.574
Visual_Reasoning: 0.6218181818181818
Overall: 3.4324848484848486

MMMU#

The MMMU dataset is a comprehensive benchmark designed to evaluate multimodal models on college-level tasks that require domain-specific knowledge and reasoning. It includes 11,500 questions sourced from college exams, quizzes, and textbooks, spanning six disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions cover 30 subjects and feature 30 types of images, such as charts, diagrams, maps, tables, and more.

For the validation set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmmu-val

The expected test results are:

{'Overall-Art and Design': {'num': 120, 'acc': 0.558}, 'Art': {'num': 30, 'acc': 0.633}, 'Art_Theory': {'num': 30, 'acc': 0.7}, 'Design': {'num': 30, 'acc': 0.633}, 'Music': {'num': 30, 'acc': 0.267}, 'Overall-Business': {'num': 150, 'acc': 0.313}, 'Accounting': {'num': 30, 'acc': 0.333}, 'Economics': {'num': 30, 'acc': 0.4}, 'Finance': {'num': 30, 'acc': 0.133}, 'Manage': {'num': 30, 'acc': 0.433}, 'Marketing': {'num': 30, 'acc': 0.267}, 'Overall-Science': {'num': 150, 'acc': 0.333}, 'Biology': {'num': 30, 'acc': 0.367}, 'Chemistry': {'num': 30, 'acc': 0.3}, 'Geography': {'num': 30, 'acc': 0.267}, 'Math': {'num': 30, 'acc': 0.4}, 'Physics': {'num': 30, 'acc': 0.333}, 'Overall-Health and Medicine': {'num': 150, 'acc': 0.393}, 'Basic_Medical_Science': {'num': 30, 'acc': 0.367}, 'Clinical_Medicine': {'num': 30, 'acc': 0.433}, 'Diagnostics_and_Laboratory_Medicine': {'num': 30, 'acc': 0.4}, 'Pharmacy': {'num': 30, 'acc': 0.367}, 'Public_Health': {'num': 30, 'acc': 0.4}, 'Overall-Humanities and Social Science': {'num': 120, 'acc': 0.542}, 'History': {'num': 30, 'acc': 0.567}, 'Literature': {'num': 30, 'acc': 0.767}, 'Sociology': {'num': 30, 'acc': 0.4}, 'Psychology': {'num': 30, 'acc': 0.433}, 'Overall-Tech and Engineering': {'num': 210, 'acc': 0.29}, 'Agriculture': {'num': 30, 'acc': 0.433}, 'Architecture_and_Engineering': {'num': 30, 'acc': 0.267}, 'Computer_Science': {'num': 30, 'acc': 0.233}, 'Electronics': {'num': 30, 'acc': 0.333}, 'Energy_and_Power': {'num': 30, 'acc': 0.2}, 'Materials': {'num': 30, 'acc': 0.233}, 'Mechanical_Engineering': {'num': 30, 'acc': 0.333}, 'Overall': {'num': 900, 'acc': 0.388}}

For the test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmmu-test

Then submit the results to the evaluation server. The expected test results are:

All subject resultes
{'Overall-Art & Design': {'num': 1163, 'acc': 0.537}, 'Art': {'num': 231, 'acc': 0.606}, 'Art_Theory': {'num': 429, 'acc': 0.59}, 'Design': {'num': 169, 'acc': 0.746}, 'Music': {'num': 334, 'acc': 0.314}, 'Overall-Business': {'num': 1428, 'acc': 0.317}, 'Accounting': {'num': 380, 'acc': 0.345}, 'Economics': {'num': 267, 'acc': 0.345}, 'Finance': {'num': 355, 'acc': 0.245}, 'Manage': {'num': 245, 'acc': 0.29}, 'Marketing': {'num': 181, 'acc': 0.392}, 'Overall-Science': {'num': 2426, 'acc': 0.282}, 'Biology': {'num': 345, 'acc': 0.357}, 'Chemistry': {'num': 603, 'acc': 0.244}, 'Geography': {'num': 565, 'acc': 0.312}, 'Math': {'num': 505, 'acc': 0.275}, 'Physics': {'num': 408, 'acc': 0.24}, 'Overall-Health & Medicine': {'num': 1752, 'acc': 0.365}, 'Basic_Medical_Science': {'num': 326, 'acc': 0.436}, 'Clinical_Medicine': {'num': 325, 'acc': 0.397}, 'Diagnostics_and_Laboratory_Medicine': {'num': 162, 'acc': 0.364}, 'Pharmacy': {'num': 430, 'acc': 0.309}, 'Public_Health': {'num': 509, 'acc': 0.346}, 'Overall-Humanities & Social Science': {'num': 947, 'acc': 0.564}, 'History': {'num': 278, 'acc': 0.579}, 'Literature': {'num': 112, 'acc': 0.804}, 'Sociology': {'num': 252, 'acc': 0.556}, 'Psychology': {'num': 305, 'acc': 0.469}, 'Overall-Tech & Engineering': {'num': 2784, 'acc': 0.28}, 'Agriculture': {'num': 287, 'acc': 0.369}, 'Architecture_and_Engineering': {'num': 551, 'acc': 0.272}, 'Computer_Science': {'num': 371, 'acc': 0.315}, 'Electronics': {'num': 256, 'acc': 0.152}, 'Energy_and_Power': {'num': 432, 'acc': 0.306}, 'Materials': {'num': 458, 'acc': 0.26}, 'Mechanical_Engineering': {'num': 429, 'acc': 0.27}, 'Overall': {'num': 10500, 'acc': 0.353}}

Leaderboard
[{'test_split': {'Art & Design': 0.537, 'Business': 0.317, 'Science': 0.282, 'Health & Medicine': 0.365, 'Humanities & Social Science': 0.564, 'Tech & Engineering': 0.28, 'Overall': 0.353}}]

MMVet (GPT-4-0613)#

⚠️ Warning: Here, we use GPT-4-0613 as the judge model, while in VLMEvalKit, GPT-4-Turbo is used as the judge model. Using different versions of GPT-4 can result in significant score variations. Therefore, testing the same model with the two codebases can lead to notable score differences.

The MM-Vet dataset is a comprehensive benchmark designed to evaluate the integrated capabilities of MLLMs. It encompasses six core vision-language (VL) capabilities: recognition, knowledge, optical character recognition (OCR), spatial awareness, language generation, and math. The dataset includes 200 images and 218 questions, each requiring one or more of these capabilities to answer. The evaluation uses an open-ended LLM-based approach, allowing assessment across various answer styles and question types.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmvet

Then, submit the results to the evaluation server. The expected test results are:

runs: [46.7]

MMBench#

The MMBench dataset is a comprehensive multi-modality benchmark designed to evaluate the fine-grained abilities of vision-language models. It contains around 3,000 multiple-choice questions covering 20 ability dimensions, structured into a hierarchical taxonomy. These dimensions include perception and reasoning abilities, further broken down into specific skills like coarse and fine-grained perception, attribute reasoning, and logic reasoning.

For the English dev / test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmbench-dev-en
GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmbench-test-en

Then, submit the results to the evaluation server. The expected test results are:

mmbench-dev-en: 76.7
mmbench-test-en: 75.4

For the Chinese dev / test set, run:

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmbench-dev-cn
GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmbench-test-cn

Then, submit the results to the evaluation server. The expected test results are:

mmbench-dev-cn: 71.9
mmbench-test-cn: 70.3

CCBench#

CCBench, a multi-modal benchmark in the domain of Chinese Culture, is designed to evaluate the performance of MLLMs on tasks specifically related to Chinese cultural content.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 ccbench-dev

Then, submit the results to the evaluation server. The expected test results are:

ccbench-dev: 43.3

SEED#

CCBench is a multimodal benchmark specifically designed to evaluate models on tasks related to Chinese culture. It is part of the larger MMBench suite of benchmarks, developed by the OpenCompass Community, and aims to provide fine-grained evaluations across various capabilities of vision-language models. CCBench includes 510 questions in a multiple-choice format, focusing on cultural knowledge and understanding.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 seed

The expected test results are:

Data type Scene Understanding: 78.63%
Data type Instance Identity: 77.44%
Data type Instance Location: 74.66%
Data type Instance Attributes: 70.04%
Data type Instances Counting: 65.79%
Data type Spatial Relation: 58.90%
Data type Instance Interaction: 77.32%
Data type Visual Reasoning: 79.15%
Data type Text Understanding: 39.53%
Data type Action Recognition: 54.34%
Data type Action Prediction: 40.82%
Data type Procedure Understanding: 37.24%
Total accuracy: 67.40%
+ Image accuracy: 73.24%
Video accuracy: 45.28%

MMVP#

The MMVP dataset is designed to benchmark the performance of multimodal large language models (MLLMs) in visual question answering tasks. This dataset focuses on identifying “CLIP-blind pairs,” which are images that appear similar to the CLIP model despite having clear visual differences. The MMVP dataset includes 300 images derived from ImageNet-1k and LAION-Aesthetics, each paired with straightforward questions to evaluate the models’ visual capabilities. It highlights the challenges these systems face, often leading to incorrect responses and hallucinated explanations.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mmvp

The expected test results are:

Evaluating MMVP ...
Results saved to results/MMVP_240725163208.jsonl
The accuracy is 0.4466666666666667

LLaVA-Bench (GPT-4-0613)#

⚠️ Warning: Here, we use GPT-4-0613 as the judge model, while in VLMEvalKit, GPT-4-Turbo is used as the judge model. Using different versions of GPT-4 can result in significant score variations. Therefore, testing the same model with the two codebases can lead to notable score differences.

The LLaVA-Bench-in-the-Wild dataset is designed to evaluate the capabilities of MLLMs in handling more complex and diverse visual tasks. It includes a set of 24 images with 60 associated questions, covering a range of indoor and outdoor scenes, memes, paintings, and sketches. Each image is paired with detailed, manually curated descriptions and questions that test the model’s generalizability to novel domains.

export OPENAI_API_KEY='your openai key'
GPUS=1 sh evaluate.sh pretrained/InternVL-Chat-V1-1 llava-bench

The expected test results are:

all *72.8* 87.7 63.8
llava_bench_complex [8.75, 6.643] 75.9
llava_bench_complex 75.9 87.5 66.4
llava_bench_conv [9.0, 6.118] 68.0
llava_bench_conv 68.0 90.0 61.2
llava_bench_detail [8.533, 6.2] 72.7
llava_bench_detail 72.7 85.3 62.0

RefCOCO Series#

RefCOCO, RefCOCO+, and RefCOCOg are datasets used for tasks involving referring expression comprehension, segmentation, and generation. These datasets are built upon the MSCOCO dataset, and they are essential for evaluating models in natural language processing and computer vision.

GPUS=8 sh evalulate.sh pretrained/InternVL-Chat-V1-1 refcoco

The expected test results are:

RefCOCO val, 84.7
RefCOCO testA, 89.9
RefCOCO testB, 78.6
RefCOCO+ val, 78.5
RefCOCO+ testA, 85.6
RefCOCO+ testB, 70.1
RefCOCO‑g val, 81.0
RefCOCO‑g test, 81.4

MVBench#

MVBench is a comprehensive multimodal video understanding benchmark developed to evaluate the temporal comprehension capabilities of MLLMs. It includes 20 challenging video tasks that require temporal understanding and cannot be effectively solved using a single frame. The benchmark uses a novel static-to-dynamic method, transforming static tasks into dynamic ones to systematically generate video tasks that demand a wide range of temporal skills, from perception to cognition.

We evaluate our models on MVBench by extracting 16 frames from each video, and each frame was resized to a 448x448 image.

GPUS=8 sh evaluate.sh pretrained/InternVL-Chat-V1-1 mvbench --load-in-8bit

The expected test results are:

{"Action Sequence": 62.5, "Action Prediction": 56.49999999999999, "Action Antonym": 49.0, 
"Fine-grained Action": 41.0, "Unexpected Action": 64.5, "Object Existence": 49.0, 
"Object Interaction": 64.0, "Object Shuffle": 32.5, "Moving Direction": 39.5, 
"Action Localization": 27.500000000000004, "Scene Transition": 88.5, "Action Count": 42.0,
"Moving Count": 33.0, "Moving Attribute": 57.99999999999999, "State Change": 46.5, 
"Fine-grained Pose": 44.0, "Character Order": 59.0, "Egocentric Navigation": 38.5,
"Episodic Reasoning": 44.5, "Counterfactual Inference": 39.0, "Avg": 48.949999999999996}

Evaluation using VLMEvalKit Codebase#

Data Preparation#

VLMEvalKit will automatically download the data for evaluation, so you do not need to prepare it manually.

MathVista#

The MathVista dataset is a comprehensive benchmark for evaluating mathematical reasoning within visual contexts. It consists of three newly created datasets—IQTest, FunctionQA, and PaperQA—designed to address logical reasoning on puzzle test figures, algebraic reasoning over functional plots, and scientific reasoning with academic paper figures, respectively.

torchrun --nproc-per-node=8 run.py --data MathVista_MINI --model InternVL-Chat-V1-1 --verbose

The expected test results are:

--  ---------------------------  ----  ---  ---  -------  -------
Overall                      1000  594  363  59.4     36.3
scientific reasoning          122   95   58  77.8689  47.541
textbook question answering   158  107   74  67.7215  46.8354
numeric commonsense           144   67   56  46.5278  38.8889
arithmetic reasoning          353  143  122  40.5099  34.5609
visual question answering     179  102   80  56.9832  44.6927
geometry reasoning            239  198   63  82.8452  26.3598
algebraic reasoning           281  217   74  77.2242  26.3345
geometry problem solving      208  181   50  87.0192  24.0385
math word problem             186   74   66  39.7849  35.4839
logical reasoning              37   25    5  67.5676  13.5135
figure question answering     269  130   93  48.3271  34.5725
statistical reasoning         301  126  109  41.8605  36.2126
--  ---------------------------  ----  ---  ---  -------  -------

HallusionBench#

HallusionBench is a comprehensive benchmark designed to evaluate image-context reasoning in MLLMs, focusing on identifying issues related to language hallucination and visual illusion. The dataset consists of 346 images paired with 1,129 questions crafted by human experts. These questions are divided into two categories: Visual Dependent (VD) and Visual Supplement (VS), allowing the benchmark to assess the nuanced understanding and interpretation of visual data by MLLMs.

torchrun --nproc-per-node=8 run.py --data HallusionBench --model InternVL-Chat-V1-1 --verbose

The expected test results are:

"split","aAcc","fAcc","qAcc"
"Overall","56.256572029442694","26.011560693641616","26.153846153846157"
"VD","54.483925549915405","29.565217391304348","23.465703971119133"
"VS","59.166666666666664","18.96551724137931","30.337078651685395"
"VD_figure","70.0","51.21951219512195","41.02564102564102"
"VD_ocr","75.28089887640449","53.48837209302325","51.162790697674424"
"VD_math","47.22222222222222","5.555555555555555","18.51851851851852"
"VS_ocr","53.70370370370371","23.076923076923077","11.11111111111111"
"VS_map","54.6875","13.636363636363635","12.5"
"VS_chart","62.30769230769231","17.5","44.73684210526316"
"VD_video","42.35294117647059","10.416666666666668","5.797101449275362"
"VD_illusion","52.77777777777778","27.419354838709676","18.055555555555554"
"VS_table","60.71428571428571","21.428571428571427","30.23255813953488"

result = (56.256572029442694 + 26.011560693641616 + 26.153846153846157) / 3 = 36.1

MMStar#

The MMStar dataset is an advanced multimodal benchmark designed to evaluate the capabilities of MLLMs. It comprises 1,500 carefully selected samples that are balanced and purified to ensure they exhibit visual dependency and minimal data leakage. The dataset evaluates models across six core capabilities and 18 detailed axes, focusing on complex multimodal tasks that require advanced reasoning and understanding of visual content.

torchrun --nproc-per-node=8 run.py --data MMStar --model InternVL-Chat-V1-1 --verbose

The expected test results are:

"split","Overall","coarse perception","fine-grained perception","instance reasoning","logical reasoning","math","science & technology"
"none","0.452","0.652","0.384","0.612","0.404","0.292","0.368"

OCRBench#

OCRBench is a comprehensive evaluation benchmark designed to assess the OCR capabilities of MLLMs. It includes five components: Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). The benchmark encompasses data from 29 datasets, making it one of the most thorough OCR evaluation tools available. OCRBench aims to reveal both the strengths and weaknesses of MLLMs, particularly in handling multilingual text, handwritten text, non-semantic text, and mathematical expressions. The benchmark includes 1,000 question-answer pairs, all manually verified for precision.

torchrun --nproc-per-node=8 run.py --data OCRBench --model InternVL-Chat-V1-1 --verbose

The expected test results are:

{
    "Text Recognition": 230,
    "Scene Text-centric VQA": 157,
    "Doc-oriented VQA": 72,
    "Key Information Extraction": 71,
    "Handwritten Mathematical Expression Recognition": 0,
    "Final Score": 530,
    "Final Score Norm": 53.0
}

MMMU#

torchrun --nproc-per-node=8 run.py --data MMMU_DEV_VAL --model InternVL-Chat-V1-1 --verbose

The expected test results are:

-----------------------------------  -------------------  -------------------
split                                validation           dev
Overall                              0.4022222222222222   0.47333333333333333
Accounting                           0.36666666666666664  0.2
Agriculture                          0.4666666666666667   0.2
Architecture_and_Engineering         0.23333333333333334  0.4
Art                                  0.6333333333333333   0.8
Art_Theory                           0.7333333333333333   0.6
Basic_Medical_Science                0.4                  0.6
Biology                              0.36666666666666664  0.4
Chemistry                            0.4666666666666667   0.8
Clinical_Medicine                    0.4                  0.4
Computer_Science                     0.26666666666666666  0.6
Design                               0.6333333333333333   0.6
Diagnostics_and_Laboratory_Medicine  0.4                  0.8
Economics                            0.43333333333333335  0.8
Electronics                          0.4                  0.4
Energy_and_Power                     0.2                  0.2
Finance                              0.3333333333333333   0.8
Geography                            0.26666666666666666  0.0
History                              0.5333333333333333   0.6
Literature                           0.7666666666666667   0.4
Manage                               0.4666666666666667   0.2
Marketing                            0.26666666666666666  0.8
Materials                            0.16666666666666666  0.2
Math                                 0.43333333333333335  0.4
Mechanical_Engineering               0.26666666666666666  0.2
Music                                0.26666666666666666  0.2
Pharmacy                             0.4                  0.6
Physics                              0.3333333333333333   0.2
Psychology                           0.43333333333333335  0.6
Public_Health                        0.36666666666666664  0.4
Sociology                            0.36666666666666664  0.8
Art & Design                         0.5666666666666667   0.55
Business                             0.37333333333333335  0.56
Health & Medicine                    0.3933333333333333   0.56
Humanities & Social Science          0.525                0.6
Science                              0.37333333333333335  0.36
Tech & Engineering                   0.2857142857142857   0.3142857142857143
-----------------------------------  -------------------  -------------------

RealWorldQA#

The RealWorldQA dataset is a benchmark designed to evaluate the real-world spatial understanding capabilities of multimodal AI models. It consists of over 700 images, each accompanied by a question and a verifiable answer, focusing on various real-world scenarios, including those captured from vehicles. This dataset aims to test how well AI models comprehend physical environments and spatial relations, enhancing their ability to interpret and analyze real-world scenes.

torchrun --nproc-per-node=8 run.py --data RealWorldQA --model InternVL-Chat-V1-1 --verbose

The expected test results are:

"split","Overall"
"none","0.5803921568627451"

MMVet (GPT-4-Turbo)#

torchrun --nproc-per-node=8 run.py --data MMVet --model InternVL-Chat-V1-1 --verbose

The expected test results are:

-  -------  ---  -------
rec      187  49.9465
ocr      108  42.5
know      84  36.4286
gen       80  36.25
spat      75  42.9333
math      26  22.3077
Overall  218  44.7706
-  -------  ---  -------

Note that because the version of GPT-4 used for scoring differs from the official server, the scores tested by VLMEvalKit will be slightly different.

LLaVA-Bench (GPT-4-Turbo)#

torchrun --nproc-per-node=8 run.py --data LLaVABench --model InternVL-Chat-V1-1 --verbose

The expected test results are:

-  -------  ----  ----  ----
0  overall *64.8* 46.7  72
1  complex  64.4  47.1  73.2
2  conv     67.1  57.6  85.9
3  detail   61.7  33.3  54
-  -------  ----  ----  ----

Citation#

If you find this project useful in your research, please consider citing:

@inproceedings{chen2024internvl,
  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={24185--24198},
  year={2024}
}

Evaluation of InternVL-Chat-V1-1

Contents

Evaluation of InternVL-Chat-V1-1#

Model Preparation#

Evaluation using InternVL Codebase#

Data Preparation#

MME#

OKVQA#

TextVQA#

VizWiz#

ChartQA#

DocVQA#

AI2D#

InfographicVQA#

GQA#

ScienceQA#

POPE#

Tiny LVLM#

MMMU#

MMVet (GPT-4-0613)#

MMBench#

CCBench#

SEED#

MMVP#

LLaVA-Bench (GPT-4-0613)#

RefCOCO Series#

MVBench#

Evaluation using VLMEvalKit Codebase#

Data Preparation#

MathVista#

HallusionBench#

MMStar#

OCRBench#

MMMU#

RealWorldQA#

MMVet (GPT-4-Turbo)#

LLaVA-Bench (GPT-4-Turbo)#

Citation#