Retrieve-then-compare mitigates visual hallucination in multi-modal large language models

2.2.1. Parameter tuning

Approaches for mitigating visual hallucinations in MLLMs through parameter tuning can be categorized into supervised fine-tuning (SFT) methods and preference optimization techniques. Effective SFT-based methods include curating multi-modal instruction tuning dataset with distracting instructions ^[19], and the provision of extra supervision to promote image-text feature alignment ^[14] or the distinctiveness of visual features ^[7]. On the other hand, preference optimization methods ^{[34, 35, 36]} construct multi-modal pairwise preference data comprising accurate and erroneous responses and optimize MLLMs using the direct preference optimization (DPO) loss ^[37]. These methods incorporate inferior responses during training, which may enhance their ability to suppress hallucinations. Nonetheless, for methods relying on SFT or DPO, the training cost becomes prohibitive for large-scale MLLMs. Furthermore, excessive parameter tuning may impair some of the strengths of MLLMs, such as the capability to provide detailed descriptions ^[19], when the training recipe is suboptimal.

2.2.2. Model ensemble

Integrating knowledge from other models compensates for MLLMs' own shortcomings. Feasible methods include improving the object recognition accuracy through ensembling object detectors ^[38], and obtaining distinctive image features by ensembling various pretrained vision encoders ^[5]. Another line of work utilizes a language model to post-hoc revise visual hallucinations ^{[16, 17]}. Key challenges within this paradigm include tailoring interfaces for various task-specific models, and automating their selection based on the hallucination categories.

2.2.3. Decoding strategy

Intervening the decoding process of MLLMs is a more efficient approach compared to model training and ensemble. For instance, OPERA ^[13] directly discards the candidates that may skew subsequent content toward hallucination and reelects the others. visual contrastive decoding (VCD) ^[15] and its variants ^{[21, 39]} extend CD ^{[40, 41]}, which aims to mitigate factual hallucinations in large language models (LLMs), to the vision domain. This line of work distorts the visual input to amplify the language priors, and downgrades the candidates advocated merely by the language priors through logit subtraction. Thus, they are capable of reducing the erroneous language bias inherent in the decoder of MLLMs. However, hallucinations originating from the visual branch of MLLMs have not been examined. As explained in Section 3.2.1, this kind of hallucination may be exacerbated by VCD-based methods. In this study, we investigate the respective impact of the visual and textual input modalities on the hallucinatory content produced by MLLMs. Based on our findings, we propose an approach to mitigate visual hallucinations arising from both the visual and textual branches of MLLMs.

3.2.1. MLLMs are aware of accurate visual cues amid hallucination

To integrally decouple the contribution of the visual branch to MLLMs' predictions, $$\mathrm{\Delta }\mathit{v}\mathit{e}$$ should encapsulate the majority of visual information. To this end, we erase the visual information in the test image $${\mathit{v}}^{\tau }$$ until it is indistinguishable from Gaussian noise, following the image diffusion ^[48] process (Note that we only use the diffusion process to add noise to images, rather than employing the Denoising Diffusion Probabilistic Model (DDPM) to generate images or other content, as done in prior work^{[49, 50, 51]}.),

where $${\overline{\alpha }}_T=\prod _{i=1}^T{\alpha }_i$$ is the cumulative product of the noise schedule parameters $${\alpha }_i$$. $$T$$ is the diffusion step. $$ϵ$$ is Gaussian noise sampled from a normal distribution $$\mathcal{N}(0,I)$$, $$I$$ is the identity matrix. Then the MLLM "blindly" predicts a new score distribution (denoted as the txt scores) using $${\mathit{v}}^d$$, devoid of valid visual cues. Thus, the subtraction of the txt scores from the base scores can be interpreted as the contribution of the visual modality (denoted as the img scores).

To quantify the dependency of each predicted token on the visual input, we propose a metric that combines the Jensen-Shannon Divergence (JSD) between the base scores and the txt scores, and the img score of the top-ranked candidate (denoted as the top-1 img score). A high JSD value suggests significance difference between the predicted probabilities corresponding to the base scores and the txt scores. Consequently, the img scores will not be uniformly distributed. This indicates that some, but not all, candidates are substantially influenced (either advocated or suppressed) by the visual branch. Conversely, if both the JSD value and the top-1 img score are close to zero, we assert that the visual information has minimal impact on predictions at the current decoding step, as ablating the input visual information does not result in a significant difference in the predicted probabilities.

The results of JSD and top-1 img scores are presented in Figure 3A. At the third decoding step, where the erroneous token $$\mathrm{\_}$$green is selected, both the JSD and the top-1 img scores are significantly higher than those of the grammatical tokens (e.g., $$\mathrm{\_}$$on and $$\mathrm{\_}$$a) and are not close to zero. Figure 3B presents the predicted base scores, txt scores, and img scores. First, we observe that the predicted img scores are multimodally distributed among the top-ranked candidates. Notably, both accurate and erroneous candidates obtain relatively high img scores, e.g., +5.008 scores for accurate candidate $$\mathrm{\_}$$gray, +3.898 and +4.250 scores for erroneous candidates $$\mathrm{\_}$$green and $$\mathrm{\_}$$brown, respectively. Therefore, the MLLM (LLaVA-1.5) is not entirely disregarding accurate visual cues when it generates hallucinations in this case. On the other hand, this result also reveals that the visual branch of MLLMs may substantially promote hallucinatory token candidates. Second, certain candidates may receive minimal support or even opposition from the visual branch, e.g., candidate $$\mathrm{\_}$$black gets -1.992 img scores. According to previous studies ^{[15, 21]}, the word 'black' in this example reflects erroneous language inherited from superficial syntactical patterns in the training data. In summary, these findings supplement previous studies, indicating that the hallucinatory token candidates are not solely advocated by textual branch of MLLMs.

Figure 3. Visual hallucination analysis results. (A) the JSD is employed to measure the dependency of each generated token on the visual input. The JSD corresponding to articles and prepositions (such as $$\mathrm{\_}$$a and $$\mathrm{\_}$$on) are close to zero, while the JSD value for the erroneous token $$\mathrm{\_}$$green is significantly higher. (B) LLaVA-1.5 can identify accurate visual cues even amid hallucinations, as the visual information contributes +5.008 confidence scores to the accurate candidate $$\mathrm{\_}$$gray. However, the visual branch also mistakenly supports inaccurate candidates (e.g., +3.898 for $$\mathrm{\_}$$green and +4.250 for $$\mathrm{\_}$$brown). Additionally, images with similar semantics and appearances can induce analogous visual hallucinations. For instance, candidate $$\mathrm{\_}$$green receives high confidence scores (+15.930 and +12.352) in images that do not contain green frogs. JSD: Jensen-Shannon Divergence.

Regarding the example presented in Figure 3B, employing existing methods ^{[15, 21]}, which adds scaled img scores to base scores, can suppress erroneous language priors. For instance, the base score of the erroneous candidate $$\mathrm{\_}$$black will be reduced. However, this approach will further increase the base scores of the hallucinatory candidates $$\mathrm{\_}$$green and $$\mathrm{\_}$$brown, thereby exacerbating hallucinations. In this work, we aim to eliminate this adverse side effect to achieve better mitigation of visual hallucinations.

3.2.2. Analogous visual hallucinations occur among similar images

Having identified that MLLMs are aware of accurate visual cues even amid visual hallucinations, we then investigate whether similar images tend to induce similar hallucinations with identical textual prompts. To this end, we randomly select 100 image-question pairs from the VQAv2 ^[22] validation set to investigate the patterns of incorrect answers. For each test image, reference images with similar semantics and appearance are retrieved from the COCO Caption ^[47] dataset utilizing the image retrieval method specified in Section 4.1. We manually verify that each question and its corresponding ground truth answers strictly adhere to both the test image and the retrieved reference images. Ultimately, we obtained 100 test samples and a total of 325 images, with each test image having an average of 2.25 reference images.

LLaVA-Next's answers for each image-question pair are obtained utilizing multi-nominal sampling strategy with fixed decoding parameters. Specifically, $$N$$ answers are independently sampled for each image (both the test image and references), as illustrated in Figure 4A. Recall that token candidates with higher confidence scores are more likely to be sampled according to Equation (1). Thus, the proportion of each answer's occurrence in the sampled results reflects the model's confidence level in those answers. In practice, we set $$N=20$$ and obtain 6, 500 generated answers. The Accuracy metric defined using Exact Match (EM) evaluates the correctness of each answer. Next, we assess whether a sample exhibits hallucination and if similar images induce analogous hallucinations following the process outlined in Figure 4B. Specifically, if all $$N$$ answers fall within the set of ground truth answers, the current answers are considered to be correct. Otherwise, the test sample is classified as Has Hallucination. Furthermore, for samples exhibiting hallucinations, if at least one incorrect answer exists in the set of reference answers (predicted using the reference images), we determine that analogous hallucination has occurred among similar images. For example, the test sample in Figure 4A is classified as Analogous Hallucination, since the hallucinatory answer "Left" exists in the reference answers (both the test image and reference images show a truck on the right lane). Otherwise, we determine that the test sample has exclusive hallucinations.

Figure 4. Quantitative results supporting our hypothesis that similar images might induce analogous hallucinations. (A) Experiment pipeline for investigating the characteristics of visual hallucinations. image-question pairs are randomly selected from the VQAv2 validation set and similar images are retrieved for each test image. We then independently sample $$N$$ answers for the test image and the retrieved images. (B) We evaluate the correctness of each sampled answer, and analyze the overlap between the incorrect answers predicted using the test image and those derived from the reference images. if all $$N$$ answers are correct, then the test sample is categorized into "Correct". If at least one incorrect answer exists in the reference answers, the sample is categorized into "Analogous Hallucination". Otherwise, it is categorized into "Exclusive Hallucination". (C) LLaVA-Next exhibits hallucinations in over 70% of the selected samples from VQAv2. Among the samples where hallucinations occur, more than two-thirds demonstrate that similar images induce analogous hallucinations. VQAv2: Visual Question Answering version 2.

Experimental Phenomena: The percentage distributions of the defined categories (Correct versus Has Hallucination and Analogous hallucination versus Exclusive Hallucination) are presented in Figure 4C. Notably, LLaVA-Next exhibits hallucinations on $$75.2\mathrm{\%}(\pm 3.2\mathrm{\%})$$ of the selected samples. Among the test samples with hallucinatory answers, $$68.7\mathrm{\%}(\pm 3.4\mathrm{\%}$$) show that similar images induce analogous visual hallucinations. The results are averaged on five separate experiments with different random seeds.

These experimental results support the following assumption:

● Condition: The input textual prefix to the MLLM is identical.

● Conclusion: Similar images are likely to induce analogous visual hallucinations.

3.2.3. Analogous visual hallucinations help discern accurate content

Having established that similar images can induce analogous hallucinations, we further investigate whether these analogous hallucinations can help identify accurate token candidates by analyzing the differences in the confidence score distributions generated using the test image and the reference images. Figure 3B presents four retrieved images, along with their corresponding confidence scores (in the k-NN scores columns).

Experimental Phenomena: Upon comparing the base scores with the four k-NN scores, a notable observation emerges: The score of the accurate candidate $$\mathrm{\_}$$gray decreases significantly from 13.297 to 7.865 (on average across four references). In contrast, the score changes for visually deceptive candidates (We refer to the inaccurate candidates that are erroneously promoted by the visual branch, as visually deceptive candidates).

(e.g., $$\mathrm{\_}$$green, $$\mathrm{\_}$$brown, and $$\mathrm{\_}$$yellow) and textually deceptive candidates (We refer to the inaccurate candidates, which are opposed by the visual branch, as textually deceptive candidates). (e.g., $$\mathrm{\_}$$black) are relatively modest. For instance, candidate $$\mathrm{\_}$$green receives 15.390 and 12.352 scores in images without green frogs. Considering token candidates in the head vocabulary illustrated in Figure 3B, the average score variation for accurate candidates, i.e., $$\mathrm{\_}$$gray, $$\mathrm{\_}$$grey, $$\mathrm{\_}$$white, and $$\mathrm{\_}$$silver, is 3.68, while the average score variation for erroneous or ambiguous candidates is 0.60.

Conclusion: The example in Figure 3B demonstrates that, after replacing the input image with another globally similar image under the same text prefix, the confidence score of the accurate token candidates decreases more significantly than that of the hallucinatory candidates.

Inference: Under the same text prefix, token candidates whose confidence scores vary significantly across similar images are more likely to be the correct ones compared to candidates with moderate score changes. Note that we do not deny that candidates with moderate score changes may also represent common visual content in similar images. However, we emphasize that they are more likely to be hallucinatory content than candidates with significant score variations.

Therefore, the difference in confidence scores predicted by similar images can potentially assist MLLMs in distinguishing between accurate and erroneous token candidates.

4.1.1. Reference database

The reference database is constructed using samples from the COCO Caption dataset. Specifically, both the Karpathy train and rest-val splits ^[52] are included, totaling around 113, 000 samples. This ensures that the database encompasses a wide range of visual concepts. In our database, each sample is stored as a key-value pair for retrieval purposes. The retrieval key is the vector embedding generated by the retriever $$E_R(\cdot )$$, and the retrieval value is the filename of the image on the disk. The retrieval keys are stored in a vector database format and are clustered to facilitate efficient retrieval. Importantly, all samples that appear in test datasets are excluded from the reference database to prevent information leakage.

4.1.2. Retriever

The retriever $$E_R(\cdot )$$ extracts representations from raw inputs (images or text). The representations are then used to measure the similarity between samples. To improve the efficiency and effectiveness of visual reference retrieval, we aim to extract a compact and distinctive representation for each sample that encapsulates its global features. To achieve this, we employ feature extraction models for image and text modalities that have been pre-trained on large-scale datasets. Specifically, We ensemble the CLIP ^[53] vision transformer (ViT ^[54]), which excels at encoding semantics, and the self-supervised pre-trained DINOv2^[55] ViT, which is proficient in capturing visual details. For image captioning and open-ended VQA, the retriever extracts semantic and appearance features from the visual input $$\mathit{v}$$ into $$E_R(\mathit{v})$$ for all test images and reference images. In practice, the predicted $$[cls]$$ tokens from the penultimate transformer block of the CLIP (Available at https://huggingface.co/openai/clip-vit-large-patch14) and DINOv2 (Available at https://github.com/facebookresearch/dinov2) ViT-L14/336 models are concatenated in the feature dimension to serve as the representation for each image, as illustrated in Figure 5. For yes-or-no binary VQA, we focus on finding visual references that are semantically aligned with the question. The CLIP Transformer is used to extract semantics from the question $$\mathit{x}$$ into a vector embedding $$E_R(\mathit{x})$$ for all test samples. For hallucination evaluation benchmarks MME and POPE, we modify the question template before extracting text features, rewriting questions into narratives.

4.1.3. Similarity metric

The similarity metric $$\mathcal{F}(\cdot ,\cdot )$$ in Equation (4) is implemented using cosine similarity,

where $$n$$ is the size of the feature dimension. Cosine similarity selected for its invariance to the magnitude of vectors, which reduces the impact of numerical discrepancies between image features from different models, ensuring a balanced contribution from each vision encoder. The similarity between the retrieval query $$E_R(q)$$ (image or text features of all test samples) and all retrieval keys $$E_R(s_j)$$ (image features of all reference samples $$s_j\in \mathcal{D}$$) is evaluated using Equation (5). In practice, $$E_R(s_j)$$ and $$E_R(q)$$ are first L2 normalized, and maximum inner product search (MIPS) is conducted using FAISS ^[56], which is a library for vector clustering and similarity search. For image captioning and open-ended VQA, both semantic and appearance-level similarities between the test image and visual references are equally considered, as the normalized $$[cls]$$ tokens from CLIP and DINOv2 ViTs are concatenated in the feature dimension. For binary VQA, we utilize the cross-modal feature matching capabilities of the CLIP model to retrieve images relevant to the question. On the MME benchmark, the retrieval results are re-ordered according to the BLEU@1 ^[57] score between the question $$\mathit{x}$$ and the annotated captions of the retrieved images for better performance.

4.1.4. Efficiency

The FAISS library uses vector clustering and approximate k-nearest neighbor (k-NN) algorithms for acceleration. We evaluate the efficiency of the retrieval process on a single NVIDIA 3090 GPU. When retrieving 32 visual references for each query, calculating $$E_R(\mathit{v})$$ takes up around 40 ms, and the retrieval process takes under 5 ms (averaged on 500 queries). It is important to note that RCD does not increase the input token sequence length of MLLMs, and the additional token sequences can be stacked for batch processing to further reduce computational latency. A detailed efficiency analysis is provided in Section 5.5.4.

4.2.1. Adaptive scaling strategy

The influence of $$\{{\mathit{v}}^{NN}{\}}_k$$ and $${\mathit{v}}^d$$ should be regulated in order to coordinate the effect in mitigating hallucinations originating from both visual or textual branches of MLLMs. Specifically, in cases where MLLMs exhibit uncertainty ^[17] regarding the recognized visual cues, i.e., the confidence scores, $$\{l_{i,t}^{\delta }{\}}_{i=0}^{m-1}$$, exhibit multimodal distribution, which are calculated by

then the coefficient $${\alpha }_d^t$$ is reduced while $${\alpha }_{NN}^t$$ is increased at the current decoding step $$t$$. This adjustment ensures that the token candidates, which are erroneously promoted by the visual branch, are not further endorsed. Formally,

Following the adaptive plausibility constraint ^[41], RCD only considers $$x_i$$ that are in the head vocabulary $${\mathcal{V}}_{head}^m$$, which consists of $$m$$ top-ranked candidates that are selected based on the confidence scores predicted by $${\mathit{v}}^{\tau }$$ (i.e., the base scores). In practice, we set a cut-off value as the base score of the $$m^{th}$$-ranked candidate. Confidence scores lower than this cut-off value are set to $$-inf$$. $$m$$ is set at 50 by default. $${\alpha }_{\tau }$$, $${\beta }_d$$ and $${\beta }_{NN}$$ are hyper-parameters, which are by default set at 1.0, 0.1, and 0.1, respectively.

5.1.1. Datasets and metrics

We use publicly available hallucination benchmarks POPE ^[25], (the MLLM Evaluation benchmark) MME ^[24], WHOOPS ^[26], and LLaVA-Bench-in-the-wild ^[27] to validate the effectiveness of our proposed method. POPE is designed to assess hallucinations at the object level, while MME evaluates hallucinations at both the object level and the visual attribute level. In these benchmarks, MLLMs are prompted to answer the questions using a single word or phrase. The evaluation metrics for POPE include Accuracy and F1 score, based on whether the prediction contains the annotated answer (either "yes" or "no"). For MME, the evaluation metric is the combined score of Accuracy and Accuracy$$+$$. Given that MLLMs encode rich real-world conventions and may embed certain superficial syntactical patterns in model parameters ^[17], we hypothesize that images containing counterintuitive visual cues (visual content that contradicts common sense) can highlight the problem of visual hallucinations. Therefore, we conduct quantitative experiments using the WHOOPS benchmark, where MLLMs are instructed to describe the counterintuitive images in a single sentence. For the image captioning task, automatic evaluation metrics such as BLEU ^[57], METEOR ^[58], CIDEr ^[59], and SPICE ^[60] are employed. Additionally, we conduct qualitative experiments on the "detailed description" subset of the LLaVA-Bench-in-the-wild benchmark (LLaVA-W), which encompasses a diverse array of image styles, including real photographs, text-rich images, and sketches.

We further evaluate the effectiveness of RCD in enhancing the MLLMs' capability in real-world scene comprehension and reasoning. To this end, we develop a challenging image captioning and VQA test set, which contains images of complex real-world traffic situations and accidents. Images are sourced from copyright-free websites and high-quality images are retained through manual inspection. For the VQA task, human volunteers are enlisted to write one question for each image based on our devised question categories. Subsequently, we invite 10 volunteers with over 3 years of driving experience to annotate 10 answers for each question. Finally, we manually check all the answers to ensure consistency. In total, 300 image captioning test samples with 1, 500 annotated captions, and 200 VQA test samples with 2, 000 annotated answers are obtained. For the image captioning task, we use a multi-modal supermodel (gpt-4o-2024-08-06 model, https://platform.openai.com/docs/models/gpt-4o) to generate captions for each image, and then manually verify and revise these captions to ensure their accuracy.

For the VQA task, we use the EM metric for quantitative evaluation, following the implementation as in the VQAv2 ^[22] benchmark. We carefully devise six categories of questions that are essential for comprehending traffic scenes, encompassing basic visual recognition tasks, logical reasoning tasks, and knowledge-intensive tasks, as detailed below:

● Object Recognition: We design questions regarding the types of objects in the scene (such as the categories and sizes of obstacles on the road), their quantities, and the distances to objects. The proportion of this type of question accounts for 24.2%.

● Traffic Sign Recognition: We design optical character recognition (OCR) questions based on the traffic signs, such as asking about the speed limit, the meaning of prohibition signs, and the names and distances of destinations. We also include questions about traffic lights. The proportion of this type of question accounts for 26.8%.

● Lane Identification: This task involves questions about the number of lanes on the road, the directions of the lanes, and which lane should be selected under specific conditions. The proportion of this type of question accounts for 5.6%.

● Traffic Police Hand Signal Recognition: This is a knowledge-intensive task. The questions ask about the meaning of the traffic police gestures in each image. The proportion of this type of question accounts for 19.7%.

● Reasoning and Forecasting: This task requires the MLLM to observe the information in the image, combined with common sense, and deduce the reasons behind a certain state or action depicted in the picture, or to predict what actions a car or a pedestrian might take in the near future. The proportion of this type of question accounts for 11.1%.

● Driving Maneuver: This task involves questions about which driving behaviors are permitted or prohibited in the scenarios depicted in the image, as well as what actions should be taken to avoid the risk factors. These questions correspond to images that feature different road conditions (dry or icy), different traffic situations (congested or clear), and various traffic signal indications. The proportion of this type of question accounts for 12.6%.

Among the test samples, images depicting traffic accidents (e.g., collisions and fire) and significant risk factors (e.g., slippery roads, obstacles, and imminent collisions) account for 59.6%, nighttime images account for 17.9%, and images from the driver's first-person perspective account for 25.8%. This test set is expected to comprehensively evaluate the capability of MLLMs in comprehending real-world scenes.

5.1.2. MLLM baselines

Three state-of-the-art MLLMs are selected to implement our method, including InstructBLIP ^[3], LLaVA-1.5 ^[2], and LLaVA-Next (stronger) ^[23]. These three MLLMs integrate CLIP Vision Transformer as their visual encoder. As for the cross-modal connector, LLaVA-1.5 and LLaVA-Next use a two-layer MLP, and InstructBLIP uses a Q-former ^[6] with textual query. We use the 7B version (with around 7-8 billion learnable parameters) of the MLLMs, with Vicuna-7B ^[42] as the language decoder for LLaVA-1.5 and InstructBLIP, and LLaMA3-8B (Available at https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md) for LLaVA-Next.

5.1.3. Implementation details

All experiments are in zero-shot manner, and greedy decoding serves as the baseline strategy for reproducibility. All random seeds are set at 42, ensuring that repeated trials would yield identical results. We run all experiments on two NVIDIA 3090 GPUs. We use PyTorch ^[61] and the Huggingface Transformers library (https://github.com/huggingface/transformers) to implement our method. We compare RCD with two advanced decoding strategies VCD ^[15] and DoLa ^[62], using their official code implementations. VCD contrasts the confidence scores of the test image with the scores of a noised image, targeting at reducing erroneous language priors. For VCD, we follow the official hyper-parameters on the MME and POPE benchmarks. DoLA aims to amplify the knowledge injected by deeper layers by contrasting the confidence scores predicted by the language model's final hidden state with those predicted by its shallow layers. For DoLa, we follow its dynamic premature layer selection strategy.

5.2.1. Results on MME

Experimental results on the perception subtasks of MME are presented in Table 1. Without additional training, RCD significantly improves the total score for LLaVA-Next (+36.4 scores), LLaVA-1.5 (+56.6 scores), and InstructBLIP (+76.3 scores), surpassing current advanced decoding strategies DoLa and VCD by 64.1 and 29.8 scores on average, respectively. In particular, RCD effectively mitigates visual hallucinations in color-related perception tasks for LLaVA-1.5 (+10 scores) and InstructBLIP (+33.3 scores). RCD also improves the OCR capability for LLaVA-1.5 (+15 scores) and InstructBLIP (+30 scores).

We further investigate how RCD enhances model performance on the MME benchmark and present the results in Figure 6. We plot the predicted confidence scores corresponding to token candidates $$\mathrm{\_}$$yes and $$\mathrm{\_}$$no (We observe that in all test results, candidates $$\mathrm{\_}$$yes and $$\mathrm{\_}$$no consistently rank as the top two positions in the vocabulary). on the vertical and horizontal axis, respectively. Each test sample corresponds to a single triangle mark colored in blue (the samples' ground truth answer is $$\mathrm{\_}$$yes) or red (the ground truth answer is $$\mathrm{\_}$$no). If a blue triangle mark is below the $$y=x$$ line, or a red triangle mark is above the $$y=x$$ line, the answer is incorrect. The main observations are summarized as follows:

Figure 6. Analysis of RCD's impact on the MME benchmark. Our proposed RCD reduces the answer uncertainty of InstructBLIP, and decreases the number of false positive samples. RCD: Retrieval contrastive decoding; MME:

● Without visual references, InstructBLIP has equal confidence, i.e., high uncertainty, to the positive and negative answers. As illustrated in Figure 6, a number of test samples in the Count sub-task are close to the $$y=x$$ line.

● Without visual references, InstructBLIP tends to answer "yes" for all questions. As shown in Figure 6, almost all samples are located above the $$y=x$$ line.

Figure 6 illustrates that after integrating RCD, there are fewer red triangle marks above the $$y=x$$ line. This indicates that the number of false positive answers is reduced. Besides, the sample points demonstrate more dispersed distribution above or under the $$y=x$$ line with RCD, indicating that the level of uncertainty is reduced.

5.2.2. Results on POPE

The averaged Accuracy and F1 score across the random, popular, and adversarial splits of POPE are presented in Table 2. On three subsets of POPE, RCD boosts the overall Accuracy of LLaVA-Next (+1.98 on average), LLaVA-1.5 (+0.25 on average), and InstructBLIP (+0.91 on average), outperforming VCD and DoLA in varying degrees.

5.2.3. Results on WHOOPS

Quantitative results are presented in Table 3. RCD significantly enhances the image description accuracy of LLaVA-1.5 and InstructBLIP. For instance, RCD improves the CIDEr score of LLaVA-1.5 and LLaVA-Next by 7.6 and 5.8, respectively, surpassing existing methods VCD and DoLa. These experimental results validate the effectiveness of RCD in image captioning task and demonstrate its ability to generalize to counterintuitive images. The examples in Figure 7 illustrate the effectiveness of RCD in enhancing MLLMs' comprehension of counterintuitive visual cues. For instance, RCD enables LLaVA-1.5 to accurately identify a traffic signal with three green lights and assists InstructBLIP in correctly distinguishing rubber ducks from real ducks. These results demonstrate that RCD has strong generalization capabilities, enabling precise visual understanding in scenarios involving counterintuitive visual content.

Figure 7. RCD demonstrates robust generalization capabilities to counterintuitive images, thereby enhancing the accuracy of MLLMs’ visual understanding. Correct and hallucinatory content are highlighted in green and red, respectively. RCD: Retrieval contrastive decoding.

5.2.4. Results on LLaVA-W

The previous sections quantitatively evaluate the effectiveness of RCD. This section qualitatively assesses RCD's performance by presenting examples from the image detailed captioning task on the LLaVA-Bench-in-the-wild (LLaVA-W) benchmark, offering a more intuitive analysis of its capabilities. The examples illustrated in Figure 8 demonstrate that RCD effectively mitigates hallucinatory content in detailed descriptions for both in-door and out-door scenes that contain diverse objects, outperforming existing methods DoLa and VCD. For instance, RCD enables LLaVA-1.5 to accurately discern the spatial relationships among multiple instances in a refrigerator, and helps InstructBLIP correctly distinguish streetlights from traffic lights. These examples demonstrate that RCD effectively mitigates various categories of visual hallucination, thereby enhancing MLLMs' capabilities in comprehending complex real-world scenes.

Figure 8. RCD effectively reduces visual hallucinations in detailed image descriptions. DoLa's response is omitted when it is identical to the greedy baseline. Correct and hallucinatory contents are highlighted in green and red, respectively. RCD: Retrieval contrastive decoding.

5.3.1. Quantitative results

Table 4 demonstrates that RCD enhances the capability of MLLMs to comprehend real-world scenarios. RCD leads to notable improvements in the LLaVA model family for both image captioning and VQA. Specifically, RCD increases the CIDEr score by 3.0 for LLaVA-Next and 1.5 for LLaVA-1.5 in image captioning, and improves the EM metric by 2.4 and 1.2 for LLaVA-Next and LLaVA-1.5 in VQA, respectively. However, the performance gains for InstructBLIP are less pronounced, with only a 0.5 increase in CIDEr scores and a 0.2 increase in EM scores. This may be attributed to the loss of information in the visual token compression process executed by the Q-former ^[6] model. These results also demonstrate that RCD consistently outperforms DoLA and VCD.

5.3.2. Qualitative results

Qualitative experimental results on our proposed VQA benchmark are presented in Figure 9. Our method demonstrates higher answer accuracy than DoLa and VCD. Specifically, our approach help MLLMs accurately recognize object categories (e.g., bags and cones on the road), and interpret traffic signs (e.g., a yellow triangle sign indicating children crossing) Additionally, RCD enhances the MLLM's reasoning capabilities, such as inferring appropriate driving behaviors (e.g., proceeding at a green light intersection), and predicting potential risks in the scene (e.g., possible collision with a motorcycle on the right side). It is noteworthy that RCD also exhibits robust performance on hazy and low-light images. For particularly challenging questions, such as selecting a lane based on specific requirements, all MLLMs and methods struggle. This suggests that current MLLMs still have considerable room for improvement in accurately comprehending real-world scenarios.

Figure 9. Qualitative results on our proposed VQA benchmark. Our method enhances the accuracy of answers from three MLLMs across nearly all question categories. Human-annotated answers are provided for each test sample. Incorrect and correct answers are highlighted in red green, respectively. VQA: Visual Question Answering.

Qualitative samples from our proposed image captioning benchmark are exhibited in Figure 10. RCD enhances the accuracy of image descriptions, mitigating various types of visual hallucinations, such as incorrect physical states (e.g., an upside-down car), incorrect object categories (e.g., mistaking a pedestrian sign for a real person), and incorrect spatial relationships (e.g., perceiving a truck coming from the other side). Additionally, RCD improves the specificity of image descriptions, such as identifying three cars involved in a collision on a wet road and a police officer standing in the middle of the street. These examples provide clear evidence of the effectiveness of RCD.

Figure 10. Qualitative results on our proposed image captioning benchmark. Our method RCD improves the accuracy of image descriptions and reduces various types of visual hallucinations. Additionally, RCD enhances the specificity of the descriptions. Incorrect content is highlighted in red, while correct content is highlighted in green. RCD: Retrieval contrastive decoding.

5.4.1. Similar images are better than random images

We first ablate the image retrieval process (abbreviated img. ret.), opting to randomly sample visual references from the database instead. The average results and standard deviations from five separate trials (each with different random seeds) are presented in Exp.3. The results suggest that random images can positively influence performance (+1.2 CIDEr scores and +1.1 SPICE scores compared to Exp.1), underscoring the resilience of RCD in scenarios lacking similar images. Furthermore, when similar images are available (Exp.2), RCD leads to more substantial improvements across all metrics (e.g., CIDEr +3.0 and SPICE +1.2). Note that the noised image is retained because it aids in mitigating erroneous language priors, and the CIDEr score dropped by 0.9 in Exp.4 after it is ablated.

5.4.2. Visual concepts comparison is better than combination

We modify the visual concept comparison process (abbreviated comp.), transitioning from logit subtraction in Equation (6) into addition. The additive paradigm emphasizes the shared visual cues across similar images. Exp.5 demonstrates significant declines in all metrics (e.g., -7.7 CIDEr score and -2.1 SPICE score). These results support the superiority of contrasting visual concepts between similar images over combining them.

5.4.3. Other ablations

Exp.6 demonstrates that the adaptive scaling strategy (abbreviated as adapt.) can enhance almost all metrics. Additionally, increasing the number of reference images yields improvements on the BLEU@4 metric (from 12.3 to 13.0). However, it also leads to slight decreases on the CIDEr score (from 28.0 to 27.8) in Exp.7-9. Therefore, RCD defaults to using two visual references.

5.5.1. Analysis of various text decoding baselines

In the experiments on public benchmarks, greedy search is used as the baseline strategy. This approach selects only the token candidate with the highest confidence score, thereby excluding any randomness in text decoding. We further investigate the integration of RCD with various token sampling baselines, including multi-nominal sampling, top-$$k$$ sampling, and top-$$p$$ sampling. The experimental results are presented in Table 6. Notably, RCD integrates well with different baseline sampling strategies. However, these token sampling strategies exhibit significantly weaker performance compared to greedy decoding.

5.5.2. Analysis of similarity metrics

RCD employs cosine similarity as the similarity measure for reference information retrieval. Table 7 provides a quantitative analysis on our proposed traffic scene image captioning test set comparing cosine similarity with the L2 distance metric. These results indicate that the performance based on the Euclidean distance metric is marginally inferior to that of the cosine similarity metric, yet it consistently surpasses the baseline performance of the greedy decoding approach.

5.5.3. Analysis of the reference database's size

RCD uses a reference database containing 113k samples from the COCO Caption dataset. Table 8 demonstrates that incorporating the entire Visual Genome dataset ^[63] into the reference database doubles the size of the database but leads to a decline in the evaluation metrics of the LLaVA-1.5 model. For instance, the CIDEr score slightly decreases by 0.9 compared to using the COCO Caption data. This decline is likely attributable to the increased inaccuracy in image retrieval results, as the reference database becomes significantly more expansive. As a result, the retrieval module's robustness may require further optimization to effectively manage larger-scale reference databases.

5.5.4. Analysis of inference efficiency

RCD requires the independent prediction of an additional $$k$$+1 confidence score distribution, which introduces linear computational complexity. As shown in Table 9, the naive sequential prediction of these distributions (implemented using a for loop, denoted as RCD-Naive) increases the MLLM's inference time by approximately a factor of $$k$$. To mitigate this delay, the $$k$$+1 input sequences can be stacked for batch processing. This optimization (RCD-Optimized) reduces the $$k$$-fold increase in inference time to approximately $$0.43k$$-fold. Moreover, as demonstrated in Table 9, both implementations of RCD result in only a negligible increase in GPU memory (less than 15 MB) when $$k=2$$.