Figure2

Figure 2. Experiment pipeline for investigating the impact of visual and textual input modality on the hallucinatory output. At each decoding step $$ t $$, the test image $$ \boldsymbol{v}^{\tau} $$ is replaced with alternative images $$ \boldsymbol{v}' $$ while keeping the textual prefix constant. Next, we assess the difference in output confidence scores (i.e., logits) between $$ y_t $$ and $$ \hat{y_t} $$ to demonstrate the impact of the visual input. This test image is taken from the OpenImages validation set ^[46]. Similar images are retrieved from the COCO Caption dataset ^[47].