StorytellingSerious

IJCAI–ECAI · 5th Workshop on Human Brain and Artificial Intelligence (HBAI) ·

Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction

Sukriti Tiwari, BHVSP Subrahmanyam, Nidhi Goyal, Sai Amrit Patnaik

Accepted · Workshop · EEG & multimodal AI

An image reconstructed from brain signals can preserve meaning even when it looks different.

Abstract

EEG-to-image evaluation should distinguish visual fidelity from recoverable meaning. Yet EEG-derived reconstructions are blurry, distorted, and low-detail, causing SSIM, LPIPS, and CLIP to penalize semantically recoverable outputs or reward plausible but incorrect ones. We analyze 6,855 ground-truth/reconstruction pairs from ATM, ENIGMA, BrainVis, and DreamDiffusion using semantic probes, caption harshness and blind-spot rates, and controlled degradations. Pixel metrics show near-zero correlation with semantic consistency, while representation metrics conflate perceptual and semantic errors. We therefore introduce a BCI-aware framework in which four VLMs assess image pairs through structured questions, producing Tolerant Perceptual Alignment Scores (T-PAS) and Tolerant Semantic Alignment Scores (T-SAS). Their consensus is distilled into the BCI-Coherence Score (BCS), a compact evaluator achieving T-PAS MAE 0.079 (r = 0.700) and T-SAS MAE 0.082 (r = 0.850) on our data. Human validation shows highly reliable joint coherence judgments (κ = 0.882 ± 0.174, α = 0.882), supporting perceptual–semantic recoverability over generic visual similarity. Code and resources are available at https://sukt03.github.io/BCS/.

Findings

  • Analyzes 6,855 image pairs from ATM, ENIGMA, BrainVis, and DreamDiffusion.
  • The distilled BCI-Coherence Score achieves MAE 0.079 for perceptual alignment and 0.082 for semantic alignment on the study data.
  • Human validation supports evaluating recoverable meaning alongside visual fidelity.

Sources and citation

View BibTeX citation
@inproceedings{visual-translation-2026,
  title = {{Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction}},
  author = {Sukriti Tiwari and BHVSP Subrahmanyam and Nidhi Goyal and Sai Amrit Patnaik},
  booktitle = {IJCAI–ECAI · 5th Workshop on Human Brain and Artificial Intelligence (HBAI)},
  year = {2026},
  url = {https://arxiv.org/abs/2607.12364},
  note = {Accepted}
}

What are you in the mood for?