Reconstructing visual experiences from brain activity promises to strengthen brain-computer interfaces and deepen the basic understanding of perception. Current deep learning approaches to fMRI-based image synthesis, the authors say, are often person-specific and need substantial data to adapt to new individuals, which limits their scalability and translational potential. At the core of MindShow is a Hierarchically-Conditioned Mixture-of-Experts (HiCo-MoE) encoder that disentangles population-shared latent representations from subject-specific neural characteristics, enabling data-efficient adaptation to a target subject under limited calibration data.
Those representations pass through a Gated Perceiver Bottleneck, a gated Perceiver-style tokenization interface that adaptively maps fMRI features into distinct, fixed-size image and text latent tokens to resolve multi-scale representational misalignment. A multi-granular optimal transport loss (MOT-Align) regularizes sample- and token-level distributional alignment between brain-derived features and the latent space of a pretrained vision-language model, and a frozen diffusion model guided by the aligned embeddings synthesizes images meant to preserve the semantic content and coarse layout of what was perceived. MindShow improves high-level reconstruction metrics while maintaining competitive structural fidelity, which the authors describe as a methodological step toward scalable shared-subject neural decoding. The implementation code is public on GitHub.