BCIwiki (bciwiki.com) — Researchers from Beihang University and Tsinghua University have built a multimodal contrastive learning framework that pairs EEG with synchronized video and motion data to detect cybersickness in virtual reality (VR). According to MBrainTrain, the approach outperformed every single-modality method they tested and every prior multimodal approach, pointing toward a more reliable path for EEG-based cybersickness detection.
To collect useful data, the team built a Unity-based outdoor scene measuring 25 by 25 meters, scattered with red target spheres. 29 healthy adults navigated it with joystick controls, searching for 20 targets. EEG was recorded with mbt's Smarting PRO, a 32-channel wireless system, while participants wore a Pico 4 Ultra headset for the VR experience. First-person video was captured directly from the headset, and motion data including position, velocity, acceleration, and rotation was logged through Unity. Hardware-triggered markers synchronized all three streams down to the millisecond. Participants held down a controller trigger to mark when symptoms started and stopped, avoiding interruption and cognitive load.
The paper's central contribution is Brain Connectivity Graph Representation (BCGR), which represents connectivity between 30 electrode sites, arranged on the standard 10-20 layout, as a graph rather than a flat feature vector. EEG data produced one version, E-BCGR, trained separately for male and female participants using dynamic graph convolutional networks. The researchers then merged the two graphs into a standard version, S-BCGR, keeping only connections that appeared in both genders. The dropped connections were mostly in the prefrontal cortex, a memory region unrelated to the sensory conflict that causes cybersickness, indicating the algorithm focused on relevant signals.
Video and motion data were mapped onto the same 30-node structure using an attention-based encoder, routing visual features toward occipital and parietal nodes, and motion features toward central, motor-related nodes. An ablation test compared EEG alone, video alone, motion alone, and every combination. Video alone performed best among individual signals, but every pairing beat every single modality, and using all three together beat every pairing, with the cleanest separation between cybersick and non-cybersick states. Compared against five existing state-of-the-art methods, the fused model led on accuracy and area under the curve.
The study also demonstrates that naturalistic, actively-navigated VR tasks are possible without sacrificing signal quality. As VR hardware and mobile EEG systems continue to shrink, cybersickness detection tools could eventually run inside a headset, not a lab. The paper, titled "Multimodal Contrastive Learning for Cybersickness Recognition Using Brain Connectivity Graph Representation," was published on July 3, 2026.