EFS-NET: EEG-fNIRS multi-scale fusion network based on spatial calibration.
EFS-Net achieved an 81.69% classification accuracy for Mental Arithmetic tasks, suggesting potential for improved brain-computer interface performance.
Where it sits
this study against the rest of the dsip corpusSummary and findings
This study presents the EFS-Net, a multi-scale EEG-fNIRS fusion network designed to enhance neural signal decoding. It was evaluated on two datasets: Word Generation (WG) and Mental Arithmetic (MA), achieving classification accuracies of 77.71% and 81.69%, respectively. The study aims to improve the integration of EEG and fNIRS data for better brain-computer interface performance.
Abstract
Hybrid brain-computer interfaces (hBCIs) integrate multiple neuroimaging modalities and utilize their complementary information to address the inherent limitations of single-modality neural signal decoding. For electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) hybrid BCIs, advanced fusion algorithms are crucial to fully exploit the superior spatial localization capability of fNIRS and the millisecond-level temporal resolution of EEG. This work proposes an end-to-end spatial calibration-based multi-scale EEG-fNIRS fusion network named EFS-Net, which organically integrates EEG and fNIRS signals through a multi-scale spatio-temporal fusion architecture. The network consists of three complementary functional branches: a multi-scale temporal convolution branch for capturing rapidly changing cortical electrophysiological features of EEG, an EEG spatial branch for constructing latency-compensated cortical topographies to adapt to the delayed hemodynamic response of fNIRS, and a spatially calibrated fNIRS spatial branch for dynamically fusing spatial feature maps with EEG counterparts to generate temporally aligned and spatially enhanced neural representations. This three-branch fusion structure constructs abundant spatio-temporal feature embeddings and improves the discriminability of neural features. Evaluated on two public datasets including Word Generation (WG) and Mental Arithmetic (MA) with a rigorous subject-specific leave-one-session-out cross-validation protocol, EFS-Net achieves classification accuracies of 77.71 ± 8.23% and 81.69 ± 9.49% on the WG and MA datasets respectively, which surpasses state-of-the-art unimodal algorithms and traditional fusion models. Visualization results demonstrate that the designed alignment strategy can restore realistic cortical spatial distribution characteristics, providing a feasible solution for personalized neural signal decoding in hybrid BCIs.
Background
This paper addresses the limitations of single-modality neural signal decoding in brain-computer interfaces (BCIs) by integrating electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). Prior research has shown that combining these modalities can leverage their respective strengths: EEG's temporal resolution and fNIRS's spatial localization. The significance of this study lies in its proposed fusion network, which aims to improve the accuracy of neural signal interpretation.
Methods
The study utilized a multi-scale spatio-temporal fusion architecture for the EFS-Net, integrating EEG and fNIRS signals through three functional branches. The evaluation was performed on two public datasets, Word Generation (WG) and Mental Arithmetic (MA), using a subject-specific leave-one-session-out cross-validation protocol. Specifics regarding sample size, dose, or duration were not reported in the abstract.
Results
The primary endpoint achieved a classification accuracy of 81.69 ± 9.49% on the Mental Arithmetic dataset. This result indicates a significant improvement over traditional unimodal algorithms and fusion models, although specific p-values or confidence intervals were not provided.
Interpretation
The findings suggest that EFS-Net may enhance neural feature discriminability compared to existing methods. However, while the classification accuracies are statistically significant, the clinical relevance of these improvements remains uncertain without further validation in real-world applications. Limitations such as potential overfitting due to the specific datasets and the absence of diverse populations may confound the conclusions.
Key findings
- 77.71 ± 8.23% classification accuracy on Word Generation dataset.
- 81.69 ± 9.49% classification accuracy on Mental Arithmetic dataset.
Limitations
- Not reported in abstract.
- Specific sample size not disclosed.
- No information on follow-up duration.
- Potential overfitting due to dataset specificity.