Dual-stream temporal-frequency convolutional network with additive attention for auditory attention decoding_Journal of Biomedical Engineering

Authors：

LIN Xiao ,  ZHANG Sunjie , HUANG Shuai

School of Opto-Electronic Information & Computer Engineering, University of Shanghai for Science & Technology, Shanghai 200093, P. R. China;

Corresponding?author：

ZHANG Sunjie, Email: zhang_sunjie@126.com

Keywords：

Auditory attention decoding; Electroencephalography; Time-frequency fusion; Brain-computer interface; Attention mechanism

DOI：

10.7507/1001-5515.202509064

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

In complex auditory environments, humans can selectively attend to a target speaker while suppressing interfering sources. Electroencephalography (EEG)-based auditory attention decoding (AAD) aims to identify the speech source an individual is attending to, holding significant potential for applications in hearing aids and human-machine interaction. However, existing methods still exhibit notable limitations in modeling dynamic time-frequency features, extracting multi-scale representations, and computational efficiency.To address these challenges, we propose a dual-stream time-frequency convolutional network with additive attention. The temporal branch employs a temporal convolutional additive attention module to integrate temporal and channel attention, enhancing the modeling of key EEG time segments and feature channels. The spectral branch adopts a spatial-spectral feature extractor and a multi-scale feature enhancement module to improve sensitivity to and modeling of frequency variation patterns. The global features from both branches are ultimately fused for attention direction classification. Under a 1-second decision window, our model achieved accuracy rates of 96.5%, 86.3%, 73.1%, and 73.5% on the KUL, DTU, AVED (audio-only), and AVED (audio-visual) datasets, respectively. These results significantly outperform state-of-the-art methods and demonstrate the effectiveness and superiority of the proposed approach.

Citation： LIN Xiao, ZHANG Sunjie, HUANG Shuai. Dual-stream temporal-frequency convolutional network with additive attention for auditory attention decoding. Journal of Biomedical Engineering, 2026, 43(4): 695-703. doi: 10.7507/1001-5515.202509064 Copy

1.	Golumbic E M Z, Ding N, Bickel S, et al. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”[J]. Neuron, 2013, 77(5): 980-991.
2.	王春麗, 李金絮, 高玉鑫. 多尺度時空頻注意殘差融合的聽覺注意解碼網絡[J]. 北京郵電大學學報, 2026, 49(1): 83-90.
3.	吳文瑋. 基于腦電信號的聽覺注意力檢測方法研究[D]. 大連: 大連理工大學, 2025: 1.
4.	Maruyama Y, Yoshimura N, Rana A, et al. Electroencephalography of completely locked-in state patients with amyotrophic lateral sclerosis[J]. Neurosci Res, 2021, 162: 45-51.
5.	Mesgarani N, Chang E F. Selective cortical representation of attended speaker in multi-talker speech perception[J]. Nature, 2012, 485(7397): 233-236.
6.	Birbaumer N, Ghanayim N, Hinterberger T, et al. A spelling device for the paralysed[J]. Nature, 1999, 398(6725): 297-298.
7.	Ding N, Simon J Z. Neural coding of continuous speech in auditory cortex during monaural and dichotic listening[J]. J Neurophysiol, 2012, 107(1): 78-89.
8.	Bellier L, Llorens A, Marciano D, et al. Music can be reconstructed from human auditory cortex activity using nonlinear decoding models[J]. PLoS Biol, 2023, 21(8): e3002176.
9.	Huang S, Wang Y, Luo H, et al. SSAAD: A multi-scale temporal-frequency graph network for binary auditory attention detection with self-supervised learning[C]//ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Hyderabad: IEEE, 2025: 1-5.
10.	Das N, Bertrand A, Francart T. EEG-based auditory attention detection: boundary conditions for background noise and speaker positions[J]. J Neural Eng, 2018, 15(6): 066017.
11.	Geirnaert S, Vandecappelle S, Alickovic E, et al. Electroencephalography-based auditory attention decoding: Toward neurosteered hearing devices[J]. IEEE Signal Process Mag, 2021, 38(4): 89-102.
12.	Van Eyndhoven S, Francart T, Bertrand A. EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses[J]. IEEE Trans Biomed Eng, 2016, 64(5): 1045-1056.
13.	Geirnaert S, Francart T, Bertrand A. Unsupervised self-adaptive auditory attention decoding[J]. IEEE J Biomed Health Inform, 2021, 25(10): 3955-3966.
14.	Huang S, Wang Y, Luo H. A dual-branch generative adversarial network with self-supervised enhancement for robust auditory attention decoding[J]. Eng Appl Artif Intell, 2025, 159(Part A): 111122.
15.	Ding N, Simon J Z. Emergence of neural encoding of auditory objects while listening to competing speakers[J]. Proc Natl Acad Sci USA, 2012, 109(29): 11854-11859.
16.	O’Sullivan J A, Power A J, Mesgarani N, et al. Attentional selection in a cocktail party environment can be decoded from single-trial EEG[J]. Cereb Cortex, 2015, 25(7): 1697-1706.
17.	Crosse M J, Di Liberto G M, Bednar A, et al. The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[J]. Front Hum Neurosci, 2016, 10: 604.
18.	Geirnaert S, Francart T, Bertrand A. Fast EEG-based decoding of the directional focus of auditory attention using common spatial patterns[J]. IEEE Trans Biomed Eng, 2020, 68(5): 1557-1568.
19.	Ilias L, Askounis D, Psarras J. Multimodal detection of epilepsy with deep neural networks[J]. Expert Syst Appl, 2023, 213: 119010.
20.	Samal P, Hashmi M F. Role of machine learning and deep learning techniques in EEG-based BCI emotion recognition system: a review[J]. Artif Intell Rev, 2024, 57(3): 50.
21.	Cai S, Sun P, Schultz T, et al. Low-latency auditory spatial attention detection based on spectro-spatial features from EEG[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Mexico: IEEE, 2021: 5812-5815.
22.	Cai S, Su E, Xie L, et al. EEG-based auditory attention detection via frequency and channel neural attention[J]. IEEE Trans Hum Mach Syst, 2021, 52(2): 256-266.
23.	Cai S, Zhang R, Li H. Robust decoding of the auditory attention from EEG recordings through graph convolutional networks[C]//ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul: IEEE, 2024: 2320-2324.
24.	Fan C, Zhang H, Huang W, et al. DGSD: dynamical graph self-distillation for EEG-based auditory spatial attention detection[J]. Neural Netw, 2024, 179: 106580.
25.	XUE J, FAN C, YI J, et al. Learning from yourself: a self-distillation method for fake speech detection[C]//ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island, Greece: IEEE, 2023: 1-5.
26.	NI Q, ZHANG H, FAN C, et al. DBPNet: dual-branch parallel network with temporal-frequency fusion for auditory attention detection[C]//LARSON K, ed. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. Jeju, Korea: International Joint Conferences on Artificial Intelligence Organization, 2024: 3115-3123.
27.	Snyder J P. Map projections--a working manual[M]. Washington: US Government Printing Office, 1987.
28.	Jiang Y, Chen N, Jin J. Detecting the locus of auditory attention based on the spectro-spatial-temporal analysis of EEG[J]. J Neural Eng, 2022, 19(5): 056035.
29.	Han K, Xiao A, Wu E, et al. Transformer in transformer[J]. Adv Neural Inf Process Syst, 2021, 34: 15908-15919.
30.	Zhang T, Li L, Cao S, et al. Attention-guided pyramid context networks for detecting infrared small target under complex background[J]. IEEE Trans Aerosp Electron Syst, 2023, 59(4): 4250-4261.
31.	Zhang T, Li L, Igel C, et al. LR-CSNet: low-rank deep unfolding network for image compressive sensing[C]//2022 IEEE 8th International Conference on Computer and Communications (ICCC). Chengdu, China: IEEE, 2022: 1951-1957.
32.	Zhang T, Li L, Peng Z. Optimization-inspired cumulative transmission network for image compressive sensing[J]. Knowl Based Syst, 2023, 279: 110963.
33.	Zhang T, Li L, Zhou Y, et al. CAS-ViT: convolutional additive self-attention vision transformers for efficient mobile applications[J]. IEEE Transactions on Image Processing, 2026, 35: 1899-1909.
34.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 770-778.
35.	Liu Z, Shen Z, Savvides M, et al. ReActNet: towards precise binary neural network with generalized activation functions[C]//European Conference on Computer Vision (ECCV). Cham: Springer, 2020: 143-159.
36.	Das N, Biesmans W, Bertrand A, et al. The effect of head-related filtering and ear-specific decoding bias on auditory attention detection[J]. J Neural Eng, 2016, 13(5): 056014.
37.	Biesmans W, Das N, Francart T, et al. Auditory-inspired speech envelope extraction methods for improved EEG-based auditory attention detection in a cocktail party scenario[J]. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(2): 1-11.
38.	Fuglsang S A, Dau T, Hjortkj?r J. Noise-robust cortical tracking of attended speech in real-world acoustic scenes[J]. Neuroimage, 2017, 156: 435-444.
39.	Wong D D E, Fuglsang S A, Hjortkj?r J, et al. A comparison of regularization methods in forward and backward models for auditory attention decoding[J]. Front Neurosci, 2018, 12: 531.
40.	Zhang H, Zhang J. Based on audio-video evoked auditory attention detection electroencephalogram dataset[J]. J Tsinghua Univ (Sci Technol), 2024, 64(11): 1919-1926.

1. Golumbic E M Z, Ding N, Bickel S, et al. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”[J]. Neuron, 2013, 77(5): 980-991.
2. 王春麗, 李金絮, 高玉鑫. 多尺度時空頻注意殘差融合的聽覺注意解碼網絡[J]. 北京郵電大學學報, 2026, 49(1): 83-90.
3. 吳文瑋. 基于腦電信號的聽覺注意力檢測方法研究[D]. 大連: 大連理工大學, 2025: 1.
4. Maruyama Y, Yoshimura N, Rana A, et al. Electroencephalography of completely locked-in state patients with amyotrophic lateral sclerosis[J]. Neurosci Res, 2021, 162: 45-51.
5. Mesgarani N, Chang E F. Selective cortical representation of attended speaker in multi-talker speech perception[J]. Nature, 2012, 485(7397): 233-236.
6. Birbaumer N, Ghanayim N, Hinterberger T, et al. A spelling device for the paralysed[J]. Nature, 1999, 398(6725): 297-298.
7. Ding N, Simon J Z. Neural coding of continuous speech in auditory cortex during monaural and dichotic listening[J]. J Neurophysiol, 2012, 107(1): 78-89.
8. Bellier L, Llorens A, Marciano D, et al. Music can be reconstructed from human auditory cortex activity using nonlinear decoding models[J]. PLoS Biol, 2023, 21(8): e3002176.
9. Huang S, Wang Y, Luo H, et al. SSAAD: A multi-scale temporal-frequency graph network for binary auditory attention detection with self-supervised learning[C]//ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Hyderabad: IEEE, 2025: 1-5.
10. Das N, Bertrand A, Francart T. EEG-based auditory attention detection: boundary conditions for background noise and speaker positions[J]. J Neural Eng, 2018, 15(6): 066017.
11. Geirnaert S, Vandecappelle S, Alickovic E, et al. Electroencephalography-based auditory attention decoding: Toward neurosteered hearing devices[J]. IEEE Signal Process Mag, 2021, 38(4): 89-102.
12. Van Eyndhoven S, Francart T, Bertrand A. EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses[J]. IEEE Trans Biomed Eng, 2016, 64(5): 1045-1056.
13. Geirnaert S, Francart T, Bertrand A. Unsupervised self-adaptive auditory attention decoding[J]. IEEE J Biomed Health Inform, 2021, 25(10): 3955-3966.
14. Huang S, Wang Y, Luo H. A dual-branch generative adversarial network with self-supervised enhancement for robust auditory attention decoding[J]. Eng Appl Artif Intell, 2025, 159(Part A): 111122.
15. Ding N, Simon J Z. Emergence of neural encoding of auditory objects while listening to competing speakers[J]. Proc Natl Acad Sci USA, 2012, 109(29): 11854-11859.
16. O’Sullivan J A, Power A J, Mesgarani N, et al. Attentional selection in a cocktail party environment can be decoded from single-trial EEG[J]. Cereb Cortex, 2015, 25(7): 1697-1706.
17. Crosse M J, Di Liberto G M, Bednar A, et al. The multivariate temporal response function (mTRF) toolbox: a MATLAB toolbox for relating neural signals to continuous stimuli[J]. Front Hum Neurosci, 2016, 10: 604.
18. Geirnaert S, Francart T, Bertrand A. Fast EEG-based decoding of the directional focus of auditory attention using common spatial patterns[J]. IEEE Trans Biomed Eng, 2020, 68(5): 1557-1568.
19. Ilias L, Askounis D, Psarras J. Multimodal detection of epilepsy with deep neural networks[J]. Expert Syst Appl, 2023, 213: 119010.
20. Samal P, Hashmi M F. Role of machine learning and deep learning techniques in EEG-based BCI emotion recognition system: a review[J]. Artif Intell Rev, 2024, 57(3): 50.
21. Cai S, Sun P, Schultz T, et al. Low-latency auditory spatial attention detection based on spectro-spatial features from EEG[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Mexico: IEEE, 2021: 5812-5815.
22. Cai S, Su E, Xie L, et al. EEG-based auditory attention detection via frequency and channel neural attention[J]. IEEE Trans Hum Mach Syst, 2021, 52(2): 256-266.
23. Cai S, Zhang R, Li H. Robust decoding of the auditory attention from EEG recordings through graph convolutional networks[C]//ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul: IEEE, 2024: 2320-2324.
24. Fan C, Zhang H, Huang W, et al. DGSD: dynamical graph self-distillation for EEG-based auditory spatial attention detection[J]. Neural Netw, 2024, 179: 106580.
25. XUE J, FAN C, YI J, et al. Learning from yourself: a self-distillation method for fake speech detection[C]//ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island, Greece: IEEE, 2023: 1-5.
26. NI Q, ZHANG H, FAN C, et al. DBPNet: dual-branch parallel network with temporal-frequency fusion for auditory attention detection[C]//LARSON K, ed. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. Jeju, Korea: International Joint Conferences on Artificial Intelligence Organization, 2024: 3115-3123.
27. Snyder J P. Map projections--a working manual[M]. Washington: US Government Printing Office, 1987.
28. Jiang Y, Chen N, Jin J. Detecting the locus of auditory attention based on the spectro-spatial-temporal analysis of EEG[J]. J Neural Eng, 2022, 19(5): 056035.
29. Han K, Xiao A, Wu E, et al. Transformer in transformer[J]. Adv Neural Inf Process Syst, 2021, 34: 15908-15919.
30. Zhang T, Li L, Cao S, et al. Attention-guided pyramid context networks for detecting infrared small target under complex background[J]. IEEE Trans Aerosp Electron Syst, 2023, 59(4): 4250-4261.
31. Zhang T, Li L, Igel C, et al. LR-CSNet: low-rank deep unfolding network for image compressive sensing[C]//2022 IEEE 8th International Conference on Computer and Communications (ICCC). Chengdu, China: IEEE, 2022: 1951-1957.
32. Zhang T, Li L, Peng Z. Optimization-inspired cumulative transmission network for image compressive sensing[J]. Knowl Based Syst, 2023, 279: 110963.
33. Zhang T, Li L, Zhou Y, et al. CAS-ViT: convolutional additive self-attention vision transformers for efficient mobile applications[J]. IEEE Transactions on Image Processing, 2026, 35: 1899-1909.
34. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 770-778.
35. Liu Z, Shen Z, Savvides M, et al. ReActNet: towards precise binary neural network with generalized activation functions[C]//European Conference on Computer Vision (ECCV). Cham: Springer, 2020: 143-159.
36. Das N, Biesmans W, Bertrand A, et al. The effect of head-related filtering and ear-specific decoding bias on auditory attention detection[J]. J Neural Eng, 2016, 13(5): 056014.
37. Biesmans W, Das N, Francart T, et al. Auditory-inspired speech envelope extraction methods for improved EEG-based auditory attention detection in a cocktail party scenario[J]. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(2): 1-11.
38. Fuglsang S A, Dau T, Hjortkj?r J. Noise-robust cortical tracking of attended speech in real-world acoustic scenes[J]. Neuroimage, 2017, 156: 435-444.
39. Wong D D E, Fuglsang S A, Hjortkj?r J, et al. A comparison of regularization methods in forward and backward models for auditory attention decoding[J]. Front Neurosci, 2018, 12: 531.
40. Zhang H, Zhang J. Based on audio-video evoked auditory attention detection electroencephalogram dataset[J]. J Tsinghua Univ (Sci Technol), 2024, 64(11): 1919-1926.

Journal of Biomedical Engineering

Dual-stream temporal-frequency convolutional network with additive attention for auditory attention decoding

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content