Recently, Ji Longcheng, a doctoral student in the School of Mechanical Engineering and Automation, published an article in Medical Image Analysis, a top-tier international journal in the field of biomedical engineering. The article titled “Memory Like the Human Brain: A Framework for Decoding Multimodal Learning of Brain-Visual-Linguistic Features” lists Ji Longcheng as the first author and Professor Wang Hong of the School of Mechanical Engineering and Automation as the corresponding author. The sole affiliated institution listed in the paper is the School of Mechanical Engineering and Automation at NEU.
How does the human brain process the images and text? This study aims to teach machines to understand this as well. This article introduces the MLHuB (Memory Like the Human Brain) model which achieves deep alignment across three modalities: EEG signals, vision, and language. This model mimics the learning mechanisms of the human brain and is a method that uses memory cells to read and update learned text-image features, thereby consolidating acquired knowledge. Meanwhile, it introduces an orthogonal projection mechanism to separate shared representations from modality-specific representations, and utilizes mutual information maximization to ensure consistent alignment across modalities. Tests on multiple benchmark datasets show that MLHuB achieves state-of-the-art performance and offers a more interpretable and generalizable approach to understanding how the brain decodes visual and linguistic information.
Visual decoding technology enables machines to directly perceive human intent, offering broad application prospects in fields such as neurological rehabilitation, intelligent prosthetic control, and man-machine interaction.
