Alzheimer’s disease (AD), the most common form of dementia in elderly people, is an irreversible and incurable neurodegenerative disease that is characterized by progressive memory loss and cognitive deficits (Association, 2019). With its cause and pathology remaining under study, there is no prevention or cure for it. Therefore, early and accurate diagnosis of AD at its preclinical stage of mild cognitive impairment (MCI) becomes the most effective practice to enable possible interventions and delay its progression (Kam et al., 2019). Towards this end, intense research has been devoted to precise MCI classification with neuroimaging data.

Among various imaging techniques, the resting-state functional Magnetic Resonance Imaging (rs-fMRI) (Fan et al., 2007) is one of the most widely adopted modalities for MCI classification (Jie et al., 2014, Liu et al., 2014). While PET can also be used for early MCI diagnosis, rs-fMRI has been favored due to its non-radiation nature and affordability (Kropotov, 2016, Mier and Mier, 2015). On the other hand, structural changes observed in structural Magnetic Resonance Imaging (sMRI) often appear several years after these functional alterations (Sheline and Raichle, 2013, Hahn et al., 2019), which may delay the diagnostic process and potentially hinder timely interventions. In comparison, rs-fMRI enables early identification of functional brain abnormalities in MCI when observable structural changes are still ambiguous by building and analyzing functional brain networks (FBNs). How to construct effective FBNs and extract informative features from them for abnormality detection has been the focus of the studies in this line (Jie et al., 2018). Conventional FBN-based MCI classification methods typically consist of three separate stages: FBN construction, feature extraction, and classification. In the first stage, the cerebral cortex is divided into distinct brain regions of interest (ROIs) as nodes according to anatomical brain region templates, e.g., AAL atlas (Tzourio-Mazoyer et al., 2002). After estimating pairwise functional connectivities between nodes, FBNs are generated, and the subsequent two stages of hand-crafted feature extraction and classification are carried out. These methods suffer from inflexibility since the three subtasks in the process are conducted sequentially and lack interactions and adaptivity. This can restrict their performance. For example, the FBNs constructed in the first step may not necessarily well serve the following two steps of feature extraction and classification, especially when prior knowledge or experience is lacking.

Recent development of deep learning has revolutionized the fields of FBN construction and analysis. By feeding the built FBNs into deep models, adaptive features can be extracted and jointly optimized with the classification layers in an end-to-end manner. The work in (Kawahara et al., 2017) is among the first ones proposing convolutional neural network (CNN) models for brain connectivity network analysis. Following it, more advanced CNN-based models are developed in (Kam et al., 2019) and (Huang et al., 2020) to capture richer information from FBNs for brain function evaluation or disorder diagnosis. Besides CNN, the recent advancement of graph convolution network (GCN) also inspires intensive applications of them on FBN analysis (Gadgil et al., 2020, Li et al., 2021, Yao et al., 2021, Liu et al., 2022) since they are especially designed to handle graph data and enable flexible extraction of topological features.

Although these deep learning methods mentioned above have demonstrated performance breakthroughs in FBN-based brain disorder diagnosis, they have a major limitation that their model architectures are inherently ‘flat’, i.e., they extract and propagate features through the same number of brain nodes in different layers and do not reflect the hierarchical organization of the brain (Ying et al., 2018). Numerous studies (Sahoo et al., 2020, Meunier et al., 2009, Ferrarini et al., 2009, Park and Friston, 2013) have identified that the structure and function of the human brain are hierarchically organized. Structurally, the neural anatomy of human brain is organized over multiple orders of magnitude, ranging from the level of individual neurons, through cortical columns to lobes and systems (Fornito et al., 2016). Such hierarchical modular organization enables diverse functional interactions between different levels of brain partitions and allows nested information segregation and integration during cognitive functions across multiple spatial resolutions (Wang et al., 2019). In this case, while enjoying simplicity, only considering a single scale FBN based on predefined brain partitions, e.g., a brain atlas, would inevitably overlook the rich and complementary information that may be present at other scales (Lei et al., 2020).

In order to address the limitations of the single scale FBN, a handful of pioneering studies have been devoted to building multi-scale FBNs for joint analysis within deep learning frameworks. Among these methods, a majority of them (Yao et al., 2021, Liu et al., 2021b, Liu et al., 2022, Wang et al., 2020b, Liu et al., 2021a, Wang et al., 2022) utilize a series of brain atlases to define coarse-to-fine ROIs, and then the corresponding multi-scale FBNs can be constructed by repeatedly generating an FBN with each level of ROI definition. For example, the study in (Yao et al., 2021) employs four different atlases with different numbers of ROIs, including AAL (Tzourio-Mazoyer et al., 2002) with 116 ROIs, Craddock with 200 ROIs, Brainnetome with 273 ROIs, and Bootstrap Analysis of Stable Clusters with 325 ROIs, to construct multi-atlas FBNs as the input of GCN for feature extraction and classification. In contrast to different atlases, one atlas at multiple spatial scales, ranging from 100 to 500 ROIs, is applied in (Liu et al., 2021b, Liu et al., 2022) to form a brain network hierarchy as a representation of each subject. These methods, to a certain extent, alleviate the limitations of a single scale FBN and demonstrate the advantages of integrating multi-scale FBNs. Nevertheless, there are at least two drawbacks to this line of methods. Firstly, the flexibility and generality of these methods are restricted by the availability of brain atlases. Specifically, the ROI numbers and scale levels in the existing atlases are usually determined by the anatomical structures of the brain, so they are fixed and can hardly be adjusted for functional analysis. This may also affect the generality of these methods. For example, the methods (Liu et al., 2021b, Liu et al., 2022) using the same series of atlases at multiple spatial scales may not readily generalize to other atlases, e.g., the AAL atlas (Tzourio-Mazoyer et al., 2002), due to lack of such multiple spatial resolutions in the latter. Secondly, the multi-scale FBNs are pre-built independently for analysis in the following steps. Such multi-step approach may not be optimal due to the lack of communication between different steps. For example, the information lost in the FBN construction step can in no way be recovered in the FBN analysis step.

In addition to the deep learning multi-atlas approach mentioned above, traditional machine learning techniques have also been used to estimate hierarchical FBNs, such as hierarchical clustering of voxel-wise fMRI signals (Liu et al., 2012, Wang and Li, 2013) or cascaded factorization of correlation matrices generated from fMRI (Sahoo et al., 2020). However, their hierarchical structures are obtained by unsupervised methods primarily for general representation purpose, and, as will be demonstrated in the experiments, they usually lack discrimination and specificity when applied to the brain disorder analysis task of this paper. In sum, although it is widely accepted that the human brain is a hierarchical system with nested structures, how these hierarchies should be modeled and mapped to each other remains an ongoing research topic (Axer and Amunts, 2022).

Different from the above methods, this paper proposes an integrated deep network that simultaneously constructs and analyses hierarchical FBNs in an end-to-end manner. This is achieved by incorporating a sparse attention-based FBN construction module into the deep model and adaptively modeling the connectivity strength between brain regions according to their regional features. Sparsity (Peters et al., 2019) is introduced to reflect the fact that a brain region usually only interacts with a limited number of other regions (Song et al., 2013). Meanwhile, a node fusion module is designed to softly assign the fine-grained nodes of the current layer to coarser nodes in the next layer. In this case, a hierarchical representation of FBN can be adaptively obtained from the alternately cascaded FBN construction and node fusion modules. Within the resulting hierarchical FBNs, the functionally similar nodes are repeatedly aggregated into coarse ones layer by layer. This structure allows nested functional segregation and integration across multiple brain region scales and could uncover the brain’s intrinsic modular organizations and diverse functional interactions, providing more clues for brain disorder diagnosis. Unlike the predefined multiple brain atlases used in the existing methods, the assignment weights from finer level brain regions to those at a coarser level are learned in an end-to-end manner along with the FBN construction and feature extraction modules. This results in hierarchical structures that are better aligned with other deep network components, leading to improved discriminability towards the diagnostic tasks. Additionally, soft assignment is utilized to reflect that certain brain regions would be involved in multiple overlapping communities.

While any existing brain atlas can be used as the starting point for the hierarchical FBN construction, the effectiveness of the proposed approach, as will be demonstrated in the experiments, suggests the possibility of exploring finer brain regions beyond the predefined regions in an atlas. Therefore, we propose to overcome the scale limitation of existing atlases by using a data-driven clustering method to split each predefined region into multiple sub-regions. By doing this, it enables us to build hierarchical FBNs with a flexible number of brain regions at each scale, without being restricted by the predefined ROI numbers of the existing atlases.

The major contributions of this paper can be summarized as follows:

This paper proposes a novel attention-based deep network for adaptive and integrated construction and analysis of hierarchical FBNs. The hierarchical FBN representation is generated by progressively coarsening the brain regions layer by layer via soft assignment, which is learned jointly with other network components in an end-to-end manner. Compared to conventional methods, the proposed method is more flexible and adaptable to specific tasks;

This work overcomes the scale limitations of existing brain atlases available in the traditional methods during hierarchical FBN construction and analysis, improving the model’s generality and adaptability in multiscale FBN analysis. This is achieved in two ways: firstly, the brain regions in the coarse scales are determined by adaptive learning rather than predefined brain atlases; secondly, the starting scale, i.e., the finest level of brain regions, is not restricted by the existing brain atlas. Instead, the proposed method derives multiple sub-ROIs from each original ROI in a data-driven manner as the starting scale;

The superior performance of the proposed method over the competing methods has been consistently demonstrated in MCI classification on various data sets. In addition to better classification accuracy, the hierarchical FBNs constructed by the proposed method also provide a novel perspective to uncover the hierarchical structure of the human brain for brain function understanding and disease biomarker identification.