Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows.

To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM.

Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity.

Existing approaches to fMRI analysis fall into two categories, each with significant limitations. ROI-based methods parcellate the brain into regions of interest and compute functional connectivity matrices. While computationally efficient, they may lose fine-grained spatial information and their performance is dependent on the choice of ROI atlas.

Voxel-based methods (e.g., TFF, SwiFT) process raw 4D fMRI volumes directly, preserving spatial and temporal information. However, the massive scale of fMRI data imposes prohibitive memory demands, severely restricting the temporal "context length" that the model can process. This limits their ability to capture long-range temporal dynamics that unfold over tens of seconds.

TABLeT solves this issue by tokenizing fMRI volumes into a compact set of only 27 tokens (or even less!) per frame using an off-the-shelf 2D image autoencoder, enabling a simple Transformer to process extended sequences with limited VRAM by reducing memory consumption by over 7x than prior voxel-based methods.

TABLeT uses the encoder from a frozen Deep Compression Autoencoder (DCAE), pre-trained on natural images, to tokenize each 3D fMRI volume. Each volume is sliced along three axes (sagittal, coronal, axial), and each 2D slice is independently tokenized by the DCAE encoder. The resulting latent representations from all three axes are then aggregated according to their spatial positions, producing just 27 compact tokens per fMRI frame.

These tokens are fed into a lightweight Transformer encoder, enabling long-range spatiotemporal modeling with small memory and computation overhead. We name this method TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer).