Magnetic resonance imaging (MRI) is an important non-invasive clinical imaging modality that does not use ionizing radiation. A big benefit of MRI is its ability to capture a multitude of tissue contrasts by changing the acquisition parameters, providing complementary information to characterize and assess pathology. A typical clinical MRI protocol consists of multiple independent scans, contributing to an overall lengthy exam [1]. Long exams lead to high associated costs for patients and also reduce the overall throughput of the scanner. Moreover, it is difficult for patients (especially pediatric and elderly) to hold still for a long scanning period, hence making the images susceptible to motion-induced artifacts [2]. Even when the scans are artifact-free, the contrast weightings represent a qualitative signal intensity that must be visually inspected and reported on by a clinical radiologist, and thus only qualitative signal abnormalities can be reported.

Recently there has been substantial interest in extracting quantitative information from MRI in addition to conventional qualitative images [3,4]. In quantitative MRI, biophysical tissue and system parameters are estimated using specialized multi-contrast acquisitions that acquire the MRI signal at multiple contrast points. While conventional quantitative imaging is slow in nature, recent work has led to fast and comprehensive multi-contrast scans [[3], [4], [5], [6]]. After tissue quantification, arbitrary contrasts can be synthetically created by evaluating the MRI signal equation for a retrospectively chosen set of scan parameters in silico. A large body of work has focused on accurate quantification of T1, T2, and proton density (PD) maps [4,7]. System imperfections including magnetic and radio-frequency (RF) field inhomogeneities have also been incorporated [3,8]. However, the underlying MRI physics are more nuanced and have many confounding parameters that contribute to the final image contrast, including magnetization transfer, partial voluming, diffusion, and susceptibility, to name a few. While it is possible to extend the quantitative imaging protocol to map these additional effects, the scan time can become impractical, and the physical modeling can be challenging. As a result, the synthetic contrast image often has subtle differences when compared to the image obtained from a real acquisition. For example, synthetic T2-FLAIR contrast is known to suffer from hyperintense signal artifacts surrounding the cerebrospinal fluid (CSF) [9,10], likely due to the over-simplification of the MRI signal model.

To combat contrast-mismatch effects, researchers have recently leveraged deep learning to correct these artifacts [[11], [12], [13]]. These methods pose contrast correction as a supervised image-to-image translation, in which a particular set of experimental contrasts are acquired and treated as reference images, and a deep neural network is trained to map the incorrect contrast to the experimental contrast. While the results are striking, the approaches are limited in that they only correct a particular set of contrasts that are collected at training time, and therefore they lose the ability to synthesize arbitrary contrast-weighted images offline.

Therefore in this work, we propose a novel physics-enabled deep learning method to correct arbitrary contrast-weighted images generated by synthetic MRI. While we also pose the problem as supervised image translation, we instead aim to learn the mapping from synthetic MRI to experimental contrast as a function of scan parameters. Motivated by the main unmodeled effects in fast spin-echo (FSE) imaging, our proposed method generates a spatially dependent multiplicative correction term and a scaled apparent longitudinal relaxation time (T1). We provide several different experimental contrast-weighted images as training samples so that our network implicitly learns the relationship between scan parameters and the physical effects that are not captured by the simplified signal equation. We accomplish this using a conditional generative adversarial network (cGAN) framework complimented with perceptual loss from a separate pre-trained network [[14], [15], [16]]. To incorporate scan parameters, we include quantitative maps and synthetic contrast images along with scan parameters as additional channels for the generator network. While we demonstrate our approach for a specific multi-contrast sequence available on our scanner, we emphasize that the framework could be used for other sequences such as magnetic resonance fingerprinting (MRF) [3].

The main contributions of our paper are as follows: i) we propose a novel deep learning method that explicitly incorporates the scan parameters to correct arbitrary synthetic MRI contrast-weighted images derived from a multi-contrast sequence; ii) we show how such models can be trained using a 2D multi-contrast sequence as proof of principle, so that the pipeline can be used for other custom multi-contrast sequences; iii) we evaluate our method on subjects and MRI contrasts that were not included in the training set to give evidence that the approach implicitly accounts for unmodeled physical effects. The proposed model gives better numerical error metrics than the direct and residual correction model. We will make our data and implementation available publicly upon publication.

Image-to-image translation has been highly successful in computer vision, in which conditional generative models are trained to map from one image style to an output image style [14,17]. Analogous to these tasks, researchers in medical imaging have also proposed image-to-image translation models; for example, computed tomography (CT) to MRI [18], positron emission tomography (PET) to CT [19], medical image segmentation [20], low-to-high gadolinium dose contrast-enhanced brain MRI [15], etc. Specifically in MRI, translating from one MRI contrast-weighted image to another contrast-weighted image is a well investigated problem. Authors in [16] propose multi-contrast synthesis through cGANs and demonstrate the applicability by translating from T1-weighted to T2-weighted images and vice versa. A multi-stream approach was also proposed to join information from one-to-one and many-to-one translation streams using a fusion block [21]. The work in [22] proposes MRI motion correction through cGANs by translating from motion-corrupted to motion-free images by incorporating FSE acquisition dynamics. Authors in [23] proposed a multi-input, multi-output GAN network to generate missing MRI sequences using the redundant information from other available sequences. The work in [24] proposed a hybrid-fusion network to generate target MRI contrasts from source images. The overall network consisted of two small sub-networks, the first network to learn representations from each input modality and the second network to fuse the common latent representation and synthesize target images. The authors in [25], proposed an edge-aware network that captures the textural details of MR images to improve the overall final image quality in cross-modality MR synthesis. A novel method to generate 3D brain MRI from learned representations using variational auto-encoder and GAN was proposed in [26] that generates high-quality image data from limited training data. Authors in [27] applied the cGAN framework to reconstruct patient faces from anonymized T1 sagittal slices using unsupervised training and were able to recover patient information from both face blurred and face removed data. Similarly, work in [28] showed the multimodal image synthesis using GANs on glioma patients. Authors in [29] proposed DiamondGAN to do non-aligned cross-modality synthesis and performed a radiologist evaluation study to show that trained radiologists were not able to differentiate between experimental and synthetic MRI images.

The other MRI translation task frequently considered is to correct the synthetic MRI contrast. The authors in [13] looked at direct contrast synthesis from a multi-contrast scan using a temporal-convolutional network on a per-pixel basis. The work in [11] extended the direct contrast synthesis to operate on the whole input image and output the desired contrast using cGANs whereas work in [30] presented a convolutional encoder-decoder network to directly generate multiple contrast image from base multi-echo sequence. Authors in [31] proposed a deep learning method to improve the T2-FLAIR contrast generation directly from base multi-contrast images. On the same track, the work in [12] corrected the synthetic contrasts of T2-FLAIR on a per-pixel basis using convolutional neural networks. Similarly, authors in [10] used a generative network to translate directly from the echo images of the base multi-contrast images to the FLAIR contrasts. The review paper [32] discusses the synthetic MRI methods to generate multiple contrast images along with some of the limitations in synthetic contrast generation; in particular, they report lower quality in synthetic FLAIR images. Common to all these works is the need to collect the ground-truth experimental contrasts that are desired, and the restriction to only correcting those contrasts. In comparison, our goal is to maintain the ability to synthesize contrasts corresponding to arbitrary scan parameters. All of the discussed works follow the approximate framework shown in Fig. 1 to directly translate from base multi-contrast images to final target contrasts, which deviate from the premise of synthetic MRI to generate arbitrary contrasts.

Several clinical validation studies have shown the benefits of synthetic MRI, while also highlighting its limitations. In [33], clinicians rated all synthetic contrasts to be inferior to the conventional scans except T2 weighted images. Prior to that, authors in [34] conducted a clinical validation study using the Multi Delay Multi Echo (MDME) sequence. It was reported that overall image quality was similar for all contrasts except for FLAIR where a conventional scan was still clinically necessary. Similarly, work in [35] reported satisfactory image quality for all contrasts except T2 FLAIR. The thread of these works is that the inversion recovery contrast images were hard to correctly synthesize, with implications on their clinical use.