Three-dimensional (3D) models of bones are used in various clinical areas. For orthopedic surgeons who specialize in treating disorders of the bones and joints, inferring the 3D shape of the patient's bone is crucial. Moreover, during surgical planning or preparation of surgical navigation for fracture reduction, which provides intuitive information on the current state of the fracture through augmented/virtual reality, the precise shape of the patient's bone is essentially required [1], [2]. Conventionally, computed tomography (CT) is a common method for obtaining a 3D model of a patient's bone. Although CT can provide an accurate and detailed 3D bone model, it exposes the patient to high radiation, in addition to being expensive and time-intensive [3].

As an alternative to CT, 2D-3D reconstruction has been investigated [3], [4], [5], [6], [7], [8]. It constructs a 3D model of a patient's bone, called a patient-specific model, from several of the patient's 2D X-ray images. Most approaches in 2D-3D reconstruction utilize a statistical parametric model, such as a statistical shape model (SSM) or statistical shape and intensity model (SSIM). Because the statistical parametric model is a deformable geometric model derived through statistical shape analysis of the objects to be modeled, it can express semantically similar objects by an average shape of many 3D objects and their variations [9], [10], [11], [12]. Thus, as deforming the statistical parametric model by using features inferred from the patient's 2D X-ray images, the patient's bone can be reconstructed in 3D [13].

To deform the statistical parametric model to the patient-specific model in 2D-3D reconstruction, several procedures with multiple images are required, such as 2D-3D registration, calibration, and optimization for 3D reconstruction. Although these sub-procedures are imperative for 2D-3D reconstruction, they lead to limitations in complexity and reproducibility [4], [5], [6], [14], [15], [16], [17], [18], [19]. As a technique for estimating the relationships between a 3D model and its 2D image, 2D-3D registration is still a challenge for convincing performance due to ambiguity in similarity metrics and uncertainty in verification despite substantial efforts over the past decades. To mitigate the problem in practical situation, some manual interventions such as designing and selecting landmarks for initialization would be required [10], [20], [21]. Calibration of the C-arm, which is the process of identifying the intrinsic characteristics of the X-ray imaging device, is an essential prerequisite for 2D-3D registration and the 3D reconstruction of corresponding points in bi-planar X-ray images [2], [5], [6]. However, calibration is a cumbersome and inconvenient task because it deviates from the conventional clinical routine. It also requires extra devices, such as a calibration phantom. Accordingly, it makes the procedure of 2D-3D reconstruction complex and time-consuming [22]. 3D reconstruction is conducted by optimization based on inferred features from images by the aforementioned procedures. For optimization, the similarity in the 3D or 2D coordinates between the true and predicted models is repeatedly compared. This process also requires a proper initialization to avoid local maxima and high computational cost [23], [24].

Recently, deep neural networks (DNN) have been used for 2D–3D reconstruction. Although deep learning approaches have been developed to reconstruct 3D shapes from images captured by a camera, such as 3D-R2N2, Pix2Vox, and 3D-VAE-GAN, they are focused on informative feature representation for 3D objects in computer vision field [25], [26], [27], [28]. In medical imaging, the 3D reconstruction of vertebrae and knee bones from bi-planar X-ray images has been achieved using 3D convolutional layers [23], [29]. However, the network structure of 3D reconstruction is more complex than that of 2D convolution layers, and the dependency on the training data is also high, which leads to a sophisticated network structure with 3D convolutional layers. 3D reconstruction by SSM with 2D convolution layers has also been suggested to reconstruct a 3D spine from a calibrated bi-planar radiograph [14]. By refining the landmark positions of vertebra that are detected and localized by 2D convolution layers, a 3D model is reconstructed. However, bi-planar X-ray image with calibration and an iterative process are still required for deformation.

In the proposed method, a proximal/distal 3D bone model of the patient is constructed using a deep transfer learning network based on the SSM with a single X-ray image acquired at an invariant viewpoint position. Thus, despite the absence of multiple-input images and notorious processes including calibration, 2D-3D registration, and, optimization, which are prerequisites for conventional statistical parametric model-based methods, as well as 3D convolutional layers with extensive 3D training dataset, the 3D model of the patient can be effectively constructed. The novelty of the present study is that features extracted from a single 2D X-ray image by a deep transfer learning network uniquely identify deformation parameters in the SSM which determine the 3D shape. In order to learn distinctive features, which imply highly correlated 3D shape information related to deformation parameters in the SSM, from a single 2D X-ray image, a specific target of femur from the unique pose of the X-ray source by reflecting anatomical structure analysis was suggested. Moreover, simulated X-ray images, which are digitally reconstructed radiographs (DRRs), are used as the training images. After diversely deformed 3D femur models are constructed by adjusting the deformation parameters in the SSM, simulated X-ray images of their 3D models are generated from the designated pose of the X-ray source. Through the training the DNN with both simulated X-ray images and their corresponding deformation parameters, the deformation parameters could be predicted by the single image.