Genomes are organised through hierarchical structures, ranging from local kilobase-scale cis-regulatory contacts to large chromosome territories. Most notably, (sub)-compartments partition chromosomes according to transcriptional activity, while topologically associating domains (TADs) define cis-regulatory landscapes. The inactive X chromosome in mammals has provided unique insights into the regulation and function of the three-dimensional (3D) genome. Concurrent with silencing of the majority of genes and major alterations of its chromatin state, the X chromosome undergoes profound spatial rearrangements at multiple scales. These include the emergence of megadomains, alterations of the compartment structure and loss of the majority of TADs. Moreover, the Xist locus, which orchestrates X-chromosome inactivation, has provided key insights into regulation and function of regulatory domains. This review provides an overview of recent insights into the control of these structural rearrangements and contextualises them within a broader understanding of 3D genome organisation.