Adverse childhood experiences have been linked to detrimental mental health outcomes in adulthood. This study investigates a potential neurodevelopmental pathway between adversity and mental health outcomes: brain connectivity. We used data from the prospective, longitudinal Adolescent Brain Cognitive Development (ABCD) study (N ≍ 12.000, participants aged 9–13 years, male and female) and assessed structural brain connectivity using fractional anisotropy (FA) of white matter tracts. The adverse experiences modeled included family conflict and traumatic experiences. K-means clustering and latent basis growth models were used to determine subgroups based on total levels and trajectories of brain connectivity. Multinomial regression was used to determine associations between cluster membership and adverse experiences. The results showed that higher family conflict was associated with higher FA levels across brain tracts (e.g., t(3) = −3.81, β = −0.09, pbonf = 0.003) and within the corpus callosum (CC), fornix, and anterior thalamic radiations (ATR). A decreasing FA trajectory across two brain imaging timepoints was linked to lower socioeconomic status and neighborhood safety. Socioeconomic status was related to FA across brain tracts (e.g., t(3) = 3.44, β = 0.10, pbonf = 0.01), the CC and the ATR. Neighborhood safety was associated with FA in the Fornix and ATR (e.g., t(1) = 3.48, β = 0.09, pbonf = 0.01). There is a complex and multifaceted relationship between adverse experiences and brain development, where adverse experiences during early adolescence are related to brain connectivity. These findings underscore the importance of studying adverse experiences beyond early childhood to understand lifespan developmental outcomes.