Animals can cope with isolated stressful situations without enduring long-term consequences. However, when exposure to stressors becomes recurrent, behavioral symptoms of anxiety and depression can emerge. Yet, the neuronal mechanisms governing responsivity to isolated stressor remain elusive. Here, we investigate synaptic adaptations following mild stress in the lateral habenula (LHb), a structure engaged in aversion encoding and dysfunctional in depression. We describe that neuronal depolarization in the LHb drives long-term depression of inhibitory, but not excitatory, synaptic transmission (GABA LTD). This plasticity requires nitric oxide and presynaptic GABAB receptors, leading to a decrease in probability of GABA release. Mild stressors such as brief social isolation, or exposure to novel environment in the company of littermates, do not alter GABA LTD. In contrast, GABA LTD is absent after mice experience a novel environment in social isolation. Altogether, our results suggest that LHb GABAergic plasticity is sensitive to stress accumulation, which could represent a threshold mechanism for long-term alterations of LHb function.