Hydrogels (HGs) and nanogels (NGs) have been recently identified as innovative supramolecular materials for many applications in biomedical field such as in tissue engineering, optoelectronic, and local delivery of active pharmaceutical ingredients (APIs). Due to their in vivo biocompatibility, synthetic accessibility, low cost, and tunability, peptides have been used as suitable building blocks for preparation of HGs and NGs formulations. Peptide HGs have shown an outstanding potential to deliver small drugs, protein therapeutics, or diagnostic probes, maintaining the efficacy of their loaded molecules, preventing degradation phenomena, and responding to external physicochemical stimuli. In this review, we discuss the possible use of peptide-based HGs and NGs as vehicles for the delivery of the anticancer drug doxorubicin (Dox). This anthracycline is clinically used for leukemia, stomach, lung, ovarian, breast, and bladder cancer therapy. The loading of Dox into supramolecular systems (liposomes, micelles, hydrogels, and nanogels) allows reducing its cardiotoxicity. According to a primary sequence classification of the constituent peptide, doxorubicin-loaded systems are here classified in short and ultra-short peptide-based HGs, RGD, or RADA-peptide-based HGs and peptide-based NGs.