Infertility has a detrimental impact on the sexual function of involuntarily childless couples. However, the effect of male sexual function on female partner's sexual function in infertile couples remains unknown. In order to explore the association between these couples' sexual function, 324 consecutive couples with infertility and 326 female-age-matched fertile couples were enrolled in this study. The female sexual function index (FSFI) and sexual history were evaluated in female partners. In addition to the international index of erectile function-15 (IIEF-15) and sexual history, self-reported intravaginal ejaculation latency time was evaluated in male partners. The demographics and depression status of couples were also recorded. We observed a higher occurrence of female sexual dysfunction (FSD) in infertile females than in fertile females (58.6% vs. 50.3%, p = 0.033). FSD was associated with a number of risk variables, including partner's erectile dysfunction (ED), depression and sexual frequency. The overall FSFI score was positively associated with the IIEF-15 score (r = 0.347, p < 0.001). Besides, six FSFI domains were positively correlated with five IIEF-15 domains (p < 0.05). In conclusion, it is critical to assess and address FSD as well as the sexual problems in male partners in infertile couples.