Yellow fruit flesh color, resulting from the accumulation of carotenoids, is one of the most important commercial traits of peach. Yellow flesh is controlled by a single locus (Y), with white flesh dominant over yellow flesh. In this study, the Y locus was narrowed to a 2.6-cM interval flanked by two markers, SSRy andW2691. SSRy, which is located on the first exon of a gene encoding carotenoid cleavage dioxygenase (CCD4), was cosegregated with the Y locus in two peach F1 populations. RNA-Seq and qRT-PCR analysis revealed transcript level of CCD4 was consistent with carotenoid degradation in peach fruits. All these results suggest that CCD4 is responsible for white and yellow coloration of peach fruit flesh. In fruits of white-fleshed peach, carotenoids are synthesized but subsequently degraded into colorless compounds, leading to the formation of white color. CCD4 is likely to utilize β-carotene as the substrate in peach. Interestingly, CCD4 also controls white and yellow coloration of leaf midveins of peach. Moreover, LCYE was highly expressed in peach leaves, whereas its transcript was not detectable in fruits. This suggests the difference of carotenoid biosynthesis between peach fruits and leaves. Our study not only shows for the first time the pleiotropic effects of CCD4 gene in peach but also provides a morphological marker for easy selection of new peach cultivars with desirable white or yellow flesh colors.