Torreya Arn., a small genus of Taxaceae, consists of six species occurring in North America and eastern Asia. Several phylogenetic studies have previously been undertaken to reveal relationships within this genus, although only a few DNA segments or species were used. In the present study, we sequenced five Torreya plastomes and combined these with two existing plastomes from the genus to investigate plastome evolution and phylogenetic relationships within Torreya. All sequenced Torreya plastomes shared the same complement of 82 protein-coding genes, 4 ribosomal RNA genes, and 31 transfer RNA genes. Phylogenetic inference using a maximum likelihood framework consisted of an 82-gene, 17-taxon dataset, including all species of Torreya, resolved Torreya as a monophyletic clade. Strongly supported relationships within the genus include the position of the early diverging T. jackii Chun, the two sister pairs T. fargesii Franch.–T. nucifera (L.) Siebold & Zucc. and T. grandis Fortune ex Lindl.–T. californica Torr., and the monophyly of the clade including T. fargesii var. yunnanensis, T. fargesii, and T. nucifera. In addition to the inference of species relationships, divergence time estimation and biogeographical analysis were carried out. The diversification of Torreya was estimated to be approximately 8.9 Ma. Ancestral state reconstruction of the geographical area suggested China/eastern North America as the most likely ancestral region for the six extant Torreya species.