Japanese people are widely believed to be the descendants of the Jomon and Yayoi people. The dual-structure model, which attempts to explain the formation of the Japanese population, hypothesizes that the indigenous Jomon people migrated from the northern island of Hokkaido and the Ryukyu Islands to the other islands of Japan, where they resided before the Yayoi started to arrive in Kyushu (the westernmost main island of the Japanese archipelago) from the Korean peninsula. Regarding Y chromosome DNA polymorphisms, it is assumed that Jomon males frequently belong to haplogroups C or D, while Yayoi males frequently belong to haplogroup O. These findings suggest that the frequencies of haplogroup C, D, and O might differ between Hokkaido and northern Kyushu males and exhibit geographical gradients in Japan. However, the data of Y chromosome haplogroup frequencies in modern Japanese males is still limited. Here, we investigated whether the frequency of Y chromosome haplogroups differs between males from different regions of Japan. We recruited 2390 males from nine populations in seven cities in mainland Japan and typed their Y chromosome haplogroups. We did not detect any marked variability in the frequencies of these haplogroups among Japanese males, except for a difference between Nagasaki and Kawasaki students. In conclusion, modern Japanese males appear to be genetically homogenized in mainland Japan because of genetic drift and recent frequent gene flow.