Building an organized Yukawa structure of quarks and leptons is an essential mission to understand fermion mass hierarchy and flavor mixing in particle physics. Inspired by the similarity of CKM and PMNS mixings, a common mass pattern for up-type and down-type quarks, charged leptons, and Dirac neutrinos is realized in terms of hierarchal masses. An organized structure of Yukawa couplings can be expressed on a Yukawa basis. The CKM mixing and PMNS mixing have the same mathematical forms due to $SO(2)$ family symmetry. By fitting the CKM/PMNS mixing experiment, the flat pattern becomes a successful flavor structure. The fundamental fermions can be explained by a new picture to show the relationship between gauge structure and flavor structure.