Gotta say I'm not a linguist but I've never once ever heard a guy refer to himself as "petite", in fact I'm almost positive that it has for the most part always been a female only adjective due to its french origins as romance languages have gendered nouns and adjectives; sorta like "el niño" for boys and "la niña" for girls in Spanish. Tbh I'm not even sure if there is even a male equivalent in french.