After an 18th Century hiatus where the tricorn and bicorn (also known as the cocked hat) supplanted the high hat as the fashion of the day, the high hat, in its new iterations, most notable the stove pipe shape that we now know as the top hat, returned to rule the day in the very late 1700s.
Freudian psychologists had their own interpretations on these brown top hats and those who wore them regarding them as obvious phallic symbols. As funny and impractical as top hats may seem to our modern ideas on fashion, they have stood the test of time.