Brewer's: Fancy Man
(A). A cavalier servant or cicisbeo; one selected by a married lady to escort her to theatres, etc., to ride about with her, and to amuse her. The man she “fancies” or likes.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894