A game similar to court-tennis; the hand, however, is used instead of a racket. Said to be so called because the game is three fives (15).
“He forgot that cricket and fives are capital training for tennis.” —T. Hughes: Tom Brown at Oxford, chap. ii.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Fives from Infoplease: