Brewer's: Too Many for [Me]
or One too many for [me]. More than a match. “Il est trop fort pour moi.”
“The Irishman is cunning enough; but we shall be too many for him.” —Mrs. Edgeworth.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894