“Mr. X. will not yield. He has taken the bit between his teeth, and is resolved to carry out his original measure.” —Newspaper paragraph, April, 1886.
Money. The word is used in the West Indies for a half pistareen (fivepence). In Jamaica, a bit is worth sixpence, English; in America, 12 1/2 cents; in Ireland, tenpence.
The word is still thieves' slang for money generally, and coiners are called bit-makers. In English we use the word for a coin which is a fraction of a unit. Thus, a shilling being a unit, we have a six-penny bit and threepenny bit (or not in bits but in divers pieces). So, taking a sovereign for a unit, we had seven-shilling bits, etc.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894