An old bird is not to be caught with chaff. An experienced man, or one with his wits about him, is not to be deluded by humbug. The reference is to throwing chaff instead of bird-seed to allure birds. Hence—
You are chaffing me.
Making fun of me. A singular custom used to exist in Notts and Leicestershire some half a century ago. When a husband ill-treated his wife, the villagers emptied a sack of chaff at his door, to intimate that “thrashing was done within,” which some think to be the origin of the word.
“To chaff, ” meaning to banter, is a variant of chafe, to irritate.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894