The thorn in the side, the bitter in the cup, the spoke in the wheel, the black sheep, the object of aversion. A black sheep has always been considered an eyesore in a flock, and its wool is really less valuable. In times of superstition it was looked on as bearing the devil's mark.
“The Dutch sale of tin is the bête noire of the Cornish miners.” —The Times.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894