Something that is offensive to the sight. Sore is the Anglo-Saxon sar (painful) or swær (grievous). It is painful or grievous to the eye.
“Mordecai was an eye-sore to Haman.” -
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Eye-sore from Infoplease: