A curtailed oath. “Go to the devil!” or some such phrase.
Cassius: I [am] able than yourself To make conditions. Brutus: Go to! You are not, Cassius.'
GO TO BANFF, and bottle skate.
GO TO BATH, and get your head shaved.
GO TO BUNGAY, and get your breeches mended.
GO TO COVENTRY. Make yourself scarce.
GO TO HEXHAM. A kind of Alsatia or sanctuary in the reign of Henry VIII.
GO TO JERICHO. Out of the way. (See Jericho.)
And many other similar phrases.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894