Brewer's: Leave out in the Cold
(To). To slight, to take little or no interest in a person; to pass by unnoticed. The allusion is to a person calling at a house with a friend and the friend not being asked to come in.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894