The soul or spirit was often supposed in olden times to assume a zoömorphic form, and to make its way at death through the mouth of man in a visible form, sometimes as a pigeon, sometimes as a mouse or rat. A red mouse indicated a pure soul; a black mouse, a soul blackened by pollution; a pigeon or dove, a saintly soul.
Exorcists used to drive out evil spirits from the human body, and Harsnet gives several instances of such expulsions in his Popular Impositions (1604).
No doubt pigeons were at one time trained to represent the departing soul, and also to represent the Holy Ghost.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Mouse from Infoplease: