To give the reins. To let go unrestrained; to give licence.
(The). The kidneys, supposed by the Hebrews and others to be the seat of knowledge, pleasure, and pain. The Psalmist says (xvi. 7), “My reins instruct me in the night season,” i.e. my kidneys, the seat of knowledge, instruct me how to trust in God. Solomon says (Prov. xxiii. 16), “My reins shall rejoice when [men] speak right things,”i.e. truth excites joy from my kidneys; and Jeremiah says (Lam. iii. 13), God “caused His arrows to enter into my reins,” i.e. sent pain into my kidneys. (Latin, ren, a kidney.)
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Reins from Infoplease: