The palace of the Turkish sultan, situated in the Golden Horn, and enclosed by walls seven miles and a half in circuit. The chief entrance is the Sublime Gate; and the chief of the large edifices is the Harem, or “sacred spot,” which contains numerous houses, one for each of the sultan's wives, and others for his concubines. The black eunuchs form the inner guard, and the white eunuchs the second guard. The Seraglio may be visited by strangers; not so the Harem.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Seraglio from Infoplease: