Within the pale of my observation- i.e. the scope thereof. The dominion of King John and his successors in Ireland was marked off, and the part belonging to the English crown was called the pale, or the part paled off.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Pale from Infoplease: