Out of all notch. Out of all bounds. The allusion is to the practice of fitting timber: the piece which is to receive the other is notched upon; the one to fit into the notch is said to be notched down.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Notch from Infoplease: