Lady Berkley was made by Queen Mary a justice of the peace for Gloucestershire and appointed to the quorum of Suffolk. Lady Berkley sat on the bench at assizes and sessions, girt with a sword. Tony Lumpkin says of Mr. Hardcastle
“He'll persuade you that his mother was an alderman and his aunt a justice of the peace.” —Goldsmith: She Stoops to Conquer.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Lady Magistrate from Infoplease: