Who eats oysters on St. James's Day will never want. St. James's Day is the first day of the oyster season (August 5th), when oysters are an expensive luxury eaten only by the rich. By 6, 7 Vict., c. 79, the oyster season begins September 1, and closes April 30.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Oysters from Infoplease: