It was once supposed that little worms were bred in the fingers of idle servants. To this Shakespeare alludes—
A round little worm Pricked from the lazy finger of a maid.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Idle Worms from Infoplease: