From the Spanish lavandera (a laundress), the plant used by laundresses for scenting linen. The botanical name is Lavandula, from the Latin lavo, to wash. It is a token of affection.
He from his lass him lavender hath sent. Showing his love, and doth requital crave. Him rosemary his sweetheart, whose intent Is that he should her in remembrance have.
“Je veux que tu sois chez moi, commc dans du coton.” —La Muscotte, i. 2.
“The poor gentleman paies so deare for the lavender it is laid up in,
that if it lies long at the broker's house he seems to buy his apparel
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894