He is quite down in the mouth. Out of spirits; disheartened. When persons are very sad and low-spirited, the corners of the mouth are drawn down. “Down in the jib” is a nautical phrase of the same meaning.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Down from Infoplease: