To be in my black books. In bad odour; in disgrace; out of favour. The black books were those compiled in the reign of Henry VIII. to set forth the scandalous proceedings of the English monasteries, and were so called from the colour of their binding. We have similarly the Blue Book, the Red Book, and so on.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894