The OED gives ‘Gloomy, serious’ as a figurative sense of brown, but only cites it in the context of brown study. It’s very old:
c1555 Lacke of company wyll son lead a man into a brown studdy.
When A L Rowse proposed the identification of Emilia Lanier as Shakespeare’s Dark Lady, he cited a remark by a contemporary that Lanier was ‘very browne in youth’ as evidence that she was dark enough to have been so described. I remember one of the reviewer of his book pointing out that brown in 16th-century English was also used in this figurative sense, so that the remark was just as likely to mean that she had been a thoroughly mumpish adolescent.