The ill-advised pronouncement of an idiot I know to the effect that this phrase was first used by a female British novelist in the 1920s prompted me to look into it. Obviously there are many earlier examples of the phrase where it merely refers to soapdodgers; the earliest examples I can find in the extended sense are from Painted Veils, a novel by the American James Huneker, in 1920: Yes, he quoted Stendhal to her, and after he had related a certain anecdote of Stendhal’s life in Milan, she put fingers in her ears; “Don’t tell me another thing about the dirty old man, or I’ll hate you. and in what appears to be a short story in the literary magazine The Smart Set, also American, from 1923: When he was with Jane he knew his Uncle Rufus to be a dirty old man and often wished to break with him. . . . But somehow he didn’t. . . .
I haven’t looked all that hard, though, and I have the feeling there are better ways to do it than via the rather frustrating Google Books search.