This term is American in origin and dates to at least 1937. The exact reference is uncertain, but it is most likely to be evocative of the dust bowl and the extreme poverty and unclean conditions in which many had to live during the Depression.
From a Time magazine review of the Boris Karloff movie Night Key in the 26 April 1937 issue:
Nearly blind and dirt-poor, Inventor Dave Mallory (Karloff) devises a burglar alarm worked by electric eyes.
The bit of internet lore about Life in the 1500s claims that dirt poor dates to Shakespearian England where finished floors were rare. This is utterly false.
(Source: Oxford English Dictionary, 2nd Edition)
Copyright 1997-2015, by David Wilton