A bad penny is a false or snide coin and there a numerous cites to them being common, and in the nineteenth century ‘causing misery’ presumably because people accepted them in good faith and then couldn’t spend them - a big problem if you were poor to start with. The earliest cite for someone being compared to one is from the 14th century ‘lettered men are likened to a bad penny’ The earliest cite in OED to ‘coming back like a bad penny’ is 1937, they don’t suggest why but I suppose they were an unwanted nuisance that were hard to avoid.