I think that nutter is derivative in much the same way that “looney” is from lunatic (which, by the way, has a rather interesting pedigree all its own).
I think I’ll put my money on it coming from the sense of ‘off his head’, as the use of ‘nut’ for head appears to predate the off his nut usage by about a decade in the mid 19thC.