The OED2 has a good write-up under -ster. To summarize:
In Old English the masculine and feminine agentive suffixes were -ere and -estre. Usage in OE was strictly along sex lines, with the only crossing of the lines with the feminine applied to men being in translations from Latin masculine terms for professions that were dominated by women in England (e.g., the Latin pistor became bæcestre [bakester, baxter]).
In the Middle English period, the usage split along dialectal lines. But in Scotland and the north, words with the -ster ending began to commonly be used in reference to men. And in Scotland, most of the -ster words even switched grammatical gender, becoming masculine. In the south, the sex distinctions remained, and new -ster words with the sex distinction were coined. But more and more the French -eresse came to be used when a feminine designation was required, and starting in the 16th century the -ster suffix began to acquire the masculine in the south of England as well.
Teamster and gangster are relatively new coinages (18th and late-19th centuries, respectively) and have always been masculine. Spinster is just about the only one of the -ster words that remains exclusively feminine in modern English (and no longer refers to the occupation).