Does anyone know when the word “swagger” first appeared in literature, and who used it first?

No, but you could try the OED.

It was certainly in widespread use in Shakespeare’s time as he uses the word often in his plays.

For example:

CASSIO: I will rather sue to be despised than to deceive so good a commander with so slight, so drunken, and so indiscreet an officer.  Drunk? and speak parrot? and squabble? swagger? swear? and discourse fustian with one’s own shadow?  O thou invisible spirit of wine, if thou hast no name to be known by let’s call thee devil.

Othello, II, iii

First cite in OED of the verb swagger is Shakespeare.

1590 SHAKES. Mids. N. III. i. 79 What hempen home-spuns haue we swaggering here, So neere the Cradle of the Faierie Queene?

[app. f. SWAG v. + -ER5. Cf. the following:
1598 CHAPMAN Achilles Shield To the Vnderstander B2, Swaggering is a new worde amongst them, and rounde headed custome giues it priuiledge with much imitation, being created as it were by a naturall Prosopopeia without etimologie or deriuation.]

And for SWAG:

1. intr. To move unsteadily or heavily from side to side or up and down; to sway without control.  a. of a pendulous part of the body, or of the whole person.

[The existence of this verb is perh. attested for the 15th cent. in swaggyng (s.v. SWAGGING vbl. n. note), and in SWAGE v.2 Its immediate source is uncertain, but it is prob. Scandinavian: cf. Norw. dial. svagga and svaga to sway (see SWAY v. etym.).