Wordorigins.org

Strangely enough, kangaroo court, meaning a criminal proceeding that is conducted for show and where the defendant is inevitably going to be found guilty, is not of Australian origin. The earliest use of the term was recorded in Texas, of all places. From Philip Paxton’s (S.A. Hammett’s) 1853 A Stray Yankee in Texas:

By a unanimous vote, Judge G— was elected to the bench and the “Mestang” or “Kangaroo Court” regularly organized.

The reasoning behind the choice of kangaroo is not known. It may be a reference to bouncing from the court to the gallows. One 1862 source suggests that it is because such courts defy the law, as a kangaroo defies the laws of nature. Or it could be a reference to trying jumpers of mining claims in such courts.

(Source: Oxford English Dictionary, 2nd Edition; Historical Dictionary of American Slang; Macquarie Dictionary)