Wordorigins.org

The phrase is almost certainly an allusion to the fact that the first joint an adult thumb measures roughly one inch, literally a rule (or ruler) of thumb. Since human dimensions vary, any measurement so taken would be only a rough approximation and not to be trusted where precision was required. Rule of thumb dates to 1692, when it appears in William Hope’s The Compleat Fencing-Master:

What he doth, he doth by rule of Thumb, and not by Art.