In addition to being a church official, a penitentiary was “a place of penitential discipline or punishment for ecclesiastical offences.” The OED gives this a separate entry from that for the official, and the first citation for this sense is given as “a[nte]1500(▸?1421)” where “▸ indicates date of composition for this text (as opposed to date of documentary evidence).”
A later sense was “a refuge for ‘fallen women’; a home for reformed prostitutes, unmarried mothers, etc. Now hist.” Apparently that sense was used in the UK, since two of the citations (including the first) refer to the London Female Penitentiary. An 1891 newspaper citation reads: “The change of title..from the ‘London Female Penitentiary Society’ to the ‘London Female Guardian Society’ has been universally approved of… When the society was founded eighty-four years ago the term ‘Penitentiary’ was well understood to mean a voluntary asylum for the reception of those resolving on amendment of life.”
The American usage of the word as a place of incarceration of either sex for a variety of crimes seems to have arisen at about the same time as the previous sense (early 19th cent.), but the OED notes “In early use applied. spec. to facilities having some reformatory or correctional purpose, but now usually taken to be synonymous with ‘prison’.”
“House of correction”, btw, goes back to the 16th century.