OED references the French critic Sainte-Beuve (presumably the originator?) in an 1837 work, and then gives a 1911 cite for an appearance of the phrase in English. That’s a long gap. Surely there must have been an English translation of the Sainte-Beuve work in the 19th century (’one of the major figures of French literary history’ according to Wikipedia)?

[1837 SAINTE-BEUVE Pensées d’Août, à M. Villemain 152 Et Vigny, plus secret, Comme en sa tour d’ivoire, avant midi rentrait.] 1911 BRERETON & ROTHWELL tr. Bergson’s Laughter iii. 135 Each member [of society] must be ever attentive to his social surroundings..he must avoid shutting himself up in his own peculiar character as a philosopher in his ivory tower.

There’s a Big List entry here.

The 1911 date in the OED is of a figurative use. I doubt the OED editors would find a simple translation worthy of a citation--after all, it’s not really an “English” usage. The term’s popularity in English comes from the title of a 1916 Henry James work (which is also cited in the OED). That’s what cemented the term in English usage.

It seems in part to be an echo of the biblical phrase turris eburnea (Hebrew migdal ha-shen), used in the Song of Songs 7:5, to describe a woman’s neck: “Thy neck is as a tower of ivory.” An excerpt from the poem, “A M. Villemain”, in which Saint-Beuve used the phrase in its modern meaning can be found here in the original French.

Il la maintient encore ; et Vigny plus secret,
Comme en sa tour d’ivoire, avant midi, rentrait.

I tried to investigate this using Google Books’ Advanced search, setting the date pre-1850, but gave up in disgust after wading through reams of 20th-century journal hits disguised by 19th-century first-issue dates.  When oh when will they fix Google Books?

A bit of an antedate in English:

I suppose that in that work he [Tolstoy] will
not fail to clearly bring to light a final
cause of that immorality which we
can look upon as inherent in the very
principle of art. I mean a condition
which seems to be imposed on the ar-
tist, and which consists, in order to as-
sure his originality, not precisely in his
cutting himself off from the society of
other men, and shutting himself in his
“ivory tower,” but in his distinguish-
ing himself from the crowd.

--"Art and Morality” by Ferdinand Brunetiere
in The Living Age, issue 2844, Jan 7, 1899
accessed at MOA-Cornell.

Also used, again in quotes, in an article by the same writer, titled “French Poetry in the Nineteenth Century”, in the Feb 3, 1900 issue of the same magazine.

And then there’s this from an article called “Contemporary French Literature. II.” by Elie Reclus, published in Putnam’s monthly magazine of American literature, science and art. / Volume 14, Issue 24, Dec. 1869

Afterward Romanticism, which has
reposed in a cradle, watched over by
the Genie du Christianisme, which has
grown up in the sanctuary of Notre
Dame, which has sounded the horn
of Hernani, from the “Ivory Tower”
of Alfred de Vigny, Romanticism, dis-
abused, disgusted, but not disintoxi-
cated, slinks away like Rolla, like
Paulus, to breathe its last on the dirty
pillows of the Bonapartist Empire.

(They don’t write ‘em like that anymore!)

FWIW, Wikipedia’s article on Vigny says “After the death of his mother in 1838 he inherited the property of Maine-Giraud, near Angoulême, where it was said that he had withdrawn to his ‘ivory tower’ (an expression Sainte-Beuve coined with reference to Vigny). “

_The Poetical Works of Matthew Prior_ (1779): v. 2, p. 32 (from Google Books):

[I guess Solomon is building a palace]

//

....
The workmen here obey’d the master’s call,
To gild the turret, and to paint the wall;
To mark the pavement there with various stone;
And on the jasper steps to rear the throne:
The spreading cedar that an age had stood,
Supreme of trees, and mistress of the wood,
Cut down and carv’d, my shining roof adorns,
And Lebanon his ruin’d honour mourns.
A thousand artists shew their cunning power,
To raise the wonders of the ivory tower.
....

//

I don’t know the date of the poem but Prior died in 1721 according to Wiki.

In the 18th century one can find mention of a legendary “ivory palace” of Menelaus. In the Bible, I think, there is mention of Ahab’s “ivory palace”. It may be assumed that “ivory palace” refers to a palace ornamented with (not constructed entirely of!) ivory; likewise for “ivory tower”, I suppose.

A year or two ago I was in DC and walking across the campus of George Washington University.  I about fell over laughing when I discovered that they actually have a building called Ivory Tower.

That’s pretty good, certainly named with tongue firmly in cheek. It’s a new building. It wasn’t there when I went to school there fifteen (aak!) years ago.

I’m dismayed, however, that the description of the building lists Dunkin’ Donuts as one of the “dining options.”