A bit of an antedate in English:
I suppose that in that work he [Tolstoy] will
not fail to clearly bring to light a final
cause of that immorality which we
can look upon as inherent in the very
principle of art. I mean a condition
which seems to be imposed on the ar-
tist, and which consists, in order to as-
sure his originality, not precisely in his
cutting himself off from the society of
other men, and shutting himself in his
“ivory tower,” but in his distinguish-
ing himself from the crowd.
--"Art and Morality” by Ferdinand Brunetiere
in The Living Age, issue 2844, Jan 7, 1899
accessed at MOA-Cornell.
Also used, again in quotes, in an article by the same writer, titled “French Poetry in the Nineteenth Century”, in the Feb 3, 1900 issue of the same magazine.
And then there’s this from an article called “Contemporary French Literature. II.” by Elie Reclus, published in Putnam’s monthly magazine of American literature, science and art. / Volume 14, Issue 24, Dec. 1869
Afterward Romanticism, which has
reposed in a cradle, watched over by
the Genie du Christianisme, which has
grown up in the sanctuary of Notre
Dame, which has sounded the horn
of Hernani, from the “Ivory Tower”
of Alfred de Vigny, Romanticism, dis-
abused, disgusted, but not disintoxi-
cated, slinks away like Rolla, like
Paulus, to breathe its last on the dirty
pillows of the Bonapartist Empire.
(They don’t write ‘em like that anymore!)
FWIW, Wikipedia’s article on Vigny says “After the death of his mother in 1838 he inherited the property of Maine-Giraud, near Angoulême, where it was said that he had withdrawn to his ‘ivory tower’ (an expression Sainte-Beuve coined with reference to Vigny). “