Yiddish was not always a Jewish language. It was originally a Germanic tongue spoken by the people of Franconia, the kingdom of the Franks. Around the 9th or 10th centuries, Jews from Franconia began to emigrate from Franconia to Eastern Europe (what is today Poland, Ukraine, Russia, &c.), whose populations had not yet fully converted to Christianity, and where Jews were relatively free from religious harassment. The language they spoke, and took with them when they moved, was Franconian. In Eastern Europe they were the only people who spoke it, and they (and the locals) called it “the Jewish language” (yiddishe spruche) or simply “Jewish” (yiddish).
Yiddish is a regional phenomenon. The only people speaking it, for a long time, were Jews of Eastern Europe*. The people who speak Yiddish today are the descendants of these, most of whom now live in parts other than Eastern Europe. Yiddish was, and remains, a foreign language unknown to, and not spoken by, the Jewish populations indigenous to Africa, the Near and Far East, Italy, Greece, pre-Columbian Spain, etc.
*(meanwhile, back at the ranch in Franconia, other languages developed)