VIRTUAL CLASSROOM. Rensselaer Polytechnic Institute (RPI)
graduate student Xiangyang Mou practices tai chi with an avatar
in a campus studio at RPI in Troy, New York, in this August 22,
2018 file image taken from video. The "Mandarin Project" is a
joint venture of RPI and IBM. Cognitive and Immersive Systems
Laboratory researchers are developing a sort of smart room that
can understand students’ words, answer their questions, and
perceive their gestures. (AP Photo/Michael Hill)
From The Asian Reporter, V28, #18 (September 17,
2018), page 8
Virtual learning: Using AI, immersion to teach
By Michael Hill
The Associated Press
TROY, N.Y. — To learn Chinese in this room, talk to the
floating panda head.
The Mandarin-speaking avatar zips around a 360-degree
restaurant scene in an artificial intelligence-driven
instruction program that looks like a giant video game.
Rensselaer Polytechnic Institute (RPI) students testing the
technology move inside the 12-foot-high, wraparound projection
to order virtual bean curd from the panda waiter, chat with
Beijing market sellers, and practice tai chi by mirroring moves
of a watchful mentor.
"Definitely less anxiety than messing it up with a real human
being," says Rahul Divekar, a computer science graduate student
working on the project. "So compared to that anxiety, this is a
lot more easy."
The "Mandarin Project" is a joint venture of RPI and IBM.
Cognitive and Immersive Systems Laboratory researchers are
developing a sort of smart room that can understand students’
words, answer their questions, and perceive their gestures.
Lessons are presented as games or tasks, like ordering a meal
Divekar orders Peking duck — Beijng kaoya — and the
panda fetches the virtual dish. Divekar says the food was good —
Cai hen hao chi — but he can’t pay the bill. No problem,
the panda replies — ni keyi xi pan zi — you can wash the
Other scenes include an outdoor market and a garden, each a
high-tech twist on cultural immersion.
"Our plan is to complete several scenes of real life in
China, to let the student be able to have a virtual trip over
there," said Hui Su, director of the lab at RPI.
Tests on the room with students studying Mandarin will
continue this school year as they work on additional scenes,
including an airport. A six-week course is being readied for the
Artificial intelligence, or AI, is ubiquitous in everything
from call-center chat-bots to home assistants. Even some
language instruction products on the market feature AI or
The Mandarin Project is notable for its scale and
sophistication. Computers simultaneously interpret speech and
gesture to keep a dialogue going. When a student points to a
picture and asks "What’s that?" computers can come up with an
And feedback is immediate. When Divekar orders dou fu
— or tofu — a voice responds "here’s how close you got" and
illustrates it with a graph of his intonation. Another voice
gives the precise pronunciation.
Still, language teachers need not fear for their jobs just
yet. Developers of the Mandarin Project say it isn’t
sophisticated enough right now to completely replace classroom
RPI president Shirley Ann Jackson foresees the same type of
technology being applied to other spaces, such as corporate
boardrooms. When the executives discuss a potential acquisition,
the room will follow the group discussion and produce relevant
information seamlessly into the debate.
"We’re not at the end of the line," Jackson says, "but closer
to the beginning."