The MUSE project seeks to turn the spoken word into computer generated visual narratives
Everybody loves stories. The challenge is to make them compelling, memorable and easy to understand for everyone. Virtual 3D environments have great potential for exceptional immersive storytelling, as exemplified by the latest video games. However, creating complex 3D environments is a technical and time-consuming process that is out of reach for most would-be storytellers.
How could 3D storytelling be made available for everyone with a few clicks of a mouse? Enter MUSE (Machine Understanding for interactive StorytElling), an European FET (Future and Emerging Technologies) project based at the University of Leuven, Belgium that is developing an innovative text-to-virtual-world translation system capable of automatically converting written text in to 3D virtual worlds.
The research team is currently working on advancing technologies that allows natural language – the way we use words that has evolved naturally in humans through use and repetition without conscious planning or premeditation – to be better understood and processed by computers. The algorithms under development are learning to make associations between natural language and 3D visual data so that a real-time ‘visual thesaurus’ can be created. Think of it as a ‘Google Translate for virtual drawings’, where you can input a sentence that describes a scene, such as “a man waiting for a train at the station”, and that scene would then be automatically rendered in a 3D environment.
Making language corpora that can be associated to a database of virtual tools for rendering 3D environments in an automated way is a monumental challenge. However, the project’s methodology is currently being tested and showcased in two scenarios: one for creating immersive children’s stories from text, and another that enables medical patients to interact with educational materials. See the demos at: http://www.muse-project.eu/
The benefits of text visualisation arise from our characteristic of being visual creatures
MUSE’s technology has the potential for countless applications. Visualising language could be a particularly useful to help people understand complex instructions or specific medical treatments. And besides games, people with limited mobility can explore scenarios and places that they never had the chance to encounter in real life.
The technology could power a whole new realm of creativity and business. Like the first graphic interfaces empowered those who were less computer-literate to get to grips with computing, individuals with little graphic ability may one day become creative designers or visual storytellers. As author and technology visualisation expert Tom Wujec put it: “We are visual creatures. When you doodle an image that captures the essence of an idea, you not only remember it, but you also help other people understand and act on it – which is generally the point of meetings in the first place.”
Cover image: via Maxpixel
Research projects in this story
The MUSE project targets the creation of 3D virtual words from written texts. This is based on the development of an algorithm which takes natural language as input, associates the content to 3D visual data stored in a dedicated data base, and generates the described virtual scenario. The outcome of the MUSE project may have applications in a variety of fields, from facilitating the explanation of instructions to the improvement of medical treatments.