Virtual creatures called "Kobito" interact with real objects and real people.

One common way to create imaginary, virtual creatures is to overlay computer graphics images on real scenes. But this method is not sufficient, because it allows people to only see the imaginary creatures. In Kobito: Virtual Brownies, imaginary creatures interact with the real world. They move real objects, and people interact with them through the real objects. The real objects function as a kind of "haptic interface." This technology can be used in the fields of design, amusement, and healthcare because it conveys haptic information in addition to the visual information that is delivered in current artificial life systems. 

The goal of this project is to make people feel Kobitos as "creatures that exist in our real world." To make Kobitos exist virtually and naturally, it is necessary to avoid using markers, head-mounted displays, or wearable sensors, which are very common in virtual reality systems but unfamiliar in daily life. And users must be able to interact with Kobitos in various ways. 

Each component of this system is based on existing technology, but the application and the combination of components are innovative. The interaction between the Kobitos and a tea caddy is realized with physical simulation. Usually, physical simulation deals only with imaginary objects, but in this case, it includes a real object (the caddy). This is realized by synchronizing the movement between the real caddy (in the real world) and the imaginary caddy (in the physical-simulation world). Attendees can see the interaction through the Kobito window, which displays a combined image of the Kobito CG and the image of the real scene. No previous virtual reality system has enabled imaginary creatures to move real objects. 

Kobitos can be good playmates, because they are invisible sometimes and visible at other times. When they are invisible, and they move something in the real world, they generate a sense of wonder. Interaction with such a creature could be a new type of entertainment. Beyond entertainment, this technology could be very useful in daily life. Kobitos are intermediate between physical agents and imaginary agents, so they could act as agents between human beings and machines. Imaginary agents that live in computers can do many things automatically, but they exist only in digital machines. You have to turn on the computer and view the display. On the other hand, robots (physical agents) always exist. So you can ask them to do something whenever you want. But, they exist even when you don't want, the same as a real partner. Kobitos are a new type of agent. Usually, Kobitos don't bother you. Whenever you want to see them, you can see them through the Kobito window. The most important thing is that Kobitos work even when they are invisible.