This corner is written by Hiroki Yamashita, who presides over the web media "Seamless" that introduces the latest research in technology. Mr. Yamashita picks up highly novel scientific papers and explains them.
``Sketch Dance Stage Online'': Three-dimensional CG projection of hand-drawn characters to real space and interaction, developed by the research team of Shinji Mizuno's laboratory at Aichi Institute of Technology, is a collection of hand-drawn sketches by online users. It is a system that displays a 2D character as a 3D CG character in a distant real space with a projector and allows interaction with local people.
Since the CG characters move according to the movements of the local people, it is possible for the local people to be surrounded by the CG characters and control them with gestures to dance together. Online users can remotely observe their dancing with their drawings.Following the movement of the person dancing in the middle, the surrounding CG characters also dance in the same wayThe CG characters all jump together as the person jumps >
The system consists of two PCs, a projector, a camera, and a Kinect sensor. One PC converts hand-drawn characters into CG characters for projection, and the other PC is used to deliver live video from a camera placed near the projector. The projector projects a character onto the floor, and the Kinect captures the positions and movements of people and objects surrounded by the character.Overview of the system
Online users access the web page on a PC or tablet and use pens of various colors and thicknesses to draw characters on the canvas. The drawn character image is divided into parts (divided into 10 parts such as head, body, limbs, etc.) and rigging are automatically set (manual correction is also possible), and movements are randomly assigned from the data group obtained in advance. .Draw a character on the web page and set the rigging
Project the created character onto the floor of a remote real space with a projector. Since normal projection is only a two-dimensional display on the floor, the viewpoint of the CG character is set based on the position and optical axis of the projector, and projection is performed based on the viewing angle of the projector. As a result, the projected image has the effect of anamorphosis, and the CG character can be seen three-dimensionally from the position of the projector.
In addition, by installing a camera near the projector, online users can observe live images of their drawings as CG characters with a three-dimensional effect from a distance. It is also possible to remotely control CG characters from a PC by operating buttons.
The projected CG character moves around in a circle around the center, so when a person stands in the middle of it, the CG characters appear to be dancing around them. The movement of the person standing in the center captured by Kinect is reflected in the CG character, and gesture operations (up and down of limbs, twisting of the waist, jumping, etc.) are possible, so it is possible to provide an experience of dancing with the CG character. .
At that time, we have to consider the occlusion between the two that occurs when the CG character passes behind the person. Therefore, by generating a 3D model of a moving person with the same color as the background, rearranging it in the CG space together with the CG character, and generating an image for projection, only the area of the character that overlaps with the person is erased, making it appear as if the person is behind the person. It is expressed as if it is going around.Occlusion is taken into account so that the CG character can wrap around people.
Source and Image Credits: Sora Ahn and Shinji Mizuno. 2021. “Sketch Dance Stage Online” : Three-dimensional CG projection of hand-drawn characters to real space and interaction. In SIGGRAPH Asia 2021 Posters (SA '21 Posters). Association for Computing Machinery, New York, NY, USA, Article 23, 1-2. DOI: https://doi.org/10.1145/3476124.3488640
Copyright © ITmedia, Inc. All Rights Reserved.
Log in to the mail magazine "ITmedia NEWS Anchor Desk Magazine" with your registered ITmedia ID to read more. Or register for "ITmedia NEWS Anchor Desk Magazine" to read more.