Blog

Developed by AIT, a system that lets you dance along with handwritten illustrations Illustrations respond to people's movements: Innovative Tech

Innovative Tech:

This corner is written by Hiroki Yamashita, who presides over the web media "Seamless" that introduces the latest research in technology. Mr. Yamashita picks up highly novel scientific papers and explains them.

``Sketch Dance Stage Online'': Three-dimensional CG projection of hand-drawn characters to real space and interaction, developed by the research team of Shinji Mizuno's laboratory at Aichi Institute of Technology, is a collection of hand-drawn sketches by online users. It is a system that displays a 2D character as a 3D CG character in a distant real space with a projector and allows interaction with local people.

 Since the CG characters move according to the movements of the local people, it is possible for the local people to be surrounded by the CG characters and control them with gestures to dance together. Online users can remotely observe their dancing with their drawings.

Following the movement of the person dancing in the middle, the surrounding CG characters also dance in the same wayThe CG characters all jump together as the person jumps >

 The system consists of two PCs, a projector, a camera, and a Kinect sensor. One PC converts hand-drawn characters into CG characters for projection, and the other PC is used to deliver live video from a camera placed near the projector. The projector projects a character onto the floor, and the Kinect captures the positions and movements of people and objects surrounded by the character.

Overview of the system

 Online users access the web page on a PC or tablet and use pens of various colors and thicknesses to draw characters on the canvas. The drawn character image is divided into parts (divided into 10 parts such as head, body, limbs, etc.) and rigging are automatically set (manual correction is also possible), and movements are randomly assigned from the data group obtained in advance. .

Draw a character on the web page and set the rigging

 Project the created character onto the floor of a remote real space with a projector. Since normal projection is only a two-dimensional display on the floor, the viewpoint of the CG character is set based on the position and optical axis of the projector, and projection is performed based on the viewing angle of the projector. As a result, the projected image has the effect of anamorphosis, and the CG character can be seen three-dimensionally from the position of the projector.

 In addition, by installing a camera near the projector, online users can observe live images of their drawings as CG characters with a three-dimensional effect from a distance. It is also possible to remotely control CG characters from a PC by operating buttons.

 The projected CG character moves around in a circle around the center, so when a person stands in the middle of it, the CG characters appear to be dancing around them. The movement of the person standing in the center captured by Kinect is reflected in the CG character, and gesture operations (up and down of limbs, twisting of the waist, jumping, etc.) are possible, so it is possible to provide an experience of dancing with the CG character. .

Dancing system with hand-drawn illustrations Developed by AIT, illustrations respond to people's movements: Innovative Tech

 At that time, we have to consider the occlusion between the two that occurs when the CG character passes behind the person. Therefore, by generating a 3D model of a moving person with the same color as the background, rearranging it in the CG space together with the CG character, and generating an image for projection, only the area of ​​the character that overlaps with the person is erased, making it appear as if the person is behind the person. It is expressed as if it is going around.

Occlusion is taken into account so that the CG character can wrap around people.

Source and Image Credits: Sora Ahn and Shinji Mizuno. 2021. “Sketch Dance Stage Online” : Three-dimensional CG projection of hand-drawn characters to real space and interaction. In SIGGRAPH Asia 2021 Posters (SA '21 Posters). Association for Computing Machinery, New York, NY, USA, Article 23, 1-2. DOI: https://doi.org/10.1145/3476124.3488640

Related Articles

Related Links

Copyright © ITmedia, Inc. All Rights Reserved.

Log in to the mail magazine "ITmedia NEWS Anchor Desk Magazine" with your registered ITmedia ID to read more. Or register for "ITmedia NEWS Anchor Desk Magazine" to read more.

Hot Articles

How to Save Websites as PDF on iPhone or PC | Business Insider Japan

How to Save Websites as PDF on iPhone or PC | Business Insider Japan

Sign up for a free e-mail newsletter We'll send you a Business Insider Japan e-mail newsletter at 17:00 on weekdays. Check the terms of use You can save the website as a PDF from various web browsers including Safari on iPhone. Photo: Takuma Imamura Web page suddenly ...

READ MORE READ MORE
 It's okay if you forget to record the news!How to see the famous scenes of the Olympics later on your smartphone

It's okay if you forget to record the news!How to see the famous scenes of the Olympics later on your smartphone

Explaining how to use the archive distribution The Tokyo Olympics attracts attention not only for players' play but also for unique commentary. Even if you miss it even though it became a hot topic, or if you did not record it, you can do it at your favorite timing later ...

READ MORE READ MORE
Yahoo! News Digitalizing the traditional "small pattern dyeing" pattern Crisis of disappearance, challenge of long-established president

Yahoo! News Digitalizing the traditional "small pattern dyeing" pattern Crisis of disappearance, challenge of long-established president

In the file in front of Mr. Atsushi Tomita, a well-preserved paper pattern is included so that it is not exposed to the air as much as possible. To prepare for digitization and prevent deterioration = Taken by Hiroyuki Kondo on the morning of December 10, 2021 at Tomita Dyeing Crafts in Shinjuku-ku, Tokyo ...

READ MORE READ MORE

Related Articles