Audio-Driven Speech Animation with Text-Guided Expression
| dc.contributor.author | Jung, Sunjin | en_US |
| dc.contributor.author | Chun, Sewhan | en_US |
| dc.contributor.author | Noh, Junyong | en_US |
| dc.contributor.editor | Chen, Renjie | en_US |
| dc.contributor.editor | Ritschel, Tobias | en_US |
| dc.contributor.editor | Whiting, Emily | en_US |
| dc.date.accessioned | 2024-10-13T18:04:06Z | |
| dc.date.available | 2024-10-13T18:04:06Z | |
| dc.date.issued | 2024 | |
| dc.description.abstract | We introduce a novel method for generating expressive speech animations of a 3D face, driven by both audio and text descriptions. Many previous approaches focused on generating facial expressions using pre-defined emotion categories. In contrast, our method is capable of generating facial expressions from text descriptions unseen during training, without limitations to specific emotion classes. Our system employs a two-stage approach. In the first stage, an auto-encoder is trained to disentangle content and expression features from facial animations. In the second stage, two transformer-based networks predict the content and expression features from audio and text inputs, respectively. These features are then passed to the decoder of the pre-trained auto-encoder, yielding the final expressive speech animation. By accommodating diverse forms of natural language, such as emotion words or detailed facial expression descriptions, our method offers an intuitive and versatile way to generate expressive speech animations. Extensive quantitative and qualitative evaluations, including a user study, demonstrate that our method can produce natural expressive speech animations that correspond to the input audio and text descriptions. | en_US |
| dc.description.sectionheaders | Human and Character Animation | |
| dc.description.seriesinformation | Pacific Graphics Conference Papers and Posters | |
| dc.identifier.doi | 10.2312/pg.20241290 | |
| dc.identifier.isbn | 978-3-03868-250-9 | |
| dc.identifier.pages | 12 pages | |
| dc.identifier.uri | https://doi.org/10.2312/pg.20241290 | |
| dc.identifier.uri | https://diglib.eg.org/handle/10.2312/pg20241290 | |
| dc.publisher | The Eurographics Association | en_US |
| dc.rights | Attribution 4.0 International License | |
| dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | |
| dc.subject | CCS Concepts: Computing methodologies → Animation; Neural networks | |
| dc.subject | Computing methodologies → Animation | |
| dc.subject | Neural networks | |
| dc.title | Audio-Driven Speech Animation with Text-Guided Expression | en_US |