Enhancing Human Optical Flow via 3D Spectral Prior
| dc.contributor.author | Mao, Shiwei | en_US |
| dc.contributor.author | Sun, Mingze | en_US |
| dc.contributor.author | Huang, Ruqi | en_US |
| dc.contributor.editor | Chen, Renjie | en_US |
| dc.contributor.editor | Ritschel, Tobias | en_US |
| dc.contributor.editor | Whiting, Emily | en_US |
| dc.date.accessioned | 2024-10-13T18:05:21Z | |
| dc.date.available | 2024-10-13T18:05:21Z | |
| dc.date.issued | 2024 | |
| dc.description.abstract | In this paper, we consider the problem of human optical flow estimation, which is critical in a series of human-centric computer vision tasks. Recent deep learning-based optical flow models have achieved considerable accuracy and generalization by incorporating various kinds of priors. However, the majority either rely on large-scale 2D annotations or rigid priors, overlooking the 3D non-rigid nature of human articulations. To this end, we advocate enhancing human optical flow estimation via 3D spectral prior-aware pretraining, which is based on the well-known functional maps formulation in 3D shape matching. Our pretraining can be performed with synthetic human shapes. More specifically, we first render shapes to images and then leverage the natural inclusion maps from images to shapes to lift 2D optical flow into 3D correspondences, which are further encoded as functional maps. Such lifting operation allows to inject the intrinsic geometric features encoded in the spectral representations into optical flow learning, leading to improvement of the latter, especially in the presence of non-rigid deformations. In practice, we establish a pretraining pipeline tailored for triangular meshes, which is general regarding target optical flow network. It is worth noting that it does not introduce any additional learning parameters but only require some pre-computed eigen decomposition on the meshes. For RAFT and GMA, our pretraining task achieves improvements of 12.8% and 4.9% in AEPE on the SHOF benchmark, respectively. | en_US |
| dc.description.sectionheaders | Human I | |
| dc.description.seriesinformation | Pacific Graphics Conference Papers and Posters | |
| dc.identifier.doi | 10.2312/pg.20241314 | |
| dc.identifier.isbn | 978-3-03868-250-9 | |
| dc.identifier.pages | 10 pages | |
| dc.identifier.uri | https://doi.org/10.2312/pg.20241314 | |
| dc.identifier.uri | https://diglib.eg.org/handle/10.2312/pg20241314 | |
| dc.publisher | The Eurographics Association | en_US |
| dc.rights | Attribution 4.0 International License | |
| dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | |
| dc.subject | CCS Concepts: Computing methodologies → Spectral methods | |
| dc.subject | Computing methodologies → Spectral methods | |
| dc.title | Enhancing Human Optical Flow via 3D Spectral Prior | en_US |