40-Issue 7

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/2633093

Browse

Now showing 1 - 20 of 21

Consistent Post-Reconstruction for Progressive Photon Mapping
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Choi, Hajin; Moon, Bochang; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Photon mapping is a light transport algorithm that simulates various rendering effects (e.g., caustics) robustly, and its progressive variants, progressive photon mapping (PPM) methods, can produce a biased but consistent rendering output. PPM estimates radiance using a kernel density estimation whose parameters (bandwidths) are adjusted progressively, and this refinement enables to reduce its estimation bias. Nonetheless, many iterations (and thus a large number of photons) are often required until PPM produces nearly converged estimates. This paper proposes a post-reconstruction that improves the performance of PPM by reducing residual errors in PPM estimates. Our key idea is to take multiple PPM estimates with multi-level correlation structures, and fuse the input images using a weight function trained by supervised learning with maintaining the consistency of PPM. We demonstrate that our technique boosts an existing PPM technique for various rendering scenes.
Deep Learning-Based Unsupervised Human Facial Retargeting
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Kim, Seonghyeon; Jung, Sunjin; Seo, Kwanggyoon; Ribera, Roger Blanco i; Noh, Junyong; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Traditional approaches to retarget existing facial blendshape animations to other characters rely heavily on manually paired data including corresponding anchors, expressions, or semantic parametrizations to preserve the characteristics of the original performance. In this paper, inspired by recent developments in face swapping and reenactment, we propose a novel unsupervised learning method that reformulates the retargeting of 3D facial blendshape-based animations in the image domain. The expressions of a source model is transferred to a target model via the rendered images of the source animation. For this purpose, a reenactment network is trained with the rendered images of various expressions created by the source and target models in a shared latent space. The use of shared latent space enable an automatic cross-mapping obviating the need for manual pairing. Next, a blendshape prediction network is used to extract the blendshape weights from the translated image to complete the retargeting of the animation onto a 3D target model. Our method allows for fully unsupervised retargeting of facial expressions between models of different configurations, and once trained, is suitable for automatic real-time applications.
Diverse Dance Synthesis via Keyframes with Transformer Controllers
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Pan, Junjun; Wang, Siyuan; Bai, Junxuan; Dai, Ju; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Existing keyframe-based motion synthesis mainly focuses on the generation of cyclic actions or short-term motion, such as walking, running, and transitions between close postures. However, these methods will significantly degrade the naturalness and diversity of the synthesized motion when dealing with complex and impromptu movements, e.g., dance performance and martial arts. In addition, current research lacks fine-grained control over the generated motion, which is essential for intelligent human-computer interaction and animation creation. In this paper, we propose a novel keyframe-based motion generation network based on multiple constraints, which can achieve diverse dance synthesis via learned knowledge. Specifically, the algorithm is mainly formulated based on the recurrent neural network (RNN) and the Transformer architecture. The backbone of our network is a hierarchical RNN module composed of two long short-term memory (LSTM) units, in which the first LSTM is utilized to embed the posture information of the historical frames into a latent space, and the second one is employed to predict the human posture for the next frame. Moreover, our framework contains two Transformer-based controllers, which are used to model the constraints of the root trajectory and the velocity factor respectively, so as to better utilize the temporal context of the frames and achieve fine-grained motion control. We verify the proposed approach on a dance dataset containing a wide range of contemporary dance. The results of three quantitative analyses validate the superiority of our algorithm. The video and qualitative experimental results demonstrate that the complex motion sequences generated by our algorithm can achieve diverse and smooth motion transitions between keyframes, even for long-term synthesis.
A Dynamic Mixture Model for Non-equilibrium Multiphase Fluids
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Jiang, Yuntao; Lan, Yingjie; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
We present a dynamic mixture model for simulating multiphase fluids with highly dynamic relative motions. The previous mixture models assume that the multiphase fluids are under a local equilibrium condition such that the drift velocity and the phase transport can be computed analytically. By doing so, it avoids solving multiple sets of Navier-Stokes equations and improves the simulation efficiency and stability. However, due to the local equilibrium assumption, these approaches can only deal with tightly coupled multiphase systems, where the relative speed between phases are assumed stable. In this work we abandon the local equilibrium assumption, and redesign the computation workflow of the mixture model to explicitly track and decouple the velocities of all phases. The phases still share the same pressure, with which we enforce the incompressibility for the mixture. The phase transport is calculated with drift velocities, and we propose a novel correction scheme to handle the transport at fluid boundaries to ensure mass conservation. Compared with previous mixture models, the proposed approach enables the simulation of much more dynamic scenarios with negligible extra overheads. In addition, it allows fluid control techniques to be applied to individual phases to generate locally dynamic and visually interesting effects.
Fluidymation: Stylizing Animations Using Natural Dynamics of Artistic Media
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Platkevic, Adam; Curtis, Cassidy; Sýkora, Daniel; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
We present Fluidymation-a new example-based approach to stylizing animation that employs the natural dynamics of artistic media to convey a prescribed motion. In contrast to previous stylization techniques that transfer the hand-painted appearance of a static style exemplar and then try to enforce temporal coherence, we use moving exemplars that capture the artistic medium's inherent dynamic properties, and transfer both movement and appearance to reproduce natural-looking transitions between individual animation frames. Our approach can synthetically generate stylized sequences that look as if actual paint is diffusing across a canvas in the direction and speed of the target motion.
Geometric Sample Reweighting for Monte Carlo Integration
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Guo, Jerry Jinfeng; Eisemann, Elmar; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Numerical integration is fundamental in multiple Monte Carlo rendering problems. We present a sample reweighting scheme, including underlying theory, and analysis of numerical performance for the integration of an unknown one-dimensional function. Our method is simple to implement and builds upon the insight to link the weights to a function reconstruction process during integration. We provide proof that our solution is unbiased in one-dimensional cases and consistent in multi-dimensional cases. We illustrate its effectiveness in several use cases.
Global Illumination-Aware Stylised Shading
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Doi, Kohei; Morimoto, Yuki; Tsuruno, Reiji; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Our aim is to convert an object's appearance to an arbitrary colour considering the light scattering in the entire scene, which is often called the global illumination. Existing stylisation methods convert the colour of an object with a 1-dimensional texture for 3-dimensional computer graphics to reproduce a typical style used in illustrations and cel animations. However, they cannot express global illumination effects. We propose two individual methods to compute the global illumination and convert the shading to an arbitrary colour. The methods reproduce reflections in other objects with the converted colour. As a result, we can convert the colour of illumination effects that have not yet been reproduced, such as soft shadows and refractions
Interactive Analysis of CNN Robustness
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Sietzen, Stefan; Lechner, Mathias; Borowski, Judy; Hasani, Ramin; Waldner, Manuela; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
While convolutional neural networks (CNNs) have found wide adoption as state-of-the-art models for image-related tasks, their predictions are often highly sensitive to small input perturbations, which the human vision is robust against. This paper presents Perturber, a web-based application that allows users to instantaneously explore how CNN activations and predictions evolve when a 3D input scene is interactively perturbed. Perturber offers a large variety of scene modifications, such as camera controls, lighting and shading effects, background modifications, object morphing, as well as adversarial attacks, to facilitate the discovery of potential vulnerabilities. Fine-tuned model versions can be directly compared for qualitative evaluation of their robustness. Case studies with machine learning experts have shown that Perturber helps users to quickly generate hypotheses about model vulnerabilities and to qualitatively compare model behavior. Using quantitative analyses, we could replicate users' insights with other CNN architectures and input images, yielding new insights about the vulnerability of adversarially trained models.
A Lagrangian Particle-based Formulation for Coupled Simulation of Fracture and Diffusion in Thin Membranes
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Han, Chengguizi; Xue, Tao; Aanjaneya, Mridul; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
We propose a Lagrangian particle-based formulation for simulating deformation, fracture, and diffusion in thin membranelike structures, such as aluminium foil, rubbery films, and seaweed flakes. We integrate our model with diffusion processes and derive a unified framework for simulating deformation-diffusion coupled phenomena, which is applied to provide realistic heterogeneity induced by the diffusion process to fracture patterns. To the best of our knowledge, our work is the first to simulate the complex fracture patterns of single-layered membranes in computer graphics and introduce heterogeneity induced by the diffusion process, which generates more geometrically rich fracture patterns. Our end-to-end 3D simulations show that our deformation-diffusion coupling framework captures detailed fracture growth patterns in thin membranes due to both in-plane and out-of-plane motions, producing realistically wrinkled slit edges, and heterogeneity introduced due to diffusion.
Line Art Colorization Based on Explicit Region Segmentation
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Cao, Ruizhi; Mo, Haoran; Gao, Chengying; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Automatic line art colorization plays an important role in anime and comic industry. While existing methods for line art colorization are able to generate plausible colorized results, they tend to suffer from the color bleeding issue. We introduce an explicit segmentation fusion mechanism to aid colorization frameworks in avoiding color bleeding artifacts. This mechanism is able to provide region segmentation information for the colorization process explicitly so that the colorization model can learn to avoid assigning the same color across regions with different semantics or inconsistent colors inside an individual region. The proposed mechanism is designed in a plug-and-play manner, so it can be applied to a diversity of line art colorization frameworks with various kinds of user guidances. We evaluate this mechanism in tag-based and referencebased line art colorization tasks by incorporating it into the state-of-the-art models. Comparisons with these existing models corroborate the effectiveness of our method which largely alleviates the color bleeding artifacts. The code is available at https://github.com/Ricardo-L-C/ColorizationWithRegion.
Luminance Attentive Networks for HDR Image and Panorama Reconstruction
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Yu, Hanning; Liu, Wentao; Long, Chengjiang; Dong, Bo; Zou, Qin; Xiao, Chunxia; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
It is very challenging to reconstruct a high dynamic range (HDR) from a low dynamic range (LDR) image as an ill-posed problem. This paper proposes a luminance attentive network named LANet for HDR reconstruction from a single LDR image. Our method is based on two fundamental observations: (1) HDR images stored in relative luminance are scale-invariant, which means the HDR images will hold the same information when multiplied by any positive real number. Based on this observation, we propose a novel normalization method called " HDR calibration " for HDR images stored in relative luminance, calibrating HDR images into a similar luminance scale according to the LDR images. (2) The main difference between HDR images and LDR images is in under-/over-exposed areas, especially those highlighted. Following this observation, we propose a luminance attention module with a two-stream structure for LANet to pay more attention to the under-/over-exposed areas. In addition, we propose an extended network called panoLANet for HDR panorama reconstruction from an LDR panorama and build a dualnet structure for panoLANet to solve the distortion problem caused by the equirectangular panorama. Extensive experiments show that our proposed approach LANet can reconstruct visually convincing HDR images and demonstrate its superiority over state-of-the-art approaches in terms of all metrics in inverse tone mapping. The image-based lighting application with our proposed panoLANet also demonstrates that our method can simulate natural scene lighting using only LDR panorama. Our source code is available at https://github.com/LWT3437/LANet.
Manhattan-world Urban Building Reconstruction by Fitting Cubes
(The Eurographics Association and John Wiley & Sons Ltd., 2021) He, Zhenbang; Wang, Yunhai; Cheng, Zhanglin; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
The Manhattan-world building is a kind of dominant scene in urban areas. Many existing methods for reconstructing such scenes are either vulnerable to noisy and incomplete data or suffer from high computational complexity. In this paper, we present a novel approach to quickly reconstruct lightweight Manhattan-world urban building models from images. Our key idea is to reconstruct buildings through the salient feature - corners. Given a set of urban building images, Structure-from- Motion and 3D line reconstruction operations are applied first to recover camera poses, sparse point clouds, and line clouds. Then we use orthogonal planes detected from the line cloud to generate corners, which indicate a part of possible buildings. Starting from the corners, we fit cubes to point clouds by optimizing corner parameters and obtain cube representations of corresponding buildings. Finally, a registration step is performed on cube representations to generate more accurate models. Experiment results show that our approach can handle some nasty cases containing noisy and incomplete data, meanwhile, output lightweight polygonal building models with a low time-consuming.
A Motion-guided Interface for Modeling 3D Multi-functional Furniture
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Chen, Minchan; Lau, Manfred; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
While non-expert 3D design systems are helpful for performing conceptual design, most existing works focused on modeling static objects. However, the 3D modeling interfaces can include more interactions between the user and the models that are dynamic (and can be interacted with). In this paper, we propose a 3D modeling system for the conceptual design of interactable multi-functional furniture. Our contribution is in the design and development of a motion-guided interface. The key idea is that users should create interactable furniture components as if they are interacting with them with their hands. We conducted a preliminary user study to explore users' preferred hand gestures for creating various dynamic furniture components, implemented a 3D modeling system with the preferred gestures as a basis for the motion-guided user interface, and conducted an evaluation user study to demonstrate that our user interface is user-friendly and efficient for novice designers to perform conceptual furniture designs.
A Multi-pass Method for Accelerated Spectral Sampling
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Ruit, Mark van de; Eisemann, Elmar; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Spectral Monte Carlo rendering can simulate advanced light phenomena, such as chromatic dispersion, but typically shows a slow convergence behavior. Properly sampling the spectral domain can be challenging in scenes with many complex spectral distributions. To this end, we propose a multi-pass approach. We build and store coarse screen-space estimates of incident spectral radiance and use these to then importance sample the spectral domain. Hereby, we lower variance and reduce noise with little overhead. Our method handles challenging scenarios with difficult spectral distributions, many different emitters, and participating media. Finally, it can be integrated into existing spectral rendering methods for an additional acceleration.
Neural Sequence Transformation
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Mukherjee, Sabyasachi; Mukherjee, Sayan; Hua, Binh-Son; Umetani, Nobuyuki; Meister, Daniel; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Monte Carlo integration is a technique for numerically estimating a definite integral by stochastically sampling its integrand. These samples can be averaged to make an improved estimate, and the progressive estimates form a sequence that converges to the integral value on the limit. Unfortunately, the sequence of Monte Carlo estimates converges at a rate of O(pn), where n denotes the sample count, effectively slowing down as more samples are drawn. To overcome this, we can apply sequence transformation, which transforms one converging sequence into another with the goal of accelerating the rate of convergence. However, analytically finding such a transformation for Monte Carlo estimates can be challenging, due to both the stochastic nature of the sequence, and the complexity of the integrand. In this paper, we propose to leverage neural networks to learn sequence transformations that improve the convergence of the progressive estimates of Monte Carlo integration. We demonstrate the effectiveness of our method on several canonical 1D integration problems as well as applications in light transport simulation.
Real-time Denoising Using BRDF Pre-integration Factorization
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Zhuang, Tao; Shen, Pengfei; Wang, Beibei; Liu, Ligang; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
Path tracing has been used for real-time renderings, thanks to the powerful GPU device. Unfortunately, path tracing produces noisy rendered results, thus, filtering or denoising is often applied as a post-process to remove the noise. Previous works produce high-quality denoised results, by accumulating the temporal samples. However, they cannot handle the details from bidirectional reflectance distribution function (BRDF) maps (e.g. roughness map). In this paper, we introduce the BRDF preintegration factorization for denoising to better preserve the details from BRDF maps. More specifically, we reformulate the rendering equation into two components: the BRDF pre-integration component and the weighted-lighting component. The BRDF pre-integration component is noise-free, since it does not depend on the lighting. Another key observation is that the weighted-lighting component tends to be smooth and low-frequency, which indicates that it is more suitable for denoising than the final rendered image. Hence, the weighted-lighting component is denoised individually. Our BRDF pre-integration demodulation approach is flexible for many real-time filtering methods. We have implemented it in spatio-temporal varianceguided filtering (SVGF), ReLAX and ReBLUR. Compared to the original methods, our method manages to better preserve the details from BRDF maps, while both the memory and time cost are negligible.
Relighting Humans in the Wild: Monocular Full-Body Human Relighting with Domain Adaptation
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Tajima, Daichi; Kanamori, Yoshihiro; Endo, Yuki; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
The modern supervised approaches for human image relighting rely on training data generated from 3D human models. However, such datasets are often small (e.g., Light Stage data with a small number of individuals) or limited to diffuse materials (e.g., commercial 3D scanned human models). Thus, the human relighting techniques suffer from the poor generalization capability and synthetic-to-real domain gap. In this paper, we propose a two-stage method for single-image human relighting with domain adaptation. In the first stage, we train a neural network for diffuse-only relighting. In the second stage, we train another network for enhancing non-diffuse reflection by learning residuals between real photos and images reconstructed by the diffuse-only network. Thanks to the second stage, we can achieve higher generalization capability against various cloth textures, while reducing the domain gap. Furthermore, to handle input videos, we integrate illumination-aware deep video prior to greatly reduce flickering artifacts even with challenging settings under dynamic illuminations.
Seamless Satellite-image Synthesis
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Zhu, Jialin; Kelly, Tom; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
We introduce Seamless Satellite-image Synthesis (SSS), a novel neural architecture to create scale-and-space continuous satellite textures from cartographic data. While 2D map data is cheap and easily synthesized, accurate satellite imagery is expensive and often unavailable or out of date. Our approach generates seamless textures over arbitrarily large spatial extents which are consistent through scale-space. To overcome tile size limitations in image-to-image translation approaches, SSS learns to remove seams between tiled images in a semantically meaningful manner. Scale-space continuity is achieved by a hierarchy of networks conditioned on style and cartographic data. Our qualitative and quantitative evaluations show that our system improves over the state-of-the-art in several key areas. We show applications to texturing procedurally generation maps and interactive satellite image manipulation.
UprightRL: Upright Orientation Estimation of 3D Shapes via Reinforcement Learning
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Chen, Luanmin; Xu, Juzhan; Wang, Chuan; Huang, Haibin; Huang, Hui; Hu, Ruizhen; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
In this paper, we study the problem of 3D shape upright orientation estimation from the perspective of reinforcement learning, i.e. we teach a machine (agent) to orientate 3D shapes step by step to upright given its current observation. Unlike previous methods, we take this problem as a sequential decision-making process instead of a strong supervised learning problem. To achieve this, we propose UprightRL, a deep network architecture designed for upright orientation estimation. UprightRL mainly consists of two submodules: an Actor module and a Critic module which can be learned with a reinforcement learning manner. Specifically, the Actor module selects an action from the action space to perform a point cloud transformation and obtain the new point cloud for the next environment state, while the Critic module evaluates the strategy and guides the Actor to choose the next stage action. Moreover, we design a reward function that encourages the agent to select action which is conducive to orient model towards upright orientation with a positive reward and negative otherwise. We conducted extensive experiments to demonstrate the effectiveness of the proposed model, and experimental results show that our network outperforms the stateof- the-art. We also apply our method to the robot grasping-and-placing experiment, to reveal the practicability of our method.
Write-An-Animation: High-level Text-based Animation Editing with Character-Scene Interaction
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Zhang, Jia-Qi; Xu, Xiang; Shen, Zhi-Meng; Huang, Ze-Huan; Zhao, Yang; Cao, Yan-Pei; Wan, Pengfei; Wang, Miao; Zhang, Fang-Lue and Eisemann, Elmar and Singh, Karan
3D animation production for storytelling requires essential manual processes of virtual scene composition, character creation, and motion editing, etc. Although professional artists can favorably create 3D animations using software, it remains a complex and challenging task for novice users to handle and learn such tools for content creation. In this paper, we present Write-An- Animation, a 3D animation system that allows novice users to create, edit, preview, and render animations, all through text editing. Based on the input texts describing virtual scenes and human motions in natural languages, our system first parses the texts as semantic scene graphs, then retrieves 3D object models for virtual scene composition and motion clips for character animation. Character motion is synthesized with the combination of generative locomotions using neural state machine as well as template action motions retrieved from the dataset. Moreover, to make the virtual scene layout compatible with character motion, we propose an iterative scene layout and character motion optimization algorithm that jointly considers characterobject collision and interaction. We demonstrate the effectiveness of our system with customized texts and public film scripts. Experimental results indicate that our system can generate satisfactory animations from texts.

Browse

Browsing 40-Issue 7 by Subject "Computing methodologies"

Results Per Page

Sort Options