Browsing by Author "Averbuch-Elor, Hadar"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collections
    (The Eurographics Association and John Wiley & Sons Ltd., 2024) Dudai, Chen; Alper, Morris; Bezalel, Hana; Hanocka, Rana; Lang, Itai; Averbuch-Elor, Hadar; Bermano, Amit H.; Kalogerakis, Evangelos
    Internet image collections containing photos captured by crowds of photographers show promise for enabling digital exploration of large-scale tourist landmarks. However, prior works focus primarily on geometric reconstruction and visualization, neglecting the key role of language in providing a semantic interface for navigation and fine-grained understanding. In more constrained 3D domains, recent methods have leveraged modern vision-and-language models as a strong prior of 2D visual semantics. While these models display an excellent understanding of broad visual semantics, they struggle with unconstrained photo collections depicting such tourist landmarks, as they lack expert knowledge of the architectural domain and fail to exploit the geometric consistency of images capturing multiple views of such scenes. In this work, we present a localization system that connects neural representations of scenes depicting large-scale landmarks with text describing a semantic region within the scene, by harnessing the power of SOTA vision-and-language models with adaptations for understanding landmark scene semantics. To bolster such models with fine-grained knowledge, we leverage large-scale Internet data containing images of similar landmarks along with weakly-related textual information. Our approach is built upon the premise that images physically grounded in space can provide a powerful supervision signal for localizing new concepts, whose semantics may be unlocked from Internet textual metadata with large language models. We use correspondences between views of scenes to bootstrap spatial understanding of these semantics, providing guidance for 3D-compatible segmentation that ultimately lifts to a volumetric scene representation. To evaluate our method, we present a new benchmark dataset containing large-scale scenes with groundtruth segmentations for multiple semantic concepts. Our results show that HaLo-NeRF can accurately localize a variety of semantic concepts related to architectural landmarks, surpassing the results of other 3D models as well as strong 2D segmentation baselines. Our code and data are publicly available at https://tau-vailab.github.io/HaLo-NeRF/.
  • Thumbnail Image
    Item
    What's in a Decade? Transforming Faces Through Time
    (The Eurographics Association and John Wiley & Sons Ltd., 2023) Chen, Eric Ming; Sun, Jin; Khandelwal, Apoorv; Lischinski, Dani; Snavely, Noah; Averbuch-Elor, Hadar; Myszkowski, Karol; Niessner, Matthias
    How can one visually characterize photographs of people over time? In this work, we describe the Faces Through Time dataset, which contains over a thousand portrait images per decade from the 1880s to the present day. Using our new dataset, we devise a framework for resynthesizing portrait images across time, imagining how a portrait taken during a particular decade might have looked like had it been taken in other decades. Our framework optimizes a family of per-decade generators that reveal subtle changes that differentiate decades-such as different hairstyles or makeup-while maintaining the identity of the input portrait. Experiments show that our method can more effectively resynthesizing portraits across time compared to state-of-theart image-to-image translation methods, as well as attribute-based and language-guided portrait editing models. Our code and data will be available at facesthroughtime.github.io.