Cybernetics And Systems Analysis logo
Editorial Board Announcements Abstracts Authors Archive
Cybernetics And Systems Analysis
International Theoretical Science Journal
Volume 59 >>> № 6 NOVEMBER — DECEMBER 2023
-->

UDC 004.8, 004.62
V. Savin1, O. Kolodiazhna2


1 National Technical University “Igor Sikorsky Kyiv Polytechnic Institute,” Kyiv, Ukraine

 vladimyr.savin@gmail.com

2 National Technical University “Igor Sikorsky Kyiv Polytechnic Institute,” Kyiv, Ukraine

 kolodyazhna.lena@gmail.com

ADAPTING NEURAL RADIANCE FIELDS (NeRF) FOR 3D SCENE
RECONSTRUCTION UNDER DYNAMIC ILLUMINATION CONDITIONS

Abstract. We address the problem of novel view synthesis using Neural Radiance Fields (NeRF) for scenes with dynamic illumination. NeRF training utilizes photometric consistency loss that is pixel-wise consistency between a set of scene images and intensity values rendered by NeRF. For reflective surfaces, image intensity depends on the viewing angle, and this effect is taken into account by using ray direction as the NeRF input. For scenes with dynamic illumination, image intensity depends not only on the position and viewing direction but also on time. We show that this factor affects NeRF training with standard photometric loss function and decreases the quality of both image and depth rendering. To cope with this problem, we propose to add time as additional NeRF input. Experiments on ScanNet dataset demonstrate that NeRF with modified input outperforms the original model version and renders more consistent 3D structures. The results of this study could be used to improve the quality of training data augmentation for depth prediction models (e.g., depth-from-stereo models) for scenes with non-static illumination.

Keywords: computer vision, neural radiance fields, dynamic illumination, view synthesis, 3D scene reconstruction.


full text

REFERENCES

  1. Tardon L., Barbancho I., Alberola-Lopez C. Markov random fields in the context of stereo vision. In: Advances in Theory and Applications of Stereo Vision. Bhatti A. (Ed.). IntechOpen, 2011. 366 p. https://doi.org/10.5772/12953 .

  2. Zhu S., Yan L. Local stereo matching algorithm with efficient matching cost and adaptive guided image filter. The Visual Computer. 2017. Vol. 33, Iss. 9. P. 1087–1102. https://doi.org/10.1007/s00371-016-1264-6 .

  3. Bleyer M., Rhemann C., Rother C. PatchMatch stereo - stereo matching with slanted support windows. Proc. 2011 British Machine Vision Conference (BMVC 2011) (29 August – 2 September 2011, Dundee, UK). Dundee, 2011. 11 p. URL: https://api.semanticscholar.org/CorpusID:1798946.

  4. Laga H., Jospin L.V., Boussaid F., Bennamoun M. A survey on deep learning techniques for stereo-based depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020. Vol. 44, N 4. P. 1738–1764. http://dx.doi.org/10.1109/TPAMI.2020.3032602.

  5. Watson J., Aodha O.M., Turmukhambetov D., Brostow G.J., Firman M. Learning stereo from single images. Proc. 16th European Conference on Computer Vision (ECCV 2020) (23–28 August 2020, Glasgow, UK). Glasgow, 2020. Lecture Notes in Computer Science. Vol. 12346. P. 722–740. https://doi.org/10.1007/978-3-030-58452-8_42.

  6. Mildenhall B., Srinivasan P.P., Tancik M., Barron J.T., Ramamoorthi R., Ng R. NERF: Representing scenes as neural radiance fields for view synthesis. Proc. 16th European Conference on Computer Vision (ECCV 2020) (23–28 August 2020, virtual). Lecture Notes in Computer Science. 2020. Vol. 12346. P. 405–421. https://doi.org/10.1007/978-3-030-58452-8_24 .

  7. Moreau A., Piasco N., Tsishkou D., Stanciulescu B., de La Fortelle A. LENS: Localization enhanced by NeRF synthesis. Proc. 5th Conference on Robot Learning (8–11 November 2021, London, UK). London, 2021. Vol. 164, P. 1347–1356. ttps://doi.org/10.48550/arXiv.2110.06558 .

  8. Godard C., Aodha O.M., Brostow G.J. Unsupervised monocular depth estimation with left-right consistency. Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (21–26 July 2017, Honolulu, HI, USA). Honolulu, 2017. P. 6602–6611. .

  9. Li H., Gordon A., Zhao H., Casser V., Angelova A. Unsupervised monocular depth learning in dynamic scenes. Proc. 2020 Conference on Robot Learning (16–18 November 2020, virtual). 2021. Vol. 155. P. 1908–1917. https://doi.org/10.48550/arXiv.2010.16404.

  10. Wang K., Zhang Z., Yan Z., Li X., Xu B., Li J., Yang J. Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10–17 October 2021, Montreal, QC, Canada). Montreal, 2021. P. 16035–16044. https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.01575.

  11. Kolodiazhna O., Savin V., Uss M., Kussul N. 3D scene reconstruction with neural radiance fields (NeRF) considering dynamic illumination conditions. Proc. 11th International Conference on Applied Innovations in IT (ICAIIT 2023) (9 March 2023, Kothen, Germany). Kothen, 2023. Vol. 11. P. 233–238. https://doi.org/10.25673/101943.

  12. Dai A., Chang A.X., Savva M., Halber M., Funkhouser T., Niebner M. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (21–26 July 2017, Honolulu, HI, USA). Honolulu, 2017. P. 2432–2443. https://doi.org/10.48550/arXiv.1702.04405.

  13. Waechter M., Moehrle N., Goesele M. Let there be color! Large-scale texturing of 3D reconstructions. Proc. 13th European Conference on Computer Vision (ECCV 2014) (6–12 September 2014, Zurich, Switzerland). Zurich, 2014. Lecture Notes in Computer Science. Vol. 8693. P. 836–850. https://doi.org/10.1007/978-3-319-10602-1_54.

  14. Jiang C.M., Sud A., Makadia A., Huang J., Niener M., Funkhouser T. Local implicit grid representations for 3D scenes. Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (13–19 June 2020, Seattle, WA, USA). Seattle, 2020. P. 6000–6009. https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.00604.

  15. Penner E., Zhang L. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics. 2017. Vol. 36, Iss. 6. P. 1–11. https://doi.org/10.1145/3130800.3130855 .

  16. Lin C.-H., Ma W.-C., Torralba A., Lucey S. BARF: Bundle-adjusting neural radiance fields. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10–17 October 2021, Montreal, QC, Canada). Montreal, 2021. P. 5721–5731. http://dx.doi.org/10.1109/ICCV48922.2021.00569.

  17. Chen A., Xu Z., Zhao F., Zhang X., Xiang F., Yu J., Su H. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10–17 October 2021, Montreal, QC, Canada). Montreal, 2021. P. 14104–14113. https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.01386 .

  18. Wei Y., Liu S., Rao Y., Zhao W., Lu J., Zhou J. NerfingMVS: Guided optimization of neural radiance fields for indoor multi-view stereo. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10–17 October 2021, Montreal, QC, Canada). Montreal, 2021. P. 5590–5599. https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.00556.

  19. Schnberger J.L., Frahm J.-M. Structure-from-motion revisited. Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (27–30 June 2016, Las Vegas, NV, USA). Las Vegas, 2016. P. 4104–4113. https://doi.org/10.1109/CVPR.2016.445.

  20. Roessle B., Barron J.T., Mildenhall B., Srinivasan P.P., Niebner M. Dense depth priors for neural radiance fields from sparse input views. Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (18–24 June 2022, New Orleans, LA, USA). New Orleans, 2022. P. 12882–12891. https://doi.ieeecomputersociety.org/10.1109/ CVPR52688.2022.01255.

  21. Li Z., Niklaus S., Snavely N., Wang O. Neural scene flow fields for space-time view synthesis of dynamic scenes. Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (20–25 June 2021, Nashville, TN, USA). Nashville, 2021. P. 6498–6508. https://doi.org/10.1109/CVPR46437.2021.00643.

  22. Roberts M., Ramapuram J., Ranjan A., Kunar A., Bautista M.A., Paczan N., Webb R., Susskind J.M. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10–17 October 2021, Montreal, QC, Canada). Montreal, 2021. P. 10892–10902. https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.01073.

  23. Kajiya J.T., Herzen B.V. Ray tracing volume densities. ACM SIGGRAPH Computer Graphics. 1984. Vol. 18, N 3. P. 165–174. https://doi.org/10.1145/964965.808594.

  24. Kusupati U., Cheng S., Chen R., Su H. Normal assisted stereo depth estimation. Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (13–19 June 2020, Seattle, WA, USA). Seattle, 2020. P. 2186–2196. http://dx.doi.org/10.1109/CVPR42600.2020.00226.




Editorial Board Announcements Abstracts Authors Archive
about scope office editorial board instructions for authors subscriptions contacts
© 2023 Kibernetika.org. All rights reserved.