Computational Imaging
and Vision Laboratory


Polarimetric Three-View Geometry

This paper theorizes the connection between polarizationand three-view geometry. It presents a ubiquitous polarization-inducedconstraint that regulates the relative pose of a system of three cameras.We demonstrate that, in a multi-view system, the polarization phase obtained for a surface point is induced from one of the two pencils ofplanes: one by specular reflections with its axis aligned with the incident light; one by diffusive reflections with its axis aligned with the surface normal. Differing from the traditional three-view geometry, we show that this constraint directly encodes camera rotation and projection, and is independent of camera translation. In theory, six polarized diffusive point-point-point correspondences suffice to determine the camera rotations. In practise, a cross-validation mechanism using correspondences of specularites can effectively resolve the ambiguities caused by mixedpolarization. The experiments on real world scenes validate our proposed theory.(ECCV2018 pp.20-36)

>read more

Coded Illumination and Imaging for Fluorescence Based Classification

The quick detection of specific substances in objects such as produce items via non-destructive visual cues is vital to ensuring the quality and safety of consumer products. At the same time, it is well known that the fluorescence excitation-emission characteristics of many organic objects can serve as a kind of “fingerprint” for detecting the presence of specific substances in classification tasks such as determining if something is safe to consume. However, conventional capture of the fluorescence excitation-emission matrix can take on the order of minutes and can only be done for point measurements. In this paper, we propose a coded illumination approach whereby light spectra are learned such that key visual fluorescent features can be easily seen for material classification. We show that under a single coded illuminant, we can capture one RGB image and perform pixel-level classifications of materials at high accuracy. This is demonstrated through effective classification of different types of honey and alcohol using real images.(ECCV2018 pp.502-516)

>read more

Variable Ring Light Imaging: Capturing Transient Subsurface Scattering with an Ordinary Camera

Subsurface scattering plays a significant role in determining the appearance of real-world surfaces. A light ray penetrating into the subsurface is repeatedly scattered and absorbed by particles along its path before reemerging from the outer interface, which determines its spectral radiance. We introduce a novel imaging method that enables the decomposition of the appearance of a fronto-parallel real-world surface into images of light with bounded path lengths, i.e., transient subsurface light transport. Our key idea is to observe each surface point under a variable ring light: a circular illumination pattern of increasingly larger radius centered on it. We show that the path length of light captured in each of these observations is naturally lower-bounded by the ring light radius. By taking the difference of ring light images of incrementally larger radii, we compute transient images that encode light with bounded path lengths. Experimental results on synthetic and complex real-world surfaces demonstrate that the recovered transient images reveal the subsurface structure of general translucent inhomogeneous surfaces. We further show that their differences reveal the surface colors at different surface depths. The proposed method is the first to enable the unveiling of dense and continuous subsurface structures from steady-state external appearance using ordinary camera and illumination.(ECCV2018 pp.598-613)

>read more

Deeply Learned Filter Response Functions for Hyperspectral Reconstruction

Hyperspectral reconstruction from RGB imaging has recently achieved significant progress via sparse coding and deep learning. However, a largely ignored fact is that existing RGB cameras are tuned to mimic human trichromatic perception, thus their spectral responses are not necessarily optimal for hyperspectral reconstruction. In this paper,rather than use RGB spectral responses, we simultaneously learn optimized camera spectral response functions (to be implemented in hardware) and a mapping for spectral reconstruction by using an end-to-end network. Our core idea is that since camera spectral filters act in effect like the convolution layer, their response functions could be optimized by training standard neural networks. We propose two types of designed filters: a three-chip setup without spatial mosaicing and a single-chip setup with a Bayer-style 2x2 filter array. Numerical simulations verify the advantages of deeply learned spectral responses compared to existing RGB cameras. More interestingly, by considering physical restrictions in the design process, we are able to realize the deeply learned spectral response functions byusing modern film filter production technologies, and thus construct data inspired multispectral cameras for snapshot hyperspectral imaging.(CVPR2018 pp.4767-4776)

>read more

From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping

Spectral analysis of natural scenes can provide much more detailed information about the scene than an ordinary RGB camera. The richer information provided by hyperspectral images has been beneficial to numerous applications, such as understanding natural environmental changes and classifying plants and soils in agriculture based on their spectral properties. In this paper, we present an efficient manifold learning based method for accurately reconstructing a hyperspectral image from a single RGB image captured by a commercial camera with known spectral response. By applying a nonlinear dimensionality reduction technique to a large set of natural spectra, we show that the spectra of natural scenes lie on an intrinsically low dimensional manifold. This allows us to map an RGB vector to its corresponding hyperspectral vector accurately via our proposed novel manifold-based reconstruction pipeline. Experiments using both synthesized RGB images using hyperspectral datasets and real world data demonstrate our method outperforms the state-of-the-art.(ICCV2017 pp.4705-4713)

>read more

A Microfacet-Based Reflectance Model for Photometric Stereo with Highly Specular Surfaces

A precise, stable and invertible model for surface reflectance is the key to the success of photometric stereo with real world materials. Recent developments in the field have enabled shape recovery techniques for surfaces of various types, but an effective solution to directly estimating the surface normal in the presence of highly specular reflectance remains elusive. In this paper, we derive an analytical isotropic microfacet-based reflectance model, based on which a physically interpretable approximate is tailored for highly specular surfaces. With this approximate, we identify the equivalence between the surface recovery problem and the ellipsoid of revolution fitting problem, where the latter can be described as a system of polynomials. Additionally, we devise a fast, non-iterative and globally optimal solver for this problem. Experimental results on both synthetic and real images validate our model and demonstrate that our solution can stably deliver superior performance in its targeted application domain.(ICCV2017 pp.3162-3170)

>read more

Separation of Transmitted Light and ScatteringComponents in Transmitted Microscopy

In transmitted light microscopy, a specimen tends to be observed as unclear. This is caused by a phenomenon that an image sensor captures the sum of these scattered light rays traveled from different paths due to scattering. To cope with this problem, we propose a novel computational photography approach for separating directly transmitted light from the scattering light in a transmitted light microscope by using high-frequency lighting. We first investigated light paths and clarified what types of light overlap in transmitted light microscopy. The scattered light can be simply represented and removed by using the difference in observations between focused and unfocused conditions, where the high-frequency illumination becomes homogeneous. Our method makes a novel spatial multiple-spectral absorption analysis possible, which requires absorption coefficients to be measured in each spectrum at each position. Experiments on real biological tissues demonstrated the effectiveness of our method.(MICCAI2017 pp.12-20)

Wetness and Color from A Single Multispectral Image

Visual recognition of wet surfaces and their degrees of wetness is important for many computer vision applications. It can inform slippery spots on a road to autonomous vehicles, muddy areas of a trail to humanoid robots, and the freshness of groceries to us. In the past,monochromatic appearance change,the fact that surfaces darken when wet, has been modeled to recognize wet surfaces. In this paper, we show that color change, particularly in its spectral behavior, carries rich information about a wet surface. We derive an analytical spectral appearance model of wet surfaces that expresses the characteristic spectral sharpening due to multiple scattering and absorption in the surface. We derive a novel method for estimating key parameters of this spectral appearance model, which enables the recovery of the original surface color and the degree of wetness from a single observation. Applied to a multispectral image, the method estimates the spatial map of wetness together with the dry spectral distribution of the surface. To our knowledge, this work is the first to model and leverage the spectral characteristics of wet surfaces to revert its appearance. We conduct comprehensive experimental validation with a number of wet real surfaces. The results demonstrate the accuracy of our model and the effectiveness of our method for surface wetness and color estimation.(CVPR2017 pp.3967-3975)

>read more

Visibility enhancement of fluorescent substance under ambient illumination using flash photography

Many natural and manmade objects contain fluorescent substance. To visualize the distribution of fluorescence emitting substance is of great importance for food freshness examination, molecular dynamics analysis and so on. Unfortunately, the presence of fluorescent substance is usually imperceptible under strong ambient illumination, since fluorescent emission is relatively weak compared with surface reflectance. Even assuming that surface reflectance could be somehow blocked out, shading effect on fluorescent emission that relates to surface geometry would still interfere with visibility of fluorescent substance in the scene. In this paper, we propose a visibility enhancement method to better visualize the distribution of fluorescent substance under unknown and uncontrolled ambient illumination. By using an image pair captured with UV and visible flash illumination, we obtain a shading-free luminance image that visualizes the distribution of fluorescent emission. We further replace the luminance of the RGB image under ambient illumination by using this fluorescent emission luminance, so as to obtain a full colored image. The effectiveness of our method has been verified when used to visualize weak fluorescence from bacteria on rotting cheese and meat.(ICIP2017 pp.1622-1626)

>read more

Light transport component decomposition using multi-frequency illumination

Scene appearance is a mixture of light transport phenomena ranging from direct reflection to complicated effect such as inter-reflection and subsurface scattering. To decompose scene appearance into meaningful photometric components is very helpful in scene understanding and image editing. However, it has proven to be a difficult task. In this paper, we explore the difference of direct components obtained by multi-frequency illumination for light transport component decomposition. We apply independent vector analysis (IVA) to this task with no fixed constraints. Experiment results have verified the effectiveness of our method and its applicability to generic scenes.(ICIP2017 pp.3595-3599)

>read more

Direct and global component separation from a single image using basis representation

Previous research showed that the separation of direct and global components could be done with a single image by assuming neighboring scene points have similar direct and global components, but it normally leads to loss of spatial resolution of the results. To tackle such problem, we present a novel approach for separating direct and global components of a scene in full spatial resolution from a single captured image, which employs linear basis representation to approximate direct and global components. Due to the basis dependency of these two components, high frequency lighting pattern is utilized to modulate the frequency of direct components, which can effectively resolve the ambiguity between the basis representation for direct and global components, and contributes to achieving robust separation results. The effectiveness of our approach is demonstrated on both simulated and real images captured by a standard off-the-shelf camera and a projector mounted in a coaxial system. Our results show better visual quality and less error compared with those obtained by the conventional single-shot approach on both still and moving objects.(ACCV2016 pp.99-114)

>read more

Spectral Reflectance Recovery with Interreflection Using a Hyperspectral Image

The capture of scene spectral reflectance (SR) provides a wealth of information about the material properties of objects, and has proven useful for applications including classification, synthetic relighting, medical imaging, and more. Thus many methods for SR capture have been proposed. While effective, past methods do not consider the effects of indirectly bounced light from within the scene, and the estimated SR from traditional techniques is largely affected by interreflection. For example, different lighting directions can cause different SR estimates. On the other hand, past work has shown that accurate interreflection separation in hyperspectral images is possible but the SR of all surface points needs to be known a priori. Thus we see that the estimation of SR and interreflection in its current form constitutes a chicken and egg dilemma. In this work, we propose the challenging and novel problem of simultaneously performing SR recovery and interreflection removal from a single hyperspectral image, and develop the first strategy to address it. Specifically, we model this problem using a compact sparsity regularized nonnegative matrix factorization (NMF) formulation, and introduce a scalable optimization algorithm on the basis of the alternating direction method of multipliers (ADMM). Our experiments have demonstrated its effectiveness on scenes with a single or two reflectance colors, containing possibly concave surfaces that lead to interreflection.(ACCV2016 pp.52-67)

>read more

Shape from Water: Bispectral Light Absorption for Depth Recovery

This paper introduces a novel depth recovery method based on light absorption in water. Water absorbs light at almost all wave-lengths whose absorption coefficient is related to the wavelength. Based on the Beer-Lambert model, we introduce a bispectral depth recoverymethod that leverages the light absorption difference between two near-infrared wavelengths captured with a distant point source and ortho-graphic cameras. Through extensive analysis, we show that accuratedepth can be recovered irrespective of the surface texture and reflectance,and introduce algorithms to correct for nonidealities of a practical imple-mentation including tilted light source and camera placement and non-ideal bandpass filters. We construct a coaxial bispectral depth imagingsystem using low-cost off-the-shelf hardware and demonstrate its use forrecovering the shapes of complex and dynamic objects in water. Exper-imental results validate the theory and practical implementation of thisnovel depth recovery paradigm, which we refer to as shape from water.(ECCV2016 pp.635-649)

>read more