Neural Sensors: Learning Pixel Exposures for HDR Imaging and Video Compressive Sensing with Programmable Sensors
Abstract
Camera sensors rely on global or rolling shutter functions to expose an image. This fixed function approach severely limits the sensors’ ability to capture high-dynamic-range (HDR) scenes and resolve high-speed dynamics. Spatially varying pixel exposures have been introduced as a powerful computational photography approach to optically encode irradiance on a sensor and computationally recover additional information of a scene, but existing approaches rely on heuristic coding schemes and bulky spatial light modulators to optically implement these exposure functions. Here, we introduce neural sensors as a methodology to optimize per-pixel shutter functions jointly with a differentiable image processing method, such as a neural network, in an end-to-end fashion. Moreover, we demonstrate how to leverage emerging programmable and re-configurable sensor–processors to implement the optimized exposure functions directly on the sensor. Our system takes specific limitations of the sensor into account to optimize physically feasible optical codes and we demonstrate state-of-the-art performance for HDR and high-speed compressive imaging in simulation and with experimental results.
Videos
ICCP talk (12 min)
System overview
Illustration of our end-to-end neural sensor framework.
The exposure program of a sensor (physical layer) is learned end-to-end with a decoder (digital layer) for applications like video compressive sensing. Here, we show a single coded exposure captured with our prototype camera and several frames of the high-speed video reconstructed from this image showing an exploding balloon.
Hardware Setup
 | Real captures of the coded images using the optimized shutter functions
are performed with SCAMP-5, a programmable sensor processor. SCAMP-5 is a 256×256 pixel array in which each pixel is equipped with a small processor and a few registers. |
Results: real captures
| Coded Exposure Image | Reconstructed Video |
|---|---|
| Water Droplet (video contains 64 total frames from 4 coded frames) | |
 |  |
| Rotating Fan (video contains 64 total frames from 4 coded frames) | |
 |  |
| Jumping Frog (video contains 64 total frames from 4 coded frames) | |
 |  |
| Oscilloscope (video contains 16 total frames from 1 coded frame) | |
 |  |
| Balloon (video contains 32 total frames from 2 coded frames) |  |  |
On Twitter
Neural Sensors: We teach a programmable sensor how to "optimally" expose for high dynamic range imaging and high-speed imaging. Our work was published in the July issue of IEEE T-PAMI.
— Julien N.P. Martel (@jnpmartel) August 6, 2020
Paper and videos:https://t.co/QtjTktaUjb
with L. Muller, S. Carey, P. Dudek, @wetzste1. pic.twitter.com/Ig0qeHwtWO