Steuart Systems

Photorealistic Digital 3D Model Creation

Omnidirectional Scene Acquisition IEEE Research Paper

Digital 3d/360 Degree Camera System Patent

The Prototype-4.x family of 3D-360s is based on a camera that we have been developing for over a year.  While several areas of enhancement are still left to be implemented, the new camera is ready to be compared against the Canon 5D.  Prototype-3 used eight Canon 5Ds, and the new camera in Prototype-4 needs to meet or exceed the 5D’s performance.

One significant difference between our camera and the Canon 5D is that the 5D (and all other color cameras) uses tiny color filters arranged in a Bayer pattern on top of the individual pixels inside of the camera.  While the 5D has 12 million pixels, only 3 million are RED, 6 million are GREEN, and 3 million are BLUE.  Our camera is arguably a 15 million pixel sensor because it cycles through three large filters with the 5 million pixel monochrome sensor to produce 5 million RED pixels, 5 million GREEN pixels, and 5 million BLUE pixels. Our camera is immune to color artifacts caused by the Bayer patterns, but taking a picture takes three times longer because the filters must be rotated into place between shots. Fortunately our system automatically changes between filters in less than one second.  In the future we may want to add filters for other parts of the spectrum including infrared (IR) and ultra violet.

The purpose of this test is to compare the color reproduction, noise, and Bayer pattern artifacts between the two cameras. The 5D has a 14mm Canon lens, and the FOV is similar to our custom lens. Here is the test procedure:

1) Take a picture with each camera in RAW mode

2) Use minimal automatic processing on each image.  For the 3D-360 Photoshop was used for color balance and sharpening.  For the Canon 5D the image was processed with DxO

3) Compare the cropped images at actual size and zoomed to 600%

Here are the results:

Above is the shot from the Prototype-4 camera,

And below is the shot from the Canon 5D.

The two shots show that our camera compares well to the Canon 5D.  A slight BLUE halo is visible to the left of some objects, but this may be caused by a dirty or warped Wratten filter.

Below is a zoomed comparison of the areas the GREEN circles.

Close inspection shows that the 3D-360 camera has less noise and fewer Bayer pattern artifacts, but the 5D seems a little sharper.  The difference in sharpness could be related to the dynamic range of the two images.  The raw 3D-360 image covers a linear range of 24 bits, but the 5D covers a smaller range of only 12 bits.  We use a combination of linear and logarithmic curves to squeeze the 24 bits per pixel per color channel down to 16 bits per pixel per channel.  To improve contrast we may reduce our range from 24 bits to 22 bits.

I am pleased with this early test, and we are currently implementing upgrades that should make the difference even more dramatic.

This is the first color image produced by the new camera & lens combination. The bilinear rectification routine that we completed last week was automatically applied to correct chromatic aberration.  In the future bicubic interpolation will make the image even sharper.  The original 16-bit image had levels and curves adjusted in Photoshop, and the result was converted to the 8-bit JPG below.

Stereo reconstruction works by identifying similar features within two images, and we will use any technique that enhances small features.  As a first step in our stereo reconstruction pipeline we currently use bilinear interpolation to rectify/dewarp images.  While bilinear interpolation is easy to code and does a good job, there are many other types of interpolation worth considering. The two images below have been modified with bicubic interpolation and bilinear interpolation. The results confirm that bicubic is sharper, so we will eventually migrate to bicubic interpolation.

Wikipedia has some more examples.

We spent the last year designing and building a camera and software that can capture images with pixels that are 16-bits deep.  It isn’t easy to view these images since most tools expect 8-bit images, so the following routine is used to squeeze the 65,536 values in the 16-bit image down to the 256 values of an 8-bit image.  There are thousands of ways to compress a 16-bit image, and this approach is specifically for our machine vision/stereoscopic needs.

This approach to compressing pixel intensities is based on the octave relationship, and it is similar to the way a piano’s keys represent a wide range of frequencies. Each “octave” in this case is light intensity that is either twice as bright or half as bright as its neighboring octave.  Each octave of light intensity is broken into 20 steps, and this is similar to the 12 keys (steps) in each octave of a piano keyboard.  Below is a table and chart that illustrate the conversion from 16-bit images to 8-bits. Each red dot in the chart represent an octave, and there are 20 steps inside each octave.  The approach outlined here allows an 8-bit image to evenly cover 12 octaves: almost the full dynamic range of a 16-bit image.

This curve will probably be modified many times with different numbers of divisions per octave, but the basic approach will stay the same.  Below is an example  of an original 16-bit linear image, and an 8-bit version of the same image after application of the above logarithmic curve.  The pictures are not pretty, but they illustrate how details can be pulled from the shadows.  The 16-bit linear image is on the left, and the curve-adjusted 8-bit image is on the right.

The image at the right allows you to see the details in the shadows (notice the wires in the upper right) as well as details in the bright areas.   An image editing program could be used to manually adjust brightness and extract details from the 16-bit image, but the curve described here can do a good job automatically.

Next post: Rectification.

This weekend we assembled the first of the new lenses (the 3D-360 will use 10 of them). The new lens is bright and sharp.

We also assembled a variant of the LED Light Engine for military evaluation.