Steuart Systems

Making 3D models is time consuming. Recent programs like Google’s SketchUp (it’s free) have simplified the process of making digital 3D models, but SketchUp is definitely not automatic.

To make a 3D model look photorealistic, real world pictures can be “projected” onto a SketchUp model. While this technique can add realism, SketchUp is still a manual approach that can take hours, weeks, or even months to produce good results.

 

Many in the 3D and animation world would like an automatic process that can produce 3D models from a series of 2D pictures. Our goal is to create a system that automatically produces photorealistic digital 3D models that can be processed in existing 3D programs like 3D Studio Max, GeoMagic, or SketchUp.

The Microsoft Photosynth project can automatically create 3D-like effects (some call it 2.5D) by automatically processing 10s to 100s of 2D images. While this process is automatic, it does not produce a 3D model that can be used by other programs.

Garbage in…….. Garbage out.

A challenge for Photosynth and other automatic stitching/panoramic approaches is that they often use regular uncalibrated cameras. While this is convenient, it forces the programs to analyze each camera image to determine the field of view and other essential lens/camera characteristics: the cameras are essentially calibrated during processing. Precisely calibrating a camera is challenging in a lab setting, so it is reasonable to expect that on-the-fly calibration results will not be very precise. Any errors in the camera calibration step will build on each other and cause problems later in the process. While calibration problems cause annoying alignment errors in panoramic 2D & 2.5D images, they cause unacceptable distortion in 3D models. Here is a list of variables that must be determined before using a 2D image to create an accurate 3D model:

Camera Variables that must be determined for Precise Stereoscopic 3D Reconstruction
- The exact center of the image sensor behind the lens: sensors are normally a few pixels off-center
- Camera Horizontal & Vertical Field of View to within 1/100 degree
- Camera lens distortion correction variables: Pincushion, barrel, radial.
- Camera horizontal orientation (0.00 to 360.00 degrees) to within 1/100 of a degree
- Camera vertical orientation (tilt, roll) to within 1/100 degree
- Camera location for each shot: X, Y, and Z coordinates to within one millimeter
- Camera dynamic range and gamma

The quality of a 3D model is limited by the quality of the 2D pictures used to make it. Here’s how we calibrate our camera system:

1) Design and build a calibration routine/facility to determine the key camera variables.
2) Design and build a system of cameras that can be easily calibrated.

The important point is that the camera system and the calibration system need to be built for each other: they literally fit together like a lock and key. As we see it, a calibrated system produces “clean” images that simplify and speed up the 3D reconstruction process. Our current 8-camera system (Proto-4F) has been designed to produce sets of calibrated images, and these images are used to automatically produce 3D models.

We are currently refining the calibration of Proto-4F, and another model should be completed by the end of April.

Photorealistic 3D scanning has unique requirements when compared to regular photography. Because a 3D model will be viewed from multiple perspectives, the 3D scanner must be able to capture multiple perspectives. Not only must you move left and right when scanning, you must also move up and down. This scissor jack was built to allow us to easily move the 3D-360 up and down. Tomorrow we begin scanning with this new capability.

The cameras are finally calibrated, and the communications and power systems are installed and working. Now I can finally begin producing scans to test and fine tune the software.

Today I scanned part of the lab, and the animated GIF illustrates the 3D nature of the scan. When producing a 3D model, multiple perspectives must be captured to fill in occlusions (blind spots). For this model, three scans from different locations were merged to produce a point cloud. The GIF consists of 7 different screen-shots of the point-cloud. While there are still occlusions, many have been filled. For example, notice that you can see both above and below the table.

The original 32-bit software that we use to turn pictures into 3D models is almost 5 years old, and it runs on 32-bit Windows XP. The old software often crashes when processing high resolution images because the 2GB memory limit isn’t enough to process the gigabytes of data that our scanner can quickly produce. Today’s scan was made on a computer running 64-bit Windows 7, and we are currently replacing the old 32-bits software with more advanced 64-bit code. The new software runs much faster in 64-bit mode because it can keep temporary files in RAM instead of writing them to and reading them from a slow disk. Even using a Solid State Drive (SSD) wastes minutes of unnecessary processing.

COMING UP: Much better scans processed by SketchUp & posted into Google Earth.

The Prototype-4.x family of 3D-360s is based on a camera that we have been developing for over a year.  While several areas of enhancement are still left to be implemented, the new camera is ready to be compared against the Canon 5D.  Prototype-3 used eight Canon 5Ds, and the new camera in Prototype-4 needs to meet or exceed the 5D’s performance.

One significant difference between our camera and the Canon 5D is that the 5D (and all other color cameras) uses tiny color filters arranged in a Bayer pattern on top of the individual pixels inside of the camera.  While the 5D has 12 million pixels, only 3 million are RED, 6 million are GREEN, and 3 million are BLUE.  Our camera is arguably a 15 million pixel sensor because it cycles through three large filters with the 5 million pixel monochrome sensor to produce 5 million RED pixels, 5 million GREEN pixels, and 5 million BLUE pixels. Our camera is immune to color artifacts caused by the Bayer patterns, but taking a picture takes three times longer because the filters must be rotated into place between shots. Fortunately our system automatically changes between filters in less than one second.  In the future we may want to add filters for other parts of the spectrum including infrared (IR) and ultra violet.

The purpose of this test is to compare the color reproduction, noise, and Bayer pattern artifacts between the two cameras. The 5D has a 14mm Canon lens, and the FOV is similar to our custom lens. Here is the test procedure:

1) Take a picture with each camera in RAW mode

2) Use minimal automatic processing on each image.  For the 3D-360 Photoshop was used for color balance and sharpening.  For the Canon 5D the image was processed with DxO

3) Compare the cropped images at actual size and zoomed to 600%

Here are the results:

Above is the shot from the Prototype-4 camera,

And below is the shot from the Canon 5D.

The two shots show that our camera compares well to the Canon 5D.  A slight BLUE halo is visible to the left of some objects, but this may be caused by a dirty or warped Wratten filter.

Below is a zoomed comparison of the areas the GREEN circles.

Close inspection shows that the 3D-360 camera has less noise and fewer Bayer pattern artifacts, but the 5D seems a little sharper.  The difference in sharpness could be related to the dynamic range of the two images.  The raw 3D-360 image covers a linear range of 24 bits, but the 5D covers a smaller range of only 12 bits.  We use a combination of linear and logarithmic curves to squeeze the 24 bits per pixel per color channel down to 16 bits per pixel per channel.  To improve contrast we may reduce our range from 24 bits to 22 bits.

I am pleased with this early test, and we are currently implementing upgrades that should make the difference even more dramatic.

This is the first color image produced by the new camera & lens combination. The bilinear rectification routine that we completed last week was automatically applied to correct chromatic aberration.  In the future bicubic interpolation will make the image even sharper.  The original 16-bit image had levels and curves adjusted in Photoshop, and the result was converted to the 8-bit JPG below.