Abstract
Looking for tips on obtaining the best depth data from your 3D camera? This article will describe a few key characteristics of the Intel® RealSense™ camera R200 and the Intel® RealSense™ SDK that can affect depth data. You’ll also find a methodology for collecting raw data, including sample code (available from GitHub.com) as well as an attached zip file containing typical data.
We suggest testing under your specific lighting, texture, and distance parameters to provide some working boundaries for your scenes.
Overview
Figure 1 Intel(R) RealSense(TM) Camera R200 Reference Developer Kit
This analysis uses data obtained from the world facing Intel RealSense camera R200. The R200 is the first generation of Intel’s rear-facing 3D cameras based on active stereoscopic depth technology. The camera implements an IR laser projector, a pair of stereoscopic IR cameras and a 2 MP RGB color camera. You can purchase a developer kit for this camera from http://click.intel.com/intel-realsense-developer-kit-r200.html or a system with an integrated R200 camera.
Considerations - Camera and Object
The construction and behavior of the camera components can affect the resulting data, even in the case of fixed indoor lighting with stationary camera and target. Features and functions of the camera that affect depth data include:
- Temporal (frame-to-frame) and Spatial (within a frame) variation
- Sensor to sensor alignment and each sensor’s field of view (FOV)
- The environment’s temperature (effect on sensors)
- The environment’s ambient IR light
- Any other physical effect on a sensor including physical impact or vibration
The location and type of target, especially the object’s IR reflectivity and texture (sheen, dimples, and curvature) will also affect the depth data. For simplicity, this data collection used a fixed target of single texture.
These tests were performed using Windows* 8.1 64 bit with August update and v6 (R4) of the Intel RealSense SDK available at https://software.intel.com/en-us/intel-realsense-sdk/download.
Considerations - Geometry
For the simplest test, the camera was pointed directly at a flat target, a typical inside white wall with standard office fluorescent ceiling lighting. The camera was mounted firmly, at a fixed height, on a rail that can travel linearly (one axis only) from 600-2100 mm from the target. The distance is measured by a consumer-grade laser meter mounted flush with the front of the camera.
The camera face is aligned parallel to the wall and the rail axis is aligned perpendicular to the wall.
Figure 2: Test measurement set up with Intel® RealSense™ camera mounted for stabilization
While the camera moves between measurements, it was sitting still while collecting data.
In these tests, the motion of the rail induced significant vibrations that would affect the data so a large delay was introduced to allow the vibrations to cease before any frames were captured. The rail motion, time delays, and data capture were automated to provide consistent, repeatable results (and to cut down on labor).
Tip: Alignment of the rail and camera with the target is critical. You can align the rail axis using physical measurement from the end of the rail to reference points on the wall. Aligning the camera face is more difficult. For these tests, we adjusted the camera alignment based on a spatially uniform pattern observed in the raw depth image from an Intel® RealSense™ SDK sample, with the contrast adjusted by scaling to make it easier for the human eye to see variations across the field of view. See Figure 3 for a depiction of a non-spatially uniform pattern and Figure 4 for an aligned target image. Note that the black area on the left is an artifact of stereoscopic imaging and does not reflect an alignment issue.
TIP: All stereoscopic cameras will have areas where data is missing, due to the difference in field of view between the sensors. Figure 4 above shows such an area for the R200 camera in the black bar on the left. The effect of this camera characteristic decreases with distance but needs to be considered at closer ranges.
Test Range
Using the above test configuration, we collected a series of raw depth buffers at distances ranging from 600 mm to 2100 mm. 600mm is the minimum distance for the Intel R200 camera and 2-3 meters is the maximum distance indoors (longer range outdoors). Remember, range ultimately depends on multiple factors including lighting and object textures.
TIP: Keep within the range of the depth camera. At distances less than 600 mm, the R200 camera may not return any depth data, depending on the configured resolution.
TIP: For testing, keep the input scene simple (e.g. a flat wall), as it is easier to detect general trends.
Quality and Precision Related to Distance
Our tests looked at two factors - the quality of the depth data and the precision of the depth data as functions of distance from the target. For all depth cameras, accuracy and precision of depth measurements decreases as distance increases.
Tips:
For more accurate, precise data, stay as close to the object as possible. If moderate accuracy is needed, just stay within the recommended range of the camera.
Capturing Depth Data
A sample script is provided at https://github.com/IntelRealSense/rsdatacollector But to illustrate the flow, here’s a quick walkthrough of the basic steps to capture the camera data as individual frames using the Intel RealSense SDK:
- Create and initialize an instance of PXCSenseManager (see PXCSenseManager::CreateInstance()).
- Use sense manager to enable the depth and/or color stream specifying resolution and frame rate.
senseManager->EnableStream(PXCCapture::STREAM_TYPE_COLOR, 320, 240, 60);
senseManager->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, 320, 240, 30); - Use sense manager to acquire / lock an available frame.
camStatus = senseManager->AcquireFrame(true); - Use sense manager to query the available samples
PXCCapture::Sample *sample = senseManager->QuerySample(); - Access the Sample's specific PXCImage members to extract the desired data .
colorImage = sample->color;
depthImage = sample->depth; - Use member functions of the PXCImage object, such as QueryInfo(), to get the parameters of the color or depth data.
NOTE: Understanding the image format, dimensions, and other details is critical when traversing the raw image buffers. The parameters are documented in the SDK and the sample code associated with this paper shows an example of how those parameters are used.
The data buffer is organized in planes and pixels and specific pitches that correspond to the parameters in the ImageInfo mentioned above.Once you understand the format of the data buffer, you can directly manipulate the values for various analysis or write it out to disk for analysis in other tools.
- 7. To access the raw image buffers, you must use the PCXImage::AcquireAccess() function passing it your PCXImage::ImageData pointer to store the buffer location. The planes and pitches of the ImageData should be initialized to empty arrays.
8. After obtaining the desired image data, you must release access to the data buffer (PXCImage::ReleaseAccess()) and release the frame (PXCSenseManager::ReleaseFrame())
The Intel® RealSense™ SDK offers several abstractions for dealing with the image data, allowing easy access to and direct handling of the data buffers. These include the PXCPhoto::SaveXDM() function, which will simply write the depth enhanced JPEG file format. Additionally Background Segmentation (BGS), Enhanced Photography and Video (EPV) and Scene Perception (SP) modules in the SDK have options to provide additional filtering of the depth data. EPV and BGS share the same flags while SP flags are fixed.
Much more information is available in the SDK documentation at https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_introduction.html
Conclusion
Intel® RealSense™ Technology provides developers with depth data useful for extracting distance and size of objects from images and the ability to reconstruct a scene in 3D. Since the quality of depth images varies due to many factors including time, lighting and distance, it is important to characterize the factors that may impact your application as described in this paper. Read the Intel RealSense SDK documentation to familiarize yourself with available SDK algorithms and sample code for capture, filtering, and processing.
Further Information
The Intel Developer Zone (IDZ) website for Intel RealSense technology includes the SDK and full documentation plus many articles, code samples, videos, blogs by software developers, links to events, and a forum for discussion and questions. Start at https://software.intel.com/realsense to check out the R200 camera features and videos.
About the Authors
 | Tom Propst is a Software Engineer at Intel focusing on enabling new use cases in business environments. He received his B.S. in Electrical Engineering from Colorado State University. Outside of work, Tom enjoys playing bicycle polo and tinkering with electronics. |
| Teri Morrison is a Software Architect at Intel focusing on the Intel® RealSense™ SDK. She has worked on NASA projects, graphics driver products and physics, graphics and perception computing samples. She received her M.S. in Applied Mathematics with minor in Computer Science from the University of Colorado. |
