5.1 General calibration steps and parameters

In order to perform stereo camera reconstruction we need a mathematical formula or model that relates particular combinations of 2D pixel coordinates from each view to 3D coordinates. The mathematical model used by StereoMorph is the DLT model direct linear transformation; Abdel-Aziz & Karara 1971). With the DLT method, each calibrated camera has a set of 11 coefficients that relate each unique 3D coordinate in the calibration space to their corresponding (non-unique) 2D pixel coordinates in that particular camera view; modified forms of DLT use additional coefficients to account for lens distortion but StereoMorph uses just 11.

Note that if you only have the coefficients for a single camera view you can only go one way: you can only project 3D points to pixel coordinates, not the other way around. This is because a point with particular pixel coordinates in an image can fall anywhere along a line in 3D space. But if you have the DLT coefficients from two or more camera views you can combine the pixel coordinates of a single point in both views from multiple camera views to find the corresponding 3D coordinate. You can think of this as finding the intersection of the two lines from each camera view in 3D space. Thus, the objective of the camera calibration step is to determine these 11 DLT coefficients for each camera.

Just as there are multiple mathematical models for 3D reconstruction there are also multiple methods with which to determine the DLT coefficients. Typically, DLT coefficients are determined using what's referred to as a calibration or reference object. This is usually a 3D, cube-shaped structure filled with markers having known 3D positions relative to one another. The markers are digitized in each camera and the set of corresponding 2D and 3D coordinates are used to calculate the DLT coefficients.

Because of the difficulties in designing and building a 3D calibration, and the time-consuming task of digitizing the calibration cube markers, StereoMorph determines the DLT coefficients using the internal corners automatically detected from a checkerboard pattern. A checkerboard is photographed from both camera views at different positions and angles within the calibration space. Rather than estimate the DLT coefficients directly, StereoMorph estimates the six transformation parameters (3 translation, 3 rotation) required to transform the first checkerboard into each subsequent checkerboard in 3D space by minimizing the reconstruction error. These transformation parameters are then used to generate 3D coordinates (the equivalent of a calibration object) and calculate the DLT coefficients.

Whether you are using stereo photographs or video, be sure that you position the checkerboard at different positions and angles throughout the calibration space. This provides a sampling of points throughout the space for the calibration. If you only photograph the checkerboard in one area of the calibration volume, reconstruction errors could be relatively higher in other areas. Similarly, if you only photograph the checkerboard at a particular angle (e.g. 45 degrees), you won't have good sampling of points along each dimension of the space (since a checkerboard is a flat surface it can only sample two dimensions at any one time). This can causes reconstruction errors to be higher along particular dimensions than along others.

Calibration in StereoMorph is accomplished using a single function, calibrateCameras(). As of version 1.6, this function works with both photographs and synchronized video. While the parameters will differ depending on the camera arrangement and the type of input (i.e. photograph versus video), calibrateCameras() follows four general steps:

1. Detect checkerboard corners: The function will try to find the specified number of checkerboard corners in each input image.
2. Estimate undistortion coefficients: (Optional) If specified, the function will solve for coefficients that minimize standard lens distortion. This will not undistort the original photographs or video. But these coefficients will be used in the calibration and reconstruction process to minimize error due to lens distortion.
3. Estimate DLT coefficients: The function will estimate a set of optimized DLT coefficients that can be used to reconstruct points and curves from each view into 3D.
4. Estimate calibration accuracy: The function will estimate the calibration, using a separate set of images within the calibration set (if a sufficient number of images are supplied).

Because steps 1-3 are fairly time-consuming, the function will save the results of these steps. If you decide you would like to re-run any of these steps you can simply call calibrateCameras() again. Before running each of these three steps, you will be prompted whether you would like to keep the results from the previous run or re-run the step.

The calibrateCameras() function also has a general set of parameters used for both stereo photography and videography.

• img.dir: A folder containing the calibration images or videos.
• sq.size: The size of one checkerboard square (length along any one side) including the units (e.g. mm).
• nx: The number of internal corners along one dimension.
• ny: The number of internal corners along the other dimension (the choice of which is nx and ny is arbitrary but must be consistent throughout).
• cal.file: A file where the calibration results will be saved (the file will automatically be created if one doesn't already exist).
• corner.dir: A folder where the corners will be saved (the folder will be automatically be created if one doesn't already exist).
• verify.dir: (Optional) A folder where images will be saved that show the detected corners (the folder will be automatically be created if one doesn't already exist).
• error.dir: (Optional) A folder where several error diagnostic plots will be saved (the folder will be automatically be created if one doesn't already exist).
• undistort: A logical (TRUE or FALSE) indicating whether undistortion coefficients should be estimated. Currently, this is only recommended for video input.

The following sections will show how to call calibrateCameras() using photographs and video. Before proceeding, make sure the StereoMorph library is loaded into the current R session.

# Load the StereoMorph package
library(StereoMorph)