Car Surround view with fisheye cameras

[mrp1234567]

Hi,

I need to use libxcam to stitch the frames from 4 cameras around the car.

Can someone help me with detailed steps i need to do ? tools to be used, required changes to any of the source files of libxcam etc

https://sites.google.com/site/scarabotix/ocamcalib-omnidirectional-camera-calibration-toolbox-for-matlab does this tool generate extrinsic parameters ?

I built libxcam and used sample images provided in ocamcalib zip file for calibration and it generated calib_results.txt and Omni_Calib_Results.mat files

[zongwave]

Hi, the 4 cameras surround view stitching using Scaramuzza's oCamCalib tool to do camera calibration, you should generate extrinsic & intrinsic parameters, saved parameters into text files as:
https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L35
https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L42

libXCam implemented calibration parser at:
https://github.com/intel/libxcam/blob/master/xcore/calibration_parser.h#L38

[mrp1234567]

@zongwave
Thank You for the response.

I generated the calibration file from oCamCalib tool which has intrinsic parameters(calib_results.txt), how do I generate extrinsic parameters ?

In their documentation they mentioned RRFin has extrinsic parameters for each calibrated frame, not sure how to use RRFin to get extrinsic parameters in the following format, appreciate if you can help on this ?

#extrinsic fisheye camera calibration, positions in millimeters, rotations in degrees
#Position x
x
#Position y
y
#Position z
z
#Rotation x (roll)
r
#Rotation x (pitch)
p
#Rotation x (yaw)
yy

[zongwave]

There's a discussion about RRFin, you can check it out.
#616

[mrp1234567]

@zongwave

I have the extrinsic parameters (rotations in degrees) and intrinsics from oCamCalib but the output of four input images(in nv12 format) is still not good

I am using the following command, 4 cameras are fisheye (stereographic lens)

./test-surround-view --module soft --input frame_0_0_0.nv12 --input frame_0_0_1.nv12 --input frame_0_0_2.nv12 --input frame_0_0_3.nv12 --output test-surround-view_output.nv12 --in-w 1280 --in-h 960 --in-format nv12 --fisheye-num 4 --save true --save-topview true --loop 1

What else to look for to debug ?

What difference does --cam-model make ? [cama2c1080p/camb4c1080p/camc3c8k/camc6c8k/camd3c8k], default: camb4c1080p

[zongwave]

cama2c1080p is used to identify 2 cameras with 1080p and parse the stitching parameters,
camb4c1080p is for 4 cameras with 1080p, camc3c8k is for 3 cameras with 8k stitched output, camc6c8k is for 6 cameras with 8k stitch output.

We tested stitching function with video clips recorded by these cameras', you can find these parameters at:
https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368

[zongwave]

please enable stitching intermedia image dumping:
https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34
https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47

and if you enable opecv feature match, also enable here:
https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

[mrp1234567]

camb4c1080p

1. Our cameras output 1280x960 resolution frames, is it still ok to use camb4c1080p model ?

2. I do not see camb4c1080p in https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368, am I supposed to make any code changes other than giving input calibration parameters and 4 images from 4 cameras to generate surround view ?

3. You mentioned you used video clips but as start up test, I am giving just 4 images to stitch, is that fine or videos are mandatory ?

[mrp1234567]

please enable stitching intermedia image dumping: https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34 https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47

and if you enable opecv feature match, also enable here: https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

1. Can I know the list of all configurations that are supposed to be done ?
2. Is there a need to configure vehicle measurements information? If so, where do i need to configure ?
3. is ZYX convention for Euler Angles correct ?

Can you please verify the "Steps Followed" are correct ?
Steps Followed:
1. Used oCamCalib to generate intrinsic parameters(calib_results.txt) for all the 4 cameras
2. From RRfin, cross product of column 1 and column2 to generate column 3, then rotation matrix is converted to Euler Angles**(ZYX)**
3. Eluer Angles(in degrees)(roll - x, pitcy - y, yaw - z) are used in corresponding extrinsic files
4. Last column from RRfin of oCamCalib tool are used as Position x, Position Y and Position Z in corresponding extrinsic files
5. RRfin of an image where checkerboard is lying flat on the ground is taken for extrinsic parameter for each camera
6. Following is RRfin for one camera and calculations and Euler Angles
val(:,:,11) =

1.0e+02 *

-0.000093441273664 0.004960374800235 0.246741453624884
-0.009943325487015 0.000873493262281 0.364232264603431
0.001059031155184 0.008638963558318 -2.959915923124880

Euler Angles(ZYX) from rotation matrix:
radians: [ x: 1.0527373, y: -0.1061021, z: -1.5801934 ]
degrees: [ x: 60.3174068, y: -6.0792012, z: -90.5384151 ]

MatLab:
LeftRRFinColumn1 =
-0.009344127366400 -0.994332548701500 0.105903115518400
LeftRRFinColumn2 =
0.496037480023500 0.087349326228100 0.863896355831800
LeftRRFinColumn3 = cross(LeftRRFinColumn1, LeftRRFinColumn2)

LeftRRFinColumn3 =
-0.868250831094161 0.060604272128646 0.492410008533592

7. Content of extrinsic_camera_left.txt based on above RRfin
#Position x
24.6741453624884
#Position y
36.4232264603431
#Position z
-295.9915923124880
#Rotation x (roll)
#1.0527373
60.3174068
#Rotation x (pitch)
#-0.1061021
-6.0792012
#Rotation x (yaw)
#-1.5801934
-90.538415189

8. Following files are placed in FISHEYE_CONFIG_PATH
extrinsic_camera_front.txt extrinsic_camera_rear.txt intrinsic_camera_front.txt intrinsic_camera_rear.txt
extrinsic_camera_left.txt extrinsic_camera_right.txt intrinsic_camera_left.txt intrinsic_camera_right.txt

[mrp1234567]

stitcher-debug files.zip
test-surround-view_output.zip

Attached the debug files generated with XCAM_TEST_STREAM_DEBUG and DUMP_STITCHER enabled and also the generated output file

Can you please let me know, what might be going wrong that output is not good.

[zongwave]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.

BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ?

[zongwave]

camb4c1080p

1. Our cameras output 1280x960 resolution frames, is it still ok to use camb4c1080p model ?
2. I do not see camb4c1080p in https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L368, am I supposed to make any code changes other than giving input calibration parameters and 4 images from 4 cameras to generate surround view ?
3. You mentioned you used video clips but as start up test, I am giving just 4 images to stitch, is that fine or videos are mandatory ?

this enum camb4c1080p is to define bowl model config https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L128,
and each camera's fov https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L154
and feature match configuration: https://github.com/intel/libxcam/blob/master/tests/test_sv_params.h#L240

You should config these parameters according to your own camera hardware configuration.

You can apply still images in your development at the beginning.

[zongwave]

please enable stitching intermedia image dumping: https://github.com/intel/libxcam/blob/master/tests/test_stream.h#L34 https://github.com/intel/libxcam/blob/master/modules/soft/soft_stitcher.cpp#L47
and if you enable opecv feature match, also enable here: https://github.com/intel/libxcam/blob/master/modules/ocv/cv_feature_match.cpp#L29

1. Can I know the list of all configurations that are supposed to be done ?
2. Is there a need to configure vehicle measurements information? If so, where do i need to configure ?
3. is ZYX convention for Euler Angles correct ?

Can you please verify the "Steps Followed" are correct ? Steps Followed: 1. Used oCamCalib to generate intrinsic parameters(calib_results.txt) for all the 4 cameras 2. From RRfin, cross product of column 1 and column2 to generate column 3, then rotation matrix is converted to Euler Angles**(ZYX)** 3. Eluer Angles(in degrees)(roll - x, pitcy - y, yaw - z) are used in corresponding extrinsic files 4. Last column from RRfin of oCamCalib tool are used as Position x, Position Y and Position Z in corresponding extrinsic files 5. RRfin of an image where checkerboard is lying flat on the ground is taken for extrinsic parameter for each camera 6. Following is RRfin for one camera and calculations and Euler Angles val(:,:,11) =

1.0e+02 *

-0.000093441273664 0.004960374800235 0.246741453624884 -0.009943325487015 0.000873493262281 0.364232264603431 0.001059031155184 0.008638963558318 -2.959915923124880

Euler Angles(ZYX) from rotation matrix: radians: [ x: 1.0527373, y: -0.1061021, z: -1.5801934 ] degrees: [ x: 60.3174068, y: -6.0792012, z: -90.5384151 ]

MatLab: LeftRRFinColumn1 = -0.009344127366400 -0.994332548701500 0.105903115518400 LeftRRFinColumn2 = 0.496037480023500 0.087349326228100 0.863896355831800 LeftRRFinColumn3 = cross(LeftRRFinColumn1, LeftRRFinColumn2)

LeftRRFinColumn3 = -0.868250831094161 0.060604272128646 0.492410008533592

7. Content of extrinsic_camera_left.txt based on above RRfin #Position x 24.6741453624884 #Position y 36.4232264603431 #Position z -295.9915923124880 #Rotation x (roll) #1.0527373 60.3174068 #Rotation x (pitch) #-0.1061021 -6.0792012 #Rotation x (yaw) #-1.5801934 -90.538415189

8. Following files are placed in FISHEYE_CONFIG_PATH extrinsic_camera_front.txt extrinsic_camera_rear.txt intrinsic_camera_front.txt intrinsic_camera_rear.txt extrinsic_camera_left.txt extrinsic_camera_right.txt intrinsic_camera_left.txt intrinsic_camera_right.txt

you need import your cameras intrinsic & extrinsic parameters, and map the four camera images into a bowl model, https://github.com/intel/libxcam/blob/master/xcore/interface/data_types.h#L188
The extrinsic parameters represents your vehicle measurements information in some extent.

[mrp1234567]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.

BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ?
orig_fisheye_and_Output_files.zip

Attached the orig files and output files

[zongwave]

All the 4 files "stitcher-geomap-xx.NV12.yuv" are not correct. I don't expect the output image is good. these images should be correction images by pixel mapping from lookup table "BowlFisheyeDewarp::gen_table". this image distortion correction is the first step of stitching process. the camera intrinsic & extrinsic calibration parameters help to generate the geomap LUT, after distortion correction the four images should be basically aligned at the edge of images.
BTW, did you see some images named as "orig_fisheye_x_0/1/2/3.jpg" ?
orig_fisheye_and_Output_files.zip

Attached the orig files and output files

did you set your input & output frame size correctly? you input four 1280x960 images, after stitching output same resolution images. basically, the output image's width should be larger than height, because after stitching we got 360 degree fov in horizontal direction and less than 180 degree fov in vertical direction.
here's the default input & output resolution settings:
https://github.com/intel/libxcam/blob/master/tests/test-surround-view.cpp#L560