In this project, we calibrated a camera to generate information that can be used to project an image onto a given target. In our case, the given target is an asymmetrical checkerboard (or chessboard, if you prefer). The checkerboard works well as a target because a) the origin point is easy to find due to the high contrast between the black and the white, and b) the fact that the board's dimensions are 6x9 doesn't allow for another corner where the black squares meet in the same way. As for the image to project, we made a method to project a rectangular prism as well as a method to project fish.
The first task was to make a program that can extract corners from a checkerboard. Luckily, OpenCV already has a built-in function for finding corners on a checkerboard, and the only thing we needed to provide to this method, findChessboardCorners, was the dimensions of the board and the image to use. In the case of a live video stream, every frame of the video is a new image. We pass in each frame to the method, and let it write to a corner_set, which is a vector of 2D points that correspond to image pixel locations. We then put the corners through another built-in function, cornerSubPix, to get the corner locations to be more accurate. Then, we return the the corner_set. Since the checkerboard is 6x9, there are 54 corners on the board, and we have that many sets of points in corner_set.
For the next task, we made a method that converts the corner_set coordinates from image-relative coordinates to coordinates relative to the real world, point_set. We defined the real-world coordinates in terms of 0-6 and 0-9, where the first corner is (0,0). This effectively means that we are measuring the world in units of checkerboard squares. After this, we store all of the corners and points in vectors corner_list and point_list, which can have even more corners and points added to them. These will hold the saved points and corners from each image. The user can add points/corners to the lists by pressing S, which also saves as an image the frame of the video at that moment. Ideally, the images saved should be taken at different angles for a more effective calibration. Here is an image of one of our calibration images:
If the user presses C, the program will use all of the accumulated sets of corners and points to calibrate the camera. By "calibrate the camera", we just mean that our program will print out the camera matrix, distortion coefficients, and reprojection error for testing purposes. For this step, we also need to give the built-in method calibrateCamera the initial camera matrix, the frame size, and data structures to store the distortion coefficients, rotation vectors, and the translation vectors it calculates. We store the rotation and translation vectors in matrices.
Our reprojection error was 0.723137.
We also enable the user to press F so that the camera matrix and distortion coefficients are written out to a text file. This info is required for the next task, in which we want to project onto a live feed. For this, we make a separate file, which reads in the text file and uses it to run the OpenCV function solvePnP, which we use to find the rotation and translation vectors. We can then use these matrices to project lines onto the chessboard.
The next step was to either draw the four outermost corners or the 3D axes (on the origin corner) onto the board. We did the latter. Here is a video of that.
After that, we had to draw something on the checkerboard. We drew a rectangular prism and fish. They are only projected if the program can find a checkerboard in its main loop. Here is a demonstration of the fish.
The last task was to find and display another feature onto objects found in the video feed. We made a separate file and found the Harris corners. The process was similar to finding the checkerboard corners, but we had to play around with the k value a lot to get the right amount of points. Here is a demonstration of that being used on pixel art renditions of a bird and a cat. If we were to project images onto these like we did with the checkerboard, it would be a matter of treating the harris corners like the corners on the checkerboard (i.e. assigning them coordinates) and getting the translation and rotation vectors. This is assuming we use a good, asymmetrical image that has corners setup in a way that would enable the program to tell how wide one block is. The origin point would just be the upper-left-most corner. Overall, the process would be very similar to that of the checkerboard, but we would just have to change the math where appropriate.
We also did extensions.
We added support for prerecorded video files as well as image files to our arSystem program. This means that we do not necessarily need a live feed from a camera anymore. The user can just specify the file type when calling the program from the command line. The main method in the program checks the extension and decides which method to use based on that. If it is an image, the program essentially does the same thing is does for a live feed, except just for one frame. For a prerecorded video, it does the same loop it does for a video, except that the VideoCapture variable now takes in the video name, and we also need to check when the video has ended to decide when to close.
Note that this task may randomly segfaults about half the time when trying to run it, but after looking into it for a while and asking Professor Maxwell, we were told that this has to do with the OpenCV installs on the machines in the robotics lab, which is something we have no control over.
Another extension we did was testing out several cameras and comparing their results. For our tasks in the project, we used the red camera the whole time. For the extension, we tested two more cameras: one with 3 lights on each side of the lens, and one that says it is 1080p. The reprojection error estimate for the former was pretty close to the red one; we got 1.10618 for it. For the other camera, we got .564699, which is a lot better than the other two. We think that this is due to it being an HD camera, while the other cameras were of lower quality. We also tried another camera, but it wouldn't even focus, so we gave up.
A last extension that we "did" was make another OpenCV program with OpenGL support in order to project 3D models into the video. As it seems like the robotics lab computers have some issue with running OpenCV with OpenGL, this is currently a WIP until we hear back from Randy.