HDR with Tone Mapping, Seam Carving and Insertion
For this part of the project, I used images that I took myself from my DSLR camera (Bells and Whistles for own images). I based the camera on a tripod to avoid exposure blur and motion blur through shaking the camera. Some motion blur is still visible though because objects in the images moved. I did not implement the object alignment for this project.
I started by following the MatLab code from the slides and working to translate it into Python to generate response curves for each of the images. This followed from the equation 3 in the Debevec paper on HDR. This worked by sampling points from the independent channels of the images (r,g,b) and then using linear algebra to obtain the response curve.
After obtaining the response curve for each channel, I built a radiance map for each of the channels. To visualize what the current state was, I also stacked the radiance maps to obtain an intermediate image. The radiance maps were obtained by implementing the equation 6 in the Debevec paper.
After recovering the radiance maps, we were nearly there to complete the pipeline. The two remaining steps were tone mapping with gamma correction and intensity adjustment. The tone mapping algorithm I used simply took a gamma factor and exponentiated the float (0,1) pixel value to the gamma value. The intensity asjustment then multiplied the pixel values by a factor of the average of the middle exposure image over the average of the current image.
This to me was not hugely successful so I further implemented an extended tone mapping algorithm by including a histogram equalization to the image to improve brightness and local contrasts. (Bells and Whistles).
In the following section, I display some of the results of my work. I also display the intermediary steps for certain images to show the full pipeline at work.
First I show an example of the text file which holds the exposure lengths of the images. I take the reciprocal in the program to simplify storage of values.
Next, I display the four source images used.
Next up are the three recovered radiance maps for each of the color channels, [red, green, blue]. For reference, the water bottle and the notebook on the table are both blue. The fourth image is the radiance map stacked.
To wrap things up, the tone mapped and intensity adjusted image.
And finally, the second tone mapping with histogram equalization.
What is really quite neat to see is that the color of the clothes on the bed is not visible at all in any of the original frames, they are significantly darker due to the lack of dynamic range. However, in the HDR image, we can make out the purple color of the sweatshirt and some other clothes as well. Along with some detail on the lanyard and the presence of the headphones.
Tone Mapped Results
My second project chosen was the seam carving and insertion (Bells and Whistles).
This was a dynamic programming problem, so my implementation was rather memory expensive due to keeping track of various array states throughout the project. I started by loading the image and calculating an energy function on each channel and summing the energies of each channel to create an energy map. I experimented with two different energy functions (Bells and Whistles), the Scharr gradient and the Sobel gradient to see if they made a difference on the chosen seams (they did, discussed later).
Removing a seam involves taking the energy map, calculating a cost function across the elements of said energy map, and then selecting the min and traversing up the rows. To remove rows, I used np.rot90 to rotate the image, apply the same remove function, and then rotate back.
The next step after finding the seam was to identify each pixel in the seam and remove it from the output to build a smaller image. To remove multiple seams, simply repeat this process as many times as necessary.
After accomplishing the seam removal, the next attempt was for seam insertion (Bells and Whistles). The technique here is to calculate the seams that would have been removed from the image for an n seam removal, and then insert exact copies of those seams to expand the image. Examples and visualizations are shown in the following section.
The Scharr energy function applied to the image.
Removing 50 seams from the image. Size=(450, 333)
Adding 50 seams to the image. Size=(550, 333)
The Scharr energy function applied to the image.
Removing 10 seams vertical, 10 seams horizontal from the image. Size=(490, 323)
Removing 50 seams from the image.
The Scharr Energy Function, seams selected, and output
The Sobel Energy Function
Inserting 20 to the Image Vertically
Inserting 50 seams to the image
Removing 150 seams from the image
Both of these projects along with the Bells and Whistles were rather time consuming and difficult to work on alone, but I learned a lot about the nature of computational algorithms for tone mapping and how many of the algorithms we interact with on a daily basis work. In seam carving, although I was unable to implement an interactive version, I learned about the dynamic sized images and how those can be configured to provide elasticity in use cases. HDR was incredible to see close approximations of what our phones can do in mere seconds.