HDR with Tone Mapping

Discussing the HDR Pipeline

For this part of the project, I used images that I took myself from my DSLR camera (Bells and Whistles for own images). I based the camera on a tripod to avoid exposure blur and motion blur through shaking the camera. Some motion blur is still visible though because objects in the images moved. I did not implement the object alignment for this project.

I started by following the MatLab code from the slides and working to translate it into Python to generate response curves for each of the images. This followed from the equation 3 in the Debevec paper on HDR. This worked by sampling points from the independent channels of the images (r,g,b) and then using linear algebra to obtain the response curve.

After obtaining the response curve for each channel, I built a radiance map for each of the channels. To visualize what the current state was, I also stacked the radiance maps to obtain an intermediate image. The radiance maps were obtained by implementing the equation 6 in the Debevec paper.

After recovering the radiance maps, we were nearly there to complete the pipeline. The two remaining steps were tone mapping with gamma correction and intensity adjustment. The tone mapping algorithm I used simply took a gamma factor and exponentiated the float (0,1) pixel value to the gamma value. The intensity asjustment then multiplied the pixel values by a factor of the average of the middle exposure image over the average of the current image.

This to me was not hugely successful so I further implemented an extended tone mapping algorithm by including a histogram equalization to the image to improve brightness and local contrasts. (Bells and Whistles).

Examples of the Algorithm at Work

In the following section, I display some of the results of my work. I also display the intermediary steps for certain images to show the full pipeline at work.

A Scene from my Bedroom

First I show an example of the text file which holds the exposure lengths of the images. I take the reciprocal in the program to simplify storage of values.

Next, I display the four source images used.

Next up are the three recovered radiance maps for each of the color channels, [red, green, blue]. For reference, the water bottle and the notebook on the table are both blue. The fourth image is the radiance map stacked.

To wrap things up, the tone mapped and intensity adjusted image.

And finally, the second tone mapping with histogram equalization.

What is really quite neat to see is that the color of the clothes on the bed is not visible at all in any of the original frames, they are significantly darker due to the lack of dynamic range. However, in the HDR image, we can make out the purple color of the sweatshirt and some other clothes as well. Along with some detail on the lanyard and the presence of the headphones.

A second example, with all the intermediate steps.

A Scene from my Living Room

Sources

Radiance Maps

Tone Mapped Results

Some more examples, no intermediate steps.

Sidewalk

Sources

Outputs

Campus from Afar

Sources

Outputs

Trees

Sources

Outputs

Roommate

Sources

Outputs

Seam Carving

Discussing the Seam Carving Process

My second project chosen was the seam carving and insertion (Bells and Whistles).

This was a dynamic programming problem, so my implementation was rather memory expensive due to keeping track of various array states throughout the project. I started by loading the image and calculating an energy function on each channel and summing the energies of each channel to create an energy map. I experimented with two different energy functions (Bells and Whistles), the Scharr gradient and the Sobel gradient to see if they made a difference on the chosen seams (they did, discussed later).

Scharr Matrices

Sobel Matrices

Removing a seam involves taking the energy map, calculating a cost function across the elements of said energy map, and then selecting the min and traversing up the rows. To remove rows, I used np.rot90 to rotate the image, apply the same remove function, and then rotate back.

The next step after finding the seam was to identify each pixel in the seam and remove it from the output to build a smaller image. To remove multiple seams, simply repeat this process as many times as necessary.

After accomplishing the seam removal, the next attempt was for seam insertion (Bells and Whistles). The technique here is to calculate the seams that would have been removed from the image for an n seam removal, and then insert exact copies of those seams to expand the image. Examples and visualizations are shown in the following section.

A Detailed Example First

Mt. Hood in Oregon, Photo by Self, Size=(500, 333)

The Scharr energy function applied to the image.

Removing 50 seams from the image. Size=(450, 333)

Adding 50 seams to the image. Size=(550, 333)

The Other 5 Examples, Including Horizontal Removal and Energy Comparison

Berkeley Landmarks During 2016 Big Game Size=(500, 333)

The Scharr energy function applied to the image.

Removing 10 seams vertical, 10 seams horizontal from the image. Size=(490, 323)

A Group of People

Removing 50 seams from the image.

A Beach Town, Two Energy Functions Shown

The Scharr Energy Function, seams selected, and output

The Sobel Energy Function

Inserting 20 to the Image Vertically

Crater Lake with a Person (Me)

Inserting 50 seams to the image

Removing 150 seams from the image

A Not so great Example: Vertically adding to the Glade

Concluding Both Projects

Both of these projects along with the Bells and Whistles were rather time consuming and difficult to work on alone, but I learned a lot about the nature of computational algorithms for tone mapping and how many of the algorithms we interact with on a daily basis work. In seam carving, although I was unable to implement an interactive version, I learned about the dynamic sized images and how those can be configured to provide elasticity in use cases. HDR was incredible to see close approximations of what our phones can do in mere seconds.