Colorizing Old B&W Photos and Videos With the Help of AI

14,590

181

27

About: Engineer, writer and forever student. Passionate to share knowledge of electronics with focus on IoT and robotics.

This project is based on a research work developed at the University of California, Berkeley by Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful Image Colorization.

The idea of this tutorial will be to develop a fully automatic approach that will generate realistic colorizations of Black & White (B&W) photos and by extension, videos. As explained in the original paper, the authors, embraced the underlying uncertainty of the problem by posing it as a classification task using class-rebalancing at training time to increase the diversity of colors in the result. The Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images.

Here a photo shoot on 1906, showing one of the first tests with Santos Dumont's plane "14-bis" in Paris:

And its colorized version using the models developed using these AI technics:

The same technics could be applied to old videos. Here a 1932 B&W footage of the city of Rio de Janeiro, Brazil:

And the colorized version:

Step 1: Lab Color Space

Usually, we are used to coding a color photo using the RGB model. The RGB color model is an additive color model in which red, green and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue.

But, the model that will be used on this project is the "Lab".

The CIELAB color space (also known as CIE L*a*b* or sometimes abbreviated as simply "Lab" color space) is a color space defined by the International Commission on Illumination (CIE) in 1976. It expresses color as three numerical values, L* for the lightness and a* and b* for the green–red and blue-yellow color components.

The color space L * a * b * was created after the theory of opposing colors, where two colors cannot be green and red at the same time, or yellow and blue at the same time. CIELAB was designed to be perceptually uniform with respect to human color vision, meaning that the same amount of numerical change in these values corresponds to about the same amount of visually perceived change.

Unlike the RGB color model, Lab color is designed to approximate human vision. It aspires to perceptual uniformity, and its L component closely matches human perception of lightness. The L component is exactly what is used as input of the AI model, that was train to estimate the remained components, "a" and "b".

Step 2: The AI (Deep Learning) Process

As commented on the introduction, the Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images. In other words, millions of color photos were decomposed using Lab model and used as an input feature ("L") and classification labels ("a" and "b"). For simplicity let's split in two: "L" and "a+b" as shown in the block diagram:

Having the trained model (that is available publically), we can use it to colorize a new B&W foto, where this photo will be the input of the model or the component "L". The output of the model will be the other components "a" and "b", that once added to the original "L", will return a full colorized photo as shown here:

In short, using a broad and diverse set of objects and scenes dataset of 1.3 Millon photos from ImageNet and applying a Deep Learning algorithm (Feed-Forward CNN), final models were generated and are available at:

Zhang et al - Colorful Image Colorization - models

Step 3: The Working Envirolment

The first thing to do is to organize an environment where we will work. Let's create a folder and name it:

  • Photo_Video_Colorization

Under this main created directory, let's create sub-folders:

  • model
  • input_images
  • input_videos
  • colorized_images
  • colorized_frames
  • colorized_videos

Go to Zhang et al - Colorful Image Colorization - models and download the 3 files and load them at the created subfolder "/model". The files are:

  • colorization_release_v2.caffemodel
  • colorization_release_v2_norebal.caffemodel
  • colorization_release_v1.caffemodel

I will assume that you have Python (version 3.6) and OpenCV (4.0) installed in your machine. We will describe step by step all the process of colorization using Jupyter Notebooks. I recommend that you follow my explanation, but if you want, you can go and download the notebooks and test photos from my GitHub:

Photo_Video_Colorization

I also recommend that you review the great tutorial " Black and white image colorization with OpenCV and Deep Learning" by Dr. Adrian Rosebrok, that was the inspiration and guide for this project.

Step 4: Black & White Photo Colorization

Each one of the following steps is a specific cell at Jupyter notebook.

1. Import important Libraries:

import numpy as np
import matplotlib.pyplot as plt
import cv2

2. Define Image to be colorized:

IMAGE = "soldiers_1941"

Note: You can use any photo here. In this case, I use a 1941 B&W photo of soldiers at WWII. The photo is available on my GitHub.

3. Define Model Paths:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
image =  "./input_images/"+IMAGE

4. Load serialized black and white colorizer model and cluster:

net = cv2.dnn.readNetFromCaffe(prototxt, model)
pts = np.load(points)

5. Add the cluster centers as 1x1 convolutions to the model:

class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")]

6. Load the input image, scale it and convert it to Lab:

Note that we will first convert the image to grayscale. This step is not really necessary, but I realize that some B&W photos, especially the old ones, could have some treatment during the years, so, better to clean them a little.

image = cv2.imread(image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)

At this point, we have our original image, but in order to show it straight on a Jupyter cell, we should use a pyplot library :

plt.imshow(image)
plt.axis('off');

7. Extracting "L":

Now, let's take our " image" and proceed with the colorization process, where first should be rescaled, converted to Lab in order to extract the component "L" and center it:

scaled = image.astype("float32") / 255.0
lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)
resized = cv2.resize(lab, (224, 224))
L = cv2.split(resized)[0]
L -= 50

8. Predicting "a" and "b":

net.setInput(cv2.dnn.blobFromImage(L))
ab = net.forward()[0, :, :, :].transpose((1, 2, 0))
ab = cv2.resize(ab, (image.shape[1], image.shape[0]))

9. Creating a colorized Lab photo (L + a + b):

L = cv2.split(lab)[0]
colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)

Same as we did with the grayscaled image, let's check how the colorized image looks like:

plt.imshow(colorized)
plt.axis('off');
plt.title('colorized LAB image');

Ops, seems that Lab image cannot tell us much, let's convert it to RGB and see the result:

10. Converting to RGB:

colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2RGB)
colorized = np.clip(colorized, 0, 1)
colorized = (255 * colorized).astype("uint8")
plt.imshow(colorized)
plt.axis('off');

Uau! Pretty amazing!!!!! This is a 1941 photo that seems to really be shot in full colors! Let's save it:

11. Saving the final RGB photo:

cv2.imwrite("./colorized_images/Color_"+IMAGE, cv2.cvtColor(colorized, cv2.COLOR_RGB2BGR))

One more: Charles Darwin visiting Rio in 1832:

Great!!!! Try with other B&W photos and see how amazing Artificial Intelligence can be!

Step 5: Colorizing Videos

Once we colorized photos, colorize videos is not a complicated task. We must follow the following general steps:

  1. Get B&W footage and load it on input_video/ subdirectory
  2. Read the video one frame at a time
  3. Having a single frame, apply what we did for a photo
  4. Having a colorized frame, save it on another subfolder: colorized_video_frames
  5. Close the OpenCv windows.

Let's do a real case:

Download the notebook B_W_Video_Colorization.ipynb from my GitHub.

The first test that I did, was to download from Youtube a B&W movie, in this case :

For that, I used the free tool: VidPaw.

Applying the steps described above, at the end we will have all colorized frames stored on the subfolder colorized_video_frames. Let's do it:

1. Start defining the file that should be colorized (that should be at the input_video folder:

VIDEO = "rio_32.mp4"

2. Define Paths, constants, and video variables:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
video =  "./input_video/"+VIDEO
width = 500
vs = cv2.VideoCapture(video)

3. Load and prepare models:

net = cv2.dnn.readNetFromCaffe(prototxt,model)
pts = np.load(points)
class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")] 

4. Split video, frame by frame and apply model:

count = 0
success = True
while success:
	success, frame = vs.read()
	if frame is None:
		break

	frame = imutils.resize(frame, 500)
	frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
	frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2RGB)
	scaled = frame.astype("float32") / 255.0
	lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)

	resized = cv2.resize(lab, (224, 224))
	L = cv2.split(resized)[0]
	L -= 50
    
	net.setInput(cv2.dnn.blobFromImage(L))
	ab = net.forward()[0, :, :, :].transpose((1, 2, 0))

	ab = cv2.resize(ab, (frame.shape[1], frame.shape[0]))
	L = cv2.split(lab)[0]
	colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)

	colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR)
	colorized = np.clip(colorized, 0, 1)
	colorized = (255 * colorized).astype("uint8")

	cv2.imshow("Original", frame)
	cv2.imshow("Colorized", colorized)
    
	cv2.imwrite("./colorized_video_frames/frame%d.jpg" % count, colorized)
	count += 1
	key = cv2.waitKey(1) & 0xFF

	if key == ord("q"):
		break

vs.release()
cv2.destroyAllWindows()

The above loop process, usually takes a while. For example, the process of colorization of this video (8 minutes) having around of 14,000 frames took me around 3 hours on a MacBook Pro - 2.9GHz Core i7 with 16GB 2133MHz of RAM.

5. Once you have the file with the frames, you must "re-assemble" it to create a video. The bellow function can do it:

def convert_frames_to_video(pathIn, pathOut, fps):
    frame_array = []
    files = [f for f in os.listdir(pathIn) if isfile(join(pathIn, f))]
 
    #for sorting the file names properly
    files.sort(key = lambda x: int(x[5:-4]))
 
    for i in range(len(files)):
        filename=pathIn + files[i]
        #reading each files
        img = cv2.imread(filename)
        height, width, layers = img.shape
        size = (width,height)
        print(filename)
        #inserting the frames into an image array
        frame_array.append(img)
 
    out = cv2.VideoWriter(pathOut,cv2.VideoWriter_fourcc(*'MJPG'), fps, size)
 
    for i in range(len(frame_array)):
        # writing to a image array
        out.write(frame_array[i])
    out.release()

Note that depending on what Video Controler you have on your machine, the codec (*'MJPG'), should be changed. Please check with OpenCV documentation. At end would a "try&error" experience.

Now, it only to apply the function on the colorized frames:

pathIn= './colorized_video_frames/'
pathOut = './colorized_videos/color_rio_32.avi'
fps = 30.0
convert_frames_to_video(pathIn, pathOut, fps)

The resulting "raw" video can be seen here:

Note that the video obviously has no sound. What I did was to strip the original sound from the B&W movie and add it to the colorized one, using iMovie. The result is here:

That's it! Very cool, isn't? ;-)

Step 6: Homage to Santos Dumont

Taking the opportunity to colorize old photos and videos, I decided to honor one of the great inventors of last century, Alberto Santos-Dumont.

Santos-Dumont was a Brazilian inventor and aviation pioneer, one of the very few people to have contributed significantly to the development of both lighter-than-air and heavier-than-air aircraft.

The heir of a wealthy family of coffee producers, Santos-Dumont dedicated himself to aeronautical study and experimentation in Paris, where he spent most of his adult life. In his early career, he designed, built, and flew hot air balloons and early dirigibles, culminating in his winning the Deutsch de la Meurthe prize on 19 October 1901 for a flight that rounded the Eiffel Tower. He then turned to heavier-than-air machines and on 23 October 1906, his 14-is made the first powered heavier-than-air flight in Europe to be certified by the Aéro-Club de France and the Fédération Aéronautique Internationale. His conviction that aviation would usher in an era of worldwide peace and prosperity led him to freely publish his designs and forego patenting his various innovations.

Above you can appreciate some of the photos from the first decade of XX century that were colorized using the technics described in this tutorial. And below, a video collage is done. I hope you appreciate it.

Step 7: Conclusion

As always, I hope this project can help others find their way into the exciting world of technology!

For details and final code, please visit my GitHub repository: MJRoBot-Python4DS

Saludos from the south of the world!

See you in my next instructable!

Thank you,

Marcelo

Remix Contest

Participated in the
Remix Contest

3 People Made This Project!

Recommendations

  • Frozen Treats Challenge

    Frozen Treats Challenge
  • Pets Challenge

    Pets Challenge
  • Backyard Contest

    Backyard Contest

27 Discussions

1
None
my_grove

Question 6 weeks ago on Introduction

I have followed this script carefully and all seems to be fine down the the point:

net = cv2.dnn.readNetFromCaffe(prototxt,model)

Then it throws an error:

[libprotobuf ERROR D:\Build\OpenCV\opencv-4.1.0\3rdparty\protobuf\src\google\protobuf\text_format.cc:288] Error parsing text-format opencv_caffe.NetParameter: 7:1: Expected identifier, got: <
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.1.0) D:\Build\OpenCV\opencv-4.1.0\modules\dnn\src\caffe\caffe_io.cpp:1151: error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse NetParameter file: ./PVC/model/colorization_deploy_v2.prototxt in function 'cv::dnn::ReadNetParamsFromTextFileOrDie'

Seems to be looking for a file at D:\Build\OpenCV\ but I don';t even have such a drive on my pc. Is something hardcoded somewhere that should noty be?

Thanks


4 answers
0
None
webmastermanmy_grove

Answer 4 weeks ago

I'm having the same issue. Did you figure this out?

0
None
webmastermanwebmasterman

Reply 4 weeks ago

I figured this out. We aren't instructed to download the files colorization_deploy_v2.prototxt and pts_in_hull.npy to the model directory. After doing this and adding .jpg to my image name and updating the paths to my actual directory all is well and works great. I put my working folder under c:\source so my path is (using windows) "/source/Photo_Video_Colorization/model/colorization_deploy_v2.prototxt" as an example.

0
None
my_grovewebmasterman

Reply 4 weeks ago

Thank you. I'll follow you additional notes ansd see what happens.

0
None
webmastermanwebmasterman

Reply 4 weeks ago

Also don't forget to put your base image in the input_images folder.

0
None
Asdrubalfarina

4 weeks ago

Hello. I would like to do this. Can you please make a video tutorial for this ? I get lost... thank you !

1
None
motooka

3 months ago

Olá, Marcelo, excelente seu post!!
I'm trying to follow your instructions, but couldn't even download the models you mentioned. Can you help me? (sorry for being so dumb)

4 replies
1
None
motookadubreg

Reply 2 months ago

thank you very much.
here are the results I got:

casamento_2.jpgColor_casamento_2.jpg
0
None
redbeardtrev

2 months ago

A nice demonstration of the power of a well trained Neural Network.
but I still prefer to use the one I was born with, especially for video as this one makes up its tiny mind on a frame by frame basis what colour something should be, I found the flickering colours on the sugar loaf mountain very distracting, and basically stopped watching at that point.
I am aware it was trained for still images, and does a pretty good job there.

0
None
nemoz2017

Question 3 months ago

Thanks for this really cool project, I want to use it to colorize a lot of old family photographs.
I have Python 27 running on a Win 10 box. Will this suite run with v2.7 ? What about the libraries, especially OpenCV?
If I run it on my Linux machine, do I need Python 3?
Thanks for your help

0
None
kmpres

3 months ago

Fantastic! I have aviation history in my family and have an archive of aviation photos from WW1 and the 1920s. I wrote a 500 page book on the subject (but because I don't know if it is fair to advertise in these comments so I will only respond privately to requests for information). But pursuant to those photos was a question asked by the curator of the Deutches Museum in Munich on the exact color scheme of the only Dornier Libelle II left in the world which they possess. The plane once belonged to my grandfather and we have numerous photos of him on or in it, but no one, including the museum, knows what the original color scheme was. Dornier did a beautiful job of restoring the plane but painted it grey because that was the color in the photos they used for reference. Can this AI program bring back the original colors of this long lost airplane for us?

2 replies
0
None
mjrovaikmpres

Reply 3 months ago

I did a test with some photos that I found on the internet. The best result seems to be this one attached. Unfortunately the photo quality is low. I got the impression that fuselage is white with wings and motor gray:

libelle-1-2-big.jpgDornier_libelle_color.jpg
0
None
mjrovaikmpres

Reply 3 months ago

Good evening. I am very happy that you like the tutorial. I really do not know if can get the exact color, but I think that could be a real possibility. We can can make tests on some photos from different angles, time of the day, etc. Also, maybe we can make some kind of verification, colorizing some photos of other planes from the same period, but that you know their color at the time that the photograph were shoot. I will be more than happy to help you. My email is: rovai@mjrobot.org
Also, please tell me about your book. I love aviation history. I have had the opportunity to be twice in Munich and the second time was only to visit again the Deutsch museum. In my opinion, the best scientific and historical museum in the world (below a photo of me at the museum a few years ago).
Regards
Marcelo Rovai

0E665D64-7CE6-4704-954D-F8CB086F6E50.jpeg
0
None
Pikitote

3 months ago

Absolutely marvelous.
Congrats.