Intro: ” … the Oldest New Structure in the History of This City.”
Thank you for your interest in this project. I am an American artist using my background in visual effects in the film industry as a roto/paint artist and compositor to alter and recirculate mass-media spectacles of destruction and reconstruction. In this series, I perform an act of digital removal in order to draw attention to the complicated implications of using new technology to replicate recently destroyed historical objects from Palmyra, Syria in the midst of an ongoing civil war and humanitarian crisis in the region. I collect and alter footage of the unveiling scenes of the replicated Triumphal Arch of Palmyra (which was destroyed by ISIS in 2015) along its world tour that began in London and has since traveled to New York City, Dubai, and Florence.
For this Instructable, I will break down my visual effects work and research regarding my altering of the New York City unveiling ceremony. The magic trick of unveiling a ghostly recreation is haunted by the United States’ pending refugee ban and xenophobic mistrust of people from the region from which the arch is culled. This act of removal looks to isolate the magic trick, allowing the fullness of the unveiling spectacle to circulate while making room for emptiness, hypocrisy, and loss to be revealed.
I hope to encourage others to consider digital image alteration as means for in-depth research and slowed ingestion of rapidly-generated internet-based content. I will also point to other artists using vfx in their work. I use three main pieces of software, though you could use less and and in some cases, like the Foundry's Nuke, use educational versions for free. I'll write in a non-software-specific way in order to point out the basic principles of rotoscoping and compositing at play, which can be employed in a number of digital applications.
Step 1: Project Background #1
This mode of working began in 2014 when I altered scenes from an ISIS propaganda video depicting the hand and hammer-driven destruction of Assyrian and Hatrene sculptures at the Mosul Museum in Iraq. I was drawn to the staged motions of the destroyers and the way in which their propaganda video spoke to an inherent paradox concerning viral images of destruction; while the sculptures' destroyers labeled their actions as iconoclastic, the rapid circulation of the sculptures' likenesses via Twitter, bots, and eventually global news outlets, generated more images of the sculptures than had ever been produced.
I focused on two brief shots from the video in which men tear dusty translucent tarps from the sculptures, simultaneously introducing them to a global audience while threatening and later dismantling their physical form. I digitally removed the men from both shots in an attempt to both refocus agency towards the sculptures' likenesses and separate the eternal digital union of a video that permanently links likenesses of the destroyed with that of those of its destroyers.
Step 2: Project Background #2
This series also confronts Western complicity in creating and circulating images of terror and destruction, specifically from the Middle East. One of the readings that most informed this work is Ömür Harmanşah's ISIS, Heritage, and the Spectacles of Destruction in the Global Media. Harmanşah explains that by treating propaganda videos from this terrorist group as shareable news, our networked dissemination helps spread a more consumable form of violence (against objects) than ISIS's more graphic propaganda videos of violence against humans (which largely circulate via bots and the dark web, not as nightly news). This violence against cultural heritage is a method, however, through which the group isolates and destabilizes communities.
This article led me to think about the importance of separating the destroyed and destroyer in their eternal digital dissemination, if only to spare these two-thousand year old sculptures from the narrow context of propaganda bate. Instead of removing the men from these shots, I began to remove the sculptures' likenesses from them, leaving behind empty staged gestures.
Here is a link to the database of articles I have been collecting for the series.
Step 3: Recreating the Destroyed Arch
The Monumental Arch was built sometime between 193 and 211 AD under the reign of Roman emperor Septimus Severus in Palmyra. Palmyra's temples and architecture later functioned under Christian and then Muslim populations, until under French colonial rule in the 1930s, the local population was forced to move in order for Palmyra to become an archaeological dig site. The arch was destroyed by ISIS in October of 2015 and reappeared as a replica in the heart of London less than a year later thanks to the U.K. based Institute for Digital Archaeology, who used photogrammetry and other 3d modeling methods to create a roughly one-third scale digital model that was robotically carved in Italy.
Palmyran archaeologists have risked, and in the absolutely horrifying case of Khaled al-Asaad, lost their lives to defend the rich history of Palmyra. It is by no means an insignificant act to recreate such internationally renowned architecture from the city. While the Institute for Digital Archaeology attempts to act in laudable defiance of cultural destruction, the contrast in locations and circumstances in which we the see arch brought back are hard to ignore. It’s not the defiance of ISIS’s brutal act that I focus on in the replication efforts and their subsequent unveiling videos, but rather the discomforting easiness with which we identify the object recreation as an act of uncomplicated solidarity with the Syrian people without looking at its ironic and colonial implications.
The easiness, for instance in which Boris Johnson, in introducing the replica arch, claims to "raise two digits," presumably middle fingers, to Daesh, despite leading a xenophobic campaign of isolation in the United Kingdom...or the easiness in which western technology gets put on grand display with no mention of the heroic efforts of Syria's own archaeologists' heroic acts of cultural preservation or of an ongoing civil war and humanitarian crisis in the region from which the monument existed.. or the easiness in which on American soil we call the replication an act of solidarity with the Syrian people while the richest countries in the world, most notably the United States, close our doors to Syrian refugees.
I made this work to slow down the ingestion of this replication act and to contemplate the power of both being seen and of hiding, and to have public discussions about the complicated acts of digital witnessing, sharing, and circulating. The following steps explain the postproduction steps I took to realize these videos.
Step 4: Post-Production Process 01. Creating Clean Plates
Plate is film terminology that refers to your footage. Basically, whatever footage you are using, whether appropriated or original, is called the plate. If you want to digitally remove something from that original plate, the next step is to create a clean plate- a frame from the original plate in which you digitally (I used Photoshop) paint out the thing you are wanting to remove from the entire video. For our purposes, the plate is the original unveiling ceremony pulled from Instagram thanks to video author Claire Voon. The clean plate is a frame from that video in which I painted out the arch.
The digital removal process is not one of actual pixel deletion, but instead of covering up. We'll be covering up the areas where the arch should have been with images of what is behind it. Though the footage is hundreds of frames long, I only had to correct (paint the arch out of) one of those frames in order to realistically remove it from the entire video.
Step 5: Post Production Process 02. Multiple Layers of Clean Plates.
In later steps you will be tracking the camera movement of the original footage and then re-filming the scene in digital space with chunks from your clean plate composited (digitally collaged) on top. In order to achieve a perceived sense of reality in a scene that has camera movement (like the one we're using) you will want to isolate different parts of your clean plate on different layers of your photoshop file. In our case, this means that the sky, the building behind the arch, and the SL (screen-left) and SR (screen-right) should all be on separate layers.
Here's why you want to do that...camera tracking software works by finding points of contrast in your footage. So, for example, the software might scan frame 1 of your footage and establish the corner of a window as a good point of contrast because it has a sharp edge and is a static object. If it establishes that point of contrast on frame 1, it will look for it again on frame 2, frame 3, and so on, all the way to the end of your footage. For greater accuracy, you will often look for upwards of 300 points of contrast in the same plate. Within this piece of footage, objects that are closer to the camera will produce camera tracks with greater fluctuation than objects in the background (this principle is known as parallax). In order to have each static piece of the clean plate properly track with the camera movement, it's essential that you determine which objects are at different distances from the camera and create a unique layer for each. Sounds more mathematical than it is. For the most part you'll be able to trust your intuition and see if the part of the clean plate you are tracking into your footage sticks or not (does it appear to wiggle or float when you composite it on top of your footage).
You always want to paint more clean plate than you have to on each individual layer, especially for layers that are further away from the camera. For instance, if you are working on the layer for the sky, you want to paint the sky over a good portion of the building that is in front of it. Similarly, if you are on the building layer, you want to paint a significant amount of the building over the trees. By doing this, you again allow for nuanced articulation between different layers that will pay off in being able to replicate the proper parallax. If you ever watch those tutorials on how to turn photos into animated videos, this is essentially the same technique.
Step 6: Post Production Process #03. Frame by Frame Painting
Well, there's no way around this step....if we're going to remove the arch from the shot while keeping the white sheet, there's areas of the sheet that would be otherwise obscured by the arch that we now need to paint into our scene. I am using software called Silhouette that is built for easily cloning between one frame and the next. You could also use Nuke's built-in roto/paint node, or After Effect's paint tools, or even Photoshop. Because we're working with fabric, there's a lot of guess work in how the fabric will wrinkle. Word of wisdom though, always clone when you can- it's amazing what you can get away with when cloning from one part of an image to another as opposed to just trying to digitally paint it. It's really difficult to emulate the texture and nuance of real materials with digital brushes.
Step 7: Post Production Process #04. Rotoscoping the Sheet
No one thought this would be a quick project, right? When we've completed the time-consuming tasks of painting our clean-plates and painting out the arch, we're rewarded by having to trace (otherwise-known as rotoscoping) the outline of the sheet. This is a good time to listen to a podcast or an artist lecture...perhaps from these other amazing artists working with visual effects and media archaeology:
- Josh Azzarella Untitled #9 (W.T.P.1)
- David Tinnaple Debate Breath
- Hito Steyerl How Not to be Seen: a Fucking Didactic Educational .MOV File
- Paul Pfeiffer
- Martijn Hendriks Give Us Today Our Daily Terror
- Claire Hentschker Shining360
- Cristobal Cea Ghosts
Our reasons for wanting articulated roto of the sheet are two-fold:
1. We want to tell our camera tracker not to try finding tracking points anywhere on the sheet. Camera trackers only work with static elements, and our sheet will be moving a lot during the scene. Any tracking data it gives us would be unreliable and might mess up our camera track.
2. After we composite our cleanplate fragments back into the shot, we'll want to make sure the sheet footage is put back on top of those tracked buildings. In nuke, this is called merging.
Step 8: Post Production Process #05: Camera Tracking
Now we can track the camera all the way through the shot. I usually ask Nuke to find at least 600 points and then I get rid of any tracks that show up as red or yellow (meaning that they are either erroneous or unsolved). These colors will change based on the error thresholds we assign. For instance, if we set up our error threshold pretty low and only accept the most accurate tracks- meaning that from 1 frame to the next, to the next, and so on, that there is minimal deviation from our original point- then we will have less accepted tracks but can be assured that they are pretty accurate. In footage like the one we have, we're not going to get a huge amount of good camera tracking points because 1. the footage is shaky and handheld, and 2. the arch and sheet are moving and thus not included in our tracking. Remember, I'm inputting the alpha from my rotoscoping of the arch into my camera tracker and telling it not to look for points on this. I do this because camera trackers are only effective on static objects within a shot and because the fabric on the arch is moving rapidly it will throw off the camera track.
Once we set up a scene based on our 3d camera tracking points, we can switch into our 3d view and actually see a virtual camera moving through space. This virtual camera is the software's best guess, based on the tracking points, at how the original camera, in this case an iphone, was moving through space. It is digitally re-filming your scene, and if it's movements match your original's, then you should be able to start dropping in your static composited elements in to the scene and have them look they are realistically integrated.
In Nuke you do this by creating static cameras called "projecting cameras" and projecting your clean plate on to flat surfaces called cards. You can do this in After Effects too and I'll be demonstrating how in an upcoming Instructables.
Step 9: Post Production Process #06. Compositing and Finishing.
Now that you have all of your static images and your painted arch, it's time to merge all of these layers together. In a node-based software like Nuke you use "merge" nodes to put one element on top of another, whereas layer based compositing in After Effects is based on the layer hierarchy in your timeline. There will be a lot of tweaking in this stage, and perhaps even a pre-comp render that you will add final paint touches to. But now you get to see the effect of all the steps you've completed and work towards a final piece.
Thank you and please let me know if you any questions. I'd love to help you on your projects. I teach a workshop at Open Signal in Portland, Oregon called "Fake News: An Intro to Altering Video with VFX" if you'd like to work with me in person.