Introduction: Pumpkin Lip Sync

By

Make your own pumpkin jack o' lantern lip sync videos.

There are plenty of sources to purchase premade jack o' lantern animations: atmosfx.com, thesingingpumpkin.com. There is even a pumpkin with an integrated short throw projector, Jabberin' Jack. If you want to make a custom or DIY animation there is commercial software you can utilize such as Adobe After Effects. But now you can (with less skill) make your own using free and open source software.

I wrote this application in HTML and JavaScript. You can customize the images as you (and your talent) see fit. It utilizes an open source software package for analyzing the audio to generate an XML file of "mouth shapes", that the HTML can than load to play the audio and move the mouth accordingly. The HTML application lets you adjust/override the mouth shapes, and make the eyes move; so you can fine tune the animation.

You can either play the animation directly from you computer projected onto a pumpkin, or use a video capture tool to convert into a video that can be loaded onto your projector.

I also provided a face for animating a snowman, in addition to a jack o' lantern. Why is there a bird (cockatiel) face? My pal Launchpad has dreams of being an opera singer. (actually early test, and it amused me).

Supplies

Step 1: Record Your Audio for the Animation

You can use the audio recorder built into Windows to prepare some jokes or stories for your pumpkin to tell.

Or you can find a song for your pumpkin to sing.

Convert the audio to a WAV file.

If you're using a song, use a tool such as Acapella-Extractor.com to separate the music and vocal audio tracks.


Extract vocals from music (for better lip sync processing)

https://www.edityouraudio.com/acapella-generator (didn't work for me)

https://www.acapella-extractor.com/ (limit of two per day)

https://www.lalal.ai/

You may still need to remove/flatten sections of audio that were missed. You'll need to download a sound editor for that.


Convert both copies of the MP3/MP4A to WAV

https://convertio.co/


Step 2: Run Audio Through Rhubarb

Using the command line have Rhubarb analyze the vocal audio track, and generate a JSON file of the mouth shapes.


Install Rhubarb Lip Sync

https://github.com/DanielSWolf/rhubarb-lip-sync


Process WAV file (w/ vocals only) creating JSON output (used for English recordings)

rhubarb -o [target filename].json -f json [vocal audio filename].wav

rhubarb -o barber.json -f json baber_largo_farctum.wav


Process WAV file (w/ vocals only) creating JSON output, using phonetic interpretation (use for non-English recordings)

rhubarb -r phonetic -o [target filename].json -f json [vocal audio filename].wav

rhubarb -r phonetic -o Bella-Ciao-phonetic.json -f json La-Casa-De-Papel-Bella-Ciao-_Money-Heist_.wav


Step 3: Modify the JSON File

Modify the JSON data.metadata.soundFile to point to the original/combined audio file (vocals and music),

Using a text editor (Notepad, Notepad++, etc) modify the JSON file so the path of the audio is relative to where you will be storing it.

If use used a song that was split into music and vocals, modify the filename to the original/combined WAV audio file (both the music and vocals).

Step 4: Prepare the Initial Animation

Open the HTML Lip Sync application.

Select the JSON file that Rhubarb generated (and you modified), and press "Load".

Select the face you want to use for the animation.

Test it with the play button.

Step 5: Customize the Animation

While playing/previewing the animation, decide if the mouth shapes look correct. Remove unnecessarily mouth movements (mouth is moving when there is no talking or singing), or select different ones as you see fit. Sometimes for singing or shouting you may want the mouth open wider.

Select different eyes at times, to the jack o' lantern appears to be blinking.


You can also design your own faces/mouths to use for the animation. Just follow the guide lines on the Rhubarb site to ensure you provide all the possible mouth shapes (six to nine different shapes). Create a separate folder, name each image according to the mouth shape it represents. You'll need to go into the HTML and add the new folder name to the <select id="sltFace">

<option value="[directory name]">[Description name]</option>


Note: If customizing using a person or pet for the images, try starting with a video and saving individual frames for each referenced shape. This will help ensure they were framed similarly. Preventing jerky animation.


The editor has various features/functions available:

  • Preloads images into DIVs (basically cache the images), for smoother playback
  • Export/Save current JSON to a file, to save the current changes
  • Pause/Resume button
  • Forward and reverse button
  • Status indicator
  • Insert/Delete intervals
  • Delete
  • Using current time decide next closest range to merge with
  • Insert
  • Using current time, copy range. Old range end=current time. New range start=current time. new range end=old range end.
  • Adjust audio speed/pitch

Step 6: Capture/Record the Playback (optional)

Using a video screen capture tool (one that supports audio), record the playback of the animation. This will make playback easier later. You can take multiple animation sequences and make a playlist to cycle through.

ScreenCastify plugin for Chrome

  • Can capture the video and audio (check the additional options for enabling "Tab Audio")
  • Can crop the video in their editor
  • Export to MP4 video format (my projectors support this format on USB drives)

Step 7: Project on a Pumpkin

There are various tips available online for how to hide/disguise the the projector; to maintain the illusion.


Step 8: FUTURE/TO DO

This application is functional, but still in early stages with room for improvement. 

  • More variety of faces.
  • Modify the application to support three pumpkins simultaneously, perhaps with separate sound tracks.
  • Add support for additional facial animations.
  • Output directly to a video file, eliminating the need for screen capture tool.
  • Perhaps utilize BootStrap API to make the buttons look nicer.
  • Implement an "Undo" feature in the editor. (Keep saving mouthCues array to another array of objects, and restore as needed.)
Halloween Contest

Runner Up in the
Halloween Contest