How do I realistically match sprite animations to a video?

I've embedded an example, below. Here are some cool effects I notice:

• the animations are matched with the shaking camera shots, frame-by-frame.
• some shots have the animations changing in focus to match the camera shots
• some shots mask the animations to make them look like they're moving behind things in the video

So, how is this done? What software tools are used? What are the different effects called?

Comments

[SplineGod]

It can be done using a combination of 2d and 3d software. The 2d software can be something like after effects, fusion, nuke etc and can be used to track the video as well as do compositing (make the items appear to move behind parts of the video). 3D or other 2d animation software can be used to create the actual animations. You can also use additional software to do motion tracking for use with your 3d software like voodoo tracker which is free.

[fungus amungus]

The first one is tracking. One way to do this is to make a green square with a few black x's on it. Identify the x's in the software and it can track the movement of the box in the frame and adjust the video for it. Have seen it in Motion.

[nagutron]

So, the green square is something I make in real life and put in the camera shot?

[fungus amungus]

Yeah, that's one example. You can also track other items in the scene as well. You can probably get away with a white rock or a stick (which would work for that whale). After Effects is likely the software you want for lots of fun stuff.

[Sandisk1duo]

that video is awesome!

[mg0930mg]

I liked it too... haha. I wish I could edit videos, I'm still learning though.

[omnibot]

Yeah .. someone's been smoking something.

[omnibot]

If I was doing this I'd do the animations in 2D separately and add them frame-by-frame to the video THEN I'd add the shaking as an after-effect to make it all seem more 3D.