Context
This was an extra undertaking as part of the med.o project to add realism to the final video demonstration. I've detailed below how I achieved this composition in Blender, covering what was unique to my project.
Filming the scene
The first step is to film the scene for composition, bearing in mind where you want to add the 3D object. In this case the object is to be put on the floor. To track an object onto video, software is used to track the motion of the video camera. The camera motion can then be recreated in the 3D software with a virtual camera. The virtual camera should move exactly the same as the real camera, meaning an 3D object made in software should look like it moves with the scene.
The get a good camera track, software tracks individual points in the video and looks at how they move relative to each other. Tracking points in the foreground and background and across the whole frame captures more motion and gives more reliable solves. Also, for better results I filmed at a higher frame rate to avoid motion blur inaccuracies. If motion blur is part of your video style, for example a 180 degree shutter, this can be added after everything is rendered.
Preparing the scene
Once the camera tracking data is created, the 3D scene can be prepared. Three tracking points on the floor can be selected to identify where the virtual ground is, and to set a scale. A plane can be added, reoriented and scaled and checked against the footage to see if it moves correctly when viewed through the virtual camera. The model of the bin is then added and lined up with the floor
Lighting
The first step is to match the lighting to the real life scene. While filming, it is good to note where the light is coming from and what interacts with it. For this scene, the row of houses created a sort of channel, with the sun shinning down the street towards the camera. I modelled the sun with a sun lamp and increased the angle / size to simulate the hazy diffusion seen. Typically world lighting can be used to model sky light (world lighting is like an infinitely large uniform sphere emitting light into your scene). However because of the rows of houses, it was simpler to add sky light with another sun lamp from the opposite direction with a large angle of dispersion.
The lights' power and colour temperature needs to be adjusted to match the scene. To help with this, I looked for objects which are similarly placed or coloured to the bin and adjusted the lighting to match. For example, the window sill and top of the wall shows the bright sun light which would also appear on top of the bin. The side of the bin towards the camera should also be similar to the wall post on the right.
Scene objects
The next step was to make the 3D bin object interact with light in the scene properly. Looking at the scene, the bin is on a gravel floor, next to a wall, a round bin and opposite a tall house front. These are all affecting the real world lighting and need to be recreated in the virtual world to do the same.
Starting with the floor, I cut a square of the gravel out of the frame and saved it as an image. I used this image to create an approximate displacement map for the ground plane so that the shadows cast by the bin fall on a bumpy surface similar to the gravel. The bin can also be slightly sunk into this so that the base of the bin appears to sit in gravel. The ground plane also creates ambient occlusion to add further realism; this is the phenomenon that makes it darker in corners and where surfaces meet each other. The colour of the gravel texture also subtly bounces light back up on to the bin
Next I created a rough black bin and positioned it where the black bin was in the scene. This casts shadows on my new bin as we would expect in the real world. I also added the house front as a vertical plane to add shadows from the house and light bounces from the house onto the bin. I coloured the house plane red to match the red brick which adds a very slight redness to the light bouncing onto the bin.
Given the limited camera motion, these few reconstructions are enough to recreate the scene lighting.
Light passes
The virtual objects need to interact with the bin but not be seen themselves, this can be achieved by changing the light information the software uses. What I needed is shown in the image below, which consist of all the light for the new bin, the shadows cast on the gravel floor, the indirect lighting from the wall and the black bin cast onto the new bin, and for these objects to not be seen. To make objects invisible you can simple make them a "holdout", which essential cuts them out of the render. The two elements can be composed together. This is done with a simple overlay and enabling the bin to be rendered with a transparent background.
For added realism, the render is blurred very slightly to match the camera's imperfections. A dirt texture is also added to the bin to introduce real world imperfections.
Render output
This is however not the final step. As you can see, the bin is overlayed on top of everything, including the person it should be behind. To do this the person needs to be masked and copied in front of the bin layer. At this point I moved from Blender to DaVinci Resolve as Resolve has better video editing capabilities and masking tools. Note that not all of the person needs to be masked out, only the parts which walk in front of the bin, saving lots of time of work. The rubbish being dropped into the bin also needs to be masked so it falls in front of the back of the bin and remains behind the front of the bin.
This scene was featured in the video presentation for the med.o project which can be found here
