Video Version of this Article
Photo/Video: ‘The Mandalorian’ and StageCraft/Disney/Hollywood Insider YouTube Channel
If you’ve seen ‘Pulp Fiction’ or ‘Austin Powers’, then you’ve seen parodic examples of rear projection, the old cinematic technique wherein a moving background is projected through a screen behind the actors, often for driving scenes. It’s largely been replaced by chroma key compositing, but the basic premise of the concept seems logical and intuitive–why not use pre-existing footage to create background images that would otherwise be difficult, expensive, or impossible to produce?
Of course, all of the minor, technical hiccups accumulate–classic instances of rear projection, dating back to 1930 and ubiquitously used until the ‘60s, appear cheesy and visibly outdated to modern sensibilities. Backgrounds appear washed out due to being projected through a screen, and accurately matching the foreground with the background is no easy feat–how do you create dynamic foreground lighting that perfectly matches with the background, or cause the prop car to bump and sway with each corresponding movement of the image?
The invention of Scotchlite, a type of retroreflective sheeting developed by 3M, would lead to a new technique called front projection, prominently utilized in ‘2001: A Space Odyssey’ and ‘Superman’, that had a number of advantages over rear projection–not only was the image sharper and more saturated, but the whole set-up also occupied less space. However, it was digital compositing that ultimately replaced in-camera projection techniques as the industry standard for VFX, and “green screen” has become a household term.
‘Interstellar’–The Modern Return to In-Camera VFX
One of the main differences between digital compositing and older projection techniques is that projection techniques are done “in-camera,” meaning the visual effect exists on the original negative or recording. Green screen is capable of many things, but one definitive shortcoming is its inability to authentically interact with the actors in front of it–on the set of ‘The Hobbit’, for instance, Sir Ian McKellen cried with frustration and lamented, “This is not why I became an actor.”
Christopher Nolan, known for his preference of practical effects over CGI, would go to impressive lengths to ensure his actors wouldn’t be stuck with a green screen throughout the production of ‘Interstellar’–revamping the front projection process, Nolan and VFX company DNEG (then known as Double Negative) wrapped a projection screen 30 feet in height by 300 feet in length around the main set of the Endurance spacecraft, as well as two additional screens around smaller spacecraft. Using Barco projectors attached to a forklift, the background image could be projected anywhere on the screen.
Damien Chazelle’s ‘First Man’, whose VFX was also handled by DNEG, utilized a similar technique to shoot its spacecraft exteriors. Unlike ‘Interstellar’, however, ‘First Man’ used a 35 feet-high-by-60-feet-diameter LED screen that formed a 180-degree semicircular background. One key advantage of this set-up was that the LED screen provided its own light which fell on the actors–the horizon of the Earth reflected in Ryan Gosling’s visor, for instance, is the reflection of the CGI animation displayed on the LED.
Click here to read more on Hollywood Insider’s vision, values and mission statement here – Media has the responsibility to better our world – Hollywood Insider fully focuses on substance and meaningful entertainment, against gossip and scandal, by combining entertainment, education, and philanthropy.
Jon Favreau and Industrial Light & Magic
The films mentioned above are not the only ones to have utilized LED panels in recent years–Industrial Light & Magic (ILM), the leading VFX company initially founded by George Lucas for the production of ‘Star Wars’ in 1975, experimented with their use on the production of ‘Rogue One: A Star Wars Story’.
The pixel pitch (distance between RGB pixel clusters) of the LED panels used was too large to avoid moiré and offer photo-realistic imagery, so they were instead used purely for lighting. In doing so, ILM had realized the means of using LED panels to bypass a classic technical challenge presented by both projection techniques and digital compositing–accurate, dynamic lighting.
‘The Mandalorian’ creator Jon Favreau would bring his own experience working with similar technology on ‘Jungle Book’, virtual cameras on ‘The Lion King’, and a partial video wall for the pilot episode of ‘The Orville’ (Seth MacFarlane’s Star Trek send-up). With the additional collaboration of companies like Epic Games, Magnopus, Profile Studios, and Lux Machina, ILM would invent an integrated virtual production platform called StageCraft.
‘The Mandalorian’ and StageCraft
Over half of ‘The Mandalorian’ Season 1 was shot in the Volume, which refers to the 20-feet-high-by-180-feet-circumference LED video wall that created a 270-degree semicircular background. Serving as both background and lighting, the Volume removed not only the need for digital compositing in post but also the time-consuming task of setting up lights and sets for each shot.
However, what really sets StageCraft apart from its predecessors is its incorporation of motion-capture technology and Epic Games’ Unreal Engine (the company and engine behind ‘Fortnite’)–this allows StageCraft, unlike previous techniques, to allow camera movement. But how?
The dilemma that previous filmmakers faced was that, even if the set were completely encapsulated by photorealistic digital projections, their static nature would cause inconsistencies in perspective, also known as a failure to achieve parallax. To clarify, if there’s a ball behind a tree in real life, you can walk to the right and the ball will emerge from behind the tree. With a digital projection on screen, there’s no real 3-dimensional space and so the same effect cannot be achieved.
StageCraft circumvents this limitation through a multi-step process that first involves photographic scanning of a location to create virtual environments–thousands of photos are taken from different viewpoints to generate a 3-D photographic model. These photos are combined with computer-rendered sets through a technique commonly known as tiling or photogrammetry in order to create a photorealistic background that’s camera-ready.
Finally, the 3-D environment is rendered by Unreal Engine while the camera’s position is tracked by infrared cameras–according to the camera’s angle and position, the engine renders the image with proper 3-D parallax in real-time, finally allowing for camera movement in conjunction with in-camera compositing.
Shortcomings of StageCraft and the Future of VFX
StageCraft is undeniably a miracle in video production that not only perfects the art of in-camera compositing but also eliminates much of the time-consuming processes of lighting and production design.
However, there’s always room for improvement–as advanced as it is, the technology isn’t completely there yet. The Unreal Engine is only capable of rendering a photorealistic projection for the portion of the screen that’s on-camera, and there’s a slight delay that occurs between the camera movement and parallax adjustment–this means that it can’t yet handle fast camera movements.
Another problem is that the LED panels don’t have the power to re-create the quality or intensity of direct sunlight, and so any ‘Mandalorian’ scene that required such lighting was simply shot on a Hollywood backlot.
Nevertheless, StageCraft is an incredible technology that really brings home the level of technological sophistication that’s developed in Hollywood within our own lifetimes, and it’s incredibly exciting not only to see how different projects will utilize the technology but also how it’ll evolve moving forward. StageCraft has already been used in the production of George Clooney’s upcoming Netflix film ‘The Midnight Sky’, and is reportedly to be used in Taika Waititi’s ‘Thor: Love and Thunder’ and Matt Reeves’ ‘The Batman’.
Its arrival seems particularly timely against the backdrop of the ongoing pandemic, as productions continue to suffer from lockdowns and travel bans. ILM provides “pop up” virtual production services, as was the case with ‘The Midnight Sky’, which seems like a perfect solution for any production that’s unable to travel or shoot on location.
By Daniel Choi
Click here to read Hollywood Insider’s CEO Pritan Ambroase’s love letter to Black Lives Matter, in which he tackles more than just police reform, press freedom and more – click here.
An excerpt from the love letter: Hollywood Insider’s CEO/editor-in-chief Pritan Ambroase affirms, “Hollywood Insider fully supports the much-needed Black Lives Matter movement. We are actively, physically and digitally a part of this global movement. We will continue reporting on this major issue of police brutality and legal murders of Black people to hold the system accountable. We will continue reporting on this major issue with kindness and respect to all Black people, as each and every one of them are seen and heard. Just a reminder, that the Black Lives Matter movement is about more than just police brutality and extends into banking, housing, education, medical, infrastructure, etc. We have the space and time for all your stories. We believe in peaceful/non-violent protests and I would like to request the rest of media to focus on 95% of the protests that are peaceful and working effectively with positive changes happening daily. Media has a responsibility to better the world and Hollywood Insider will continue to do so.”
More Interesting Stories From Hollywood Insider
– Want GUARANTEED SUCCESS? Remove these ten words from your vocabulary| Transform your life INSTANTLY
– Compilation: All James Bond 007 Opening Sequences From 1962 Sean Connery to Daniel Craig
– Do you know the hidden messages in ‘Call Me By Your Name’? Find out behind the scenes facts in the full commentary and In-depth analysis of the cinematic masterpiece
– A Tribute To The Academy Awards: All Best Actor/Actress Speeches From The Beginning Of Oscars 1929-2019 | From Rami Malek, Leonardo DiCaprio To Denzel Washington, Halle Berry & Beyond | From Olivia Colman, Meryl Streep To Bette Davis & Beyond
– In the 32nd Year Of His Career, Keanu Reeves’ Face Continues To Reign After Launching Movies Earning Over $4.3 Billion In Total – “John Wick”, “Toy Story 4”, “Matrix”, And Many More
stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft, stagecraft
Daniel Choi is a writer who’s currently pursuing a BA in Film & Television from New York University. With a background in amateur film production, Daniel is fascinated by how artists’ cultural backgrounds inform their work, subconsciously or not, and how that work is then perceived by different audiences across time and space. He joined Hollywood Insider to promote its mission statement of substantive entertainment journalism, and hopes to enrich readers’ understandings of cinema through insightful analysis.