Illumination Mac Guff Meets Growth and Performance Challenges with Avere
“Real-time looking glass” analysis and vendor-neutral FXT Edge filers are an enduring hit for powerhouse French animation studio
When Illumination Entertainment began working on animated feature films in 2004, it produced one animated feature film per year. But cinema audiences have fallen in love with animated movies, generating demand that film studios are happy to try to meet. For Illumination Entertainment’s wholly-owned, Paris-based animation studio, Illumination Mac Guff, this means a 2016 production schedule of four films per year.
This represents a huge, growth-led IT challenge, especially for its storage infrastructure – one that has been successfully and successively addressed with Avere solutions for five years.
At Illumination Mac Guff, teams of artists create incredibly complex scenes frame-by-frame, employing their skills to create ever more dazzling sequences. Each frame is incredibly complex; every surface must be correctly textured and lit. It’s a highly skilled role, requiring immense concentration and as few distractions as possible. Following their train of thought instinctively, every artist needs to be able to open and use any file immediately; any delay that an artist experiences in opening a file, even of half a second, is both frustrating and a distraction that provides an opportunity for error or omission. And, of course, it does nothing for productivity.
To maximize studio output and resource utilization, another feature of the process is the immediate rendering of each frame as soon as it is finished and approved. The result: tens of thousands of rendering cores working on the same files to create the next flat image in astonishing detail, often at the same time as artists are creating and saving new frames within the same storage environment.
The constant opening and saving of frames and texture files, by artists and rendering farm alike, places intense demands on the storage arrays that underpin Illumination Mac Guff’s extraordinary capabilities.
In 2012, when the studio was working on major features such as Despicable Me and The Lorax, Illumination Mac Guff first recognized that it needed to address the disconnect between the number of spindles in its storage network, the volume of work, and the performance. Its HDS array was coming under huge strain, and the approach of purchasing new, high-performance storage hardware every time the storage array disks reached capacity had to be brought to end.
Choosing a solution
Bruno Mahe, head of technology at Illumination Mac Guff, considered several solutions to solve these problems, and ultimately chose Avere Systems FXT Edge Series filers. He cites a number of reasons that illustrate the benefits of Avere’s solution, and how it differs from alternatives.
“We felt that the FXT Edge Series filers would immediately resolve the I/O problem without adding more spindles. The immediate aim – to protect the artists – would quickly be achieved,” he states.
“But it was Avere’s ability to provide real-time, visual analysis that showed where the hot data was within the storage system as a whole that really impressed me. Before testing we didn’t know it would be such a tremendous help to production, but it was one of the aspects that sealed the deal.”
Illumination Mac Guff also wanted a vendor-agnostic solution, ensuring it could satisfy future production needs without being locked into one hardware manufacturer – something else that Avere’s solutions ensure.
Avere’s partner-led approach
French value-added reseller NewNet, the storage specialist that had introduced the existing Hitachi Data Systems (HDS) BlueArc storage arrays to Illumination Mac Guff, was called in to deploy the Avere solution. NewNet worked with Avere to fine-tune the system and balance the workloads placed on the storage.
After the NewNet team and Avere experts completed their installation, the system was brought online and there was an immediate, dramatic change. Before the Avere Edge filers were deployed, multiple flashing lights indicating heavy disk utilization could be seen on the HDS drives. But afterwards, only occasional blinks illuminated, because so much I/O was offloaded onto the Avere system.
The benefits of analysis
The next stage was to adjust the loading so as to strike the right balance between the requirements of the artists and the rendering processes.
Mahe explains: “The artists use fewer storage resources than the rendering farm, but they are more sensitive to resource contention. If they have to wait half a second to access the storage array, it creates a huge productivity issue. But a 24-hour rendering process isn’t affected too much if it takes a few milliseconds more to access to a texture file.”
Positioning the FXT Edge devices in front of the storage arrays meant the FXT cluster sees everything passing across the network. The analytics software played a major role by visualizing the flow and instantly highlighting where I/O contention bottlenecks were occurring. Rather than trawling through columns of figures, relying purely on experience, and playing hunches, the installers could see an I/O profile, allowing them to apply their skills more efficiently. This sped a successful implementation by eliminating trial and error processes – saving both time and money.
Mahe said: “Avere helped diagnose where the issues lay in our storage infrastructure: whether it was in the storage stack, in the network or elsewhere.”
Prior to the installation of the Avere FXT Edge filers, Illumination Mac Guff was inhibited by a network connectivity problem with the HDS equipment. Mahe recalls that there was no way to isolate such an important part of the network to investigate the problem without interrupting work. Instead, diagnosis and resolution took two weeks and involved building a “sandbox” to reproduce the problem. The sandbox allowed access to the data flow without interrupting the daily production processes.
“If we had had Avere and its analysis tools at that time, the two-week problem would have been resolved in two minutes by just looking at the GUI,” he said.
A solid solution for ongoing cost-reduction
In the pre-Avere environment, every time the storage array disks reached capacity, new high-performance storage hardware had to be purchased and the balance between operational SAS drives and near-line disks for archiving needed to be adjusted.
But offloading the I/O from the disk system to the FXT Edge filers resulted in important savings, both by avoiding frequent upgrades to the storage system and by providing Mahe with a strong, ongoing negotiating position whenever additional storage is required. “When dealing with hardware vendors we are in a much better position since we barely need to care about performance differentials of different disk storage options,” he said.
Integration with the wider storage strategy
Illumination Mac Guff employs a tiered data storage strategy. Even when a film has been released, the company keeps movie data in near-line storage to facilitate requests for promotional material.
This has led the company to track Avere’s development of its FlashMove® and FlashMirror® capabilities with great interest. Mahe explained, “Right now, when we move files externally between storage tiers, we have HDS for the main production, another storage vendor for near-line archive, and then there’s an even slower system for static files. The best tool we have had is the HDS duplication feature. It’s great, it’s reliable, and it’s fast, but we can only use it with HDS equipment. Avere FlashMove and FlashMirror offer alternatives to this as they are a vendor agnostic replication toolset.”
Overall, Mahe says that he could never go back to the old system of adding additional disks to a traditional storage system to scale performance, it just would not work and it’s cost prohibitive. He now has a working environment that suits all the players, plus the ability to see everything that is happening within the storage environment through the analytics capability provided by Avere – a real bonus.
“We used I/O profiling before, which was client based, but we didn’t have the overview of the complete storage network that Avere has given us, which is proving to be exceptionally helpful. It has shown us hot spots in real-time, which has been a major help with production. Now we have this real-time looking glass, which does not impact production and that’s of great value.”
The improved performance of Avere’s FXT Edge modules and the ability to continually monitor and adjust throughput using the visual analysis tools has resulted in a significant increase of throughput for Illumination Mac Guff for a relatively small expenditure outlay.
FXT 5000 Series: Upgrading the Avere solution
Since 2010, the image quality of animated movies has improved by a factor of four – which, along with the intensive production schedule, has extended the demands on Illumination Mac Guff’s storage infrastructure even further.
As a result, Illumination Mac Guff upgraded to the latest and fastest FXT Edge filers to ensure its continued success.
Bruno Mahe explains: “Avere’s solution takes all the potential issues out of the equation; its FXT Edge filers might as well have been designed specifically for us. No architectural changes were needed with the upgrade – Avere scaled up very nicely and without disruption – and, as before, the alternative of purchasing lot of expensive, high-performance NAS storage would have been financially unviable. Avere was our solution-of-choice in 2012 and it remains our solution-of-choice today.”
About Illumination Macguff
Illumination Mac Guff is one of the world's leading studios for animated feature films. Based in Paris, the company is known for such global blockbusters as Despicable Me, Despicable Me 2, The Lorax, Minions and The Secret Life of Pets – all produced by Chris Meledandri's Illumination Entertainment for Universal Pictures. The studio is currently in production on more films, including Despicable Me 3 and Dr. Seuss’ How the Grinch Stole Christmas.