Content Archive in the Cloud Eliminates Tape Backup

California Academy of Sciences Solves Latency, Complexity and Consistency Issues with Avere & AWS

In research and development, data volumes and data archives grow quickly. When you are using scientific data to accurately and creatively depict discoveries across the largest alldigital dome in the world, an incredible IT challenge must be addressed by the production engineering team. California Academy of Sciences Morrison Planetarium regularly bring scientists, visualization experts, and engineers together using high-performance compute and storage systems and processes to accurately produce shows to amaze and educate a diverse public audience on the vast, intricate stories of the natural universe.

Michael Garza, Planetarium and Production Engineering manager, oversees the team’s production IT infrastructure. “Our job in IT is to ensure sufficiently high-speed resources to handle massive data sets—content for a single show can exceed hundreds of terabytes of scene data. We use Avere Systems storage solutions to boost rendering performance during content creation and, most recently, to automate and accelerate backup/retrieval of show data to/from Amazon public-object cloud storage. Avere solutions allow us to extend the performance of our existing on-site Core-filer storage resources, as well as take advantage of new cloud services for faster data access plus administrative efficiencies.”

Challenge: Eliminate Tape Backup Latency, Complexities, and Consistency Issues 

Garza says that the Academy traditionally has relied on off-site tape-storage services to protect production data. But backup workflows became increasingly tedious, protracted, and did not offer either the reliability or responsiveness that production teams required for show projects. “Our process had been to take completed projects off the active system and use a vaulting service to move backup tapes offsite,” notes Garza. “Unfortunately, we were spending too much time in manual tape labeling and cataloging processes. Because of data-consistency issues, we also had to make and store multiple copies of our backups. Another issue was retrieval—the vaulting service could take up to seven days to locate and return requested tapes. That latency impacted our digital artists’ ability to edit or re-use content and had the potential to delay critical project schedules.”



Solution: Avere for Seamless, On-Demand Cloud Storage

Today at the Academy, a three-node Avere FXT 3200 Edge filer cluster makes it possible for the visual effects production team to deliver more visual impact in less time and at a lower cost. The Edge filer accelerates I/O to render nodes by automatically placing active data—like texture files—on solid-state media. The Edge filers meet the demands of the complex workloads, effectively eliminating queue bottlenecks. The Avere clustered solution delivers high-speed I/O to some 400 rendering nodes while ensuring maximum responsiveness to artist workstations. Avere FXT Edge filers meet current performance requirements, protect the Academy’s investment in Core filers, and support future growth with linear performance scaling to millions of operations per second and more than 100GB-per-second throughput. With the addition of Avere FlashCloud technology to the Edge filer cluster, the Academy can now take advantage of Amazon Simple Storage Service (S3) object-based storage from Amazon Web Services (AWS). Avere Edge filers with FlashCloud software integrate object storage with legacy NAS into a global namespace to enable efficient cloud-based data storage with low-latency access. Transparently to applications and users, FlashCloud translates NFS and CIFS protocols to object-based protocols for on-demand cloud storage for any NAS application, including backup and archive systems.

Benefits: Fast Access, Automation, and Protection

Within-Minutes Access to Backup/Archive Show Content

Avere FlashCloud on FXT Edge filers permits applications to read from and write data to the Amazon cloud. FlashCloud effectively hides the latency of the WAN access, providing essential responsiveness to the Academy’s production team and applications. “Using Avere FlashCloud and Amazon S3,” continues Garza, “we’re able to provide within-minutes access to cloud-based content. That gives our artists tremendous flexibility to edit video content or to re-use data sets. Rendering applications can now access data in the cloud and complete a rendered video within a day. In the past, that same process would have added days to a project schedule while we waited to restore data from off-site tapes.” 

Automation for IT Productivity

The Academy also uses the Avere and Amazon solution to automate backups and eliminate tedious, time-consuming tape labeling and cataloging processes. For those processes alone Garza estimates a time savings of at least two hours per week. “Using Avere FlashCloud for backup to the Amazon cloud, we’ll recover more than two weeks of productivity over the course of a year.”

Scale and Protection for Data-Fueled Lifescapes

Using Avere FlashCloud to integrate public-object storage also helps the Academy protect show-data integrity and access with the Amazon service designed for 99.99% availability and 99.999999999% durability. Garza summarizes, “Avere solutions play a critical role in our ability to create, store, and protect show data such as the data-fueled lifescapes of our latest production. In this show, we’re able to create and display intricate networks of plants, animals, and other life forms so that audiences can discover the beauty, variety, and immense complexity of these relationships—from the food webs of our own estuary to the beneficial partnerships in a Douglas-fir forest community and more. Our productions routinely push the boundaries of technological storytelling. This production, and those that follow, will no doubt continue to achieve unprecedented scale in the use of real-world scientific data and 3D visualization to create highly immersive, educational forays through earth and space.”

About California Academy of Sciences

The California Academy of Sciences is a world-class scientific and cultural institution based in San Francisco, California. The Academy’s newest facility, a 400,000-square foot structure, houses an aquarium, a planetarium, a natural history museum, and a four-story rainforest. Located in Golden Gate Park, the facility is also home to the Academy’s staff of world-class scientists, an education department that provides a wide range of student and teacher services, and an extensive science library with more than 28 million specimens and artifacts. The mission of the California Academy of Sciences is to explore, explain and sustain life.