S3 object storage with tape for large data volumes

Data volumes are rising fast and show no sign of slowing down. This places businesses under ever greater pressure. The rise of digitization is resulting in increasingly large volumes of data being generated across all fields of business. As a result, there is a need for flexible and scalable storage infrastructures that reliably record data and keep it safe.

Object storage systems are an ideal response to this demand, as they can store large amounts of inactive and unstructured data. Many businesses rely on the cloud to keep their data secure. But often the amount of data they hold is well beyond what hard disk-based object storage and cloud services can cost-effectively manage. The question, therefore, is this: Which storage medium is best for archiving very large quantities of inactive data? Which storage technology is suitable for this kind of storage tiering?

For reasons of both cost and efficiency, there is a strong case for the answer being tape. With LTO‑8 media, the cost are only 0.01 Euro per GB. Moreover, tape libraries can be very easily integrated into existing storage infrastructures using an S3 interface. S3, the Simple Storage Protocol, was introduced by Amazon in 2006 and has since established itself as a market standard. With the right software, tape drives can be accessed using a standard S3 interface. This allows you to use the tape library as an object storage system for large volumes of unstructured data.

Use cases

A tape-based object storage system is suitable for integrating into your storage architecture under three classical scenarios:

  • For long-term archiving of very large volumes of data,
  • To free up hard drive-based object storage systems,
  • To replicate and secure cloud-based data.

Long-term archiving for research and measurement data

The rules of good scientific practice stipulate that research data should be archived for a period of ten years. It must remain reliably available for further use and follow-up research. There are numerous requirements placed on data management and long-term archiving solutions in the fields of science and research. Tape-based S3 object storage systems are ideal for both securely and cost-effectively archiving large quantities of research, measurement and multimedia data.

Information lifecycle management and storage tiering for object storage systems

Many businesses use hard-disk based object storage systems. These on-premises storage systems need to be regularly cleaned of outdated or inactive data. This is where the concept of information lifecycle management comes in. This storage strategy selects a storage tier for data based on its current lifecycle status. Outdated, inactive, “cold” data is typically not accessed any more, but it does still need to be retained. It is removed from costly primary storage systems and, using storage tiering configured with predefined rules, moved to secondary storage instead. This reduces the load on primary hard disk-based object storage systems by transferring inactive data to tape using an S3 interface.

Backing up and replicating productive cloud-based data

The cloud is a popular option for securing data: It offers flexible, scalable storage capacity—and usually similarly flexible payment options, based on the actual capacity used. In principle, the data is available anytime and anywhere, even off-premises. Nevertheless, it is risky to assume that data in the cloud is secure. Even data centers operated by big cloud providers can fall victim to viruses or ransomware, which could bring down all the clouds they host. Businesses must therefore take additional steps to secure any data stored in the cloud. Duplicating it with another cloud provider or on hard drives is out of the question, given the costs involved. Instead, replicating it to tape is the much more cost-effective option. Businesses can use an S3 interface to directly access data replicated to tape, allowing them to continue their work without interruption. This realizes a 3-2-1 backup strategy.

How to: Store objects on tape

Tapes are therefore a future-proof storage medium for countering the challenges of rising data volumes, storing unstructured data at high speeds and acceptable cost levels. One question remains, however: how to actually save object data to tapes. Object storage systems do not offer a direct connection to tape libraries, but it is inadvisable to copy objects to a file system before storing them on tape. A number of different technical factors mean that this leads to a risk of losing information.

Instead, the S3 objects themselves must be saved directly to tape. This requires a software solution that receives data via an S3 interface and then writes it to tape. PoINT Archival Gateway is a high-performance, scalable object storage system that supports tape libraries and enables a tiered storage architecture for object storage. It comes with a standard S3 REST interface and guarantees data security via integrated erasure coding. This means that with the help of PoINT Archival Gateway, you can replicate large volumes of data from object-based primary storage systems — or from the cloud — onto tape, for secure long-term storage.