What you will learn: The term "big
data" is used anytime an enterprise produces a set of data containing critical business
information that's too large to be processed by relational databases. Determining what data is left
unstructured depends on the size and scope of the enterprise’s IT
infrastructure, but it's common for businesses of all sizes to have some amount of
information that could be considered big data. The struggle for IT administrators and business
analysts is not only how to store this data, but how to store it in a way that allows for analysis,
resulting in the identification of critical business patterns and insights.
As the IT industry continues to preach the advantages of cheap storage, businesses are
keeping more data than ever, resulting in a deep investigation into which factors matter most when
evaluating a big data infrastructure. Among the most important are capacity, latency, access,
security and cost, all of which are covered in this article.
What's driving the big data movement?
Aside from the ability to keep more data than ever before, we have access to more types of data.
These data sources include Internet transactions, social networking activity, automated sensors,
mobile devices and scientific instrumentation, among others. In addition to static data points,
transactions can create a certain “velocity” to this data
growth. As an example, the extraordinary growth of social media is generating new transactions
and records. But the availability of ever-expanding
data sets doesn’t guarantee success in the search for business value.
Data is now a factor of production
Data has become a full-fledged factor of production, like capital, labor and raw materials, and
it’s not just a requirement for organizations with obscure applications in special industries.
Companies in all sectors are combining and comparing more data sets in an effort to lower costs,
improve quality, increase productivity and create new products. For example, analyzing
data supplied directly from products in the field can help improve designs. Or a company may be
able to get a jump on competitors through a deeper analysis of its customers’ behavior compared
with that of a growing number of available market characteristics.
Storage must evolve
Big
data has outgrown its own infrastructure and it’s driving the development of storage,
networking and compute systems designed to handle these specific new challenges. Software
requirements ultimately drive hardware functionality and, in this case, big data
analytics processes are impacting the development of data storage infrastructures. This could
mean an opportunity for storage and IT infrastructure companies. As data sets continue to grow with
both structured and unstructured
data, and analysis of that data gets more diverse, current storage system designs will be less
able to meet the needs of a big
data infrastructure. Storage vendors have begun to respond with block- and file-based systems
designed to accommodate many of these requirements. Here’s a listing of some of the characteristics
big data storage
infrastructures need to incorporate to meet the challenges presented by big data.
Capacity. “Big” often translates into petabytes of data, so big data
infrastructures certainly needs to be able to scale. But they also need to scale easily, adding
capacity in modules or arrays transparently to users, or at least without taking the system down.
Scale-out storage is becoming a popular alternative for this use case. Scale-out’s clustered
architecture features nodes of storage capacity with embedded processing power and connectivity
that can grow seamlessly, avoiding the silos of storage that traditional systems can create.
Big data also means a large number of files. Managing the accumulation of metadata for file
systems at this level can reduce scalability and impact performance, a situation that can be a
problem for traditional NAS
systems. Object-based
storage architectures, on the other hand, can allow big data storage systems to expand file
counts into the billions without suffering the overhead problems that traditional file systems
encounter. Object-based storage systems can also scale geographically, enabling large
infrastructures to be spread across multiple locations.
Latency. A big data infrastructure may also have a real-time component, especially in use
cases involving Web transactions or finance. For example, tailoring Web advertising to each user’s
browsing history requires real-time analytics. Storage systems must be able grow to the
aforementioned proportions while maintaining performance because latency can produce “stale data.”
Here, too, scale-out
architectures enable the cluster of storage nodes to increase in processing power and
connectivity as they grow in capacity. Object-based storage systems can parallelize data streams,
further improving throughput.
Many big
data environments will need to provide high IOPS performance, such as those in high-performance
computing (HPC) environments. Server virtualization will drive high IOPS requirements, just as it
does in traditional IT environments. To meet these challenges, solid-state storage devices can be
implemented in many different formats, from a simple server-based cache to all-flash-based scalable
storage systems.
Access. As companies get better at understanding the potential of big
data analysis, the need to compare differing data sets will bring more people into the data
sharing loop. In the quest to create business value, firms are looking at more ways to
cross-reference different data objects from various platforms. Storage infrastructures that include
global file systems can help address this issue, as they allow multiple users on multiple hosts to
access files from many different back-end storage systems in multiple locations.
Security. Financial data, medical information and government intelligence carry their own
security standards and requirements. While these may not be different from what current IT managers
must accommodate, big data analytics may need to cross-reference data that may not have been
co-mingled in the past, which may create some new security considerations.
Cost. “Big” can also mean expensive. And at the scale many organizations are operating
their big data environments, cost containment will be an imperative. This means more efficiency
“within the box,” as well as less expensive components. Storage deduplication has already entered
the primary storage market and, depending on the data types involved, could bring some value for big
data storage systems. The ability to reduce capacity consumption on the back end, even by a few
percentage points, can provide a significant return on investment as data sets grow. Thin
provisioning, snapshots and clones may also provide some efficiencies depending on the data types
involved.
Many big data storage systems will include an archive component, especially for those
organizations dealing with historical trending or long-term retention requirements. Tape is still
the most economical storage medium from a capacity/dollar standpoint, and archive systems that
support multiterabyte cartridges are becoming the de facto standard in many of these
environments.
What may have the biggest impact on cost containment is the use of commodity hardware. It’s
clear that big
data infrastructures won’t be able to rely on the big iron enterprises have traditionally
turned to. Many of the first and largest big data users have developed their own “white-box”
systems that leverage a commodity-oriented, cost-saving strategy. But more storage products are now
coming out in the form of software that can be installed on existing systems or common,
off-the-shelf hardware. In addition, many of these companies are selling their software
technologies as commodity appliances or partnering with hardware manufacturers to produce similar
offerings.
Persistence. Many big
data applications involve regulatory compliance that dictates data be saved for years or
decades. Medical information is often saved for the life of the patient. Financial information is
typically saved for seven years. But big data users are also saving data longer because it’s part
of an historical record or used for time-based analysis. This requirement for longevity means
storage manufacturers need to include on-going integrity checks and other long-term reliability
features, as well as address the need for data-in-place upgrades.
Flexibility. Because big data storage infrastructures usually get very large, care must
be taken in their design so they can grow and evolve along with the analytics component of the
mission. Data migration is essentially a thing of the past in the big data world, especially since
data may be in multiple locations. A big data storage infrastructure is essentially fixed once you
begin to fill it, so it must be able to accommodate different use cases and data scenarios as it
evolves.
Application awareness. Some of the first big
data implementations involved application-specific infrastructures, such as systems developed
for government projects or the white-box systems invented by large Internet services companies. Application
awareness is becoming more common in mainstream storage systems as a way to improve efficiency
or performance, and it’s a technology that should apply to big data environments.
Smaller users. As a business requirement, big data will trickle down to organizations
that are much smaller than what some storage infrastructure marketing departments may associate
with big data analytics. It’s not only for the “lunatic fringe” or oddball use cases anymore, so
storage vendors playing in the big data space would do well to provide smaller configurations while
focusing on the cost requirements.
BIO: Eric Slack is a senior analyst at Storage Switzerland.
