The Information Explosion – The Sorcerer Goes to Lunch

Sat, 28 Nov 2009 03:10:09 +0000

The other day I decided it was finally time to clean – really clean – my desk. As I worked through years of accumulation, I noticed an actual pencil lying forgotten on an old pad of paper. The pencil was all dusty and unused, and the pad had yellowed and curled edges. That got me thinking about how long it’s been since pen and paper were used for daily communication and record keeping. Now, some of you may brand me an old codger for being able to remember that at all, but, bear in mind that the first usable version of Microsoft Windows, version 3.0, shipped just 18 years ago. At that time, many pencil pushers resisted transferring everything to the “computer,” but even the most active holdouts gave way in a year or two.

Oh, some of us might have used an email system back then. At Hewlett Packard, I remember sending emails using their proprietary HP Desk mail system back in the mid 1980s. However, at the time, the majority of office workers were… actual pencil pushers.

Can you find anyone today who doesn’t compulsively check their email or scroll through file listings to find what they need? OK, John McCain doesn’t, but he has “people.” I don’t know about you, but I don’t have “people.” However, I do have my fully networked system with Internet access, Instant Messaging, stunning graphics, oodles of productivity, and regular backups. And I love it!

In that eighteen years we experienced the world changing faster, and more completely than it ever had. The Internet (once the exclusive domain of the nerdy-ist of nerds – cosmological physicists) became everyone’s instant window to the world. Easy-to-use authoring tools allowed everyone to be productive. It’s as if the sorcerer’s apprentice were replicating legions of pencils instead of brooms to generate a catastrophic data deluge. Today, corporations are literally drowning in it.

Back in 2006, IDC conducted an exhaustive study (Source: The Expanding Digital Universe, IDC, March 2007) and forecasted, between 2006 through 2010, a 57% growth rate year over year in the amount of information created, captured and replicated.

So, where is all this information coming from and why aren’t companies able to deal with it? Well, it comes from everyone, and it’s a problem because most of it is unstructured. Most people aren’t aware of this, but The Enterprise Strategy Group estimates that between 80-85% of all business data is unstructured (Source: Extending Discovery to All Corporate Information, Enterprise Strategy Group, December 2007).

What is unstructured data? It consists of emails, reports, all user files (documents, spreadsheets, PPTs, PDFs), images, video, HTML/XML, MP3, etc. It varies in importance, too. The average user will save pictures of their children, emails about what a good job they are doing, CYA “email trails,” work-related spreadsheets, thick Word documents, etc.

In the book, “Tapping into unstructured data: Integrating unstructured data and structural analytics into business intelligence” (Bill Inmon and Anthony Nesavich, Prentice Hall, 2008), the authors describe the various types of unstructured data created by the typical departments in a corporation. These include: Accounting, Call Centers, Engineering, Finance, Human Resources, Legal, Marketing, Sales, Shipping and Operations. That means everyone is contributing to the challenge while they look to the data center to control it.

The Challenges of Unbridled Information Growth

Let’s take a look at some of the major challenges in dealing with this unbridled growth of information.

Factor #1: Information must be stored

The more data we generate, the more storage is required. This storage need opened up tremendous opportunities for storage vendors as customers sought to purchase more and more equipment. The storage industry introduced the moniker Information Lifecycle Management to provide more cost effective ways to deal with this growth. They also introduced the concept of tiered storage to allow companies to better manage it along various dimensions: price, performance, capacity and function. Initially, the storage cost factor was the biggest impact on corporations of this growth. However, as storage cost quickly declined, its importance became dwarfed by other factors.

Factor #2: Information can be sensitive and needs to be protected

As companies created more and more information, the importance of protecting that information and ensuring the proper access level became more apparent. While it sounds easy (i.e. making sure the right people have access to the right information), it’s not so easy to actually do, and the costs of not securing data can be astounding. Examples are:

Hefty fines under PCI, SOX and HIPAA for breaches and noncompliance Bad PR and damage to the corporate brand due to the need to publicly disclose privacy breaches Outright IP theft where trade secrets and proprietary information could fall into the hands of a competitor and materially damage the company’s business prospects
Factor #3: Information must be preserved for regulatory reasons

Every company is governed by a set of regulations that that determine the length of time that information must be stored. There are a slew of regulations that govern information retention. The more familiar of these include:

Health Insurance Portability and Accountability Act (HIPAA) of 1996 Sarbanes-Oxley Act of 2002 SEC Rule 17a-3, a-4
There are countless more. Some industries (e.g. Pharmaceutical, Finance, etc.) are more regulated than others. And, of course, with the recent Credit Crisis, we expect the number of regulations to skyrocket in the coming years.

In the good old days, retaining this information was simple. We simply put everything in a box and placed that box in a warehouse for however long. Given the explosive growth of easily replicable electronic information, it’s much more challenging.

Factor #4: Information is subject to electronic discovery

A critical event occurred in December 2006 with the passing of The Federal Rules of Civil Procedure (FRCP). The FRCP governs procedures for civil suits in United States district (federal) courts. It was amended to outline how electronic documents can be used to support litigation proceedings. The amendment also defined how electronic documents should be handled to support litigation search and discovery.

Essentially, this means that all information is discoverable, which presents a problem. Companies are not only required to keep information for a particular period of time (for regulatory purposes), but also are incentivized to get rid of it as soon as possible. It simply isn’t practical for a company to pay an attorney $400/hour to perform discovery across all of their information.

The Challenges of Managing Unbridled Information Growth (the Buckets Multiply Geometrically)

So – where does this leave us? We have too much information today. Some of this data needs to be protected because it contains sensitive information. Some of it needs to be retained for certain periods of time due to regulatory constraints, and it’s all discoverable. We are creating new information at an alarming pace, and like with the sorcerer’s apprentice bucket brigade, it’s frighteningly out of control.

I attended last month’s ARMA (Association of Records Managers and Administrators) conference in Las Vegas to get more perspective on the information conundrum. After all, records managers have had to deal with the management of information for many years, initially in physical form and more recently in electronic form. Their mantra is simple. They need to know what they have and where they have it. They need to make certain only the right people have access to the information. They need to know what to keep, and they need to keep it as long as they have to. They need to get rid of everything else. It’s a simple matter of setting up policies across the enterprise and enforcing them.

It sounds so simple. But is it? Do we know what we have? Do we know where we have it?

Unfortunately, it is easier said than done. The information we create is vast. It is stored in heterogeneous formats throughout the world.

I spoke with one Records Manager of a mid-sized company who told me, “Yes, I know what we have. I have two file shares in Des Moines with my finance, marketing and sales files. I have a user share in our corporate office with personal files. At corporate, I also have my web farm. I have an Exchange Server, one Personnel Database, one Accounting Database, and one Documentum System. Oh, and twelve SharePoint sites.”

Her problem is typical. With information stored everywhere, how can she manage it across heterogeneous systems? How can she set up consistent policies for ensuring the right access? How can she ensure that the right data is retained? How can she ensure that she gets rid of what she does not need?

Basically, she not able to address these problems, which could place her company out of compliance. This brings a risk of being heavily fined. The problem, of course, is even worse for larger companies who literally have Petabytes of information stored everywhere.

One Certified Records Manager I spoke with likes to categorize information as follows:

Type of Information and Where it Exists

Unstructured – Data File Shares, Desktops, Laptops Enterprise Content Management System – SharePoint, FileNet, Documentum Messaging – Email, Voice Mail, IM, etc. Databases – Human Resources, Order Processing, etc.

He explained, “The problem is that a single, universal system for managing information does not exist.” I visited vendors, both big and small, and confirmed what every Records Manager has known for quite some time. That being able to effectively manage information according to the Records Management mantra is truly a Herculean task.

Managing Information in a Cloud (the Sorcerer Returns from Lunch)

There is a sorcerer who can clean up, organize, and control the deluge of information, and its name is Classification Management. Only when information is classified, can it be effectively managed to address the concerns of sensitivity, retention and destruction.

The magic wand of this Classification and Management sorcerer is a sophisticated Policy Engine. It’s sophisticated because it can support different Policies for different information sources (databases, email systems, etc.). It can support different Policies for different regional regulations, and it’s flexible enough to deal not only with today’s regulations, but also with future ones.

Scalability is the other side of this magic wand because, classification alone is not sufficient to deal with the scale of the information in the average enterprise. It is geographically distributed across heterogeneous systems, and a centralized information management scheme will not and cannot scale.

Because of this, we see Enterprise Information Management as being the first enterprise application that requires a form of cloud computing. This allows all information everywhere to be managed.

What’s Next

As we proceed into 2009, only a couple things are certain: companies will create more information; government will create more regulation, and the sorcerer will have more and more data to manage.

By: Will Matlack

Finance Book » Stunning Graphics

The Information Explosion – The Sorcerer Goes to Lunch