Blog Host: Brett Cooper READ ENTIRE BIO

I have learned that the only real expert is the one that is willing to say that they don't know and go to their network of friends and counterparts in the industry and get their views before formulating their own opinion. I am not an expert, rather an individual that has had the pleasure of spending time in this great industry, which I love.

I wanted to leave my time in this blog with one thought about the solutions (products) that we see in this space and how each of us can make a difference. The final thought is feedback. I am not talking about the nice warm and fuzzy kind of feedback, but the kind of feedback that drives us all to be better and helps the partners (or vendors) that we work with deliver the solutions to our challenges. Each company works based on the premise that if they build solutions (products) based on our requirements, they can build a business and continue to offer these great products. So, go forth and scream from the hilltops to your partners about the solutions that you need and have them commit to delivering those solutions and work with the rest of their customers to ensure that the solutions will work in a broad spectrum of environments.

Thank you for reading my blog and I look forward to answering your questions and penning my next blog!
Posted by Brett Cooper Metadata products
06 APR 2006 17:37 EDT (21:37, GMT)
In my last blog I discussed how metadata can be used for managing unstructured information (AKA user files) and today I will talk about some of the solutions (products) available on the market that make this task easier.

The choices:

Freeware/shareware solutions
There are some simple freeware-shareware programs on the market that will grab metadata on a per server basis, enabling simple data collection and reporting. These $10-$50 tools provide simple text reports, and, in some cases graphics, but that is all that they provide. An administrator will need to manually pull the information together from multiple servers to create a single report and then will need to do some analysis to determine what to do with the files. Some examples are SizeExplorer and BindView. These products will be good for a couple of file servers, but aren't the best for more than a small handful, as they will require hours, if not days, to pull information together and they need to be run regularly if you want to see the benefits (or opportunities for improvement-AKA mistakes) of your information management decisions. These solutions usually don't interfere with the production environment and require very little resources on a server to collect their information.

The SRM players
Next is the storage resource management (SRM) products from such companies as IBM (Tivoli), CA, HP, Symantec-Veritas and a few others. These SRM solutions offer the more elegant reporting and information collection that you would expect from these companies. They focus on file servers and NAS products to deliver more complete, automated reports on a regular interval. These SRM solutions are more fitted to larger environments with more than 10 servers and may require more resources from a file server as well as deploying agents on remote servers to collect information. Also, the cost is going to be significantly higher than the base freeware or shareware program.

Both the freeware/shareware and SRM players only collect information about the files and fall short on actively managing/moving the files.

The global file system/HSM players
Next up are the global file system (GSM) and HSM players. These solutions provide little in the way of collecting detailed metadata to make decisions about the files, but they make up for it with their management/movement capabilities. Usually, these players rely on file stubs or on placing a flag-pointer in the file system where a file once resided that points to a new location for the file on another storage environment. While very useful for cutting down cost and end-user complexity -- or as some call it "transparency" (since the end user doesn't realize the file has moved, save for a different looking icon and possibly a longer recall time) -- the IT organization usually has to contend with larger issues, such as system performance and scalability, recall avalanches and unrecoverable files due to the larger, more complex environment that lives behind these solutions.

These solutions are usually meant for single purposes, not large-scale production environments in larger organizations. HSM is very successful in mainframe environments due to the locality of a single system and the purpose-built nature of the environment for which the solution runs on. In open systems environments HSM tends to fall into the single solution space serving a single production application or user file set well, but is not generally applicable to a larger enterprise-wide user file management deployment. GFS solutions can set up migrations and remote user environments very well, making it possible for traveling salespeople that go between offices to locally see their files without having to reconfigure their notebook/desktop environments. Also, the GFS solutions are integrated into the leading open system server platform (Windows) using DFS and Active Directory (OK, am I getting too technical?) so management for the IT team is relatively straightforward.

The information management players
Finally, we come to the information management players, who provide it all. They can gather the most metadata from the unstructured data, use it to make decisions -- which become policies -- automate those policies, and move the information, retaining the metadata along the way. These solutions can manage/move the information on their own and integrate seamlessly with other solutions to complete the information management picture. Some of them open each file, creating some challenges for data security and reliability, since the actual accessed and ownership information will change, unless the product caches the original information and then sets the file's information back to what it looked like before it was accessed. Some do not crack the content within the files. In either case you need to try these solutions out in a test environment before rolling them into production. These solutions are meant for larger enterprises with millions of files and tens of thousands of users.

The goal here is performance and scalability coupled with a back-end repository of the metadata that scales well. Imagine if you have some 100TBs of unstructured information and the metadata store to manage the environment took some 20TBs, or 20% of the overall storage that is being managed. Did this solve your cost-complexity issues? I don't think so. Also, look at the performance piece, not just of the scanning, but of the storage pool of unstructured information. If the scan is happening and the storage pool for your users is unavailable, the solution may not be applicable in your environment since users are accessing their files 24x7 these days. Also, check with your legal and risk management teams to see if there are any files that require more stringent management. There may be some organizations that require a couple of products to make the total solution. There is no one-size-fits-all solution in the unstructured information management space because each organization's goals and requirements are different.

My next blog entry will be my last, so let me know your thoughts and if you have any questions.

Thank you.
Posted by Brett Cooper The value of metadata
05 APR 2006 20:06 EDT (00:06, GMT)
To manage user files, called "unstructured information" within the industry, there are multiple solutions in the industry. When I talk about managing unstructured information, I am talking about making decisions about the information and then moving it based on the decision (based on the corporate goals) that was made. The movement could be as simple as backing it up to tape, or it could be more sophisticated; it could include using a global file system or HSM solution and migrating the file(s) to a secondary tier of storage. In either case, information is required about the files to make the decision. This information, metadata, is what is kept in the file system that tells you some information about the files themselves. Simply said, metadata is information about information (I think I read this somewhere, and a quick search of the Internet reveals my source: The National Information Standards Organization [NISO]).

Metadata can contain information about ownership of the files, access and modified dates, file types, backup-archive information and much more. Each piece of metadata can help build a picture of the files value to the organization based on the organizations goals.

Many analysts have stated that user files take up some 60-80% of the organization's total storage pool and the growth is between 140-160% year on year. One of the biggest forgotten places to gain economies of scale is user files themselves. Some keys to better manage user files include:

1. If an employee leaves the organization do their files leave with them? Usually, in an IT department form is sent to tell the IT team to delete the user from the corporate domain. Once their user ID is deleted, all of their files become unowned or orphaned. If an organization has some 10-15% turnover each year, in a few short years the unstructured stores can quickly become full with information that may no longer be of use to the organization, depending upon the corporate requirements and regulations. Time for a backup and a delete of the files.

2. Not very useful metadata -- If a migration has occurred the metadata is usually set to the date of the migration, and the ownership information all changes to administrator. When doing a migration it is critical to maintain the metadata as it is the only record of the file(s). Make sure your migration solution supports the retention of the metadata.

3. Files older than a certain date. Many organizations have rules that say "Any file older than X number of years" is automatically backed up and removed from the costly storage. In this way organizations can manage their user files based on a known timeframe.

4. Users in XYZ department have more stringed requirements. End users in the purchasing, contracts, finance, HR, legal and executive departments usually have more stringent requirements for their information. These departments require different management principles and security provision for their information.

I wonder if the same is true for the information within an organization? The answer is not a simple one, rather the answer is "it depends." If the information is governed by a government regulation or business specific policy, it must be maintained until the time or value passes and then the information can be destroyed. "Wait," you say, "destroy information?" Yes, after information has lived its life it can be destroyed, provided the organization agrees that the information is of no value.

In fact, many organizations rush to destroy information the moment after its expected lifetime has ended, as it becomes a liability to the organization. Imagine a court case in which a judge asks an organization to produce information from 20 years ago. The records from 20 years ago were destroyed, since the government regulation that apply to the information required the organization to keep it for seven years. The judge cannot ask for the information, and any litigants in the case would understand that the case is closed, having not been brought to the attention of the courts prior to the regulatory period ending.

The challenge for organizations created by the stockpiling of information is the creation of a ripple effect that I discussed in my blog a few days ago; like dropping a stone into a lake and watching the ripples go out from the entry point of the stone. The creation and management of information is all tightly linked with each process affecting one another. Each successive ripple carries with it some of the energy from the big splash. This is the same as the creation of a piece of information, there are multiple processes that are in place to protect and manage that new information: back it up, migrate it from site A to site B in the case of a disaster and more.
Posted by Brett Cooper Creating long-term archives
03 APR 2006 18:28 EDT (22:28, GMT)
Today, I thought I would discuss the idea of creating a long-term archive and what that means. I am not going to talk about a one-year archive or even a 10-year archive; rather I thought a 100-year archive discussion would fit the needs of most organizations today. The team at SNIA has a project underway to flush out exactly what is needed in a 100-year archive and is working with many organizations to create a solid solution. In fact, there will be a moderated panel discussion on the 100-year archive at the Enterprise Information World event, May 23-25, 2006 in Secaucus, NJ, that will shed some light on the topic. And while I am not an archiving expert, I have some thoughts on what information from today will look like in 100 years and some of the challenges in dealing with the information. Before I get started laying out the thoughts of the technology evolution that will occur between 2006 and 2106, I thought we should look backwards over the past 100 years to understand the evolution that has already occurred; the first automobile, the first flight, tubes spaceflight, satellites, transistors, computers -- as my statistics professors in college reminded me of on a daily basis -- punch cards, processors, microprocessors, the Internet (Thanks Al Gore, I couldn't resist!), cell phones, magnetic media, nano technologies and many other technological innovations that I have not named, and probably put in the wrong order. Many of these innovations occurred as the result of man's desire to gather information and process it faster than before.

Today, much of the information that has been created over the past 100 years has been lost or destroyed because there was no way to easily archive it, or because the information owners did not care to hold on to the information, or because of a war or natural disaster. My brother-in-law runs the National Climatic Data Center and is responsible for collecting, processing and archiving climate information from around the globe as part of an effort by the U.S. government -- our taxpayer money at work on what I believe has an excellent goal of understanding our climate and its changes as well as supporting our cooperative efforts around the globe. The archives at the National Climatic Data Center contain Ben Franklin's weather observations from hundreds of years ago.

Yet, many of the movies from the original days of Hollywood have been lost since there was no archive. Most organizations have maintained archives of their business and the Federal Government has mandated that all information from any military efforts must be preserved forever as the result of the information from our first efforts in Iraq being lost and needing to be recreated in their entirety. These are all very important archives that should be maintained in perpetuity.

You may be surprised, as I was, to find out that the largest information archive in the world is not at the Library of Congress, nor is it maintained by a financial organization, but by the Church of Latter Day Saints. The genealogy database that the Church maintains is the largest single information archive on the planet, and it is available to anyone to search through to find out about one's roots. The Church works tirelessly to gather genealogy information from any country that will let their personnel come in and document the information available into their repository. Think about the vast amounts of information that are contained in this repository…family genealogy going back hundreds of years.

Ok, now let's fast forward 100 years and think about what an archive will have to have in order for the technology of the day to be able to access it. Do you think Oracle 100i will be able to read Oracle 8i's format? Will Microsoft Office 2106 on the Windows MMCVI OS be able to read Microsoft Office 2003 files? Will the tape that I put in the vault be readable by the devices of 2016? How about my 250GB SATA hard drive? Can I just plug it in and I'll be reading my information? A lot of smarter people than I am are tackling this issue, but I have some thoughts to share.

What information will be important to save for 100 years from now? The government has provided some important guidelines for organizations, both public and private, that dictate what must be saved. For example, HIPAA regulations require all information be kept for the life of a person plus two years, so much of the information about children born today will probably be around in 100 years. Financial records are only required to be kept for seven years for most organizations.

With this said, many organizations are grappling with the question: "What information is important to document for the business in the future?" I am going to go off course here and remember my time as a child when we created a time capsule containing pieces of our world at a point in time that we were supposed to open when we graduated from college (or so I remember). We put in a couple of LPs (remember those? Vinyl records! Now I am dating myself!), some baseball cards, a school yearbook and a bunch of other things that had the date on them. This was a record of a point in time, the information of its day. What defines the information of an organization's day? Its financial records for sure are critical, including payroll, balance sheet transactions, etc. How about the contracts and agreements? OK, what else? E-mails from specific people that are records of the way business is done? All of these documents are kept in applications that need to be persevered both in the communication of the information and the information itself. Having the information is just one piece of the solution, being able to access it 100 years from now is quite another. I have heard many in the industry talk about exporting the information in XML or text-based formats so it is easily accessible, or using PDF formats as they will surely be readable in 100 years. I think that the answers are not yet apparent, and there are many standards bodies at work with the vendor and the organizational communities to provide answers.

I, for one, just want to make sure the digital pictures of my family survive into the next generation and beyond. So where should I keep them? In a safe in my home? On a DVD in a safe deposit box, or printed out in a scrap book? I think the answer is in multiple places. I back them up to a secondary media device, burn a DVD (I am up to some 5GB of digital pictures), and I also print the important ones out on archival quality paper using archival quality inks (OK, not all the time), so I know that they will last. I think that many organizations will follow the same path by keeping multiple copies of critical information in a variety of formats and locations to ensure that they are available and accessible.

Let me know what you think about a 100-year archive. Tomorrow, I will discuss the idea of deleting information.
Posted by Brett Cooper What about information security?
31 MAR 2006 17:09 EST (22:09, GMT)
Records and information management and information technology go together like peanut butter and jelly -- sharing the same goals and working together to deliver results gained by reducing the risk, cost and complexity of information within the organization. There are several other groups within the organization that play an active role in managing information, including the Information Security group within the chief security officer's team. This is a relatively new group on the timeline of information management professionals, having shown up on the landscape sometime between 10 and 15 years ago. The information security professionals are responsible for defining security mechanisms and processes through the organization. There may be a chief security officer within the organization, or the Information Security team may work directly for a manager in the chain of command leading up to the CIO. In either case, this group is the first and last stop for the decisions that surround access to information through the various physical and logical entry points to the organization. There are professional organizations and standards groups for Information Security just like other roles within the information management world, including the ITSC and the ITAA.

One of the more important security discussions that has entered the Information Management purview is the discussion around encryption of information. In the wake of the losses of tapes and computers, many information security professionals are calling for all information pertaining to an organization, whether at rest or in transit, to be encrypted. This is a controversial discussion, and I don't claim to be an expert or to know all of the various arguments for or against this integration, but I believe that there are issues that come to bear on any organization that may be looking at this technology, just like any other technology that has not been used to date. The challenges, for example, of cutting over to the new technology include: integrating it with processes and management paradigms, distributing it through the organization and insuring compliance. With this said, the benefits far outweigh the simple challenges listed since information will be encrypted and only those trusted parties will have the ability to view the information when it is needed for recovery purposes or transaction processing.

Another hot topic in the information security world is reverse firewalls or technologies that check information streams leaving the organization. In the case of digital assets these reverse firewalls check the information stream to ensure that real property is not leaving the organization. An example would be a large music company that is releasing a new album or a movie house releasing a new film, only to find that it has been posted for download to multiple Web sites. This happened because someone within the organization leaked a copy of the digital assets prior to its general release. Usually, the culprit uses e-mail or an FTP program to simply transfer the media to the Web, or they may even use a mass storage device such as USB memory key chain, MP3 player or burnable media to simply transfer the file and physically carry it out of the building. I have visited several organizations that no longer allow their users to use MP3 players or have media devices in their systems, including floppy disk drives. In the terms of digital media this can save the organization $100s of thousands or even millions of dollars in lost revenue. The same can be true for financial, health care or life sciences companies, whose lifeblood is their information.

Let me know your thoughts on the topic of information security and how you deal with it within your organization.

My next post will be on managing information. I plan to tackle the topic of the long-term archive.
Posted by Brett Cooper Bringing RIM and IT together -- a view from the top
30 MAR 2006 21:18 EST (02:18, GMT)
Today, I am going to attempt to bring the records and information managers (RIM) and IT communities organization together and help them understand what their management and their customers would like them to provide.

First, from the management team's perspective:

CEO's goals:

• Return profits to the shareholders
• Create a model company
• Serve the community in which the company is based
• Follow government regulations

• Increase profits by reducing costs
• Reduce complexity which adds to cost
• Avoid risk, unless there is a payoff
• Volunteer time in the community
• Follow the rules for managing information and products

Next, let's talk about risk-reward. Usually, when the risk is higher, the expected return is higher. While true for financial transactions, information is not something to play with. Rather, information management processes must be documented to pass audits that are required in today's regulated environment. That does not mean that the most expensive, most complex process is the right one, rather it maybe a choice, usually there is a happy medium that can be maintained to meet the goals of the organization. One CEO that I talked to stated that he wanted to be graded a solid C+ for his information management process. He wasn't concerned with getting an A, as the cost for the A was beyond his budget. While attractive, he had the sense to realize that the C+ would meet the needs of the organization and provide his business with a competitive edge. There is another example of this thinking in the disaster recovery processes that many organizations have. They usually go with a solution for their critical applications that has little if any downtime, as these are the applications that run the business, like the banking industry's cash management applications. These real-time applications are run synchronously between two or three data centers spanning hundreds of miles. While critically important for this application, the user files for the same bank may only be mirrored once a day, or backed up to tape and then shipped to a secondary facility. OK, let's check off the risk-reward discussion for the CEO.

And lastly, let's talk about creating a model company and serving the community. These are the ones that many people define as "culture." You can't see it on the balance sheet, or in the process documentation that defines how information is managed, but it is there at the water cooler, in the conference rooms within the organization and at the company softball game. It is the way in which the company and its employees conduct business, sharing their unique style with others. The goal here is to share the knowledge and success of the organization with others so they can learn and grow, avoiding the missteps that your organization may have made in getting to where it is at. The same can be said for other organizations. This is why there are so many user groups and standards bodies. This is also why so many charities enjoy corporate sponsorship. For information management professionals, it is as simple as signing up for an industry event and presenting on what you are doing within your organization, serving on a panel or volunteering time for a local charity. This is the goal that also gives back, not in terms of pay, but in terms of feeling good about what you and your company does.
Posted by Brett Cooper Adding information technology to the information management equation
29 MAR 2006 00:00 EST (05:00, GMT)
Today, I thought I would complete the picture that I have started to draw in my past two blog entries by discussing the idea of crossing an organizational chasm that has existed since the introduction of technology into today's business organizations. The chasm separates the information technology organization from the other organizations within the enterprise. This chasm is not only of physical location within the organization, but one of culture and lexicon used to conduct business. The information technology department is a cost center at most organizations; providing a service to the greater organization that helps deliver on the goals of the business. The information technology organization utilizes a combination of infrastructure, personnel, know-how and vendors/partners to deliver these services. There is a partnership in place between the information technology organization and the rest of the organization that enables the business to meet its goals. In the case of information management where everyone's goal is to better manage information, the information technology department provides the tools and solutions needed to keep the creation, management, archival and disposition of information moving.

I have heard many records and information managers call the information technology department the container managers, referring to the storage and infrastructure containers that hold the information for the organization. This isn't mocking what the services that the information technology team provides, but rather I believe that the RIM professionals are referring to the storage and infrastructure managers within the IT department having no understanding of the real information that the 0s and 1s make up on their containers. The storage managers know that the storage containers they are managing have user files, e-mail or a database contained within, but they may not understand the true value of the information since its management is done by an external group. For example, the database may have records in it from eight years ago that could be purged to save space, or some 20% of the user files are from employees that no longer are employed at the company. In these cases there needs to be a more detailed review of the information being stored to understand its value to the company and determine what the right thing to do with it is. This is accomplished using some of the best solutions available in the market to wade through the volume of information that is already within the enterprise as well as keeping up with the growth that occurs on a daily basis.

This isn't simply a question of moving information from a higher-priced, more available storage tier to a less expensive one, it is a question of working together with the information owners within the business to determine what the right process is to put in place to manage the information on an ongoing basis, which is where the RIM professionals come into play. Also, there is a ripple effect that moves out from the management process effecting every stage of the process; from data protection (backup to restore), disaster recovery and migration, capacity management, corporate governance and beyond.

Together, the information creators, IT professionals, RIM professionals and other organizations can create processes that deal with each type of information and review them on an ongoing basis against governmental and business requirements. Also, in this way, when a new application comes online, the team can create solutions utilizing best practices learned from working on other projects.

Tomorrow I will draw the complete picture and a process that links the two organizations together with the rest of the organization.
Posted by Brett Cooper Who is responsible for ILM?
28 MAR 2006 19:19 EST (00:19, GMT)
The genesis of information lifecycle management (ILM) can be found in the work of records and information managers dealing with paper records decades ago. So, my friends, ILM isn't new, rather it is steeped in tradition and has some good established processes around it. These records and information managers banded together to create an organization where they can exchange best practices, develop standards and network -- ARMA International. ARMA International was founded in 1955 and has approximately 10,000 members, including records managers, archivists, corporate librarians, imaging specialists, legal professionals, IT managers, consultants and educators (from the ARMA International Web site). This group of information management professionals leads the way in helping organizations leverage the value of records, information and knowledge as a strategic corporate asset. Don't be surprised to find that your organization has a Records and Information Management (RIM) department as part of the CIO's office.

Who is responsible for managing records and information within an organization? The answer is simple: everyone. Each employee has an important role to play in protecting the future of the organization by creating, using, retrieving and disposing of records in accordance with the organization's established policies and procedures. One of the new goals for IT personnel (and others) should be to work closely with the RIM team creating an infrastructure and processes that meet the needs of the information. You may be saying, "OK, Brett is going to suggest another layered schema as a solution." And guess what, you would be right. The closer that one gets to the information, the better one is able to manage it. If the information is a contract that is easy to read and the person reading it knows the two parties that the contract is between, then he/she can easily process it; but if the information is 0s and 1s in a container that is being managed by a storage administrator who is dealing with a capacity issue -- buying more storage again -- then it is impossible to value the contract and process it accordingly. This in itself lends to create a horizontal and vertical approach to dealing with the information. Also, the rules for processing the information are established. More on the schema in tomorrow's entry…

You may be asking, what is records management? There are a couple of definitions:

1. "A field of management responsible for the systematic control of the creation, maintenance, use, reproduction, disposition of corporate records." Society of American Archivists --1992

2. "Records Management is the planning, controlling, directing, organizing, training, promoting, and other managerial activities involving the lifecycle of information, including creation, maintenance (use, storage, retrieval) and disposal, regardless of media." The Rockley Group -- 2003

Records come in many formats:

• Physical paper in our files such as memos, contracts, marketing materials and reports.

• An electronic record is a record created, generated, sent, communicated, received or stored by electronic means. Like paper records, electronic records require a long-term records management strategy.

• Electronic messages such as e-mail content and their attachments and instant messages.

• Content on the Web site, as well as the documents that reside on PDAs, flash drives, desktops, servers and document management systems.

• Information captures in the organization's various databases.

• When there's a lawsuit, all of these -- including the copies that individuals have retained and any items deleted from the system -- may be identified as discoverable. That means they could be used against the organization in a lawsuit (What is Records Management? Why Should I care? ARMA International, Copyright 2005).

This discussion about information management varies greatly from the traditional role of the Information Technology department. Traditionally, the IT department is concerned with managing data, or the 0s and 1s that sit on containers or infrastructure throughout the organization. This data is the building blocks of information, in much the same way that proteins make up DNA or genetic code that defines a living thing -- differentiating it from other living things, the data makes up information that the organizations uses to make its larger decisions, effecting the body of the organization itself.

The first time I heard the term Information Lifecycle Management used in the storage industry, was from STK about six years ago. The team at STK had a lock on the idea that information's value changes based on the organization's measurements. A few years later the ILM idea was co-opted by EMC and turned into a major marketing campaign defining every offering in their portfolio. And finally, the Storage Networking Industry Association (SNIA) brought the vendor community together to formally define ILM for the masses:

The policies, processes, practices, services and tools policies, processes, practices, services and tools used to align the business value of information used to align the business value of information with the most appropriate and cost effective with the most appropriate and cost effective infrastructure from the time information is created infrastructure from the time information is created through its final disposition. Information is aligned through its final disposition. Information is aligned with business requirements through management with business requirements through management policies and service levels associated with policies and service levels associated with applications, metadata, and data.

--2003, The SNIA