Data Resource Quality: Turning Bad Habits into Good Practices

Read an Excerpt

Why is the data resource failing to adequately support an organization's information needs? What did organizations do to get into their current disparate data situation? What do organizations, public and private, large and small, new and old, do to ruin their data resource quality? What do organizations consistently do, or not do, to mess up one of their most critical resources? Why have people allowed this situation to happen and to continue for so long?

What is it that organizations are doing wrong that results in a low-quality data resource? What bad habits should be avoided? What good practices should be followed? People are asking me these questions with increasing regularity. The underlying theme of most of these questions is, What can organizations do to prevent any further data disparity and develop a high-quality data resource?

I started emphasizing the need to improve data quality in the early 1980s and have continued to emphasize the importance of data quality for the last 15 to 20 years. I have written several books and many articles about the current state of the data resource and how a quality data resource can be developed that truly supports the organization's business information needs. These books and articles explain the concepts, principles, and techniques for designing and building a high-quality data resource within a single, comprehensive, organization-wide common data architecture.

This book is about the ten most prominent ways I have found that organizations destroy their data resource quality or prevent development of a high-quality data resource. It is built around ten sets of bad habits organizations have that build defects into the data resource; bad habits that increase the cost of building, maintaining, and using a data resource; bad habits that waste resources on data; bad habits that use the data resource to cripple an organization; bad habits that compromise business strategies by using data as an internal weapon. If I were to set about slowly, quietly, and subtly ruining the quality of an organization's data resource, these are the things that I would do.

Not all organizations have all of these bad habits, nor are the bad habits done with the same degree of severity on all data in all organizations. An organization's data resource is a blend of the bad habits and the severity of those bad habits governs the current state of the data resource in any organization. These are not all the possible bad habits either; but, overall, they are the most frequent and most severe bad habits that lead to a low-quality data resource.

The material in this book was gained over the last 20 years as I reviewed the state of the data resource in many public and private sector organizations. In 1990 I made a statement that I never ceased to be amazed at the ingenious ways that organizations could mess up their data. Since then, I have been literally astounded at the very ingenious, and sometimes quite stupid, ways that organizations ruin a very critical resource.

Understanding the bad habits, however, does not eliminate those bad habits or improve data resource quality. It only raises the awareness about problems that cause a low-quality data resource. A more positive approach is to understand what needs to be done to replace the bad habits with good practices. The main emphasis of this book is what organizations, public or private, large or small, local or multi-national, can do to turn bad habits into good practices that lead to improved data resource quality.

The first chapter explains the current state of the data resource in most public and private sector organizations. The next ten chapters explain each of the ten sets of bad habits that contribute to a low-quality data resource, the impacts of those bad habits, how those bad habits can be turned into good practices, the benefits of the good practices, and the best practices to implement for quick results. The first five of these chapters deal with architectural issues, and the next five chapters deal with non-architectural issues that pertain largely to organizational culture and management of the data resource. The final chapter describes a new direction for improving data resource quality.

This is not a technical book and does not contain all the techniques for building and maintaining a high-quality data resource and resolving all existing data disparity. Those topics are covered quite adequately in two previous books. This is a detail book about what can go wrong with data resource quality and what can be done to stop data disparity and improve data resource quality. It contains a summary of the concepts and principles presented in previous books and presents some new concepts and principles about improving data resource quality. It emphasizes what can be done to implement an organization-wide, integrated, subject-oriented data architecture and then build a high-quality data resource within that architecture.

This book is not about the tools or technology currently available to ensure a high-quality data resource. Technology evolves, and what may be current technology when this book was written will be inappropriate or obsolete in the near future. The fundamental problems that result in a low-quality data resource and the basic principles for ensuring a high-quality data resource are, however, relatively static and are independent of the current technology. Therefore, this book concentrates on the basic principles pertaining to data resource quality and not on current technology.

Some people may perceive that this book is academic or esoteric because it does not explain the current tool set that is available for developing a high-quality data resource. These people are looking for readily-available technology that will help them solve their current problems, and anything that does not provide how-to solutions to those problems is perceived to be academic or esoteric. These people are usually looking for quick-fix solutions that resolve current problems with little concern for the future.

Some people may feel that new organizations or organizations involved in Internet and Web-based business do not need to understand what can go wrong with a data resource. These people, however, need to understand what can go wrong so they can prevent the development of a low-quality data resource. Every organization was new at one time and every organization implemented new technology at one time, yet these organizations have low-quality data resources today. There is no reason to believe that new organizations today will be immune to the evolution of a low-quality data resource. It is the case that those who do not know history are destined to repeat it! Organizations that know what can go wrong with a data resource are more likely to take steps to avoid those situations than organizations that do not know what can go wrong.

There are two major phases to improving data resource quality. The first phase is a proactive phase to stop any further data disparity; the second is a reactive phase to reduce or eliminate the existing data disparity. This book is about what needs to be done to stop the burgeoning increase in data disparity that is occurring in most organizations today. It does not explain what needs to be done to resolve any existing data disparity. A forthcoming book describes what needs to be done to formally transform existing disparate data within a common data architecture.

This book is intended to raise awareness about the reasons why data resource quality deteriorates and what can be done to prevent those problems. The orientation is toward turning bad habits into good practices that will achieve a high-quality data resource. By understanding what makes the data resource go bad, organizations can learn to recognize the situation and prevent any further data disparity. The material in this book can also be used to evaluate the current state of data resource quality and to lay plans for stopping data disparity and developing a high-quality data resource that produces early results. The material provides hope and encouragement for people who are buried in a data resource mess and perceive that there is no way out.

This book is intended for two audiences. The first audience is executives or managers who need to be aware of the data resource problems, but do not need to be involved with the details about identifying or resolving those problems. The second audience includes people who want to understand what goes wrong with data resource quality and be able to implement solutions to prevent those problems and improve data resource quality.

Chapter 1 explains that the current state of the data resource in most organizations today is one of low quality and that quality is getting lower rather than better. It sets the stage for taking a new direction for improving data resource quality. Chapter 12 is a call to action for organizations to improve their data resource quality, but it is a different type of call for action. It is not the current hype about implementing new technology that can cleanse the data resource in one pass and produce high-quality data. It is a call to action based on what can go wrong with a data resource and what can be done to stop the things that go wrong. These two chapters are useful to both audiences.

Chapters 2 through 11 explain the ten major groups of bad habits that lead to a low-quality data resource and the corresponding good practices that lead to a high-quality data resource. These chapters present the bad habits, the impacts of those bad habits, the good practices that replace the bad habits, the benefits of those good practices, and the best practices that provide early benefits with minimal effort. These chapters are useful to the second audience.

Executives, managers, and others who are interested only in the current state of the data resource and a call to action to improve data resource quality should read Chapter 1 and jump to Chapter 12. They can also refer to Appendix A that summarizes the bad habits and good practices and to Appendix B that summarizes the evaluation criteria for the current state of a data resource. Those who want to understand the specific bad habits and good practices can read the chapters in sequence or can read Chapter 1, jump to Chapter 12, and then return to Chapters 2 through 11.

This book does not explain specific impacts, because the same specific impacts cannot be used for public and private sector organizations. The public sector deals with citizens and the private sector deals with customers. Citizens, in most situations, do not have the same choices that customers have. Unlike customers, they cannot choose to shop elsewhere, wait for sales, or decide not to purchase. They are bound by regulations rather than by choice. Public sector organizations do not face situations like loss of customers, loss of market share, competitive advantage, or bottom-line profits like private sector organizations.

Each organization needs to take the guidelines provided and prepare its own quantitative measures to the impacts of bad habits and benefits of good practices. Organizations are just too varied in their size and orientation to provide general quantitative measures. For example, if a comment was made that a certain bad habit resulted in unnecessary costs of $1,000,000 a year, a large organization might consider this to be pocket change. Yet an unnecessary cost of $1,000,000 a year might be double, triple, or quadruple the annual budget of a smaller organization.

The business changes over time, and the information needed to support the business must also change. The data resource must be dynamic enough to provide the data to support those changing information needs. A formal data architecture that is oriented toward the business and integrated across the business is the only way to build a dynamic data resource. If the data architecture is allowed to deviate from this orientation, so will the data resource, the information that is based on the data resource, and the business that is based on the information. If, however, a formal data architecture is properly developed and maintained, a high-quality data resource can be developed and maintained.

You will see in the first chapter that data resource quality is defined as how well the data resource supports the current and future information needs of the organization. Using this definition of data resource quality sidesteps the hype and metrics commonly associated with data resource quality and focuses on what data resource quality really means to the current information needs of an organization. This new focus, however, opens up a broader issue about what data resource quality really means with respect to the future information needs of an organization. Since technology is evolving at an ever-increasing rate, the ultimate data resource quality is achieving data resource stability across technological change!

As we enter the new millennium, I believe that progressive organizations will become more interested in data resource quality and in the frameworks and architectures that promote improved quality. As organizations finish resolving their Y2K and Y2K-related problems, the orientation is turning toward data quality and the degree to which the data resource supports the business information needs of an organization. The Y2K problem is only the beginning of many, many more data resource quality issues. It should be the wake-up call to the state of the data resource in many organizations.

Michael H. Brackett Olympia, Washington June 2000

Read More