Agile Database Techniques

John Wiley & Sons

Chapter One

It is possible to take an agile approach to data-oriented development. The first step is to choose to work this way.

Data is clearly an important aspect of software-based systems - something we've all known for decades - and yet many organizations still struggle with their approach to data-oriented issues within their software processes.

The goal of the agile data (AD) method is to define strategies that enable IT professionals to work together effectively on the data aspects of software systems. This isn't to say that AD is a "one size fits all" methodology. Instead, consider AD as a collection of philosophies that will enable software developers within your organization to work together effectively when it comes to the data aspects of software-based systems. Although the focus of this book is proven techniques for agile software development, it's critical to define an underlying methodological foundation.

In this chapter, I help you understand the AD method by exploring the following topics:

* Why working together is currently difficult

* The agile movement

* The philosophies of agile data

* Agile data in a nutshell

* Does agile data address our problems?

Why Working Together Is Currently Hard

In many organizations, therelationship between data professionals and developers is often less than ideal. Yes, there are some organizations where these two communities work together quite well, but there are always tensions - some healthy tension exists between groups, and from these your organization can benefit, but when the tension isn't healthy these differences often lead to conflicts. The challenges that data professionals and developers must overcome can include:

Different visions and priorities. Developers are often focused on the specific needs of a single project and often strive to work as much as possible in isolation from the rest of the organization. Database administrators (DBAs) focus on the database(s) that they are responsible for, often "protecting" the databases by minimizing changes to them. Data administrators and data architects focus on the overall data needs of the enterprise, sometimes to the virtual exclusion of the immediate needs of project teams. Clearly, the scope of each group is different, their priorities are different, and the issues that the groups deal with are different. To make matters worse, your project stakeholders, including direct users all the way up to senior management, have varying priorities and visions as well.

Overspecialization of roles. Specialists have a tendency to become too narrowly focused; they can work so hard to know everything there is to know about a small slice of software development that they can become oblivious of everything else. For example, it's quite common to find senior Java developers that have never heard about data normalization (discussed in Chapter 4), or even understand why you would want to do such a thing, and data architects who can't read a Unified Modeling Language (UML) state chart diagram (discussed in Chapter 2). Because these roles are overly specialized, the people in those roles often have difficulties relating to others. At the other end of the spectrum are generalists who understand the big picture but don't have any concrete skills to offer a development team. We need to find the sweet spot between these two extremes. An underlying philosophy of Agile Modeling (Ambler 2002a) is that software developers should have a general understanding of the overall software process and have one or more specialties. Because agile modelers are generalists, they understand the broad range of issues pertinent to the "software game" and yet they still have specific, valuable skills to offer to their team.

Process impedance mismatch. One of the few things that processes such as the Unified Process (Kruchten 2000; Ambler 2001b), Extreme Programming (XP) (Beck 2000), Scrum (Beedle and Schwaber 2001), DSDM (Stapleton 1997), Crystal Clear (Cockburn 2001b), feature-driven development (FDD) (Palmer and Felsing 2002), and Agile Modeling (AM) have in common is that they all work in an evolutionary (iterative and incremental) manner. Unfortunately, many within the data community still view software development as a serial or near-serial process. Clearly, there is an impedance mismatch here, indicating that the data community needs to rethink its approach. You will see in Part II of this book that it is possible to take an evolutionary approach to data, a change that will require cultural and organizational adjustments to succeed.

Technology impedance mismatch. Developers work with objects and components, whereas data professionals work with databases and files. Software-engineering principles form the underlying foundational paradigm for objects and components, whereas set theory forms the underlying foundational paradigm for relational databases (by far the most popular database technology). Because the underlying paradigms are different, the technologies don't work together perfectly, and an impedance mismatch exists. This mismatch can be overcome, although doing so requires a significant skillset (this topic is covered in Chapter 7).

Ossified management. The technology and techniques used by software developers change rapidly, a fact that we all know very well. As people progress up the corporate hierarchy, they deal less with technology and more with people issues, the end result being that many managers have lost their technical edge. The implication is that management's previous development experiences, on which they base technical decisions, may no longer be applicable. We experienced this when we moved from procedural to object-oriented technologies - what may have been a good decision on a COBOL project often proves to be the kiss of death to a Java project. We're clearly seeing this problem once again as we move to agile software processes. Management needs to change with the times.

Organizational challenges. Common problems, such as poor communication or politics among individuals and groups, hurt the data aspects of software development just as badly as they hurt other efforts, by preventing everyone from working together effectively.

Poor documentation. Most documentation seems to be at one of the following extremes: little or no documentation or overly complex documentation that nobody reads. Mutually agreed-to development standards and guidelines, legacy system documentation, legacy database documentation, and enterprise models can be valuable resources when written well. Chapter 10 presents agile strategies for writing documentation.

Ineffective architectural efforts. Most organizations face significant challenges when it comes to enterprise architecture, the most common of which being that they don't know where to start. Biased enterprise architectures that overly focus on one view of the enterprise lead to architectures that do not adequately address the real needs of an organization. As the Zachman Framework (ZIFA 2002; Hay 2003) indicates, there are many potential views that you want to consider. These views are data/structure, function/process, network, people, time, and motivation. Ivory tower architectures - those formulated by teams that have removed themselves from the day-to-day realities of project teams - look good on paper but unfortunately fail in practice. Furthermore, developers need to accept that their efforts must reflect and conform to the constraints imposed on them by their organization's environment.

Ineffective development guidelines. Many organizations struggle to come to a collection of development guidelines that all software developers will work to. There are a large number of causes for this, including people not understanding the need to follow such guidelines, people unwilling to follow someone else's guidelines, overly complex guidelines, overly simplistic guidelines, a "one size fits all" attitude that leads to inappropriate guidelines for a specific platform, and an unwillingness to evolve guidelines over time. When you have an effective collection of guidelines available to you, and (this is key) everyone understands and applies them appropriately, you can dramatically improve the productivity of your software development efforts.

Ineffective modeling efforts. This is often the result of several of the previously identified problems. People focused on a specific aspect of development will often produce models that wonderfully reflect the priorities of that narrow view but fail to take into account the realities of other views. An enterprise data model may present an excellent vision of the data required by an organization, but an enterprise model that reflects the data, functional, usage, and technical requirements of an organization is likely to be far more useful. A UML class diagram may reflect the needs of a single project, but if it doesn't reflect the realities of the legacy data sources that it will access then it is of little value in practice. Modelers, and software developers in general, need to work together and look at the full picture to be truly effective.

Detecting That You Have a Problem

It is very easy for organizations to deny that they have a problem. It can be very difficult for senior management to detect problems until it's too late because the bad news that they need to hear is filtered out long before it gets to them. Similarly, it can be difficult for people elsewhere in the organization to detect problems - perhaps everything is going quite well in their opinion - unfortunately the value system that they're using to judge the situation isn't ideal, making them blind to the problems that they are causing.

As a consultant I have the privilege of working in a wide range of organizations, and it seems to me that about one in ten organizations is reasonably successful with its approach to data-oriented activities, about six in ten think they're doing well but really aren't, and the remaining three in ten know that they have a problem but don't know what to do about it. It doesn't have to be this way.

So how do you know you've got a problem? Enterprise data professionals, including both data architects and data administrators, will be frustrated by the fact that project developers on project teams ignore their advice, standards, guidelines, and enterprise models. Worse yet, application developers often don't even know about these people and things in the first place. Developers will be frustrated by what they perceive (often rightfully so) to be the glacial pace of enterprise data professionals to make or authorize seemingly simple changes. DBAs often find themselves stuck in between these two warring factions, trying to get their work done, while struggling to keep the peace. If one or more of these problems is common within your organization you've got a problem.

The following is a list of potential symptoms that may indicate that your organization has one or more challenges that the agile data method may help you address:

* People are significantly frustrated with the efforts, or lack thereof, of one or more groups.

* Software is not being developed, or if it is it is taking far too long or is much too expensive.

* Finger pointing occurs such that you hear things like "the data administrators are holding up progress" or "the developers aren't following corporate guidelines." Worse yet, the finger pointer typically doesn't perceive that he or she is also part of the problem.

* Political issues are given higher priority than working together to develop, maintain, and support software-based systems.

* Ongoing feuds exist between people and groups. Phrases that start with "you always" and "you never" are good indicators of this.

* Well-known problems within your organization are not being addressed. Furthermore, suggestions for improvements appear to be ignored, nothing happens, and no reason for rejection is provided.

* People are working excessively long hours with little or no reward.

* Decisions affecting teams - in particular project teams - are made in an apparently arbitrary and arrogant fashion.

We need to find a way to work together effectively. There are clear differences between the data and development communities as well as between the project and enterprise communities. The fact that we're talking about different communities is also part of the problem, arguably one of the root causes. You have a fundamental decision to make: Should you use these differences as an excuse to exacerbate existing problems within your organization or should you revel in these differences and find a way to take advantage of them? I prefer the latter approach. My experience is that the values and principles of the agile movement form the basis for an effective approach to working together.

The Agile Movement

To address the challenges faced by software developers an initial group of 17 methodologists formed the Agile Software Development Alliance (agilealliance.org), often referred to simply as the Agile Alliance, in February 2001. An interesting thing about this group is that the members all came from different backgrounds, and yet they were able to come to an agreement on issues that methodologists typically don't agree upon (Fowler 2001a). This group of people defined a manifesto for encouraging better ways of developing software, and then, based on that manifesto, formulated a collection of principles that defines the criteria for agile software development processes such as AM.

(Continues...)

---

Excerpted from Agile Database Techniques by Scott Ambler Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---