By Dr.Thomas J Mowbray Chairman & Chief Architect - iCMG

Solving complex problems with teams of people requires planning. For enterprise software systems, some of the most important planning is highly technical (i.e. planning system architecture).

Planning generates artifacts, but planning (as an activity) is much more important, than project management plans, the typical artifacts. By this, we mean that document-driven processes, which focus on paper artifacts, are often ineffective, whereas the real product of any software development project is software. Instead, we view planning in a broader context, with multiple levels of formality and technical detail. For example, architecting is planning, and so is requirements analysis, design modeling and generating plans. The level of formality should be tied to the document audience's needs, including the longer-term usefulness of the documentation.

In architecture-centered development, planning is pragmatic (Figure 1). Project plans and design models are throwaway documentation because their accuracy is short-lived. Once a plan or specification is out of date, it is essentially useless. For example, source code changes can quickly obsolesce design models.

Figure 1. Without Planning, It Becomes Apparent That Many Individual Successes Are Not Sufficient for Overall Project Success.

In addition, software methods and standards should be treated as guidelines, not mandates. Project teams are encouraged to think for themselves, and tailor the process to meet the project's needs.

Pragmatics is a fundamental principle of software modeling: for requirements, architecture, and design. Every model has a purpose and focus, suppressing unimportant details. Models should document important decisions, based upon project assumptions and priorities. Deciding what's important is an essential decision skill that is part of being a competent architect.

Architecture-Centered Process

Figure 2 shows the 10-step process for Architecture-Centered Development, that covers the full system lifecycle. This process is based upon key software standards, and best-practice patterns proven in practice.

Figure 2. Architecture Centered Development Process

A key objective is to facilitate productivity in Step 7 for parallel iterative development (i.e. coding and testing). In this discussion, the activities preceding Step 7 are emphasized, because these steps comprise the architecture planning activities where we believe the key issues reside in current enterprise development.

We emphasize that this process is inherently iterative and incremental perhaps requiring revisions to artifacts from previous steps. However, the pre-development steps do have a waterfall progression, due to their interdependencies. The entire process is quality driven, with the ultimate goal of satisfying end-user needs, by establishing a stable architecture description and a working software codebase that can accommodate change.

Step 1: System Envisioning

In discussing modeling, we mentioned the key words purpose, focus, assumptions, and priorities. These are all essential elements of a systemwide Vision Statement. If they change during system development, the project is at risk of obsolescing its own models. Therefore, the first step of architecture-centered development is to establish a Vision Statement, with the binding assumption that the Vision Statement cannot change, once development begins (Step 7). Any changes must be reflected in key project plans, in particular, the System Architecture (Step 3).

In effect, the Vision Statement is a binding agreement between the system developers and the system users. It should be short and to the point, typically less than 10 pages of text, depending on the system.

The Vision Statement establishes the context for all subsequent project activities, starting with requirements analysis.

Step 2: Requirements Analysis

The requirements should define the external behavior and appearance of the system, without designing the internal structure of the system. The external behavior includes internal actions (such as persistence or calculation) that are required to ensure desired external behavior. The external appearance comprises the layout and navigation of the user interface screens.

An effective approach for capturing behavioral requirements is through use cases. A use case comprises a top-level diagram and extensive textual description. Use case notation is one of the most effective notations ever devised for expressing complex concepts. Hence, it's great for ensuring simplicity and clarity in representing top-level requirements concepts.

For each circle in the diagram (called an individual use case), there is an extensive textual description of the relevant requirements. This write-up takes the form of a long list, containing a sequence of actions, described in domain-specific prose. The definition of use cases should be done jointly with domain experts. Without continuous involvement of domain experts, the exercise is a common antipattern called Pseudo Analysis, i.e. something to be avoided.

Use cases provide a domain model of the system for the purpose of defining architecture. Use cases also have a downstream role. In development, Step 7, use cases are extended with system-specific scenario diagrams. Eventually, these scenarios are elaborated into software tests.

The appearance, functionality, and navigation of the user interface is closely related to the use cases. An effective approach to defining the screens is called low fidelity prototyping. In this approach, the screens are drawn out with paper and pencil. Again, the end-users domain experts are continuously involved in the screen definition process.

With the use cases and user interfaces defined, we have established context for architectural planning. In addition to generating documentation (including, paper and pencil sketches), the architecture team acquires a deep understanding of the desired system capabilities in the context of the end-user domain.

A final product of requirements analysis is a project glossary which should be extended during architecture planning (Step 3).

Step 3: Architecture Planning

Architecture bridges the huge semantic gap between requirements and software. Because requirements notation is prose, requirements are inherently ambiguous, intuitive, and informal. It's left-brain stuff. Software, on the other hand, has the opposite characteristics. Software source code is a formal notation. Software is interpreted unambiguously by a machine, and it's meaning is logically unintuitive (i.e. hard to decipher). It's right-brain stuff.

Architecture's first role is to define the mapping between these two extremes. Architecture captures the intuitive decisions in a more formal manner (which is useful to programmers), and it defines internal system structure before it is hardwired into code (so that current and future requirements can be satisfied). Architecture is a plan that manages system complexity in a way that enables system construction and accommodates change. Architecture has another significant role defining the organization of the software project. (See Step 6).

Architecture planning is the key missing step in many current software projects, processes, and methods. One causes of this gap is the ongoing debate about the question: "What is architecture?" Fortunately, this question has already been answered definitively, by the software architecture profession, in a formal ISO standards for Open Distributed Processing (ODP).

ODP is a powerful way to think about complex systems, which simplifies decision-making (i.e. working smarter, not harder). It organizes the system architecture in terms of five standard viewpoints, describing important aspects of the same system. These viewpoints include business enterprise, logical information, computational interface, distributed engineering, and technology selection (Figure 3).

Figure 3 ODP Viewpoints

For each viewpoint it is important to identify conformance to architectural requirements. If conformance has no objective definition, then the architecture is meaningless, because it will have no clear impact upon implementation. ODP facilitates this process because ODP embodies a pervasive conformance approach. Simple conformance checklists, are all that's needed to identify conformance points in the architecture.

Step 4: Mockup

The screen definitions from Step 2 are used to create an on-line mockup of the system. Dummy data and simple file IO can be used to provide more realistic interface simulation in key parts of the user interface. The mockup is demonstrated to end-users and management sponsors.

End users and architects should jointly review the mockups and run through the use cases (Step 2) in order to validate requirements. Often, new or modified requirements will emerge during this interchange. Generate screen dumps of any modified screens and mark them up for subsequent development activities. Any modifications to requirements are then incorporated by the other architectural activities.

Through the mockup, management is able to see visible progress, a politically important achievement for most projects. This step is an example of an external (or vertical) increment, which is used for risk reduction, both political and requirementswise.

With rapid prototyping technologies such as screen generation wizards, mockups can be generated in less than a staff month, for most systems.

Step 5: Architecture Prototyping

Architecture-centric software development addresses the big risks up front, such as integration risks and system performance risks. Architects are concerned with system-wide behaviors and challenges that can "make or break" the system. As a result, a goal of architecture is to avoid big, expensive mistakes in systems development; whereas, many other IT disciplines are focused on "making no small mistakes". Whenever an important unknown can't be resolved by paper studies, prototyping is appropriate. The purpose of architecture prototypes is to resolve big risks early in the system development cycle.

The architecture prototype is a experimental evaluation of the system architecture. For example, system API definitions can be compiled and stub programs are written to simulate the executing system. This architectural skeleton can be used to check cross-platform integration, as well as performance. When important, a trial interface between key software packages is also implemented. The architecture prototype is used to validate the computational and engineering architectures, including flow of control and timing across distribution boundaries.

Using technologies like CORBA, a computational architecture specification can be automatically compiled into a set of programming header files with distributed stubs (calling side) and skeletons (service side). Dummy code is inserted in the skeletons to simulate processing. Simple client programs are written to send invocations across computational boundaries with dummy data. A handful of key (e.g. high-risk) use cases are simulated with alternative client programs. Prototype execution is timed to validate conformance with engineering constraints.

Changes to the computational, engineering, or technology architectures are proposed, and evaluated. The "throw away" code from the architectural prototype can be used to create an architectural template for the system construction team. This is one of the ways to ensure that the architect's intentions are actually implemented.

Step 6: Project Management Planning

As the final step in the pre-development process, project management plans are defined and validated to resolve resource issues, including staffing, facilities, equipment, and commercial technology procurement. A schedule and budget are established, according to the availability (lead time) for resources and project activities.

The schedule for Step 7 is planned in terms of parallel activities for external and internal increments. External increments (focused on user-visible functionality) support risk reduction with respect to requirements and management support (See Step 4). Internal increments support the efficient use of development resources, for example, the development of back-end services used by multiple subsystems. Internal increments are the most cost-effective, involving the least "throw-away" code, but external increments are often politically necessary, as well.

Current best practices are to perform several smaller internal increments supporting larger scale external increments, called VW staging. Ideally, several project teams of up to 4 programmers are formed, with 3-month deliverable external increments. In practice, this has proven to be the most effective team size and increment duration.

The architecture-centric process enables the parallel increments. Because the system is partitioned with well-defined computational boundaries, development teams can work independently, in parallel with other teams, within their assigned boundaries. Integration planning includes increments, which span architectural boundaries.

The detail in the project plan should not be inconsistent. The plan should be very detailed for early increments, and should include re-planning activities, for later in the project. This recognizes the reality that; project planners don't know everything up front.

A risk mitigation plan is also prepared with identification of technical backups. The development team involved in mockup and architecture prototyping should continue to develop experimental prototypes with high-risk technologies in advance of the majority of developers. This is called the "run ahead team" and is a key element of risk mitigation.

The final activity is project management planning is the architectural review and startup decision. Up to this point, the enterprise sponsors have made relatively few commitments, compared to the full-scale development (about 5% of system cost, depending on the project).

Executive sponsors of the project must make a business decision about whether to proceed with building the system. This executive commitment will quickly lead to many other commitments which are nearly impossible to reverse (such as technology lock-in, expenses, and vendor-generated publicity). At this point, the system architects are offering the best possible solution and approach, in the current business and technology context.

If the system concept still makes business sense, compared to the opportunity costs, the enterprise is in an excellent position to realize the system because they're doing software right.

Step 7: Parallel Incremental Development

Development project kickoff involves several key activities. The developers must learn and internalize the architecture and requirements. An effective way to achieve this is with a multi-day kickoff meeting, which includes detailed tutorials from domain experts and architects. The results of all previous steps are leveraged to bring the developers up to speed quickly and thoroughly. We suggest that the lectures be videotaped, so that staff turnover replacements can be similarly trained.

Each increment involves a complete development process, including design, coding, and test. Initially, the majority of the increments will be focused on individual subsystems. As the project progresses, an increasing number of increments will involve multiple subsystem integration. A project rhythm is established that enables coordination of development builds and tests.

For the most of the software development activity, the architecture is frozen, except at some planned points, where architectural upgrades can be inserted without disruption. Architectural stability enables parallel development.

For example, at the conclusion of a major external increment, an upgrade to the computational architecture can be inserted, before the next increment initiates. The increment starts with an upgrade of the software, conformant with the changes. In practice, the need and frequency of these upgrades decreases as the project progresses. The architect's goal is to increase the stability and quality of the solution, based upon feedback from development experience. A typical project would require two architectural refactorings (upgrades) before a suitably stable configuration is achieved for deployment.

Step 8: System Transition

Deployment of the system to a pilot group of end users should be an integral part of the development process. Based upon lessons learned in initial deployment, development iterations might be added to the plan. Schedule slips are inevitable, but serious quality defects are intolerable for obvious reasons. Improving quality by refactoring software (improving software structure) is an important investment in the system that should not be neglected.

An important architect's role in this step involves system acceptance. The architect should confirm that the system implementation is conformant with the specifications and fairly implements the end users' requirements. This task is called architectural certification.

In effect, the architect should be an impartial arbitrator between the interests of the end users and those of the developers of the system. If the end-users define new requirements, which impact architectural assumptions, the architect assesses the request, and works with both sides to plan feasible solutions.

Step 9: Operations and Maintenance

Operations and Maintenance (O&M) is the real proving ground for architecture-centered development. Whether or not the result of "doing software right" was effective, will be proven in this step. The majority of system cost will be expended here. And as much as 70% of the O&M cost will be due to system extensions-requirements and technology changes that are the key source of continuing development.

Typically, half of programmer's time will be expended trying to figure out how the system works. Architecture-centered development resolves much of this confusion with a clear, concise set of documentation, the system architecture.

Step 10: System Migration

System migration to a follow-on target architecture occurs near the end of the system lifecycle. Two major processes for system migration are called big bang and chicken little. A big bang is a complete, overnight replacement of the legacy. In practice, the big bang seldom succeeds; it is a common antipattern for system migration.

The chicken little approach is more effective, and ultimately more successful. Chicken little involves simultaneous, deployed operation of both target and legacy systems. The initial target system users are the pilot group (as in Step 8).

Gateways are integrated between the legacy and target systems. Forward gateways allow legacy user to have access to data that is migrated to the target system. Reverse gateways allow target system users to have transparent access to legacy data. Data and functionality are migrated incrementally from the legacy to the target system. In effect, system migration is a continuous evolution. As time progresses, new users are added to the target system and taken off the legacy environment.

In the long term, it will become feasible to switch off the legacy system. By that time, it is likely that the target system will become the legacy in a new system migration. The target system transition, Step 8, overlaps the legacy system migration, Step 10. In the chicken little approach, steps 8, 9, and 10 are part of a continuous process of migration.

Concluding Remarks

From experience, we have seen so many projects doing software wrong, that it's no wonder five out of six projects are unsuccessful. The age of the heroic programmer is coming to an end, and the age of the professional software architect has begun. Driven by escalating user expectations, business changes, and technology innovations, many organizations now realize that proper system planning generally translates into system success, and improper planning leads to system failure.

The role of the software architect is relatively new in many project cultures. What has been called architecture, informally, needs to become conformant with standards and best practice patterns, if consistent development success is desired.