Edited transcript of BriefingsDirect[TM] podcast on the book "Succeeding with SOA: Realizing Business Value Through Total Architecture," with author Dr. Paul Brown, recorded
Listen to the podcast here. Sponsor: TIBCO Software, Inc.
Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you're listening to BriefingsDirect. Today, a sponsored podcast discussion about an interesting book called "Succeeding with SOA: Realizing Business Value Through Total Architecture."
We're going to be reviewing the book and discussing some of its implications for businesses and IT departments -- that Total Architecture impacts the very notion of enterprises and IT. With us is the author of the book, Dr. Paul Brown, a principal software architect at TIBCO Software. Welcome to the show, Paul.
Paul Brown: Hello, glad to be here.
Todd Biske: Thanks, Dana.
I think that’s what you drive at with this concept of "Total Architecture." Am I getting that right, Paul?
Brown: Yes, definitely. Our enterprise business processes and our enterprise systems have become so hopelessly intertwined that we can’t really talk about designing one without designing the other. We’re moving lots of activities that used to be done manually either completely or partially into IT systems.
Yet we still have to have people involved for the exception handling, even when we try to do a fully automated system. So the idea is that you have to explore the design of both together -- and you can’t separate them.
But it seems that things are not progressing, that there are scaling and complexity issues. You say in the book that there we are "stressing" existing architectures. Can you elaborate on that?
Brown: There are a couple of things going on. The structure and organization of IT is very often not completely aligned with the business. The care and feeding of the IT infrastructure has become a focal point of a lot of the IT investment.
What we’ve lost is the connection between what’s going on in IT and what’s going on in the business. It’s somewhat analogous to when we started exploring object oriented design (OOD). One of the benefits there was that the structure of the problem now became recognizable in the structure of the solution.
We’re faced with a similar situation today. The structure of our IT solutions are difficult to map onto the structure of the business processes. It’s that realignment that we’re trying to address with SOA.
Ideally, the services that we’re building -- that have technical implementations -- are building blocks of business processes. And so we see that tie-in once again.
Brown: SOA is an architecture in which the functions that you need in your business processes become recognizable services of your IT information structure. At a lower level, we can also build services that the IT community consumes. Those are technical services. They become the lower-level building blocks for ultimately building the business services.
The real business value comes out of having a portfolio of business functionality in the form of services that you can quickly reorganize and re-orchestrate to build new business processes or to modify your existing processes. That’s where the big business win comes in.
Brown: You almost can. Most of the functionality that we need is in those applications. The problem is that the functionality wasn’t designed for reuse. Applications tend to be somewhat specialized for use in a particular situation. Their functionality is hard to get at.
It’s a COBOL mainframe transaction, and you have to be on the mainframe to invoke it. Or it’s a piece of Java code, and you need to be in Java to invoke it. One of the things we’re trying to get to here is just to make the stuff more accessible and to standardize it in such a way that it reasonably fits into different business situations.IT Misalignment
Where do you see the root causes here? What is the main problem that is preventing IT from responding today?
Biske: Every organization is going to have its own distinct challenges, but as with a lot of things you have to look at the whole notion of the enterprise -- not just IT, not just business -- the whole piece of it.
Dr. Brown points out that a lot of times IT may be misaligned and not be able to see the business processes that are spanning the silos. It may also be the case that the business isn’t either. So, you’re really setting up challenges, when you’re trying to view things from more of an enterprise perspective.
I really see things like this as an organic growth of technology over 20, 30, or 40 years. That’s what happens as you slowly add things. You’re adopting things as you go along, and you eventually get to a state where you have to step back and view it from a higher level -- to really understand what you've got, and then to start breaking it apart.
Trying to do that at the same time that you’re trying to keep the business growing and keep up with new competitors who are nimble and don’t have all of the boat anchors tying them down, is always going to be a challenge -- whether it’s SOA, business process management (BPM), OOD, or the Web.
It’s the same pattern repeating itself, but the pressures of globalization make viewing your entire enterprise and your business as a whole even more important. It’s hard to continue to operate in a niche and continue to sustain that organization for many years, whether it’s from the IT aspect or the business aspect.
The second book, which is going to be coming out later this year, is called "Implementing SOA: Total Architecture in Practice." It's more of a hands-on and how-to book, tailored toward developers, designers, and the architects themselves.
Why do you think that SOA needed to be broken up as a subject in this way?
Brown: To build on a point that Todd just made, we have to look at our business intent, which is to deliver the results of the business processes, and tie that back together with IT.
It’s distressing, when I go into a company, how infrequently I find anybody who can articulate what the end-to-end business process is that produces the key results or can tell me what the entire order-to-cash cycle looks like. There are lots of experts for fragments of it. The IT focus has traditionally been on fragments of the processes.
The pressure these days is to improve the overall business process, the response time to the customers and partners, and the overall quality of what’s going through. So we need this focus on end-to-end business process and the focus on the end-to-end system interaction that helps bring that business process to life.
The pair of books focuses on that issue. This first book is essentially a business argument, and this perspective is required. There is a needed role to carry that perspective forward. That role is the role of the architect, whether the business process architect or the systems architect.
This first book tries to make the case that we need to start taking an end-to-end perspective, and it has business implications. It has implications for how we organize and conduct projects and how we manage the business processes themselves, because very often they’re not being managed.
The second book gets down into the technical aspects of what exactly it is that the architect is doing at the business-processes and systems level. What kind of information do they need? How do they get the stakeholders involved? Essentially, it’s oriented to the execution of the architect’s role.
Evolution, Not Revolution
It seems to me that many of the points you make in the book point out the foibles of decentralization. When things are too fragmented, when the systems as well as the processes are not being addressed holistically, and that leads to eventual breakdown.
Are we really talking about a change from decentralization to command-and-control? Do we need to address all of this organizationally pretty much from the outset?
Brown: I think it's an evolution. It’s inevitable that the bulk of the resources are going to remain in the organizational silos that they’re in. They’re organized that way, because they’re grouped around a particular piece of functionality, and that functionality is not going to go away.
What we need to introduce into the picture is more thinking in terms of how these different silos play together to make the business work. That has to happen on both the business side, looking at business processes, and the technology side, looking at systems.
To me, it's a real open question as to where that responsibility actually lies. In a theoretical sense, I would think that it belongs with the chief operating officer (COO) of the company, whose job is essentially to execute the day-to-day business of the enterprise. I don’t see many companies that have evolved that far yet.
More often, you see the responsibility falling under the chief information officer (CIO), which is not necessarily inappropriate. In my book I argue that the CIO has another huge responsibility, which is simply managing the vast array of resources in the IT part of the world -- not only the systems, but the people that are there for their daily care and feeding.
That’s a huge responsibility in itself. I wonder whether a single individual has the bandwidth to manage this overall marriage of business processes and systems at the enterprise architecture (EA) level, and to also manage all those resources.
So I leave it as an open question. There are two roles that need to be played. I think each organization has to find its own solution for where those roles live in the organization.
Biske: One of the things that I liked about the book was that it introduced a traditional organizational hierarchy, but then the bulk of the book talked about this project organization. You’re trying to execute a project ... [but] what’s the structure of the organization that supports getting that project done?
Do you think that we’re going to require more fluidity in our organizations, less formal structure and more focus on this particular effort? What does the organization need to be to ensure the success of that effort?
When we go on to the next effort, you again change the organization for that effort to ensure that if it’s successful. It’s not so much just about putting out the hierarchical organization chart, and then everyone tries to operate according to those rigid silos.
Brown: I think the assembly of an ad-hoc project team, typically referred to as a cross-functional team, is very important. It’s not a new concept, and even today many projects are organized that way.
What I think is necessary to augment that, though, is active participation of the enterprise architecture group with a broader view of the end-to-end business processes and the end-to-end systems that are involved.
The project team is going to be narrowly focused on accomplishing specific tasks and achieving specific goals, but if we want our service portfolio to build out, if we want that kind of flexibility, the style in which that is done has to be integrated smoothly with the bigger picture of both enterprise business process and enterprise systems architecture.
So, we need a proactive enterprise architecture group that’s willing to roll up its sleeves and get its hands dirty, touching the individual projects that are going on. They really need to coordinate their work, so that we don’t end up with the chaos that we’re in today, and so that we have pieces that elegantly fit together and can be rearranged to achieve new business goals.
For example, Paul talks about the need for a process charter, which is sort of prime objective for a project. There needs to be a business-executive sponsor, someone who can take some ownership and have authority. There should be an IT executive sponsor, as well as the business executive.
Let’s look at just this personnel level. How do you think organizations can start to get people in a role of actually owning and having authority over an entire business process? There seems to be a vacuum, that there hasn’t been anyone in this role before.
Brown: Well, in a pragmatic sense, we sort of evolve into the business process from below. By "from below" I mean by attacking it from a system’s perspective. The reality is that anytime we touch systems, intentionally or unintentionally, we tend to alter the business processes they live in.
As a starting point, I would encourage the technical architect simply to be inquisitive about the business process and make sure he or she understands what the process is supposed to be, and acquaint the business with the implications of the changes that are being made.
You can do a lot of it through question and answer, just by asking questions: "Nobody told me what’s supposed to happen if this goes wrong at this point in the process." So by answering questions, you can raise the visibility of the business process.
That’s a far cry from actually taking control and managing that business process. We see some of the control and management arising out of workflow-oriented projects, where usually, at the departmental level, a business person is consciously trying to grab a chunk of a business process, and then structure it so that it can be controlled and managed. We see these different fragments coming into play here.
With really large projects, if you look at an enterprise resource-planning initiative, you tend to build this project team that reports in at a very high level in the organization and gains a lot of authority from where it reports in. The enterprise resource planning (ERP) projects that are most successful are the ones that consciously address both the business-process re-engineering part, as well as the systems engineering.
What we want is a model that’s very similar to that. You need the same kind of authority from that business-executive sponsor and the IT-executive sponsor to make everybody play ball. But now the projects are small. They’re not the mammoth business ERP efforts anymore.
It’s your everyday integration project or services project that spans silos. What we need is a project organization for them to report into.
In my book, I argue that organizations should be part and parcel of the same organization in which the enterprise architects live. That ideally ought to report in on the business side of the house, probably to the COO, because that’s where the authority comes from to really make changes in the business.
Architecture: An Enterprise Issue
It seems to me that you’re also saying that the business side of the house needs to understand the technology issues as well as the process issues. Is that the case?
Brown: Yes. When you design a business process, you’re making assumptions about what systems are going to do and what people are going to do. You can’t validate the assumptions until you actually start doing the technical design.
You need to have a systems architect involved when you are planning business processes. Conversely, you can’t really make modifications in the system without inadvertently changing the business processes.
So it doesn’t make sense to separate these activities. You really have to treat them as if you're architecting the business, which consists of both business processes and systems.
Biske: I think you're starting to see some organizations on the leading edge take this approach. Overall, I think enterprise architecture is still -- surprisingly -- an emerging discipline in many organizations.
I can look at some of the companies I’m familiar with here in the St. Louis area, where I’m located, and not all of them have enterprise architecture organizations. They’ve got to get over that first step of recognizing the need.
Groups that have been practicing enterprise architecture for some time are clearly growing out of the IT roots, and realizing [that the solution] is about the business and it has to incorporate business architecture.
If you go to a Gartner enterprise architecture conference, you’re going to hear the same thing. There's this momentum building around the notion of business architecture. It’s not just about what technologies I have and how do I consolidate the number of vendors that I’m dealing with, and the more IT related issues.
So I definitely agree that it is total architecture. Ultimately it has to involve the business, if the company is going to continue to be successful in the long run.
Biske: It will. It was really interesting reading the book, thinking of some of my own experiences, and realizing that this book is really written from a business perspective. I've worked in organizations where, as described in the book, there is this hand-off that occurs at some point. Whether people realize it or not, all of a sudden you’ve now got business projects and you’ve got IT projects, and you run the risk of their becoming disconnected from each other.
I kept that in mind, as I was reading the book. ... I considered whether it could be a challenge or not. If you really have the business involvement, as it’s described in this book, this makes a lot of sense.
The extent to which a business manager could read this and see it come together is going to depend on how they currently work with their IT staff, and how open are they to changing the way IT behaves.
If they’re set in their ways, are used to the traditional customer-supplier relationship, have these hand-offs, and then things get disconnected -- it may not occur to them that there is another way of doing this. But if they have recognized that they need to get better interaction with their IT departments and start to operate more in terms of a partnership, then this book can help them out a lot.
Those people who are thinking of SOA as a technology discussion will have their eyes opened by this book.
Let’s move on to the notion of "Total Architecture Synthesis." It’s a methodology that you discuss in the book, Paul.
Can you explain what you mean by "Total Architecture Synthesis," now that we’ve set the table in terms of the larger problem. Once we recognize that this is business and technology coming together, how do we then create a form of architecture that is inclusive for IT and business?
Brown: When you start getting into a project, there are a couple of traditional ways of structuring the work in the project. The oldest one is the Waterfall Model. That's where I gather all the requirements and do the business process design. Then, I would do all the systems design. Then, I would do implementation ... and so on.
The problem with that is that in a lot of our more ambitious projects, the feasibility -- and by that I mean our ability to execute the project within the cost and schedule guidelines that we’re given at the outset -- is still to be determined.
You’re well into the project, before you have enough information to recognize that you’re headed toward what Ed Yourdon would call the "Death March Project," one where the cost or schedule is off by more than 50 percent. So, the Waterfall Model is an expensive and risky approach.
Another model that’s being talked about a lot these days is the Agile Development model. Agile Development bites off a small chunk, tries to drive it to implementation, and then iterates rapidly. The issue I have with that approach is that you tend to pick off the low hanging fruit in the early iterations. You pick on the easy parts of the problem.
The situation you end up in is one in which you haven’t really thought about the architecture, because the simple parts of the problem didn’t challenge the architecture. By the time you get to the tough parts of the problem you have sunk cost into what may turn out to be an inappropriate architecture.
So I try to marry the two together. We want to take the agile approach, but just for building the architecture, not necessarily driving it to implementation. We want to take a look at the business processes that are involved in the project and figure out which ones are liable to be most challenging to the architecture. They're not hard to identify. They are the ones that have the high volume of execution, move a lot of data, require a lot of automation, and are business-critical.
If you simply inventory the business processes that you need to achieve your business goal and rank them in these four categories, the ones that are liable to challenge your architecture bubble to the top very quickly.
You then focus on gathering the requirements for just a couple of those business processes, the most challenging ones, and drive them down into a business process design sketch and an architecture sketch.
Immediately you begin evaluating your architecture: Is this something we can actually build within the cost and schedule guidelines? Does it achieve the business goals? If the answer is "no" -- you can stop right there, go back to the business, and have a business discussion: Should we be doing this project at all? Should we grow the scope? Should we cut the scope? Or, whatever needs to be done.
That’s the whole idea of Total Architecture Synthesis. We want to turn this development of the architecture into an iterative process that tackles the tough problems first -- the risk problems, the feasibility problems -- and gets them right up front.
Biske: Absolutely. I read through that section and saw a lot of things that I had seen in the past of how organizations approach things. I think it’s right-on.
The way it was put in the book that I thought was good was, when you’re in the feasibility stage, there are things that you already know how to do, and those are the things that you shouldn’t be spending your time on at that point.
You already have a good degree of confidence as to how much it’s going to cost and how many resources it’s going to take. The problems that you don’t know how to do are the ones that you’re at risk of these cost overruns.
So focusing on those larger problems and getting them the appropriate priority early on handles these things much earlier in the lifecycle, and yet can still be done in an agile manner.
I wonder about the point you make in the book, Paul, around the fact that a service being created and driven into a SOA probably won’t save money upfront. That is to say, there is a need for investment. The return on that investment (ROI) will come later as that service is used across more business processes, and those processes become more agile themselves.
What needs to happen in addition to discussing Total Architecture from the business and the systems perspective? What of incentives and the financial structure? Does that need to be adjusted as well?
Brown: Well, in order to get to a portfolio of services, to get that long-range return on investment, you need to have organizational discipline upfront to structure your IT work as services. To accept the fact that, in the very short-term, you’re probably going to pay a little bit more for services than you would for developing the same functionality in a non-service or non-reusable manner.
The way you cover that in the short-term is you simply accept a lower return on investment -- a positive return on investment -- but a lower one for the early projects. That implies that you’re structuring your projects and choosing your projects because the individual projects have inherent business value. They’re justifiable on their own, and you’re just getting a slightly smaller ROI. With this approach you can sustain the investment in services indefinitely.
We’ve seen organizations that are succeeding. I’ll use two companies as examples, Harrah’s and Con-way. They're in very different markets.
Harrah’s, prior to their SOA investment, acquired five new casinos, and it took them two years to fully integrate. What does that mean in their world? In their world, their competitive advantage is that when you earn points in one casino, you can walk across the street to another Harrah’s property and use them to buy yourself a meal at a restaurant.
That fluidity of the customers throughout all the properties was essential to Harrah’s. It took them two years to get there before their SOA investment. Two years later, they acquired Caesars, which effectively doubled the size of the company. But they had also made this two-year investment in services.
They achieved the systems integration [with the Caesars merger] in six months, rather than two years. So they gained 18 months of competitive advantage, as a result of this investment in SOA.
Concession to Reality
Brown: I mean that your business processes and your systems are intertwined, and the design of one affects the other. That’s reality.
You can choose to ignore your business processes and let them evolve by accident, as Todd was talking about earlier. That’s how we got in the bind we’re in right now. To a large extent, we’ve been focused on individual profit centers and driving down cost in individual functions. We’ve lost sight of the end-to-end business processes.
The resurgence of interest in customer loyalty, customer service, and all of that is really a reflection that, as a business, we need to stand back and look at what we’re doing to our customers and our partners.
The investments we’re making and the individual activities in the business processes are fine, but in order to remain competitive we need to stand back and take the holistic view. The holistic view simply means you need to understand what that business process looks like before you can improve it.
I’ve seen horror stories of people who have tried to improve a business process by examining a small portion and making some changes, without realizing that the changes that they’re making are having an adverse impact on the bigger business process. So while they thought they were improving things -- they actually made them worse. You can’t live with that.
Biske: I’m in complete agreement. One of the things that I wanted to call out was the ROI discussion. It was great that the book pointed out that projects are successful when they achieve the desired business benefit.
I can’t tell you how many IT projects I’ve seen whose mission statement basically said, "We have to get this thing done by this date for this much money." Their success criteria was based upon meeting the date and meeting budget -- and had nothing to do with the perceived business benefit.
Suppose suddenly we change our view on projects to say the project isn’t done until the business benefit has been achieved? Or we say if it's not going to be achieved, we made a mistake somewhere along the way and we need to cancel the effort? It puts a whole different perspective on things in terms of how these IT solutions fit into the business, and it gives that appropriate visibility that we need.
It’s not about the ROI of one service: It’s about the ROI that’s going to be achieved by a whole sequence of services coming together to support a business process, which is going to yield either those cost reductions, the ability to integrate faster, or new market opportunities.
You have to put it all into that appropriate perspective, which brings it right back up to Total Architecture.
You wind up in these debates over whether this one particular service is actually going to achieve significant cost savings or not. You can’t possibly answer that question without viewing things in a broader perspective.
So phrasing our projects in terms of the business benefits and taking that broader view is critical to understanding how these things are going to fit in, and actually achieving the goal of agility and better IT business alignment.
Brown: No -- but that’s the good news. I've gotten very positive feedback from the people that I’ve talked to who have read the book.
If there is anything that I get in the feedback, it’s that the organizational solutions that are required to make this happen are very diverse in nature. They depend as much on the current organizational structure and current culture of the enterprise as they do on any of the principles that I lay out in the book.
So I do see some creative discussion there around how to organize and execute this. But in general I’m getting very positive feedback.
Brown: Yes, although it remains to be seen. I would like to look at [the market] again a year or two from now and see if these ideas are really taking hold and starting to influence the way people think about organizing their business, their IT and their projects.
We’ve been discussing how the concept of Total Architecture can help elevate the issues and spawn some discussion about what will make SOA successful, and ultimately make businesses more productive, more agile, and perhaps even cut some IT costs in the process.
We’ve been discussing this with the author of the book, Dr. Paul Brown, principal software architect at TIBCO Software.
We’ve also been joined by Todd Biske, enterprise architect at MomentumSI.
Any parting thoughts from either of you, please?
Brown: Well, I'd just like to encourage people to take a look at the ideas in this book and keep your mind focused on delivering business value. The rest will take care of itself.
Biske: I would like to add that there is no shortage of books on SOA that deal with the technical details. But if you listen to a lot of the experts out there, all of them say that it’s not the technical details that are going to make SOA so difficult.
It’s all the cultural details that come into play. Dr. Brown’s book really does a good job in trying to discuss some of those non-technical things in the organizational culture, brings up a number of excellent points, and makes a number of excellent suggestions on how to improve that situation, and then really be successful with SOA.
I do hope that people take a look at this book. I think it does have a lot to offer.
Transcript of Dana Gardner’s podcast with Dr. Paul Brown on his book "Succeeding with SOA: Realizing Business Value Through Total Architecture." Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.