Edited transcript of SOA management trends and analysis discussion with Interarbor Solution's Dana Gardner and ZapThink's Jason Bloomberg, recorded before a live audience at the Harvard Club of Boston on Sept.14, 2007.
Listen to the podcast here. Sponsor: Tidal Software.
Our panelists consist of Jason Bloomberg, managing partner and senior analyst at ZapThink, and Dana Gardner, president and principal analyst at Interarbor Solutions. Moderating the discussion is Martin Milani, chief technology officer at Tidal Software. The in-depth presentations from the analysts are followed by questions from the live audience.
Listen as these SOA experts explore how IT management will evolve in the world of service-based applications. They delve into issues of new standards, how SOA demands that performance management and change management augment and elevate the role of systems management, and how the integrity of services delivery requires a deep and wide approach to management in total across a services lifecycle.
Now, let's hear from our moderator, Tidal Software's CTO Martin Milani.
Martin Milani: Thank you. I guess it's no secret to anyone that SOA has finally arrived, and that SOA deployments are increasing rapidly -- and far more mission critically -- in the past couple of years. It's one of the fastest-growing segments of the software industry as a whole.
So, with that I want to see if the analysts could share some of your thoughts on the industry, and then some of the challenges with SOA in general. Jason?
Jason Bloomberg: Well, thanks a lot, Martin. It’s great to be here. I’d like to start with the definition of SOA for a level-set. SOA is essentially an approach to organizing IT resources to better meet the changing needs of the business. Fundamentally, it’s an architectural approach, a set of best practices, for leveraging IT in a flexible way.
The core business motivation for SOA in most organizations is business agility, to be able to respond to changing business requirements and to leverage change for competitive advantage as well. As a result, one of the key challenges, if you are looking to architect your IT organization in terms of flexible services to meet this business agility benefit, is being able to create, manage and evolve these services over time.
One of the core challenges we’ll be talking about today is loose coupling, where you want to build services that you can control and manage independently of the consumers of those services. That’s a key part of the business agility benefit of SOA. That’s how you would actually achieve business agility in practice.
In turn, the way you achieve loose coupling is through management, and that’s something we’ll be talking about as well. Management then becomes the critical enabler for loose coupling, which is the critical enabler for business agility. That’s how it all fits together.
Milani: Dana?
Dana Gardner: Thanks Martin, and thanks to Tidal Software. I agree with Jason. I think also that SOA has some catalytic implications for companies that begin a journey toward this architecture. It can foster change, and perhaps also benefit agility, but if they have not done their homework, if they are an organization that’s very much in silos, both in technology as well as practices, there could be a lot of risk involved.
There's going to be a period of time, where people look at SOA and find that the opportunities are going to depend on how they’ve done their preparation. We've had a lot of work toward service enablement of data, cleansing data, and putting it into a form in which it can be delivered across multiple applications and processes.
We’ve seen companies begin their journey toward better integration that’s open, that fosters interoperability, and we’ve seen management, but mostly on the level of "speeds and feeds," of how to make sure that the trains run on time, when it comes to delivery of data, whether transactions are going to be fulfilled in a timely manner and if application performance is maintained.
But management for SOA needs to go a step further and take into consideration many systems and interdependencies -- perhaps involving services coming from outside the organizational boundaries, be it partners across the supply chain, even hosting organizations and commercial-services providers.
So management is going to be, as a topic, very important to SOA in a traditional sense, but also management in a new sense. If SOA benefits technology, as well as the business goals and objectives and agility, then business management, IT management, and SOA management need to have some commonality.
Relying on individual people -- "wetware," if you will -- to bind these together and to hand off one to another isn’t going to scale. It’s going to be expensive. So I'm hoping that management is propelled through SOA into a more advanced concept that bridges a gap between organizational management, IT systems management, and the management of services themselves.
Milani: It sounds like SOA is affecting the application assistance management landscape as we know it. Would you agree with that?
Bloomberg: It definitely is. The key thing to keep in mind about SOA in this context is that services are an abstraction. That is, they help to provide flexibility to the business, but they don’t actually simplify the underlying technology. Many architects are a bit surprised by this, that SOA doesn't make their jobs easier or make the job of IT any easier. If anything, it’s more complex. There's more of a challenge for IT to meet the business requirements for flexible, agile, composable, and loosely coupled services. As a result, you have this need for the IT organization to rise to the challenge of services.
This is especially true in the management area because the services essentially have to behave as advertised. These are contracted interfaces to various application functionality of data across the organization. They have to do what they're supposed to do.
That's the core of the loose coupling. So, any consumer can come along and leverage a service, and it does what it is supposed to do. If there is a problem with the service, then it’s not loosely coupled. The core challenge is essentially saying, "Well, we want to make sure these services do what they're supposed to do." That’s fundamentally a management challenge.
It’s never been the preferred way -- firefighting as the means to application performance. When you take this to the step of decoupled applications or services, you recognize that so many different systems are now supporting processes. It's not just a single model of a stack of support and platform beneath a specific application. You need to start working toward a predictive, analytical approach to the management of performance.
The implications for those dealing with applications is that you are going to service-enable those applications, decouple, and decompose them into essential core services, and then repurpose them by cross-compositing processes. What is that going to do to you, if you think you are going to go to firefighting mode when you have performance issues? It’s simply not going to work.
You need to rethink management and support, and you need to try to get proactive in how systems will be supported to head off performance issues and create insurance policies against blackouts, brownouts or other snafus. SOA is really a catalyst toward a different approach to the management and support of the services.
Bloomberg: That’s an important point. In the context of SOA, we're thinking in terms of business processes that are implemented by composing services. What’s happening here is that the definition of an application is beginning to shift, especially from the perspective of the business.
The business doesn’t want to think of an application as some big huge piece of software that costs them a lot of money and introduces a lot of risk, that doesn’t directly meet business needs, that’s not what they are asking for, but is what they’ve had to buy because it's what was available.
In the context of SOA, though, an application is what you do with the services. You compose them into composite applications or service-oriented business applications that implement processes in a flexible way to meet the needs of the business as they continue to evolve.
From the business perspective, this notion of application is whatever you are applying the services to. The root word of "application" is to apply something, and, as a result, the management challenge, especially the application management challenge, in particular, is an entirely different set of challenges.
It’s not saying, "Well I have this big enterprise application, which is some monolithic suite. I need to manage that." If you have those, you still need to manage those. So, that’s not going away, but SOA introduces this whole new concept of flexible, composable business process-based applications, which now have to be managed as compositions of services.
Services are obstructing functionality across various systems and various applications, and the management challenge becomes immensely more complex. If anything stops working, then the whole thing falls apart. Management now becomes even more important. You can’t just have a firefighting approach, and when there's a problem, you send an alert to the poor systems administrator, who has to wake up in the middle of the night to fix it. That just not going to cut it anymore.
We're talking also about business continuity issues where we are going to try to have replication of services in the event of disruptions, be it natural disasters or otherwise.
So when you think about SOA, it’s happening at a time when there are many other IT mega trends under way, and the management needs to be considered within that context as well.
Milani: Obviously, SOA is a disruptive event. There is a radical change in the architecture of the applications, as we know them. There are far more points of failure, far more pieces of the application that could reside across separate systems, separate technologies and perhaps across enterprise silos.
So the traditional management approach has been used for the past 15-20 years, and is very client-server centric. Can these traditional monitoring and management approaches handle the SOA deployments of today and tomorrow? Dana?
Therefore, you can offer a management overview, whether it’s a dashboard, or even the equivalent of screen scraping one management view with another. At least, you can assimilate them in a such way that you can start to get a total view, a comprehensive view of systems.
I do think that you are going to continue to leverage existing management approaches. I'm sure that the older-line management vendors are going to continue to augment and support more advanced processes or more complex interactions between elements of infrastructure and applications.
It needs to be looked at not just as management of discrete parts, not just trees within the forest that each stand on their own -- but the forest itself. I'd like to see that get to the point where it becomes something that can be assimilated further than just the systems -- with the business objectives as well.
Furthermore, one of the things that’s been unfortunate about systems management up until now is that it tends to be binary -- on or off, green- or red-light, it’s working or not working. SOA is going to require more of a blended approach to management.
That is to say, you are going to want to tune how your applications and services are delivered, perhaps to live up to service level agreements, or perhaps so that you can give priority to certain data, application, or services streams over others. You're going want to be able to fine tune, rather than just throw a switch on or off. That’s going to require a different level of management. It’s really about leveraging the old, finding ways to assimilate and then put a more operator- or policy-driven -- perhaps even automated -- approach on top of it.
Bloomberg: It's interesting that you say SOA is disruptive, because SOA is disruptive -- but not in the way you might think. It’s not really that disruptive on the technology side. In fact, from the technology perspective, SOA helps leverage existing technologies. It has a heterogeneity story that says, "Well, you don’t have to replace. You can leave and abstract as needed."
So, if SOA is architected properly, it doesn’t have to be disruptive on the technology side, but it is most definitely disruptive on the organizational and political-cultural side, on the human side, because what SOA tells the IT shop in particular, is that we can’t do business as usual.
We can’t simply have a siloed approach -- here are the operations people that run this, here are the Windows people that do this, here are the application people that do this. That’s not going to work anymore. In order to think about services properly and to get the benefit of SOA at all, you have to think in a more cross-functional, comprehensive, architected, best-practices way about how IT does what it's supposed to do. How IT is going to meet with the needs of the business is shifting in a broad way, and that’s how SOA is disruptive.
When it comes time to think about management at all levels -- whether it’s networks, systems, application management or the business service management -- these are being reworked, because you can’t think of any of them in the narrow silo.
It’s like saying, "Well, I have this application; I have to manage this application. I have my TCP/IP network; I have to manage the network." You still have those technologies and you still have to manage them, but now we also have this whole notion of services cutting across different parts of the IT shop, meeting different needs of different parts of the business.
A single service might abstract multiple databases, multiple underlying applications, and multiple platforms. That same service might serve multiple lines of business and might serve the needs of internal as well as external users. So, in that context, what it means to do management is being entirely disrupted by the service-oriented approach.
Milani: Traditionally, the way people have looked at the Web services in SOA management today, has been to manage or monitor the interfaces, which are actually an interface to a runtime environment that holds some sort of service.
What do you think is required to deeply understand how to monitor these services beyond what the interfaces are doing, and finding out what’s under the tip of the iceberg? Dana?
It’s working with that data in the context of a horizontal business process that’s the hard part. The type of applications that we’re going to be seeing in the future, as Jason mentioned, cutting across different aspects of the organization, are going to require redundancy. If one aspect of a process goes down, that’s the weak link in that chain, the whole chain could be at disadvantage.
In the past, you might have one application down, but people could go off and do another task, because that mainframe would be backup at
Bloomberg: It’s important to note that in the context of SOA, monitoring is just not good enough anymore. It’s not good enough to simply say, "We need to find out when we have a problem." It’s like, "Great, okay, now we know that Titanic has sunk." That’s not going to do it for you. You need to have preventative and corrective management. For a service to be loosely coupled, it has to behave the way it’s supposed to behave, regardless of what’s going on beneath the cover.
So it’s not good enough simply to say, "My management tool is telling me my service is down." You need to have a way of understanding that problems are brewing, that they are in the works. You have to be able to identify a potential problem before it actually impacts the behavior of the service and you need to take corrective action, ideally before the consumers of the service are impacted.
This is a high bar to set, an enormous challenge, and it’s not likely anybody is going to be able to do this perfectly. There is nothing perfect in this world and there will always be instances where there are some problems that a preventative, active management tool won’t be able to catch, but it’s critical for the architect to plan for this level of management, preventative corrective management.
That’s something the architect has to think of ahead of time, when they plan how they are going to build and implement those services. You can’t let it go for later. It’s not like you say, "We're going to launch our software and then let operations deal with management." There has to be something that’s part of your initial thinking when you are putting together the plan for your SOA.
There is an opportunity to learn as you go to encounter problems and then to put into place the management feedback and create the feedback loop into the remediation. Another important aspect of this is to start finding commonality between pre-production and post-production when it comes to applications and services development.
We're beginning to see some products in the market that try to take data that’s collected in the development phase and make it available to post-production to start feeding a loop of communication. So, when something is amiss in production, you can not only look to what is at issue in the systems and operation, but also look to how could we make this service or application better. So, the requirements for service and application are being impacted by operations.
For a long time, there was a significant wall between these facets of IT. It’s still very problematic, but if we can create a management feedback loop, so that -- as we get into faster iterations of development, when the test-debug cycle and build cycle is faster due to agile and lean practices -- we can start saying, "Let’s find out what’s going on in the field when something is wrong."
Do we just throw more hardware at it? Do we just add more servers? Or, do we say, "Can we actually design and engineer the service to be better," and then make that happen with a matter of weeks or months?
Milani: It sounds like the playing field again is far more complicated and complex with SOA, and is going to get more complicated from here on. You have business processes that go across multiple technology silos, multiple enterprise silos, and probably to other enterprises. So, you have different constituents, different organizations, different groups, and many different types of technologies.
This area was pretty hard to monitor and manage in even a simple three-tier architecture. Now, you have an N-tier architecture which you could almost call an "N-to-the-N-tier" architecture. Now, you have different types of services that you could be calling on, depending on the quality of service and security, trust and policy, and so on. So, it’s a different game. If you were to fast-forward five years or seven years, what do you think monitoring a management system, which could address all the issues we just talked about, would look like?
Bloomberg: It’s important to understand that management is not a practice in isolation. It’s really part of a family of capabilities that includes governance and quality, as well. You mentioned policy, security, and a variety of issues. All these are part of the SOA infrastructure story. So, to answer the question where things are going over the next several years, it’s really maturing the family of SOA infrastructure capabilities, broadly speaking.
We like to call that the governance quality management, or GQM, suite, which handles design time as well as runtime issues, creation and communication of policies in the context of governance, management in the context of runtime. It's not just thinking of runtime, in and of itself, because runtime is only part of the story. We have the full lifecycle now with design time creation of services, as well as publication and discovery of services.
The runtime part, which is the current focus of management, as well as the change time part, where you reconfigure and recompose services essentially in runtime context without any underlying co-changes, that’s where we see a lot of the activity going on in a mature SOA environment, is at that change-time configuration, composition level. So more and more of what has to happen there, from the infrastructure perspective, is this pulling together of the quality management in governance roles into full lifecycle quality management governance suite and that’s what we see happening already today, with a lot of the products in the space.
Let’s take an example of something from recent business history. Let’s say you’re a manufacturer of toys and you're told that a portion of these toys are now in the field with toxic paint that is an violation of the regulations in your country. You need to take quick action. You need to find out through your supply chain where the problem is. You need to find new suppliers. You need to go back out to the field and conduct a recall, and coordinate this with your marketing and your PR department and your investor relations department, so that you don’t lose the entire value of your company overnight.
How are you going to do this? You might do this through your IT systems. So you’re going to need to examine what’s going on in your supply chain and find out where the problem is and stop it. You’re going to say, “We don’t want to just keep the trains running on time. We want to pick up the tracks and move them somewhere else.”
IT is going to have a great opportunity to become far more valuable to their parent organizations, to be the real partnership that they should be, through the exploitation of SOA and through the proper management of IT and processes.
To answer your question: The value of IT here can be much greater. It can be an enabler, not a cost center. It can be the way in which not only is information relayed about what’s going on, but can determine what we want to happen. We want to change that supply chain. We want to change that distribution, recall these products, get a list of every single product and every serial number, and we want to relay that to our sales force.
That sounds straightforward, but if you try to do that with a lot of IT systems today, you’re going to find yourself up there with the equivalent of mimeograph and crayons, doing it by hand -- and that’s just not acceptable. So in the future, a company’s very existence could be at stake if they don’t have agility in these processes.
Bloomberg: This is a really important point. SOA is really not optional. Companies that don’t get this right will suffer the consequences. They will suffer lawsuits and suffer a competitive disadvantage. They are going to go out of business. This is an important thing to keep in mind. IT is not playing around here. You can't say, "Maybe we will do SOA, if we can figure it out, or maybe we won’t. We’ll just do things the old way, where we are siloed and we keep on going."
If you keep doing things the old way, you are going to lose out one way or the other, whether it’s some sort of regulatory or competitive pressures, some disaster like lead paint, or credit card numbers getting out on the Internet. Whatever the problem is, it’s going to happen, and you have to be prepared. Organizations that don’t get this right are not going to survive.
Milani: SOA, again, is very business-driven. One could say it’s totally business-driven. It gives business agility and adaptability. Often people call it agile enterprise, real time enterprise. Inherently, SOA means real time integration across separate applications and separate technologies.
Something we touched on a little earlier was preventative automation and correction. Can you guys elaborate on that a little bit, why it’s very important. If you’re talking about an agile, active enterprise or real-time enterprise, then that automatically means that the management cannot be reactive and the management cannot be an afterthought. Dana?
It’s the architecture of the business and the architecture of IT having some relationship to one another. We can borrow from this concept and say that this should be a "total management" approach as well, so that we have this ability to manage processes and infiltrate the systems in such a way that it becomes a two-way street.
Preemption is really about latency. You can be reactive, if you can do it in a nanosecond. But, preemptive means that you’re going to come to a time where the lights are blinking and there is something wrong, and, before you’re totally out of business or some process is brought to its knees, you want to take remediation. It’s really about balancing the latency of reaction toward a point of preemption. That's very hard to do, however.
When we have system redundancy, we have all this log data. There are probabilistic approaches. There is looking at performance states that are normal, when you compare that over time, but when we get into this area, there are many unknowns, many more variables across the supply chain. When we are dealing with services that are coming from organizations that you don’t have authority or control over, it’s going to be a difficult approach.
I don’t have a quick answer to it. I do think that the latency issue needs to be addressed -- the amount of information that's shared. As we get toward this concept of total management, we need to bring in what incentives are being applied to people and systems.
Are we going to pay for systems based on a steady stream, are we going to start putting in incentives that penalize people for down performance, while paying them higher prices for greater performance? As we see services that are acquired on a subscription basis or a pay-as-you-drink basis, we are also going to start seeing a lot more monetized incentives around performance.
So whether the systems react or not, the price could be so high that you’re going to have to make the investments. I think economics and the concept of total management need to be brought to this.
Bloomberg: There is no magic wand here. Systems aren’t perfect. Software is never a 100 percent bug free, computers are never 100 percent operational, and networks are never up 100 percent of the time. There are always problems. There are problems on the business side as well. Secrets sometimes do get out, and products sometimes are defective and whatever it is, there are always problems. This is just the way of the world, the reality of life on earth.
How do businesses deals with this traditionally? Well, you hope there aren’t any problems, and you have certain plans in place, but fundamentally, when there are problems, you step away from your technology and you deal with it however the people can deal with it. So, you run around and try to fix the problem.
We are looking at SOA, saying, "We can do a bit better with our technology in terms of dealing with the reality that there will be problems." As far as agility, we want to be able to respond to change, including changes we don’t want. That include problems with systems, with the business, with regulation, competition, or global disasters -- whatever it is.
Instead of just saying, "We can’t rely upon our technology, when problems come along," we want to say, "Yes, we have a more flexible way of dealing with problems, even though we can’t predict what they are.” That’s part of the benefit of SOA, part of the agility benefit. We are dealing with unexpected change, including various problems.
Instead of just running around and not being able to use technology, we want to have a governance plan in place, saying “This is how we’re dealing with problems. Here is how the technology will rise to these challenges." And, we make it a matter of policy. So now, instead of just having to wing it when you have some sort of issue, there is an infrastructure in place for helping you deal with issues as they come about.
A key to this is the management challenge. As management technology improves, it is less and less about just monitoring stuff, and more and more about being able to deal with issues as a matter of policy, where your policy is in place for dealing with problems that you can’t predict -- and those are the most challenging ones. That’s what we see happening over time.
Technology is getting better at dealing with these unpredictable situations. A core part of what we need from agile IT is being able to deal with the unpredictable.
Milani: Great point. It dovetails to my next point, which is businesses are moving to standard policies and standard practices in the form of a business process. More and more, they are moving the way they do business into a well-defined practice of business processes. So, using technology such as BPEL and business process management systems, why wouldn’t the management of some of these technology and software resources look like how processes are handled on the business side of the equation?
Bloomberg: Actually, they might be. You want to think about business process in the context of whatever the business does, and that applies to IT as well as it does to the business. IT should be run as a business, as part of the business. So, it’s not like IT is some different world. It’s just part of the business, one of many divisions, and it has a certain role within the context of the business.
Management, to the business side, means a whole range of things, of which technology is only a portion. It means making sure that you have a management hierarchy within the organization. It means that your business is running properly and that the business process is running properly. All of this is part of what the business means by management.
Part of that is IT enabled and part of that is specifically IT centric, but from the business perspective, it’s not like they are drawing a line and saying, "This is the stuff we mean by management in the business context, and this is the stuff we mean by management in IT context," as though they were different things. There is a spectrum, and as we move to SOA, we’re seeing that IT is becoming more of an enabler of the business in many more flexible ways. So, the term "management" now becomes much more of a business-centric context, of which IT is now an enabler, as opposed to two different kinds of management, one for the business and one for IT.
IT and business are intrinsically bound. It’s a competitive differentiator: The way in which you do IT better than your competitors. I don’t think we’re going to see a divorce here. We’re going to see more assimilation. But the management of IT is relatively new, compared to the practice of business management.
The modern corporation is over 100 years old. Mercantile economic activities are 600 years old. You can find other aspects even older than that as to how people are governed and operate as teams or individuals.
If we have things like Balance Scorecard, Six Sigma, Design for Manufacturability, Simultaneous Engineering -- these concept have evolved on the business side, and on the manufacturing side. We should expect to see the same aspects for IT, and perhaps have the same overview of a Balanced Scorecard on how the business is operating, as well as how the IT is operating -- someday, maybe even together.
What we're really talking about here is the maturation and the natural process for IT to become more like business, and not be off in its own corner. It takes time. These were complex things that happened very, very quickly. We used to call it Internet Time. It’s probably, in some respects, even faster now, when you think about Enterprise 2.0 Time.
These IT systems have evolved so quickly, and companies have implemented them in such a haphazard way with lots of different heterogeneity involved, that it’s natural that it’s going to take time. Ultimately, we're going to get to a maturation where IT catches up to the business. Then they can be governed very similarly.
Milani: Are there any standards required sooner than later? Is there any specific standard that you think can enable the deployment of these SOA environments?
Bloomberg: There are standards in the works; there is Web Services Distributed Management (WSDM) and WS-Management and a variety of others. There are some that are datacenter centric, others that are more network centric. But, by and large, I agree with Dana that this is still the early days in terms of management standards.
What’s happening in the management standards world is a pissing match between the big vendors. You have the Java guys wanting to this and then Microsoft guys wanting to do that, and nobody listens to them, because they can't agree with each other. So, they'll realize, "Hey, all of our customers are ignoring us. We'd better get our act together." It’s become this big political thing that’s just slowing whole thing down.
From the enterprise perspective, you don’t have to wait around for the vendors to grow up. You can get stuff done today. This isn’t going to stop you from being successful with SOA initiatives today. It might mean that two products you buy off the shelf might not interoperate as well you like. That has to be part of your plan. It might mean you have to come up with your own internal standards for the time being.
Ultimately in these organizations, we're not going to have people responsible for a server farm, a database, or an SAP implementation. They're going to be responsible for a order-to-bill capability. For those people to get a view into all of the aspects for which they are responsible, they're going to need a management view at a higher elevation. That’s where standards need to be applied, so that they can get that view, so that there is interoperability, sharing of data, and a canonical view of management.
Milani: We're almost out of time. I want to thank you for your time and quickly ask you if you want to close and sum up what you think people should consider, as they deploy more and more SOA implementations, and what they should consider with respect to management.
Bloomberg: SOA is not just about the technology that you can't buy from a vendor. It’s not something you buy. It’s something you do. It’s a set of best practices that are still relatively loosely defined.
You can be successful with SOA by taking it a step at a time, achieving real business value. You don’t have to set the bar so high that it becomes impossible. But, that being said, even if your SOA initiative is relatively modest and you have a few services in the context of that architecture, management is something that you have to think about very early on. In fact, you need to think about management with services even before you get to SOA.
If you just have a few services and they are not managed, then they could be redundant or they could be incompatible. Worst of all, they could be rogue services, services that are not on the radar of anybody who is running these services. They could expose confidential information or bring systems down. Who knows what the problem is?
A rogue service is one that you just have no idea what sort of damage it can cause. This is a real risk, because, if you’re building a Web service, in particular, it might just run over Port 80. It might just expose information over the firewall, and you have to think about this very early on in your services initiative, even before you get SOA.
As you get SOA, you have to think about all the services you already have and plan for loosely coupled services in the context of your architecture. All of that has to be done in the context of management, as well as governance and quality as well.
But that has stifled the ability of interoperability and of addressing things holistically, of being fleet and agile. It’s made them brittle, has made them slower, and has made things expensive. SOA forces companies to start binding what happens in pre-production to post-production, what happens in an application with what happens in an infrastructure, and what happens on a service level from an outside provider to what happens as a shared service internally.
There are great risks if you try to do SOA without growing up, but there are super opportunities if it’s done properly. It can elevate IT as a function within the organization from being an inhibitor to the absolute enablement for new business and growth and opportunity.
If you’re inside an IT department, if you are a vendor, if you’re a consultant -- consider that if you do the diligence, if you come up some standardization, it’s the methodological approaches that work. SOA will make the company better and will make IT indispensable in the best possible way.
Rod Butters: I want to thank all of you for the discussion. It’s actually ranged all the way from the value of SOA to the business and the kinds of business opportunities that enables how management keys into that and feeds into that from a technology standpoint, and the interesting ideas you shared today about how management ties with the actual SOA architecture to provide agility to the business.
So, with that, thanks very much for this great discussion today and thanks to everyone here who has attended our live debate and discussion on SOA management.
Question: I'm from Bank of America, and I was really glad the way you guys started to talk about the management top-down approach to addressing the changes that the organization will face. You also talked about agility and business demands and bringing IT management bit closer to the business management side. Business, for all intents and purposes, doesn’t really care what’s underneath the covers, right?
They have a business value proposition, they have business ideas and perhaps a very cool looking interface, if one may just look at the Internet only, a Web 2.0 experience. For them, agility and latency has a lot more meaning than what we are talking about with respect to SOA. With so much organizational change, there’s going to be some time before these things are going to get settled. How do we really achieve agility in terms of business demands and SOA, which is lot more organizationally heavy it seems? That’s one question.
Second, there is one notion of introducing and implementing SOA, which is going to take some time, and we are talking about management needs to get together into a common consensus of implementing SOA in a most efficient fashion. Now again, talking about agility in terms of maintenance and regular changes to the environment, from the business perspective, they want change to be implemented faster.
The owners define pretty much for the entire organization enterprise wide. How does business still achieve agility into their maintenance mode, when they want to make small change to an application which is now going to require a lot more changes on the back-end side?
Bloomberg: You’re quite right; the business doesn’t have visibility into the inner workings of things, but they also don’t want to know. They want SOA stuff to work the way it’s supposed to work. They will have a change. They want it to happen. They don’t want to hear about any problems. They don't want to hear a bunch of tech-speak in response to some request that they have. That’s one of the core challenges of SOA, thinking about services in the context of the business. It’s a role of IT to build and support business services that abstract the underlying technical complexity to provide the business with that flexibility that they need.
If you can achieve this with individual services, then you can achieve this with the compositions of services. You have to build services to be composable. You’re trying to enable the business to build and evolve business processes by composing, recomposing, and reconfiguring services. If you do this right, if you get the architecture and the implementation correct -- it's a big “if” because it’s a challenge -- then you can build big services out of little ones, because you can compose services into composite applications and expose that as a service as well. So, this big exposed composition of services as a service gives you two core things.
One, it’s recursive, if I can build big ones out of little ones. Second, if you were to see some sort of business service, you don’t know if it’s actually abstracting a process or not. So, in telco, you could have a user-provisioning service, where it’s actually in multiple steps. Or, in banking, you could have an open-account service. That might involve six or eight steps, whatever it is. From the business perspective, from the customer prospective, they don’t want to know.
They want to open an account. If they can push a button and the account is open, that’s great. If you can build loose coupling, not only into individual services, but into how you compose services as well, then you’re able to build flexibility into the processes themselves. So, it’s a challenge, but that's the goal -- to build this level of flexibility into the processes, because that gives the business now the ability to have the agility at that business level.
If you have a service that needs to be changed, you don’t necessarily have to rip and replace. You don’t have to shut things down. It’s not a three-year replacement cycle. You can actually recreate that service, make the changes you wanted and then slowly bring that service into production across more processes.
While one of the benefits of SOA/services enablement is reusability, you can also get redundancy. You can create services that are rather similar. You might want to not let anyone be that prolific and write too much, because then you’ve got multiple slices and dices of services. But, it allows for the ability for moving functional sets without having to recreate the entire application.
You’re going to have more iterations, smaller changes within services. You can have services that are similar, and you might well combine them to be a fuller set of requirements within a single service. Then, you can bring that into production across more and more of the organization. It’s just a more flexible approach. You cannot do this when you have brittle applications on a single silo.
Bloomberg: From the business perspective, what you’re saying is exactly right, but, from the business perspective, you can abstract a set of redundant services or revolving services. Think of those that are at a lower level. From the business perspective, they’re all just one service. You have your account-opening service, it’s just the one service, and it works the way it supposed to work.
Behind the scenes, you have redundancy. You have versioning policies. You have management infrastructure. You have all the stuff going on behind the scenes, from the perspective of the business, to get that service to be flexible, so as business requirements change, it does what it’s supposed to do.
From the business perspective, it’s an account-opening service. It opens accounts, and it works for all my lines of business, for all my different kinds of accounts, and it continues to work even if the requirements for that service change. It’s not easy, right? But, you can make it simple for the business by taking these infrastructure and architecture steps within the context of IT.
Question: You talked a lot about the active aspects of SOA management. How is that different from an enterprise service bus (ESB)?
Bloomberg: If you noticed, we didn’t talk about ESBs. Normally, we don't talk about ESBs. We didn’t talk about integration infrastructure at all. We didn’t talk about middleware at all, and that was intentional. When I talk about SOA infrastructure, ESBs are not high in the list. Middleware, in general, is not high in the list. I talk about governance quality and management as the core infrastructure capabilities that SOA requires. From the context of middleware, most organizations already have a lot of middleware. They don’t necessarily need a lot more.
Now, if you’re getting into SOA implementation and you have your architecture, you have your service design, you have your governance, and you’ve thought about governance quality and management, you may find that you need middleware at some point, because whatever middleware you have is not to the task. Then, there may be a requirement for an ESB, or other middleware solution.
But, you don’t want to start there. If you start with the ESB, because some vendor came and said, "Well, to do so you need to buy an ESB," you’re not going to end up with SOA. You’re going to end up with an ESB, actually what the blogosphere is trying to call an ESB-oriented architecture. The point is that if you start with ESB, you end up with something other than SOA. You end up with a traditional middleware-driven architecture with service interfaces.
Now, that being said, there are some perfectly good ESB products on the market, and lot of them offer management capabilities as well. So, when you get to the point of considering what management infrastructure you need, you may turn to an ESB for that capability, but you don’t want to start there. You want to start with the architecture, the business problem, the business processes. Use those two to derive your services. Look at your infrastructure, solve the governance quality management problems, and at that point an ESB might come into the story.
So ESBs do play an important role in management, and we’ll need to see more flexibility and ease of managing ESBs and what they do in applying rules and policies to ESBs. It’s just another important part of the infrastructure.
We’ve had busing and messaging for some time. The idea of trying to make that inclusive of more transport protocols and technologies makes a lot of sense, but how do you manage them? So, again it’s about bringing intelligence from a higher abstraction across more systems. So, I think ESBs will be important, and I think managing ESBs will be important.
Milani: I would add that ESBs are a very powerful piece of this puzzle, and I believe that what they really do today is interconnect multiple types of different systems, and they facilitate and orchestrate the interaction and exchange of data. So, in many ways all they’re doing is exposing existing interfaces, and they facilitate interaction within those interfaces. But, I think what's missing from the ESB picture, from the monitoring and management perspective, is they don’t have deep visibility into the runtime of the interfaces that in fact they are exposing.
So, one piece that you don’t have visibility into is the runtime of those services within ESB, but I do think ESBs will play a major role going forward to marry passive management, which is to manage and monitor what you have, with active management which is constantly "correct and adapt." And I think that’s an area where an ESB could be extremely powerful in the next few years.
Butters: I thank everyone for joining us today. This has been a very enlightening discussion. Again, thank you to our panelists and thank you all for joining us here at the Harvard Club. Have a great day.
Bloomberg: Thank you.
Edited transcript of SOA management trends and analysis discussion. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
