ITIL's Influence Extends Beyond IT Operations to Enhance SOA, Portfolio Management and Change Management

Transcript of BriefingsDirect podcast recorded at the Hewlett-Packard Software Universe Conference in Las Vegas, Nevada the week of June 16, 2008.

Listen to the podcast here. Sponsor: Hewlett-Packard.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to a special BriefingsDirect podcast recorded live at the Hewlett-Packard Software Universe Conference in Las Vegas. We are here in the week of June 16, 2008. This sponsored HP Software Universe live podcast is distributed by BriefingsDirect Network.

We now welcome to the show two folks who are dealing with the implementation of the Information Technology Infrastructure Library (ITIL) in enterprises. We are joined by Sean McClean. He is a principal at KatalystNow in Orlando, Florida. Welcome to the show, Sean.

Sean McClean: Hi. Thanks. Pleasure to be here.

Gardner: We are also joined by Hwee Ming Ng, she is a solution architect in the Consulting and Integration (C&I) unit in HP. Welcome to the show.

Hwee Ming Ng: Hi, glad to be here.

Gardner: We've been talking about ITIL quite a bit this week at the conference. For those who might not be familiar with it, why don't you give a quick overview one of what KatalystNow does and also perhaps a very brief primer on ITIL?

McClean: Okay, thank you. KatalystNow handles mentoring and learning and training solutions for ITIL and tools, principally in the HP Service Manager service support space. ITIL itself is a broad set of process concepts, much like many types of businesses. It's about the business of doing IT. It's a set of concepts focused on how do we frame and format the business of doing IT.

Gardner: And Hwee Ming, why don't you tell us a little bit about your role at HP?

Ng: I am a consultant with C&I and I focus primarily on delivering solutions at customer sites, and recently I've been working on updating the HP Reference Model to make sure it's in line with the recommendation in ITIL, version 3.

Gardner: First, let's give our listeners a little bit of history of ITIL. We've basically taken this from a technology-centric to a business-centric capability and focus, from version 2 to version 3. Can you tell us a little bit more about the progression and history of ITIL?

McClean: Sure, ITIL has kind of have an interesting level of development and actually it started out in England with the original organization, the Office of Government Commerce (OGC). In version one, what people started realizing was, whether you're talking about your faxes, your daisy-wheel printers, or your laptops of today, we still have infrastructure elements to support the business. At that point, they were simply focused -- let's make sure that we keep those fax machines or the copiers and the big computers running.

They started saying, "Well, maybe we need to get a process around how to keep them running in a way that's consistent with all of the other businesses." OGC did a great job of looking at the ongoing tasks. No matter what the organization, you do it the same way.

Gardner: The group that handles a lot of the government functions?

McClean: Yes, and essentially they were working with lots of organizations through contract who have IT, and in those engagements they started realizing more-and-more often, we're all doing the same stuff with the same equipment. So, why can't we standardize the processes?

Every other element of business has a set of rules and standards. Financial is the one we all go back to. Mostly, financial has been the same way since the abacus. We follow that set of processes, and we all agree on them.

Now, obviously, there have been some changes in that recently, but the ITIL process is the same kind of growth curve. Initially they were just going to handle the technology. Then, they started saying, "Well, the technology in an organization really supports the business." And then they started saying, "Let's take that a step a further and let's start strategically meeting the IT piece with the business. Instead of how to do the service, how do we do the service to support the business to get the job done?"

Gardner: I suppose in its history, IT sort of created an ongoing dissonance with business, because a lot of times products come out, technologies evolve, and they get adopted. Then you say, "How can we make the best use of this in the business?" It isn't always from, "Here are our business goals. Now let's go find the right processes and the right systems to support that." So, we've had it kind of backwards.

McClean: I think it's interesting how that's evolved, because I think there has been some of that. And some of it has been the case of technology is obviously complex, and so a lot of times the translation from "here is the business" to "here is the piece in the business" gets disconnected, because it becomes so involved.

If I could draw it on a map, we tend to look at knowledge evolving in pyramids. It becomes higher and higher, as you reach the apex of that area of knowledge. Maybe we're talking about biology or we're talking about IT. Even within IT, we move to the apex of networking, say, TCP/IP. You have to know all of those pieces.

By the time you get to the top of that, it's a little hard to keep track of what's going on on the other side of the world, which is the business of whatever we're doing. In the presentation we talked about today, I was saying we're in the business of selling.

A company I worked with was in the business of selling shoes. A guy used to walk around the organization, in the IT department specifically, and say, "What's your job?" If the person's answer was, "I do e-mail," they were out of the job, because the real answer is, "I do e-mail, that's supports communication between the organizations, so we can sell shoes." That piece gets disconnected over time, because it gets to higher and higher technology levels.

Gardner: I listened to your presentation earlier today and enjoyed it. One of the things that seems to be recurring theme was this notion of cultural change, and how there is a business culture and there is an IT culture, and ITIL is starting to move towards a sharing of culture or a common ground between these cultures. Why don't we get into a little bit about why and how cultures can shift, even as technologies perhaps stay the same?

Ng: With ITIL v3, there is a lot of emphasis in looking at IT as a strategic partner with business. Instead of having technology drive IT, we are really looking at what value IT delivers to the business. One of the main drivers that we see is the IT organization having to demonstrate the value that they are providing and justifying the investment in IT.

A good way to do that is to ensure that whatever the IT organization is doing aligns very closely with the overall business strategy. When you really look at ITIL v3, they start by managing the full lifecycle of service, managing it through the strategy piece of it, taking it through design, looking at the transitioning, the operation piece of it, and then having a process assuring continuous improvement to the service. So, we are looking at much broader view of the end-to-end lifecycle management of our service.

Gardner: And, when we talk about cultural change, are we talking about having the culture within the IT organization change? Are we talking about having the culture in business change towards IT or both?

Ng: I think it's a combination of both. The business would have to see that value add that is being provided by IT and that the technology that's being provided does drive a lot of the business outcomes. For example, in a bank, the technology or the infrastructure that is needed to support the bank is quite complex, and a new innovation in the IT department could generate a lot of new revenue streams. So, you want to make sure that there is that alignment between business and IT. The whole planning and strategy pieces need to come together. It's really a combination of both sides.

Gardner: Okay, we also heard you talking a little about services-oriented architecture (SOA). Now, that also requires cultural change. I have seen some studies, particularly Summit Strategies did a study, that showed the folks who had embarked upon ITIL methods and appreciate IT as a managed and matured business function were seemingly better off when it came to implementing SOA methodologies and concepts. Why do you think that's the case?

McClean: Hwee Ming and I just talked about this yesterday, and I want to have her opinion first, because I think it was one of those areas where she's got more of an understanding than I do.

Gardner: Okay, Hwee Ming?

Ng: When you talk about SOA or the infrastructure, you really need to view it not as individual IT components that you're working with or individual components that you're trying to piece together, but from the end user point of view. They want to view it as a service. So, that kind of mindset is quite important, if you are moving from an infrastructure-centric or technology-centric to a more open interface, providing value, and aligning that to a service, and that's why there is a lot of alignment in the two.

McClean: So, if you perceive IT as supporting a bunch of products, SOA may become a little bit more difficult, maybe it'll be a little bit counter-intuitive, but if you think of IT as delivering services, then SOA perhaps has a bit more continuity and alignment.

Ng: And, you also come down to the fact that when IT organization or the technical people are talking about interfacing with each other, they are no longer looking at it as point-to-point connection or an interface protocol level. They look at it as wrapping the technology in an open interface as a service, and they provide that to another organization in the same IT organization. So, it really does drive that.

McClean: In my mind, it's kind of interesting as well, because I think at this point, we transition to the idea of looking at IT as providing services. A moment ago, Hwee Ming talked about the idea that we're moving into that concept of IT providing the services. You can almost put it in the front and say that potentially now you create a technology service, which used to be simply to support the business.

Nowadays, if you look strategically, you can create new IT services that help drive the business. We're looking at, as Hwee Ming mentioned, interfaces for things that we are doing on the Web. So, we start architecting new things, which creates new business opportunities in the business of whatever you are in business of. And, you look at all the different interfaces to drive new business that wasn't there before.

Gardner: Okay, we talked about services, but there are still products out there and underneath there that allow a lot of this to take place. You brought HP Service Manager to market. I believe it's been updated this week. Can you tell us a little bit about how HP Service Manager fits into this progression and the shifts going on in IT?

Ng: With Service Manager, one of the big shifts is in closer alignment to the data processes, and we're also looking at some of the SOA piece of it to wrap service interfaces. So, one of the great pushes is to federate the model like a CMS or distributing a process. As IT organizations get more complex, maybe a change management process will not be in one single tool. So, you really look at Service Manager 7 as bringing together configuration management and project and portfolio management (PPM) and things like that.

So you really look at individual products providing the services, the Web-service interface that other products had leveraged. Previously, we were doing a lot of point-to-point integration at the application level or the data level. Now, we are trying to bring it up and integrate it through the other service level.

Gardner: How about you, Sean? Do you have anything to offer on the role of HP Service Manager, particularly the newest edition?

McClean: If you look at HP Service Manager, and also the processes -- as Hwee Ming being the senior architect of the reference model was starting to get to that point -- previously we would provide a tool and we would ask you, "Now what do you need?" and we would architect that tool.

Now we are really driving more and more towards an area where business are saying, "Look, we need to continue doing the business. We don't have time to tell you what we need. Why don't you do it the way you did it in whatever other organizations you gave us this product for?"

If you start combining the ITIL processes, and the more detailed level of ITIL processes you see in ITIL v3 with the Service Manger tool, you are starting to see people saying, "Okay, instead of one piece over here and one piece over here, let's integrate them, even if they are separate tools. Let's look at them and how do they fit into the larger picture of the processes of ITIL v3 and maturation process of that. And, let's make sure that Service Manger matches and aligns with that bigger picture so that the businesses don't have to go through all the detail of bit-by-bit designing something for themselves."

Gardner: In your presentation today here at the Software Universe conference, you said that subscriptions are key for supporting life cycle, what did you mean by that?

Ng: If you really look at it, to be able to manage your service efficiently you really need to know the service that you are offering is adding value to your business. One good way of measuring that is by being able to look at who is using the service and the demand for the service.

Having a subscription managed through the whole lifecycle of a service is key. When you are rolling out new services, subscribers actually subscribe to it, requesting changes to the subscriptions, and managing that all the way to the end of the service lifecycle. They cancel their subscription, when they no longer need the service due to changes in business environment or just changes in the role.

Gardner: I see. So, it's the business model that shifts how IT conceives of itself, and with subscriptions, where people can opt-in or opt-out, there is even a market force in effect. It's supply and demand, and if you're not offering the value that people think is commensurate with the subscription, then they cancel it. That really creates a whole new dynamic of response and refinement in IT.

Ng: It's actually quite a key communication of management of service from the IT point of view. It also helps in planning services. If you see a pick-up in the subscription, you could actually plan for expansion of the service, having the infrastructure and the capacity to support that. And, the reverse. If you see a service having not enough subscription, you might want to scale down the infrastructure that's behind that supporting it. The other aspect is in planning or budgeting -- which areas you should focus in, where you're seeing demand, and things like that.

Gardner: And, that's little different from a customer perspective, as well, instead of just having an application or a set of applications thrown at them, saying "Here, use it, you are going to get charged for it no matter what." That might even leave people with a little sour taste from the start, when it comes to IT.

McClean: I always find it interesting. IT has grown and matured. We always looked at the IT department as a black box. Someone wanders into the organization and says, "We need 'X' amount of dollars for a server that does this." And, you go, "Uh-oh. They are IT. I don't know what they do, but it must be important."

When you start move into this model, as Hwee Ming explained, where we are going through subscriptions, we're just trying to get to people being able to focus on, "Oh, they do work just like the rest of the business." And, that has to happen, because it helps the businesses embrace IT as well. Once everybody starts realizing it's an aspect of business, that makes more sense to everybody.

Gardner: It's a more natural relationship. This is how most people actually do business?

McClean: Yeah.

Gardner: You also mentioned that knowledge management (KM), in the context of ITIL, is important. I didn't know what you meant by KM. Are you talking about KM like we need to know who is good at what in our company, or we need to take both structured and unstructured content and data and make it available for people when they are doing research, or is this KM in the context of IT something else?

Ng: A quote we have been using quite often is that the focus on KM is really to be able to get the right content to the right people at the right time. This is what we are trying to do. We're trying to provide the information that is needed by the service desk to perform their work, or provide the information via Web to the end user, so that they can troubleshoot their own problem. It's about providing the right information to the right people.

Gardner: I see. So, it's the knowledge for more self-help, which takes the burden off the help desk, and then it's more knowledge in the right context to the help desk, so that they can better provide services to the people when they call in.

McClean: Yeah, I think it's interesting, because people will say "Well, we're in the information age." Then, you go forward with that and say, "Okay, if information were moving a little bit faster, then knowledge is becoming more of a commodity." What do you do with a commodity? You have to organize it and make it easily distributable, because you've got to get it to everybody that wants it.

Gardner: It's the information age, where I still can't get my VPN to work.

McClean: Right. So, we need to be able to put that in such a way in an organizational structure that you can get it as quickly as possible when you need it.

Gardner: Okay. I guess we should close up our discussion of ITIL, where we look at what's to come next? Obviously, there is a lot of adoption going on, different companies and regions and industries, and verticals adopting these things at different rates. We hear a lot about version 3 of ITIL, what should we expect with something around version 4 and when?

McClean: I think whether we are talking ancillary tools. In ITIL version 2, the thing that people constantly would drive back in training classes was, "Okay, this all makes sense to me." Some people would even say, "Well, this is common sense. So how do I do it? I get that it's important to do these certain aspects of a process, and it makes sense, but how."

Now, in ITIL version 3, we're saying, "Here's how we need to apply it more to the business." Now we're going to start seeing the design of those concepts and processes and alignment of those to the tools, so that when you look at a tool and you go to use this tool, you can quickly check here's how and here's why. We'll see that with the solutions like the blueprint and solutions like the Reference Model, where we're diving down into the ITIL process and getting more in depth. Other architects are looking at the tools and saying, "Here is where that tool connects to this process." So, the business can see those connections as well.

Gardner: Hwee Ming, where do you see the next advances in ITIL coming?

Ng: I see v3 is still relatively new in the adoption phase, and we are seeing a lot more customers wanting to go to ITIL and adopt v3. The next iteration, if we want to do it, will be driven from the field experience, collaboration, and leveraging the best practices that people see in the field. So, I see that as a more collaborative effort in the next revision.

Gardner: I see, so more toward how to manage the teams that you've already put in place that are more business-oriented and more process-oriented?

Ng: I think the next revision of ITIL probably would get a lot of content from consultants or the organization, looking at how ITIL v3 has been used in the organization and what are the best practices and improvements, and driving that back to the business model.

McClean: It's new to me, because that again drives you back to the business model where you say, "We have to figure out what our constituents want so we can continue to provide it to them." When you start researching, the business needs drive more of that.

Gardner: And, that makes sense, rather than "We'll figure out what you are supposed to do and tell you what to do. Maybe we'll have a discussion about it and come up with the best solution."

McClean: Right, and we'll take the information that you provide us in terms of the business and start applying it to what we do on our side to support that.

Gardner: Well great, we've been looking at ITIL through the lens of people who are in the field implementing it and some of the issues that customers for HP and KatalystNow are adjusting to ITIL and perhaps what they'll be doing with their future IT departments culturally and in terms of process.

Well, I want thank Sean McClean, he is a principal at KatalystNow. Thank you for joining and sharing your insights.

McClean: Thanks very much. It was a pleasure.

Gardner: We've also been talking with Hwee Ming Ng, a solution architect in the Consulting and Integration Group at HP in San Francisco. Thanks.

Ng: Thanks a lot. Glad to be here.

Gardner: This comes to you as a sponsored HP Software Universe live podcast recorded at the Venetian Resort in Las Vegas. Look for other podcast from this HP event at hp.com website, under "Software Universe Live Podcasts," as well as, through the BriefingsDirect Network. I would like to thank our producers on today’s show, Fred Bals and Kate Whalen, and also our sponsor Hewlett-Packard.

I'm Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening, and come back next time for more in-depth podcasts on enterprise software infrastructure and strategies. Bye for now.

Listen to the podcast. Sponsor: Hewlett-Packard.

Transcript of BriefingsDirect podcast recorded at the Hewlett-Packard Software Universe Conference, in Las Vegas, Nevada. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.

Apache Camel Addresses Need for Discrete Infrastructure for Services Mediation and Routing

Edited transcript of BriefingsDirect[tm/sm] podcast with Dana Gardner, recorded July 27, 2007.

Listen to the podcast here. Podcast sponsor: IONA Technologies.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion on a top level Apache Software Foundation project, known as Apache Camel.

It’s a way to help developers do better integration and mediation for routing in software-intense environments, and moving lots of objects and component services around, but also relying on rules and patterns to do so.

To help us understand more about Apache Camel, we’re joined today by James Strachan, technical director of engineering at IONA Technologies. Welcome, James.

James Strachan: Hi.

Gardner: Tell us a little bit about Apache Camel. It seems to be a subset of many other integration and infrastructure projects in the open-source community. Tell us a little bit about the history of how Apache Camel came to be.

Strachan: Earlier this year, Apache Camel grew organically from code and ideas from a bunch of other Apache projects, particularly Apache ActiveMQ and Apache ServiceMix. We found that people wanted to create and use patterns from the "Enterprise Integration Patterns" book in many different scenarios.

Some people wanted to use these patterns inside an Enterprise Service Bus (ESB), some people wanted to use these patterns inside a message broker, and other people wanted to use these patterns inside an application itself or to talk between messaging providers. Still other people wanted to use them inside a Web services framework or some other communication platform. So, rather than tie this routing code to a particular message broker or ESB, we tried to extract this code to be a standalone framework that can be used in any project.

Gardner: I assume that this is to simplify it and make it easier for developers? Is that right?

Strachan: Definitely. What we tried to do with Camel was give it the smallest footprint possible, so that it can be reused anywhere, whether in a servlet, in the Web services stack, inside a full ESB, or a messaging application.

Gardner: And, this is a plain old Java object (POJO), I believe, also built in Java. Is that correct?

Strachan: Absolutely.

Gardner: Tell us a bit about the team behind this and perhaps a little background for yourself?

Strachan: I'm an Apache committer. I’ve been in Apache for over five years now, and I am an Apache member for a few years too. I can’t remember how many. The Camel team is comprised mostly of people from the ActiveMQ team and also from ServiceMix and Apache CXF communities. Plus, a bunch of other people have joined since then.

Gardner: Tell us a little bit about your relationship with Apache. You’re involved with a number of activities there. Is that right?

Strachan: I've been around Apache for many years now, and I am involved in all kinds of different projects -- ActiveMQ, ServiceMix are the big ones -- but also on a bunch of others, like Jakarta Commons, Maven, and many others.

Gardner: There are a number of terms we’ve been bandying around. Let’s assume that not everyone listening is that deeply into technology. When we talk about mediation and patterns, let’s flesh some of these out. Mediation and routing, how is that different from what people might understand generally under enterprise integration?

Strachan: I definitely recommend people read Gregor Hohpe's book "Enterprise Integration Patterns." He offers a really good patterns catalog of how people should do mediation and integration. Rather like the original Gang of Four "Design Patterns" book, which describes low-level programming things, Gregor’s book describes very well how enterprise integration patterns (EIPs) can work and gives us a language for describing them.

Aggregate, re-sequencer, message filter, and message translator generally describe patterns people commonly want to do within an integration solution. One of the simplest patterns is "content-based router," where you want to route messages around your network, based on the content. It might be, for example, that for gold-level customers you want to use the big, shiny, fast machines in your data center, but for bronze-level customers you use the crappy, old, worn Linux box in the corner of the basement.

Gardner: You mentioned earlier ESB, enterprise service bus. Explain to me how this works in terms of other ESBs. This is ESB-agnostic. Is it something that people will use to complement ESBs or perhaps replace them in some instances? What’s the relationship between Camel and an ESB?

Strachan: That’s a good question. In some ways the term "ESB" is a little bit like the terms "object" and "component," in that ESB is used for many different things. It has kind of lost a lot of its meaning. We typically don’t tend to think of Camel as being an ESB. We think of it as a rules-based routing and mediation framework that you may deploy inside an ESB.

We work very closely with the ServiceMix community at Apache which has created a complete Java Business Integration (JBI)-compliant container of JBI components. Camel can be deployed within the ServiceMix ESB among JBI components, but some people don’t use JBI and they may just use Java Message Service (JMS) or they may just use Web services or they may just be JAX-WS clients or whatever. So, we try to make Camel agnostic to technologies. You can use it within patterns like Spring, or JBI or OSGi, or you can use it within any application.

Gardner: I’ve also heard you refer to this as a domain-specific language project. What do you mean by that?

Strachan: Programming languages are often very general purpose. They can be used to solve any kind of problem. Domain-specific languages have become popular lately, and they’re basically tied to typically a business domain, such as investment banking, telco, or whatever. In Camel’s case, we’ve made a domain-specific language that focuses purely on the EIPs.

The language itself describes how to do the various things you typically do in the patterns, such as wire tap, aggregator, content-based router. These are first-class concepts within the language itself. We’ve implemented this domain-specific language in both Java and XML, so you can stick to a Java IDE if you prefer, or, if you’d rather, you can use XML to describe this language and deploy it inside your Spring XML.

Gardner: So, we’ve had vertical industry domain-specific languages. This seems to be more of a functionally domain-specific language. Is that a correct statement?

Strachan: Yes. The EIP is the functional domain, or rather the business domain.

Gardner: And, this is all about getting more granularity, more specialization for integration that nonetheless can play across a more general or horizontal capability. Is that fair?

Strachan: Definitely. Across all of IT we’re seeing increased specialization in many different areas, where the specialization helps us solve a problem at a higher level. If you spend your life solving problems, with, say, JavaBeans everywhere, you have a lot of code to comprehend and understand, and things can get quite complicated. The higher level of abstraction helps us solve problems easily.

Gardner: And, because we’re doing it an open-source environment where there’s a community involved, the more likelihood that this will be applied across many other different types of platforms and technology?

Strachan: Exactly. The other benefit of being with Apache is that we have a very liberal license, so this can be embedded in any commercial product very easily. We've seen lots of partners using this technology.

Gardner: Right. A few moments ago, we talked about ESBs. There has been, as you say, some divergence in the term ESB. There are also a number of products and approaches in the field. How about for Camel as a rules-based routing and mediation engine? Are there many others of those? Who else is trying to solve the same problem, or can we safely say that the Camel is unique?

Strachan: It depends on how wide you cast that net. Camel is unique in a number of ways. What we’re doing with Camel is defining a high-level language to describe EIPs, which I don’t think anybody else has done before. The other thing that’s unique is that this language very closely maps to components that work inside Spring 2.

We use a lot Spring 2 features, such as declarative transactions, inversion of control configuration, and various utility classes for working, with such things as JMS and JDBC and Java Persistence API (JPA). What we’re doing is raising the abstraction level to make things very simple, reducing the amount of XML we have to write, but still exposing the wire-level access if you need to do the really hard stuff and roll your sleeves up and get down and dirty.

Gardner: Let’s move this a bit closer to the business side. I wonder what sort of a problem set it is that we’re facing here. Is this a new approach to older problems or a new approach to newer problems? What's the problem solution fit here in the market?

Strachan: The problem we’re trying to address is the routing and mediation problem, which lots of people have. They're taking data from various components and sources -- whether it’s files, databases, message queues, Web services, instant messaging, or other data systems, integrating them together, formatting them, and connecting them to the systems. From a higher level perspective, this could be for legacy integration of systems, for smart routing, performance, monitoring, or testing or monitoring transaction flows.

In general, integration is a very old problem. As soon as we relate to couple of different computer programs, we already have an integration problem and that just grows with time. The thing that’s new is the approach that’s used. We're increasing the abstraction level by making it very easy to work at the EIP layer.

People don’t have to worry about the low-level details of how to use JMS, how to use JBI, or how to wire together the Spring components correctly, and so forth. We're giving people a nice, simple, high-level abstraction, but yet we are exposing all the power of frameworks like Spring and still exposing the low-level details if you need them.

Gardner: Over the last couple of years, because of the popularity and use of Web services, the business side wants to expand the interactions across not only applications, but organizations, and perhaps in a supply chain setting. It sounds like this helps solve the issue of inclusiveness across domains and businesses, and then adds the opportunity to start moving toward event-driven computing. Does that sound fair?

Strachan: Definitely. And, as soon as people start moving toward distributed systems, service oriented architecture (SOA), event-driven architectures, and all these other terms, the one thing that keeps happening is that people need a pervasive technology they can apply anywhere. Camel is lightweight, and can be used anywhere, in a smart end point, in an ESB, or a message broker.

ActiveMQ5 is coming out in [August, 2007]. That's going to have EIPs integrated in the client and the broker by Camel. ServiceMix has integration with Camel and CXF, as a project at Apache, which implements the JAX-WS standard that’s got EIP integration via Camel. We’re seeing a large number of projects that have EIP integration via Camel. It’s already proven in the open-source world that it’s very easy to reuse this routing and mediation technology anywhere. Hopefully, we'll see this just grow and grow.

Gardner: It’s curious to me. It seems like yesterday, but I suppose we're going on 10 years now, that a lot of these functions -- singular, distributed, perhaps even proprietary -- were combined into what became conceptually the application server. Now, it seems like we’re decoupling things. That one large application server approach might not be the best fit, as we get more types of routing and messaging in these integration patterns.

Can you expand on that? What's been the larger trend, in terms of architecture?

Strachan: There’s been a definite move away from J2EE apps, the big application servers, and people are looking for small, lightweight solutions that solve only the problem they have and not problems they don’t have. A lot of software vendors suffered from the kitchen-sink syndrome, where they tried to have one of everything in a big box. That just led to lots of complexity and redundancy.

People really want small and simple-to-use components that solve the problems they have. I've seen that throughout all of our customers. People ask for very specific solutions. They don’t say, "Give me a SOA." They say, "I need a message router," "I need a message bus," "I need an ESB," or "I need a services framework." Often, people have very specific requirements and are very much cherry-picking the best-of-breed components from the open-source tool set.

Gardner: So, it’s the Swiss Army Knife approach.

Strachan: Exactly.

Gardner: That works well for developers. I wonder over time how Camel might find itself being used in terms of operators, specifiers, and architects on the operational side of things. Do you see this moving in that direction? What’s the short-, medium-, and long-term implementation roadmap that you foresee for Camel?

Strachan: Today it's very much developer-focused. The current Camel is aimed at developers who are savvy with Java, XML, or Spring. What we want then is to gradually raise the abstraction level, so the architects and operational people can monitor services, monitor business processes, perform operational behavior, plus be able to design and deploy EIPs in a simple way.

For example, building tools and making it easier to design EIPs, as well as to choose a monitoring tool, are important.

Gardner: How would the specifying roadmap shake out? Do you think this is something that would get bundled in or integrated? How would this be something easily rationalized in production environments for those people that are going to be looking at large data center implementations?

Is this something that would be part of another grouping of products, other Apache kind of projects? How would this combine to make a whole larger than the sum of the parts?

Strachan: What we’ve seen over the last few years is a huge diversity in how people actually deploy products. Maybe five years ago, people assumed the world was J2EE app servers and everything would deploy in a J2EE deployment unit. These days, we see people make their own deployment units. We're seeing a lot of people move towards OSGi bundles. So we see a huge difference in the ways in which people deploy software now.

One of the reasons we’ve taken extreme steps to split Camel up and make it a very simple and easy to use bundle is that we know that there is a huge range of deployment platforms. Some people want to use Camel within their own smart end points. Some people want to host it inside the message broker, in the ESB and so forth. So, we see it going into production in many guises and many different forms.

Gardner: In IONA’s case, how would this be used in the context of its offering? I'm assuming that, as with other IONA offerings, it will be the commercial open-source support approach.

Strachan: Yes. Camel’s pretty much the core foundation for smart routing and mediation within an entire product suite. We have a product called FUSE, an open-source SOA backplane that provides a services framework, a message broker, an ESB, and a mediation and routing engine. The mediation and routing engine is based very heavily on Apache Camel, and we're using Apache Camel throughout other projects as well.

Gardner: Are there forums or community sites? How are some of the dialog and collaboration around this taking place?

Strachan: If you go to http://open.iona.com, there is a whole raft of documentation online forums, a wiki, and so forth.

Gardner: I know this is fairly early, but maybe this will be a good time to lay out some of the milestones. Are we expecting some significant developments and/or visions within the overall Camel project? Can you give us a bit of a timeline?

Strachan: It’s still fairly early for the timeline, but the next version of Camel 1.2 should be out fairly soon, and it's going to offer improved monitoring and management tools and particularly improved visualization and tooling.

Toward the end of the year, we’re hoping to improve various other capabilities we’ve added recently for extract, transform and load (ETL). It’s a way of loading data into systems and load testing systems with data. Plus, another area we’ve been developing what we call business activity monitoring (BAM), which helps monitor transaction flows across different systems.

For example, you might want to track that a purchase order in a system maps through to an invoice that comes out of another system. You might be processing a 100,000 of those a day, but there is one that goes along every few hours, and you want to find out which one that is within a small time window. That’s another area where we are extending the reach of Camel to do very complex and powerful monitoring.

Gardner: That's interesting, because this provides the actual routing mechanisms that provide a window into what's good, and also what's not good about how those processes are being built and used.

Strachan: Definitely. We’ve found that as soon as you have this very flexible and reusable integration patterns, it’s very easy to build on that, to build higher level abstractions, whether this is powerful testing frameworks, stimulation frameworks, ETL or BAM. We’re seeing the levels of abstraction increase to remove layers and layers of complexity.

Gardner: Is this something that could be applied to the ongoing -- and I imagine increasingly arduous -- task of dealing with semantic issues in terms of data across applications? That is to say, helping to define a mediation and a rules-based approach, such as this help in how companies can deal with that diversity and then come up with more common approaches to data and content that can then be injected into a business process.

Strachan: Definitely. This is a very large and complex topic. There are just a few bullet points on the top, and we’ve already got a range of transformation and validation capabilities in Camel, but we’re looking to improve this in various ways.

IONA recently acquired a company, called C24, which has a whole range of technologies for doing semantic mapping and transformation of different data formats. For example, they've got models for pretty much all the standard financial formats, like SWIFT, FpML, FIX, and all these other things. And, they've got other models for telcos. We’re looking to open source some of that and integrate that into Camel for doing lots of data service type integration.

Another area that’s kind of similar, but slightly different, is in the business activity monitoring ground, where you sometimes want to just check the effect of systems. So, it’s a little bit like I am doing real time reconciliation systems, and you’re doing all this transformation between systems, but you want to make sure of the end results, like that final purchase order or that final settlement instruction does contain the values that you expect. For example, we might want to check that the purchase order amount matches the invoice amount, or whatever.

Gardner: As you say, it’s early in the evaluation of Camel, but are there instances where we can see how it's being used or how developers have identified this as a productivity benefit, or are there metrics of productivity? How is it helping in terms of getting a better job done?

Strachan: We’ve got a bunch of different customers using Camel in different ways. We seem to have a few different kinds of customers finding Camel for the first time. Some come from an ActiveMQ background. They're doing lots of messaging, and they want something a bit more powerful to do more complex routing within the message broker itself.

Some come from the ServiceMix background. They are very ESB focused. They love JBI. They like the idea of standards-based ESB. They’ve got a bunch of standards-based integration components, but they’re working together, and they just want a slightly more powerful routing engine to work with.

Some of our customers have been using Mule. They found Camel a little bit easier, and they’ve migrated across. Finally, there are people who’ve just found Camel, that's all they know, and they just dive straight into it.

One of the things we’re seeing from customers is that using APIs, like JMS and JBI in particular, and to a lesser extent JAX-WS, could often become complicated, but people find that wiring things together with Camel is almost trivial. People get things going really quickly without having to read pages and pages of specifications for things like JMS or JBI, or dealing with the lower-level details of Spring.

We see quite a rapid productivity gain, and it means that people can then use lots of different technologies, whether it’s JMS or JBI without having to spend a long time figuring out things in the Spring manuals.

Gardner: Among these early users are there any vertical industries popping out, where it seems to be making sense first, perhaps financial services?

Strachan: Yes, finance and telcos definitely are big markets, but we’re also fairly strong in the e-tailers. Those areas will be our big three markets right now.

Gardner: Can you explain the business problems that they're using it for?

Strachan: Pretty much all of our customers have similar kinds of problems. They have lots of silos in applications. There are lots of messages being exchanged between systems, whether those are messages via flat files, email, JMS, legacy message brokers, or web services. They generally want to do one or more of the patterns, whether counter-based routing, or message transformations. We see more people starting to investigate a business activity monitoring feature of Camel, and that’s something I'm quite excited about.

Lots of companies, once they realize they can tie together messages to business processes, start running rules off the back of that. That has been extremely useful for people. One thing about SOA that’s quite difficult is knowing what’s really happening in your system.

Today, people have often done such things as monitor message queues or use a JMX-based management console to look at numbers and statistics. Being able to tie together all the different things flowing through your SOA to actual business processes is extremely useful and valuable for customers.

Gardner: There seems to be some murkiness in how feedback will occur, when you have events-driven architectures, and how to create common approaches to runtime and design time with this BAM emphasis that we are seeing associated with Camel. Do you expect that Camel could fill some of that role in terms of a balancing act between what takes place in operations and what needs to be brought into a service level agreement of some kind?

Strachan: Definitely. I have worked with quite a few customers on various BAM-related problems, and it does seem that BAM solves those problems. It’s quite hard to have an off-the-shelf tool to do everything people need. Almost as soon as people start doing any kind of BAM, they find that they want to put their own code in there. You might be in financing, and maybe there is a trade in U.S. dollars. You want to take that trade to the settlement instruction, but you want to do a foreign exchange conversion in there.

Almost as soon as you start doing the “hello world,” BAM-type scenarios, you tend to want to roll your sleeves up and actually do some coding of some kind, to do custom reconciliation, custom calculations, custom visualizations. BAM is very much a framework-driven thing, rather then a black-box tool. Certainly, we see people take on the Camel framework itself and then expand and extend it to do more powerful things.

Gardner: I suppose that’s part and parcel of open source, community-driven projects. They tend to be pushed out in a variety of directions. Do you have any sense of what other functional parts or functional capabilities we might see brought into Camel?

Strachan: A big area is visualization. It’s incredibly hard for people to follow what’s happening in general. Once you can create a wire tap into any system, it’s very easy to glean information from it and find out what’s happening, what its throughput is, what it is actually doing. You can then do things like trace messages around the system, correlate messages to business processes, reverse engineer whole business flows from legacy systems.

That whole area is a visualization and tooling problem. I see that as a very large area of research with the Camel project. We hope to be to be rolling out in Q4 the first batch of visualization tools. It's very much a work in progress in the next few years to get even better, more powerful operational controls and visualization engines.

Gardner: By getting more visual, it seems like we can bring more people into the process, and that maybe elevates us beyond BAM into business intelligence. Is that fair?

Strachan: Definitely. Absolutely.

Gardner: I think a lot of business people would like to get more of a real-time sense of what’s going on among these processes to provide more confidence, as they move processes from manual and older message into this new SOA approach.

Strachan: Definitely. Plus people want early warning notice of any kind of failure. One of the nice things is that you can be alerted that maybe something's going on that hasn't been found yet. Maybe it's running a bit slow today, or maybe it’s just a heavy trading day. But, it might not be, and maybe something is really wrong. So, rather than waiting for some extreme failure that results in a fine or extra interest being paid, you can respond more rapidly than that.

Gardner: Let’s just circle back quickly. Developers seem to be the core short- to medium-term audience for this. What sort of developers specifically should be more aware of what Camel’s capable of?

Strachan: I’d say anyone who’s doing any kind of SOA or distributed programming. They should at least take a good look. Definitely, anybody who ever read the EIP book. If you read that book, you’d probably just see Camel and you’ll immediately just figure out exactly what it’s doing. It’s very simple.

If you're doing Web services, you might need mediation. If you're doing any kind of messaging using multiple databases, if you have any kind of integration problems, and certainly if you’re using any kind of ESB, then definitely take Camel for a ride.

Gardner: Now, when we look toward getting more information on this, we have the resources at Apache Foundation, and also, as you mentioned, http://open.iona.com?

Strachan: Yes.

Gardner: Any other resources? You mentioned the book. Are there any other resources on this that people should be aware of?

Strachan: Those are the big three. There’s also my blog: http://macstrac.blogspot.com/.

Gardner: Very good. Well, we’ve been discussing a budding Apache Foundation project known as Apache Camel. It’s a rules-based routing and mediation engine for POJO-based implementation of EIPs.

Discussing this with us today, we had James Strachan, technical director of engineering at IONA Technologies.

I'm your host and moderator Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening and learning more about Apache Camel.

Listen to the podcast here. Podcast sponsor: IONA Technologies.

Transcript of Dana Gardner’s BriefingsDirect podcast on Apache Camel. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.

Transcript of BriefingsDirect SOA Insights Edition Vol. 9 Podcast on TIBCO's SOA Tools News, ESBs as Platform, webMethods Fabric 7, and HP's BI Move

Edited transcript of weekly BriefingsDirect[TM] SOA Insights Edition, recorded Jan. 19, 2007.

Listen to the podcast here. If you'd like to learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Dana Gardner at 603-528-2435.

Dana Gardner: Hello, and welcome to the latest BriefingsDirect SOA Insights Edition, Volume 9. This is a weekly discussion and dissection of Services Oriented Architecture (SOA) related news and events with a panel of IT industry analysts. I’m your host and moderator, Dana Gardner, principal analyst at Interarbor Solutions, ZDNet blogger, and Redmond Developer magazine columnist.

This week, our panel of independent IT analysts includes show regular Steve Garone. Steve is an independent analyst, a former program vice president at IDC and the founder of the AlignIT Group. Welcome back, Steve.

Steve Garone: Hi, Dana. It's great to be here again.

Gardner: Also joining us is Joe McKendrick, an independent research consultant and columnist at Database Trends, as well as a blogger at ZDNet and ebizQ. Welcome back to the show, Joe.

Joe McKendrick: Hi, Dana.

Gardner: Next Neil Ward-Dutton, research director at Macehiter Ward-Dutton in the U.K., joins us once again. Hello, Neil.

Neil Ward-Dutton: Hi, Dana, good to be here.

Gardner: Jim Kobielus, principal analyst at Current Analysis, is also making a return visit. Thanks for coming along, Jim.

Jim Kobielus: Hi, everybody.

Gardner: Neil, you had mentioned some interest in discussing tools. We’ve discussed tools a little bit on the show, but not to any great depth. There have been some recent announcements that highlight some of the directions that SOA tools are taking, devoted toward integration, for the most part.

However, some of the tools are also looking more at the development stage of how to create services and then join up services, perhaps in some sort of event processing. Why don’t you tell us a little bit about some of the recent announcements that captured your attention vis-a-vis SOA tools?

Ward-Dutton: Thanks, Dana. This was really sparked by a discussion I had back in December -- and I think some of the other guys here had similar discussions -- with TIBCO Software around the announcement that they were doing for this thing called ActiveMatrix. The reason I thought it was worth discussing was that I was really kind of taken by surprise. It took me a while to really get my head around it, because what TIBCO is doing with ActiveMatrix is shifting beyond its traditional integration focus and providing a rear container for the development and deployment of services, which is subtly different and not what TIBCO has historically done.

It’s much more of a development infrastructure focus than an integration infrastructure focus. That took me by surprise and it took me a while to understand what was happening, because I was so used to expecting TIBCO to talk about integration. What I started thinking about was, "What is the value of something like ActiveMatrix?" Because at first glance, ActiveMatrix appears to be something with JBI, a Java Business Integration implementation, basically a kind of standards-based plug-and-play ESB on steroids. It's probably a crass way of putting it, but you kind of get the idea.

Let’s look at it from the point of view of a development theme. What is required to help those guys get into building high-quality networks of services? There are loads of tools around to help you take existing Java code, or whatever, right-click on it, and create SOAP and WSDL bindings, and so on. But, there are other issues of quality, consistency of interface definitions, and use of schemas -- more leading-edge thinking around using policies, for example. This would involve using policies at design time, and then having those enforced in the runtime infrastructure to do things like manage security automatically and help to manage performance, availability, and so on.

It seems to me that this is the angle they’re coming from, and I haven’t seen very much of that from a lot of the other players in the area. The people who are making most of the noise around SOA are still approaching it from the point of view: "You’ve got all this stuff already, all these assets, and what you’re really doing is service-enabling and then orchestrating those services." So, I just want to throw that out there. It would be really interesting to hear what everyone else thinks. Is what TIBCO is doing useful? Are they out ahead or are there lots of other people doing similar things?

Gardner: TIBCO’s heritage has been in middleware messaging, which then led them into integration, Enterprise Application Integration (EAI), and now they’ve moved more toward a service bus-SOA capability. Just to clarify, this tooling, is it taking advantage of the service bus as a place to instantiate services, production, and management? And is it the service bus that’s key to the fact that they’re now approaching tooling?

Ward-Dutton: That’s how I believe it, except it extends to service bus in two ways. One is into the tooling, if you think about what Microsoft is doing with Windows Communication Framework. From a developer perspective, they’re abstracting a lot of the glop they need to tie code into an ESB, and TIBCO is trying to do something similar to that.

It’s much more declarative. It’s all about annotations and policies you attach to things, rather than code you have to write. On the other side, what was really surprising to me was that, if I understand it right, [TIBCO] are unlike a lot of the other ESB players. They are trying to natively support .NET, so they actually have a .NET container that you can write .NET components in and hook them into the service bus natively. I haven’t really seen that from anywhere else, apart from Microsoft. Of course, they’re .NET only. I think there’s two ways in which they’re moving beyond the basic ESB proposition.

Gardner: So, the question is about ESB as a platform. Is it an integration platform that now has evolved into a development platform for services, a comprehensive place to manage and produce and, in a sense, direct complex service integration capabilities? Steve Garone, is the definition of ESB, now, much larger than it was?

Garone: I think it is. I agree with Neil. When I looked at this announcement, the first thing that popped into my mind was, "This is JBI." When Sun Microsystems talked about JBI back in 2005, this is what they were envisioning, or was part of what they were envisioning. Basically, as a platform, that raises the level of abstraction to where current ESB thinking was already. At the time was confusing users -- and still is -- because they didn’t quite understand how, or why, or when they should use an ESB?

In my opinion, this raises that level of abstraction to eliminate a lot of the work developers have to do in terms of coding to a specific ESB or to a specific integration standard, and lets them focus on developing the code they need to make their applications work. But, I would pull back a little bit from the notion that this is purely, or at a very high percentage, a developer play. To me, this is a logical extension of what companies like TIBCO have done in the past in terms of integration and messaging. However, it does have advantages for developers who need to develop applications that use those capabilities by abstracting out some of the work that they need to do for that integration.

Gardner: How about you, Joe? Do you see this as a natural evolution of ESB? It makes sense for architects and developers and even business analysts to start devoting logic of process to the ESB and let the plumbing take care of itself, vis-à-vis standards and module connectors.

McKendrick: In terms of ESBs, there’s actually quite a raging debate out there about the definition of an ESB, first of all, and what the purpose of an ESB should be. For example, I quote Ann Thomas Manes . . .

Gardner: From Burton Group, right?

McKendrick: Right. She doesn’t see ESB as a solution that a company should ultimately depend on or focus on as mediation. She does seem to lean toward the notion of an ESB on the development side as a platform-versus-mediation system. I've also been watching the work of Todd Biske, he is over at MomentumSI. Todd also questions whether ESBs can take on such multiple roles in the enterprise as an application platform versus a mediation platform. He questions whether you can divide it up that way and sell it to very two distinct markets and groups of professionals within the enterprise.

Gardner: How about you, Jim Kobielus? Do you see the role of ESB getting too watered down? Or, do you see this notion of directing logic to the ESB as a way of managing complexity amid many other parts and services, regardless of their origins, as the proper new direction and definition of ESB?

Kobielus: First of all, this term came into use a few years back, popularized by Gartner and, of course, by Progress Software as a grand unification acronym for a lot of legacy and new and emerging integration approaches. I step back and look at ESB as simply referring to a level backplane that virtualizes the various platform dependencies. It provides an extremely flexible integration fabric that can support any number of integration messaging patterns, and so forth.

That said, looking at what TIBCO has actually done with ActiveMatrix Service Grid, it's very much to the virtualization side of what an ESB is all about, in the sense that you can take any integration logic that you want, develop it to any language, for any container, and then run it in this virtualized service grid.

One of the great things about the ActiveMatrix service grid is that TIBCO is saying you don’t necessarily have to write it in a particular language like Java or C++, but rather you can compose it to the JBI and Service Component Architecture (SCA) specifications. Then, through the magic of ActiveMatrix service grid, it can get compiled down to the various implementation languages. It can then get automatically deployed out to be executed in a very flexible end-to-end ESB fabric provided by TIBCO. That’s an exciting vision. I haven’t seen it demonstrated, but from what they’ve explained, it’s something that sounds like it’s exactly what enterprises are looking for.

It’s a virtualized development environment. It’s a virtualized integration environment. And, really, it’s a virtualized policy management environment for end-to-end ESB lifecycle governance. So, yeah, it is very much an approach for overcoming and taming the server complexity of an SOA in this level backplane. It sounds like it’s the way to go. Essentially, it sounds very similar to what Sonic Software has been doing for some time. But TIBCO is notable, because they’re playing according to open standards that they have helped to catalyze -- especially the SCA specifications.

Gardner: Now, TIBCO isn’t alone in some releases since the first of the year. We recently had webMethods with its Fabric 7.0. Has anyone on the call taken a briefing with webMethods and can you explain what this is and how it relates to this trend on ESB?

Kobielus: I've taken the briefing on Fabric 7.0, and it’s really like TIBCO with ActiveMatrix in many ways. It's a strong development story there and it’s a strong virtualization story. In the case of webMethods Fabric 7.0, you can develop complex-end-to-end integration process logic in a high-level abstraction. In their case, they’re implementing the Business Process Modeling Notation (BPMN) specification. Then, you can, within their tooling, take that BPMN definition, compile it down to implementation languages like BPEL that can then get executed by the process containers or process logic containers within the Fabric 7.0 environment.

It’s a very virtualized ESB/SOA development environment with a strong BPMN angle to it and a very strong metadata infrastructure. WebMethods recently acquired Infravio, and so webMethods is very deep now both on the UDDI registry side and providing the plumbing for a federated metadata infrastructure that’s necessary for truly platform agnostic ESB and SOA applications.

Gardner: And, I believe BEA has come out through its Liquid campaign with the components that amount to a lot of this as well. I'm not sure there are standards in interoperability, based on TIBCO's announcement, but clearly I think they have the same vision. In the past several weeks, we’ve discussed how complexity has been thrown at complexity in SOA, and that’s been one of the complaints, one of the negative aspects.

It seems to me that this move might actually help reduce some of that by, as you point out, virtualizing to the level where an analyst, an architect, a business process-focused individual or team can focus in on this level of process to an ESB, not down to application servers or Java and C++, and take advantage of this abstraction.

Before we move on to our next topic, I want to go back to the panel. Steve Garone, do you see this as a possible way of reducing the complexity being thrown at complexity issue?

Garone: Yes, I do. A lot of it's going to depend on how well this particular offering -- if you're talking about TIBCO or webMethods, but I think we were sort of focusing mostly on TIBCO this morning.

Gardner: I think I’d like to extend to the larger trend. Elements that IBM is doing relates to this. Many of the players are trying to work toward this notion of abstracting up, perhaps using ESB as a platform to do so. Let's leave it on more general level.

Garone: That’s fine a good point. You’re right. IBM is doing some work in this area, and logically so, although they come at this even though they have a lot of integration products. I consider them a platform vendor, which means their viewpoint is a little more about the software stack than a specific integration paradigm.

I think the hurdle that we’ll need to get over here in terms of users taking a serious look at this is the confusion over what an ESB actually is and what it should be used for by customers. The vendors who talk to their customers about this are going to have to get over a perception hurdle that this is somewhat different. It makes things a lot easier and resolves a lot of those confusion points around ESBs. Therefore, it's something they should look at seriously, but in terms of the functionality and the technology behind it, it's the logical way to go.

Gardner: Joe McKendrick, how about you in this notion of simplicity being thrown at complexity? Are we going to retain that? Is this the right direction?

McKendrick: Ah, ha. Well, I actually have fairly close ties with SHARE, the mainframe user group, and put out a weekly newsletter for them. The interesting point about SOA in general is that TIBCO, webMethods and everybody are moving to SOA. They have no choice. They have to begin to subscribe to the standards they agree upon. What else would they do?

When we talk about what was traditionally known as the Enterprise Application Integration (EAI) market, it’s been associated with large-scale, expensive integration projects. What I have seen in the mainframe market is that there is interest in SOA, and there is a lot of experimentation and pilot projects. There are some very clear benefits, but there is also a line of thinking that says, "The application we have running on the mainframe, our CICS application transaction system, works fine. Why do we need to SOA-enable this platform? Why do we need to throw in another layer, an abstraction of service layer, over something that works fine, as-is?"

It may seem archaic or legacy. You may even have green-screen terminals, but it runs. It’s got mainframe power behind it. It’s usually a two-tier type of application. The question organizations have to ask themselves is, Do we really need to add another layer to an operation that runs fine as-is?

Gardner: If they only have isolated operations, and they don’t need to move beyond them, I suppose it's pretty clear for them from cost-benefit analysis to stay with what works. However, it seems that more companies, particularly as they merge and become engaged in partnerships, or as they ally with other organizations and go global, want to bring in more of their assets into a business process-focused benefit. So, that's the larger evolution of where we’re going. It's not islands of individual applications churning away, doing their thing, but associating those islands for a higher productivity benefit.

Kobielus: The notion of what organizations have to examine is right on the money, but I think that’s more of a fundamental issue around SOA in general. I think the question you asked was how does something like this affect the ease with which one can do that, and will it figure into the cost-benefit analysis that an organization does to see if in fact that's the right way to go.

Gardner: Neil, this was your topic. How do you see it? Does this larger notion strike you as moving in a direction of starting to solve this issue of complexity being thrown a complexity? That is to say, there’s not enough clear advantage and reduced risk as an organization for me to embrace SOA. Do you think what you’re seeing now from such organizations as TIBCO and webMethods is ameliorating that concern?

Ward-Dutton: Yes and no. And I think most of my answers on these podcasts end up like that, which is quite a shame. The "no" part of my answer is really the cynical part, which is that, at the end of the day, too much simplicity is bad for business. It’s not really in any vendor’s interest to make things too easy. If you make things too easy, no one’s going to buy any more stuff. And the easiest thing to do, of course, for the company is to say, "You know what? Let’s just put everything on one platform. We’ll throw out everything we’ve got, and rebuild everything from the ground up, using one operating system, one hardware manufacturer, one hardware architecture, and so on."

If the skills problem would go away overnight, that would be fantastic. Of course, it’s not about technology. It’s all of our responsibility to keep reminding everyone that, while this stuff can, in theory, make things simpler, you can’t just consider an end-state. You've got to consider the journey as well, and the complexity and the risk associated with the journey. That’s why so many organizations have difficulties, and that's why the whole world isn't painted Microsoft, IBM, Oracle, or webMethods. We’re in a messy, messy world because the journey is itself a risky thing to do.

So, I think that what's happening with IBM around SCA, and what TIBCO is doing around ActiveMatrix, and what webMethods is doing, have the capability for people with the right skills and the right organizational attributes. They have the ability to create this domain, where change can be made pretty rapidly and in a pretty manageable way. That's much more than just being about technology. It’s actually an organizational, cultural process, an IT process, in terms of how we go about doing things. It's those issues, as well as a matter of buying something from TIBCO. Everything’s bound up together.

Gardner: To pick up on your slightly cynical outlook on vendors who don’t want to make it too simple, they do seem to want to make things simpler from the tooling perspective, as long as that requires the need for their run time, their servers, their infrastructure, and so on.

TIBCO has also recently announced BusinessWorks 5.4, which is a bit more complex-turnkey-platform approach that a very simplified approach to tools might then engender an organization to move into. I guess I see your point, but I do think that the tooling and the simplification is a necessary step for people and process to be the focus and the priority, and that the technology needs to help to bring that about?

Ward-Dutton: You’re absolutely right, Dana, but I think the part of the point you made when you were asking your question a few minutes ago was around do we see less technical communities getting more heavily involved in development work. This is the kind of the mythical use of programming thing I remember from Oracle 4GL and Ingress 4GL. That was going to be user-programming, and, of course, that didn’t happen either. I do see the potential for a domain where it’s easier to change things and it’s more manageable, but I don’t see that suddenly enabling this big shift to business analysts doing the work -- just like we didn’t do with UML or 4GLs.

Gardner: We’re not yet at the silver-bullet level here.

Kobielus: Neil nailed it on the head here. Everybody thinks of simplicity in terms of, "Well, rather than write low-level code, people will draw high-level pictures of the actual business process, not that technical plumbing." And, voila! the infrastructure will make it happen, and will be beautiful and the business analysts will drive it.

Neil alluded to the fact that these high-level business processes, though they can be drawn and developed in BPMN, or using flow charting and all kinds of visual tools, are still ferociously complex. Business process logic is quite complex in it’s own right, and it doesn’t simply get written by the business analyst. Rather, it gets written by teams of business and IT analysts, working hand in hand, in an iterative, painful process to iron out the kinks and then to govern or control changes, over time, to various iterations of these business processes.

This isn’t getting any simpler. In fact, the whole SOA governance -- the development side of the governance process -- is just an ongoing committee exercise of the IT geeks and the business analyst geeks getting together regularly and fighting it out, defining and redefining these complex flow charts.

Gardner: One of the points here is around how the plumbing relates to the process, and so it’s time and experience that ultimately will determine how well this process is defined. As you say, it’s iterative. It’s incremental. No one’s just going to sit there, write up the requirements, and it’s going to happen. But it’s the ability to take iterations and experience in real time and get the technology to keep up with you as you make those improvements that's part of the “promise” of SOA.

McKendrick: The collaboration is messy. You’re dealing with a situation where you’ve got collaboration among primarily two major groups of people who have not really worked a lot together in the past and don’t work that well together now.

Gardner: Well, that probably could be said about most activities from last 150,000 years. All right, moving onto our next topic: IBM came out with its financials this week, we’re talking about the week of January 15, 2007, and once again, they had a strong showing on their software growth. They had 14 percent growth in software revenues, compared to the year-ago period. This would be for the fourth quarter of 2006, and that's compared to the total income growth for the company of 11 percent -- services growing 6 percent, and hardware growing only 3 percent.

So, suddenly, software, which does include a lot at IBM, but certainly a large contribution form WebSphere and middleware and mainframes. Mainframes are still growing, but not great -- 5 percent. Wow. The poster child at IBM is software. Who'd have thunk it? Anybody have a reaction to that?

Ward-Dutton: Of course, one of the things that's been driving IBM software growth has been acquisitions. I know I’m a bit behind the curve on this one, but the FileNet acquisition was due to close in the fourth quarter. If that did happen, then that probably had quite a big impact. I don’t know. Does anyone else know?

Gardner: I guess we’d have to do a bit more fine-tuning to see what contribution the new acquisition’s made on a revenue basis, but the total income growth being a certain percentage and then the software, as a portion of that, I suppose, is the trend. Even so, if they’re buying their way into growth, software is becoming the differentiator in the growth opportunity for IT companies, not hardware, not necessarily even professional services.

That does point out that where companies are investing, where enterprises are investing, and where they're willing to pay for a high-margin and not fall into a commodization pattern, which we might see in hardware, is in software.

Kobielus: Keep in mind, though, in the fourth quarter of 2006, IBM had some major product enhancements. Those happened both in the third and the fourth quarter in the software space, and those were driving much of this revenue growth. In July, they released a DB2 Version 9, formerly code-named Viper, and clearly they were making a lot of sales of new licenses for DB2 V9. Then, in the beginning of the fourth quarter, they released their new Data Integration Suite. That's not so new, but rather enhancements to a variety of point integration tools that they’ve had for a long time, including a lot of these software products that they'd acquired with Ascential.

Gardner: That’s the ETL stuff, right?

Kobielus: Not only that, it's everything, Dana. It’s the ETL, the EII, the metadata, the data quality tools, and the data governance tools. It’s a lot of different things. Of course, they also acquired FileNet during that time. But also in the late third quarter IBM released at least a dozen linked solo-product upgrades. In the late third quarter, they were clearly behind much of the revenue growth, and in the fourth quarter for the software group. In other words, the third and fourth quarters of this past year had announcements that IBM had primed the pump for in terms of the customers’ expectations. And, clearly, there were a lot of pent-up orders in hand from customers who were screaming for those products.

Gardner: So you're saying that this might be a cyclical effect, that we shouldn't interpret the third and fourth quarter software growth as a long-term trend but perhaps as beneficial but nonetheless a "bump in the road" for IBM.

Kobielus: Oh, yeah. Just like Microsoft is finally having a bump, now that it’s got Vista and all those other new products coming downstream. These few quarters are going to be a major bump for Microsoft, just like the last two were a major bump for IBM.

Gardner: Let’s take that emphasis that you have pointed out, and I think is correct, on the issue of data -- the lifecycle of data, and how to free it and expose it to wider uses and productivity in enterprise. IBM has invested quite a bit in that. We just also heard an announcement this week from Hewlett-Packard that they are going to be moving more aggressively into business intelligence (BI) and data warehouse activities, not necessarily trying to sell databases to people, but to show them how to extract, and associate, and make more relevant data that they already have -- a metadata-focused set of announcements. Anyone have reaction to that?

Garone: I don’t know too much about this announcement, but from what I’ve read it seems as if this is largely a services play. HP sees this as a professional services opportunity to work with customers to build these kinds of solutions, and there's certainly demand for it across the board. I’m not so sure this is as much products as it is services.

Kobielus: HP, in the fourth quarter of 2006, acquired a services company in the data warehousing and BI arena called Knightsbridge, and Knightsbridge has been driving HP's foray into the data warehousing market. But, also HP sees that it’s a major hardware vendor, just as Teradata and IBM are, and wants to get into that space. If you look at the growth in data warehousing and BI, these are practically the Number 1 software niches right now.

For HP it’s not so much a software play. They are partnering with a lot of software vendors to provide the various piece parts, such as overall Master Data Management (MDM), data warehousing, and business intelligence product sets. But, very clearly, HP sees this as a services play first and foremost. If you look at IBM, 50 percent of their revenues are now from the global services group, and a lot of the projects they are working on are data warehousing, and master data management, and data integration. HP covets all that.

They want to get into that space, and there’s definitely a lot of room for major powerhouse players like them to get into it. Also, very interestingly, NCR has announced in the past week or so that it’s going to spin off Teradata, which has been operating more or less on an arms-length basis for some time. Teradata has been, without a doubt, the fastest growing product group within NCR for a long time. They're probably Number 1 or a close Number 2 in the data warehousing arena. This whole data warehousing space is so lucrative, and clearly HP has been coveting it for a while. They’ve got a very good competency center in the form of Knightsbridge.

They have got a good platform, this Neoview product that they are just beginning to discuss with the analyst community. I’m trying to get some time on their schedule, because they really haven't made a formal announcement of Neoview. It’s something that’s been trickling out. I’ve taken various informal briefings for the last six months, and they let me in on a few things that they are doing in that regard, but HP has not really formally declared what its product road map is for data warehousing. I expect that will be imminent, because, among other things, there is a trade show in February in Las Vegas, the Data Warehousing Institute, and I’m assuming that they -- just like Teradata and the others -- will have major announcements to share with all of us at that time.

Gardner: Well, thanks for that overview. Anyone else have anything to offer on the role of data warehousing?

McKendrick: Something I always found kind of fascinating is that the purpose and challenges of data warehousing are very much parallel to those of SOA. The goal of data warehousing is to abstract data from various sources or silos across the enterprise and bring it all into one place. And the goal of SOA is to take these siloed applications, abstract them and make them available across the enterprise to users in a single place. The ROI formula interestingly is the same as well.

When you start a data warehouse, you’re pumping in a lot of money. Data warehouses aren't cheap. You need to take a single data source, apply the data warehouse to that, and as that begins to generate some success, you can then expand the warehouse to a second data source, and so forth. It’s very much the same as SOA.

Kobielus: I agree wholeheartedly with that. Data warehouses are a platform for what’s called master data management. That's the term in the data-management arena that refers to a governance infrastructure to maintain control over the master reference data that you run your business on -- be it your customer data, your finance data, your product data, your supply chain data and so forth.

If you look at master data management, it’s very much SOA but in the data management arena. In other words, SOA is a paradigm about sharing and re-using critical corporate resources and governing all that. Well, what's the most critical corporate resource -- just about the most critical that everybody has? It's that gospel, that master reference data, that single version of the truth.

MDM needs data warehousing, and data warehousing very much depends on extremely scalable and reliable and robust platforms. That’s why you have these hardware vendors like HP, IBM, Teradata, and so forth, that are either major players already in data warehousing or realizing that they can take their scalable, parallel processing platforms, position them into this data warehousing and MDM market, and make great forays.

I don’t think HP, though, will become a major software player in its own right. It’s going to rely on third-party partners to provide much of the data integration fabric, much of the BI fabric, and much of the governance tooling that is needed for full blown MDM and data warehousing.

Gardner: Great. I'd like to thank our panel for another BriefingsDirect SOA Insights Edition, Volume 9. Steve Garone, Joe McKendrick, Neil Ward-Dutton, Jim Kobielus and myself, your moderator and host Dana Gardner. Thanks for joining, and come back next week.

If any of our listeners are interested in learning more about BriefingsDirect B2B informational podcasts or to become a sponsor of this or other B2B podcasts, please fill free to contact me, Dana Gardner at 603-528-2435.

Listen to the podcast here.

Transcript of Dana Gardner’s BriefingsDirect SOA Insights Edition, Vol. 9. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.