Wednesday, April 22, 2015

ECommerce Portal Avito Uses Big Data to Master Just-in-Time Ad Fraud Detection

Transcript of a BriefingsDirect discussion on how a Russian ecommerce and search engine site leverages big data analytics to identify fraud.

Listen to the podcast. Find it on iTunes. Get the mobile app for iOS or Android. Download the transcript. Sponsor: HP.

Dana Gardner: Hello, and welcome to the next edition of the HP Discover Podcast Series. I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host and moderator for this ongoing sponsored discussion on IT innovation and how it’s making an impact on people’s lives.
Become a member of MyVertica
Register now
And gain access to the Free HP Vertica Community Edition.
Our next innovation case study interview highlights how Avito, a Russian eCommerce site and portal, uses big data technology to improve fraud detection, as well as better understand how their users adapt to new advertising approaches.

Gardner
To learn more about how big data offers new insights to the eCommerce portal user experience, please join me in welcoming Nikolay Golov, Chief Data Warehousing Architect at Avito in Moscow. Welcome.

Nikolay Golov: Hi.

Gardner: Tell us a little bit about your site and your business at Avito. It sounds like the Craigslist of Russia.

Golov: Yes, Avito is a Russian Craigslist. It's a big site and also the biggest search engine for some goods. We at Avito have more searches, for example, from iPhones than Google or Yandex. Yandex is a Russian Google.

Golov
Gardner: Does Avito cover all types of goods, services, business-to-business commerce?

Golov: On Avito, you can sell almost anything that can be bought in the market. You can sell cars, you can sell houses, or rent them, for example. You can even find boats or business jets. We right now have about three business jets listed.

Gardner: So quite a diversity. What are your big data needs? It sounds as if in a country as large as Russia -- with that many goods and services -- you have a high-volume-of-data issue.

Size advantage

Golov: The main advantages of Avito is firstly its size. Everybody in Russia knows that if you want to buy or sell something, the best place for it is Avito. It’s first.

http://www.hp.com/Second is speed. It is very easy to use it. We have a very easy interface. So we must keep these two advantages. But there are also some people who want to use Avito to sell weapons, drugs, and prohibited medicines. It's absolutely critical for Avito to keep it all clean, to prevent such items from appearing in the queries of our visitors.

We're growing very fast, and if we use moderators we'll have to increase our expense on moderation in a linear progressions as we grow. So, the only solution to avoid a linear increase in expenses is to use automation.

Gardner: In order to rapidly decide which should or should not be appearing on your site, you’ve decided to use a data warehouse that provides a streaming real-time data automation effect. Tell me what your requirements are for that technology?

Golov: We have various requirements. For example, we need to be able to perform fast fraud detection. The warehouse has to have very little delay. Hours are not permitted, it must be 10 minutes, no more.
Our data warehouse has to be big. It has to store months, possibly years, of data.

Second, we have to have data for long periods of time to learn our data mining algorithms, to create reports, and to analyze trends. So our data warehouse has to be very big. It has to store months, possibly years, of data. It has to be fast, or only slightly delayed, and it has to be big.

Third, we're developing very quickly. We're adding some new services, and we're integrating with partners. Not long ago, for example, we added information from Google AdWords to optimize banners. So the warehouse must be very flexible. It must be able to grow in all three ways.

Gardner: How long have you been using HP Vertica and how did you come to choose that particular platform?

Golov: Well over a year now. We chose Vertica for two two main advantages. First, speed of load and data. The I/O speed provided by Vertica is awesome.
Become a member of MyVertica
Register now
And gain access to the Free HP Vertica Community Edition.
Second is its ability to upgrade, thanks to the commodity hardware. So if you have some new requirements that require you to increase performance, you can just buy new hardware -- commodity hardware -- and its power just increases.

It’s great and it can be done really fast. Vertica was the winner.

Measuring the impact

Gardner: Do you have a sense of what the performance and characteristics of Vertica and your data warehouse have gotten for you? Do you have a sense of reduced fraud by X percent or better analytics that have given you a business advantage of some sort? Are there ways to measure the true impact?

Golov: During last year, Avito grew really fast. We have a moderation team of about 250 persons at the beginning of this process. Now, we have the same moderation team, but the number of items has increased two-fold. I suppose that's one of the best measures that can be used.

Gardner: Fair enough. Now, looking to the future, when you're working in a business where your margins, your business, your revenue comes from the ability to provide advertisement placements, improving the performance and the value on the actual distribution of ads and the costs associated is critical.

In addition to rapid fraud detection and protection, is there a value from your analytics that refines the business algorithms and therefore the retail value to your customers?
We're starting few more products. The main aim of them is to create our own tool for optimizing the directions of advertising.

Golov: We're creating more products. The main aim of them is to create our own tool for optimizing the directions of advertising. We have banners, marketing campaigns, and SMS. So we've achieved some results in our reporting and in fraud prevention. We'll continue to work in that direction, and we are planning to add some new types of functionality to our data warehouse.

Gardner: It certainly seems that a data warehouse delivers a tactical benefit but then over time moves to a strategic benefit. The more data, inference, and understanding you have of your processes, the more powerful you can become as a total business.

Golov: Yes. One of my teachers in data warehouses explained the role of data warehouses in an enterprise. It’s like a diesel engine inside a ship. It just works, works, and works, and it’s hot around it. You can create various tools to increase it, to make it better.

But there must always be something deep inside that continuously provides all of the associated tools with power and strong data services from all sides of the business.

Gardner: I wonder for others who are listening to you and saying, "We really need to have that core platform in order to build out these other values over time." Do you have any lessons that you have learned that you might share. That is to say, if you're starting out to develop your own data warehouse and your own business intelligence (BI) and analytics capabilities, do you have any advice?

Be flexible

Golov: First, you have to be flexible. If you ask a business about changing, they'll tell you that they can’t. It will be absolutely this, every time. And in two months, it will still change. If you're not ready to change using your data warehouse to get needed data and analytics, it would be a disaster. That's first.

Second, there always will be errors in data, there will be gaps, and it's absolutely critical to start building a data warehouse together with an automated data quality system that will automatically control and monitor the quality of all the data. This will help you to see the problems when they occur.
If you're not ready to change the ratio of your data warehouse to get such data, it would be a disaster.

Gardner: I'm afraid we'll have to leave it there. We've been discussing how Avito, a large e-commerce portal and super retail site in Moscow, has been deploying a data warehouse and BI capability to not only prevent fraud, but also to grow its business through a better understanding of its customers and processes.

So, a big thank you to our guest, Nikolay Golov, Chief Data Warehousing Architect at Avito. Thank you so much.

Golov: Thanks a lot.
Become a member of MyVertica
Register now
And gain access to the Free HP Vertica Community Edition.
Gardner: And I'd like to thank our audience as well for joining us today for our special big data innovation discussion.

I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this ongoing series of HP-sponsored discussions. Thanks for listening, and come back next time.

Listen to the podcast. Find it on iTunes. Get the mobile app for iOS or Android. Download the transcript. Sponsor: HP.

Transcript of a BriefingsDirect discussion on how a Russian ecommerce and search engine site leverages big data analytics. Copyright Interarbor Solutions, LLC, 2005-2015. All rights reserved.

You may also be interested in:

Wednesday, April 15, 2015

Enterprise Architecture Leader John Zachman on Understanding and Leveraging Synergies Among the Major EA Frameworks

Transcript of a live panel discussion at February's The Open Group San Diego 2015.

Welcome to a special BriefingsDirect presentation and panel discussion from The Open Group San Diego 2015, which ran Feb. 2 through Feb. 5. Download a copy of the transcript.
The following presentation and panel discussion, which together examine the role and benefits of how Enterprise Architecture (EA) frameworks can co-exist well, are provided by moderator Allen Brown, President and Chief Executive Officer, The Open Group; John Zachman, Chairman and CEO of Zachman International, and originator of the Zachman Framework, and Steve Nunn, vice president and chief operating officer of The Open Group. [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Allen Brown: John Zachman is founder of The Zachman Framework, Chairman of Zachman International, and Executive Director of the FEAC Institute, as well as Chairman of The Zachman Institute. I have had the privilege of seeing him present on a couple of occasions and introducing him on a couple of occasions. It’s always fun, it’s always entertaining, and I am pleased to say that we have a very good friend in John Zachman.

John Zachman: Allen, thank you, I appreciate that. A friend of mine did a survey of 108 CEOs, around mostly North America. I was shocked when I saw that the survey said that the biggest problem facing the enterprise is change.

Zachman
And when I heard that, I thought I had to explain to the CEOs of the world that a big problem is change. It turns out they already knew that. You have to be fairly smart to be the CEO. I wasn’t giving them credit to be who they were actually.

But when I saw that the biggest problem facing the enterprise is change, 108 out of 108, my reaction was, well, if the CEO thinks the biggest problem facing the enterprise is change, where is the "executive vice president in-charge of change management"? If nobody is in-charge, the high probability is low to zero that you are going to be able to accommodate change.

There are two reasons why I do architecture. One is complexity and the other one is change.

So if you want to create a building, you are going to describe it before you can create it. So you have a complexity. You don’t start with an axe in a forest and start chopping down trees. You start with architecture. If you can’t describe it, you can’t create it.

Baseline is architecture

Now, once you get it created, if you want to change it, the baseline for managing change is architecture. If the Westin Hotel takes an order to put 10,000 people in this room tomorrow morning, banquet style seating at 8 o'clock in the morning, big fancy party, well, they couldn’t put 10,000 people in this room. Something has to change.

Brown
So, audience, how do you do that? Well, you call up the engineering department and say, "Hey, tomorrow morning we have 10,000 people showing up. We have to have banquet style seating, 10 people a table. So we have to change the building. Send up the architecture so we can figure out what to do."

And what happens if the engineering department says, "Send up the what architecture, you mean building architecture? You mean building-wide architecture? You mean building-wide architecture at excruciating level of detail that shows where all the wires are and the outlets and the pipes and the plumbing and so on? We can't do it." You can’t do it. It would take too long and would cost too much.

To create building-wide architecture at excruciating level of detail, even if we had spent the time and money to create that, it wouldn’t be any good for what you guys want to do, because you guys keep changing the building. We can’t keep the architecture up-to-date anyway.

The architectural construct we have won't represent the reality of the enterprise. It wouldn’t be any good to do what you want to do anyway. So basically we don’t have any architecture.

Okay, so what do we do? You have three possibilities. We already have an instantiated building; it already exists. You can get a bunch of guys with axes and sledgehammers to come in here and start knocking out the walls. Not with me in this room.

Nunn
You can make some relatively small changes and you can lose the whole thing. This is not a very tasteful metaphor to use right now, but you can take the members out of a 100-story building, the 90th floor elevator shaft, and that thing could implode on you. Holy smoke, we don’t want to do that. Maybe we better call an architect and have him help us.

You call the architect, the architect comes in, and you say, "Hey, tomorrow morning we have 10,000 people, banquet style seating, and we have got to change the building. When can you start to work"? Architect says, "Well, I can start right now, but where is your architecture"? You say, "We don’t have any architecture actually; that’s why I called you."

He says, "What, you want me to change your building and you don’t have any architecture? Are you crazy?" Nobody in their right mind is going to change this building without architecture. The architect will be out here with drills and tape measures trying to recreate the architecture representation from the standing edifice. He will be reverse engineering, the as-is configuration from the existing instantiation.

They will do all kinds of research. They will go to the library. They will try to find pictures of this thing in a newspaper. They will do all kinds of research before they ever touch this thing. That takes time and costs money. We don’t have time to do that.

Well, here’s your third possibility. It’s obvious this building has exhausted its use of life. You might as well tear this one down and build a new one, this time a building with a bigger room.

Create the architecture

So those are the three options you have if you want to change something that already exists. The basic point is that if you want to change something that already exists, you are going to have architecture -- one way or another. You have to create the architecture.

Now, the reason that I am saying this is if 108 out of 108 CEOs -- of course those were high visibility CEOs -- said the biggest problem facing the architecture is change, who ought to own Enterprise Architecture? I would submit it has to be a general management responsibility.

It needs to be the executive vice president. If the CEO thinks the biggest problem facing the enterprise is change, the CEO ought to be in-charge of Enterprise Architecture. If he or she is not going to be in-charge, then it ought to be the person they see every morning when they come in the office ... "Hey, Ralph, how is it going on the architecture?" So it probably should be a general management responsibility. That’s where I would take it.

Now, I am rushing a little bit. I promised Allen I am going to do this in an hour. I am packing about four days of material into one hour. I am going to rush through it a little bit, because I want to bring you to some conclusions about what I would do relative to putting the two things together.
This same misconception about enterprise is what leads people to misconstrue Enterprise Architecture as being big, monolithic, static, inflexible, and unachievable and it takes too long and costs too much.

I put TOGAF® together with my framework and, in fact, I kind of integrate them. In fact, I have known Allen Brown for a number of years, and he was in Johannesburg a number of years ago. He introduced me to a TOGAF conference and he said, "For a lot of years, I thought it was either Zachman or TOGAF." He said that’s incorrect. "Actually it’s Zachman and TOGAF."

That basically is where I am going to take this: It’s Zachman and TOGAF. But I have to do a little bit of tutorial work. I know there’s a number of people here who have seen me talk about this subject before and I don’t want to bore all you guys to death. At the same time, there are a number of people who have never heard me talk about this.

So I will give you a little bit of a tutorial kind of introduction to Enterprise Architecture. Then, I will try to build enough of this rationale, the logic, so I can provide us some conclusions. Basically, how would you integrate TOGAF and The Zachman Framework, which I obviously think is where we want to go to.

In any case, a little introduction. The first question turns out to be what is architecture? What is it? Some people think this is architecture: the Roman Colosseum. Some people think that is architecture.

Now, notice that is a common misconception. This is not architecture. This same misconception about enterprise is what leads people to misconstrue Enterprise Architecture as being big, monolithic, static, inflexible, and unachievable and it takes too long and costs too much.

If you think that is architecture, I am going to tell you, that’s big and monolithic, and static. It took a long time and it cost a lot of money. How long do you think it took them to build that thing? Not a day, not a week, not a year, not a decade. It was a couple of decades to build it.

In fact, the architecture had to be done long before they ever created the Roman Colosseum. They couldn't have even ordered up the stones to stack on top of each other until somebody did the architecture. The architecture had to be done long before they ever created the Roman Colosseum.

Result of architecture

Now, that is the result of architecture. In the result, you can see the architect’s architecture. The result is an implementation and instance. That is one instance of the architecture. Now, they could have built a hundred of these things, but they only built one.

I was in New Zealand a few years ago and I said that they could have built a hundred of these things, but they only built one. Some guy in the back of the room said they actually built three. I said I didn’t know that. He even knew where they were. I was really impressed.

I was in Rome last June and I am talking to these guys in Rome. I said, you guys could have built a hundred of these things, you only built three. And the guys in Rome said, "We built three; I thought we only built one." Actually I felt a lot better. I mean, you can build as many as you want, but this just happens to be one instantiation. And in fact, that is not architecture. That’s just the result of architecture.

Architecture is a set, it's not one thing, it's a set of descriptive representations relevant for describing a complex object (actually, any object) such that an instance of the object can be created and such that the descriptive representations serve as the baseline for changing an object instance (assuming that the descriptive representations are maintained consistent with the instantiation). If you change the instantiation and don't change the descriptive representations, they would no longer serve as a baseline for ensuing change of that instantiation. In any case, architecture is a set of descriptive representations.
Basically, they are answering six primitive interrogatives: what, how, where, who, when and why. That's been known since the origins of language about 7,000 years ago.

Now, you can classify those descriptive representations in two dimensions. One dimension is what I call it Abstractions. I don't want to digress and say why I happened to choose that word. But if you look at architecture for airplanes, buildings, locomotives, battleships, computers, tables or chairs, or XYZ, they are all going to have Bills of Materials that describes what the thing is made out of.

You have the Functional Specs that describe how the thing works. You have the Drawings or the Geometry that describes where the compound is relative to another. You have the Operating Instructions that describes who is responsible for doing what. You have the Timing Diagram that describes when thing happens, and the Design Objectives that describes why they happen.

So it doesn't make any difference what object you are looking at. They are going to have bills of material, Functional Specs, the Drawings or Geometry or Operating Instructions and so on. You are going to have that set of descriptive representations.

Now, they didn't happen to stumble across that by accident. Basically, they are answering six primitive interrogatives: what, how, where, who, when and why. That's been known since the origins of language about 7,000 years ago. And the linguists would observe for you that's the total set of questions you need to answer to have a complete description of whatever you want to describe; a subject, an object, or whatever.

It's okay if you don't answer them all, but any one of those questions that you don't answer, you are authorizing anybody and everybody to make assumptions about what the answers are that you don't make explicit. So if don't make those answers explicit, people are going to make assumptions. You are authorizing everybody to make assumptions.

The good news is, if the assumptions are correct, it saves you time and money. On the other hand, if the assumptions are incorrect, that could be horrendous, because incorrect assumptions are the source of defects. That’s where defects are, or miscommunications, or discontinuity. That's where you have the defects come from. So you don't have to answer all the questions, but there is a risk associated with not answering all the questions.

And I did not invent that, by the way. That's a classification that humanity has used for 7,000 years. Actually, it's the origins of language basically. I did not invent that. I just happened to see the pattern.

Parts and part structures

Now, there is one other thing I have to tell you, in a Bill of Materials, you have descriptions of parts and part structures. There is no expression of functional specification in the Bill of Materials, there is no expression of Drawings in the Bill of Materials, nor expression of Operating Instructions. There is no expression of Time or Design Objectives. There are parts and part structures.

In the Functional Specs there is Functional Specs. There is no expression of parts or part structures. There is no expression of Geometry or Drawings. There is no expression of operating responsibility, time, or Design Objectives. There are Functional Specs.

In the Geometry, there is no expression of parts and part structures, there is no expression of Functional Specs, operating responsibility, time, or Design Objective. There are the Drawings or the Geometry.

I am not going to do it anymore; you get the idea. If you are trying to do engineering kind of work, you want one, and only one, kind of thing in the picture. You start putting more and more kinds of thing in the picture, and that picture is going to get so complicated that you will never get your brain around it actually.
You want to minimize any potential discontinuity, any kind of disorder. You want to normalize everything.

And if you are going to do engineering work, what you want to do is normalize everything. You want to minimize any potential discontinuity, any kind of disorder. You want to normalize everything. In order to normalize everything you have to see all the parts relative to the whole object. You have to see them all, so that you can get rid of any re-occurrence or any kind of redundancy.

You want to normalize, get the minimal possible set. You only want to look at all the Functional Specs, but you want to look at it for the whole object. Get it down to minimize, minimize the complexity. You want to minimize the redundancy. You don’t want any redundancy showing up and so on. You want to minimize everything. Minimum possible set of components to create the object.

You don't want extraneous anything in that object, whatever that object happens to be,  airplane, building, or whatever. So I just made that observation.

Now I am going to digress. I am going to leap forward into the logic a little bit for you. There is the engineering view. If you want to do engineering work, you want to see the whole. You only want to see one type of fact, but you want to see the whole set for the whole object, for the whole product.

So when you are doing engineering work, you want to see the whole thing, because you have to see how all the parts fit together basically.

Now, if you are doing manufacturing work, however, that's not what you need. You want to take one part, you want to decompose the object down to as small parts as possible and then you want to see the characteristics. You take one part and you need to know the part or part structure. You have to know the functionality for that part, you have to know the Geometry for that part, the operating responsibility, for that part, the Timing Diagram for that part, and the Design Objective for that part. So if you are doing manufacturing, you want to see all the variables relative to one part.

Different models

There are two different kinds of models that are required here. You want the engineering models, which are, in fact, a normalized set. You want to see one fact for the whole object. And the manufacturing model, you want to see for one part. You want to see all the variables for one part. So there are two different kinds of descriptive representations that are relevant for describing the object.

Now, I would just observe this, engineering versus manufacturing. Engineering work requires single-variable, ontologically defined descriptions of the whole of the object, which I would call a primitive.

In contrast, manufacturing work requires multi-variable, holistic descriptions of parts of the object, what I would call a composite. That’s the implementation; that’s the composite.

The interesting phenomenon is -- and somebody talked about this yesterday too -- in manufacturing, this is analysis. You break it down into smaller and smaller pieces. In fact,  it's a good approach if you want to classify, you want to deal with complexity. The way humanity deals with complexity is through classification.
If you just do analysis, and you are doing manufacturing or implementation work, you are going to get disintegration. If you are doing engineering work, you want to deal with the issue of synthesis.

A one-dimensional classification for manufacturing is to decompose it down to various small parts. The reason why that becomes useful, is that it’s cheaper and faster to manufacture the part. The smaller the part, the faster and cheaper it is to manufacture it.

So basically if you go back to The Wealth of Nations by Adam Smith, the idea was to break it down into smaller parts so you can manage the parts, but in doing that, basically what you are doing is you are disintegrating the object, you are disintegrating it.

In contrast, in engineering work, you need to look at synthesis. It you take a one-dimensional classification, you are disintegrating the object. The same content can show up in more than one category, the bottom of the tree. If you want to do engineering work, you want to see how all the parts fit together. That’s a synthesis idea.

So if you just do analysis, and you are doing manufacturing or implementation work, you are going to get disintegration. If you are doing engineering work, you want to deal with the issue of synthesis.

So it’s not an either-or thing; it’s an “and” kind of a thing. And the significant issue is that this is radically different. In fact, it was Fred Brooks who said, programming is manufacturing, not engineering. So those of us who come from the IT community have been doing manufacturing for the last 65 or 70 rears basically. In contrast, this is different; this is a standard. This stuff appears radically different.

So the reason why we build implementations and we get frustration on the part of the enterprise is because the implementations are not integrated, not flexible, not interoperable, not reusable, and not aligned. They are not meeting expectations. Fundamentally, if we use a one-dimensional classification, you're going to end up with disintegrating the thing. It’s not engineering. It’s implemented, but not engineered.

Two-dimensional classification

If you want the thing to be engineered, you have to have a two-dimensional classification. You have to have a schema, a two-dimensional classification, because you have to have two-dimensional in order to normalize things.

I don’t want to digress into that, but Ted Codd was floating around with the relational model. Before Ted Codd and the relational model, we didn’t even have the word normalization at that point in time. But to try to manage the asset you are trying to manage, you have to have a normalized structure.

If you want to manage money, you have to have chartered accountants. If you want to manage an organization, you have to have allocation responsibilities. If you want to manage whatever you want to mange, you have to have a normalized structure.

So if you want the thing to be engineered, integrated, flexible, interoperable, reusable, and so on, then you have to do the engineering work. Those are engineering derived characteristics.

You don't get flexibility, integration and so on from implementation. Implementation is what you get, which is really good. I am not arguing; that’s really good, but on the other hand, if you need integration, flexibility and so on, then you have to do engineering work. So it takes you beyond merely the manufacturing and the implementation.

I gave you one dimension of the classification of descriptive representations, which I called abstractions; the other dimension I call perspectives. Typically, I would take a few minutes to describe this for you, but I'm just going to kind of net this out for you.
You have to take those apart to create the descriptive representations in such a fashion that we can do engineering work with it.

Back in the late 1960s time frame, we had methodologically defined how to transcribe the strategy of the enterprise, but we had to transcribe it. We knew at the time we had to transcribe it in such a fashion that you can do engineering work with it.

It's not adequate to transcribe the strategy in such fashion to say make money or save money or do good or don't do, whatever, or feel good, or feel bad, or go west or go east. Those are all relevant, but you have to take those apart to create the descriptive representations in such a fashion that we can do engineering work with it.

These are in the late 1960s time frame. We knew how to transcribe this strategy in such a fashion that we could do engineering work. What we didn't know is how to transform that strategy into an instantiation such that the instantiation bears any resemblance to what the strategy was fundamentally.

So the problem is that in those days, we tended to describe this in a somewhat abstract fashion: make money or save money, whatever, but down here, you're telling a person how to put a piece of metal in a lathe and then how to turn it to get whatever you're trying to create. Or it could be telling a machine what to do, in which case you're going to have a descriptive representation, like 1,100 and 11,000. So it's a long way from "make money" to "11,000." We didn’t know how to make the transformation of the strategy to the instantiation, such that the instantiation bears any resemblance to the strategy.

We knew architecture had something to do with this, but, if you go back to the late 1960s time frame, we didn’t know what architecture was.

Radical idea

I had a radical idea one day. I said, "What you ought to do is ask somebody who does architecture for something else, like a building, an airplane, a computer, an automobile, or XYZ. Ask them what they think architecture is." If we could figure out what they think architecture is, maybe we can figure out what architecture is for enterprises. That was my radical idea back in those days.

A friend of mine was an architect who built buildings actually. So I went to see my friend Gary, Gary Larson, the architect, and I said, "Gary, talk to me about architecture." He said, "What do you want to know?" I said, "I don't know what I want to know; just talk to me and maybe I'll learn something."

He said, "Somebody came in my office, and said I want to build a building." I said, well, what kind of building do you have in mind? Do you want to work in it? Do you want to sell things in it? Do you want to sleep in it? Is it a house? What are you going to do with it? Are you going to manufacture things in it? What’s the structure of it: steel structure, wood structure, stucco, glass, or whatever?

I have to know something about the footprint. Where are you going to put this thing? What’s the footprint? I have got to know what the workflow is. If you're going to eat in the thing and sleep in the thing, you put the eating and the cooking near each other, you put the sleeping someplace else. You have to know something about the workflows.
By the way, we learned about that a long time ago, those of us who are in IT; separate the independent variables.

I have to know something about the Timing Diagrams. Am I going to design a building that has elevators. It has an up button, you go up, and the down button, you go down. I have to know something about the Design Objectives.

Do you want to change this building, you want flexibility. If you want to change this building after I get a bill, then don't hard bind the wall to the floor. Separate the independent variables. If you want flexibility, you separate the independent variables.

By the way, we learned about that a long time ago, those of us who are in IT; separate the independent variables. I haven’t heard this for 30 or 40 years, but it’s like binding. You don’t want to bind anything together.

You want to bind independent variables together so you collect relationship knowledge. That’s why you bind them together, because as soon as you fix two things together, independent variables, if you want to change one, you have to change them all -- throw the whole thing and you have to start over again.

So if you want to change things, you separate the independent variables. How do you like this for an idea by the way? You have the data division and a procedure division? That’s pretty interesting. You can change one data element and all the instructions. You want to change one data element and all the instructions. So you separate the independent variables if you want to change them.

Implementation

Now, for manufacturing purpose, you want to hard bind them together. That’s the implementation.

So Gary says, "I have to know whether they want flexibility or whatever. I have to know the Design Objectives. sketch up my bubble charts. I have to understand what the boundaries are here, so I don't get blindsided in effect."

"If I'm going to build a 100-story building, a huge building, then I'll live with the owners for a couple of years, so I find their aesthetic values, what they're thinking about, what their constraints are, what they really want to do, how much money they have, what their purpose is. I have to understand what the concept of that building is."

"I transcribe the concepts of the building. And this is really important. I can take this down to an excruciating level of detail. Actually, I have to build the scale model. It has light bulbs that go on or off. I have water that runs through the pipes. I can build a scale model, so that the owners can look at this and say, 'That is really great; it’s exactly what I had in mind', or 'Whoa, it’s not what I had in mind.'"

"It's really important because if the owner. If this is what they have in mind and they say, 'Okay, chief, sign here, press hard on the dotted line, you have got to go through three copies.'"
So the architect defined these models, then they transformed it into the instantiation. He built the building, but it’s not what the owner had in mind. And it’s a massive lawsuit.

"I have an architect friend right now, who's in the middle of a massive lawsuit. The owners of the building did not want to sit down and define these models up here. They said, 'You know what we have in mind so go ahead and define it. We don’t have the time to think about this or whatever.'"

"So the architect defined these models, then they transformed it into the instantiation. He built the building, but it’s not what the owner had in mind. And it’s a massive lawsuit.

"I said to my architect friend, 'I went out to your website and I figured out, I found out why you're having this lawsuit. They were not involved in defining what the concepts are."

Now, Gary would say, "Once I get the concepts, I have to transform those concepts into design logic for the buildings, because I haven’t got the building design, I only have the concepts transcribed. Now I have to deal with pressure per square inch, metallurgical strength, weight of water to move the water around. I have to deal with earthquakes. I have got to deal with the whole bunch of other stuff."

"I may have some engineering specialization to help me transform the requirement concepts into the engineering design logic." In manufacturing, they call this the as-designed representation. Gary called that the architect’s plans. He called these architect’s Drawings. He called these the architect’s plans.

"Now, I have to get the architect’s plans. "I have to negotiate with the general contractor, because the general contractor may or may not have the technology to build what I have designed. So I have to transform the logic of the design into the physical design. I have got the schematics here, but I have to have the blueprints."

Making transformations

"So we have to negotiate and make the transformations, have some manufacturing engineers help me make that transformation. And in manufacturing they would call this as designed and this as planned."

"I make the transformation, so the implementation. They have the technology to implement, the design. Then, this contractor goes to the subcontractors who have the tooling, and they have to configure the tools or express precisely what they want somebody to do in order to create it and then you build the building."

That’s pretty interesting. You notice, by the way, there are some familiar words here: concepts, logic, and physics in effect. So you have the owner’s view thinking about the concept; the designer’s view thinking about the logic; and the builder’s view thinking about the physics in effect. You have the concepts, the schematics, and the blueprints. Then you have the configuration and the instantiation. That's the other dimension of a classification.

Now, there is a two-dimensional classification structure. That’s an important idea. It’s a really important idea. If you want to normalize anything, you have to be looking at one fact at a time. You want to normalize every fact. You don’t want anything in there that’s extraneous. You want to normalize everything.
The original databases typically were either flat files or hierarchical databases. They're not any good for managing data; they're good for building systems.

So it’s a two-dimensional schema, not a one-dimensional schema, not a taxonomy or a hierarchy or a decomposition; this is a two-dimensional schema.

If you folks go back to the origins in the IT community, the original databases typically were either flat files or hierarchical databases. They're not any good for managing data; they're good for building systems. You break it down, decompose it onto small parts, and they're good for building systems. They're not good for managing data.

So then you had to have a two-dimensional classification and normalization. Ted Codd showed up, and so on. I don’t want to digress into that, but you get the idea here. It’s a two-dimensional classification.

And I was in Houston at one time, talking about the other dimensional classifications. Some guy in the back of the room said, "Oh, that’s reification." I asked what that was. Reification? I never heard the word before. It turns out it comes out of philosophy. 

Aristotle, Plato, and those guys knew the ideas that you can think about are one thing but the instantiation of the ideas is a completely different thing. If you want the instantiation to bear any resemblance to what the idea is, that idea has to go through a well-known set of transformations.

You have to identify it and name it. So you're going to have to dialogue about it. Then you define it and you have the semantic structures. They have to have their representations -- all the interior designs are done with representations -- and then you have to specify it based upon the implementation technology. Then, you configure it based upon the tooling and then you instantiate it. And if it goes through that set of well-known transformations, then the end result will bear some resemblance to the outset.

Set of transformations

If you don’t go through that, you may or may not look out, and say, "A blind thing finds a solution every now and then." Well that’s pretty good, but on the other hand, you won’t have any degree of assurance that whatever you’re going to end up with bears any resemblance to what you had in mind of the outset. It has to go through that set of transformations.

By the way, I didn't define those; those came out about a couple of thousand years ago as reification. The etymology of the word "re" is Latin; it means thing. So you’re taking an idea and transforming it into a thing. That’s the other dimension of classification in any case.

This is the framework for Anything Architecture, okay? They are going to bills of material, the Functional Specs, the Geometry, or Drawing, Operating Instructions, Timing Diagrams, Design Objectives. That’s one dimension. For the other dimension, you have the scoping representation, the boundaries, requirement concepts, the design logic, the plan physics, the tooling configurations, and then you get the instantiations. So that’s the framework for Anything Architecture.

And I don’t care whether you’re talking about airplanes, buildings, locomotives, battleships, tables, chairs, or whatever. It’s anything in effect. That's just a framework for Anything Architecture.
I didn't define those; those came out about a couple of thousand years ago as reification. The etymology of the word "re" is Latin; it means thing. So you’re taking an idea and transforming it into a thing.

Now all I did was I put enterprise names on the same descriptive representation relevant prescribing anything.

Okay, we produced a Bill of Materials, too. We would call these the Inventory Models, actually that's the business name for them. The technical name would be Entity Models. Now what's an entity, what's a set? What's important about sets? Well how many members are in the set? Are they all present or not? It is actually that the business cares less about entity. They don't care about entity; they care about inventories.

So let's call them by their business name. It's the Inventory Model. The technical name is be Entity Model, but there is Inventory Model. Now the system Entity Model would be the logic entity. In fact, we would call it a Logical Model, but that would be sitting right there. But the Bill of Materials we would call them Inventory Models.

The Functional Specs we call it the Process Models, those are processes. It takes on something different, or the input process output.

The Drawings or the Geometry we would call the Geography, the distribution models, the locations where you store things and transport things around. That would be the distribution models or the Geometry of the enterprise. Maybe Geography would be our name.

The Operating Instructions, we call the Responsibility Models, the workflow models. You know what responsibilities are going to assign to various roles within the enterprise; responsibility of workflow.

The Timing Diagrams, we would call Timing Models. Some people say the Dynamics Models. Jay Forrester at MIT basically wrote the book Industrial Dynamics in 1959. They were tracing resource flows in the enterprise. They were using manufacturing concepts in human systems and so they call the Dynamics Model, but a lot of times we will call them Timing Models.

Motivation models

The Design Objectives we might call motivation models. So all I was doing was putting enterprise names on the same concepts. By the same token the Scope Contexts we would call Scope Lists. We are just scoping out. Give me a list of inventory, give me a list of processes.

The Requirement Concept, we would call Business Models; those are models of the business. And the design logic, we call system models. Those are the Logic Models, they are the System Models and we call systems.

The plan physics we call technology models, the technology constraint. The part configuration, we call tooling models and then product instances we call the enterprise implementation.
I calculated 176 different plausible definitions for business architecture . . . So you have to get definitive about it, or else you are like freight trains passing in the night.

The enterprise is sitting down here. Actually all this is architecture, but the instantiation is down here.

Allen Brown made some really good observations about business architecture. I have a whole other observation about business architecture. Now the question is when you say business architecture, what do you mean?

I was talking at a big business architecture conference. They were having animated discussions and they were getting real passionate about it, but the fact of the matter is they weren’t defining business architecture the same way; they were all over the board.

I said this yesterday. I calculated 176 different plausible definitions for business architecture. For those guys, you could be talking about any one of those, but if you don’t define which one you are talking about, whoever you're talking to may be hearing any one of the other 175. So you have to get definitive about it, or else you are like freight trains passing in the night.

I will tell you, there are various combinations of these models up here that somebody can articulate as business architecture. Which one are you talking about when you say business architecture. Are you talking about the business process? Are you talking about the objectives and strategies. Or are you talking about the infrastructure distribution structure?

Or are you talking about some combination? You have to talk about the inventories and the processes and see those together. You can put together whatever combinations you want. There are 176 possibilities basically.

I would have what I would call the primitive components defined and then, depending upon what you want to talk about, I would construct whatever definition you want to construct.

Enterprise names

Now, I just put the enterprise names on it again. So here is The Framework for Enterprise Architecture and I populated this. Here is the Bill of Material, here are the functional specs, here is the Geometry or the Geography, here are the Operating Responsibilities, here are the Timing Diagrams and here is the Design Objectives, and here are the Scoping Representations, here are the Concepts Models, the Requirement Concepts, here are the Design Logic, here is the Building Physics in effect, the as planned, here are the Tool Configuration, and there is the Instantiation. So that’s The Framework for Enterprise Architecture.

I just put the enterprise names on it, You obviously saw what I was doing. You can read The Framework for Anything, you can read The Framework for Enterprise, but I was telling you The Framework for Anything. So it’s all basically the same thing. This is Enterprise Architecture.

Now, I have some of these framework graphics. For anybody who wants to go to the workshop this afternoon, we will make sure you have a copy of it, and anybody who doesn't go to the workshop, we will have them out at the table. So you can pick up a copy.

I wrote a little article on the back of  that John Zachman’s Concise Definition of Zachman Framework.

Actually somebody asked me if I had ever read what Wikipedia said about my Framework? I said no, I had never read it. I don’t need to read Wikipedia to find out the definition of The Zachman Framework. So they said, "You better read it, because whoever wrote it has no idea what you're talking about."
It’s architecture for every other object known to humankind. It’s architecture for airplanes, buildings, locomotives, computers, for XYZ. It doesn't make any difference.

So I read it, and they were right. They had no idea what I was talking about. So I fixed it. I wrote the article and put it out there. A couple of months later some friend of mine said, "Did you ever read what they wrote on Wikipedia about your Framework?" I said I wrote it. He said, "What? You wrote it? I don't believe it. It’s not what you talk about."

So I read it and some yo-yo had changed it back. So I changed it back. And a couple of months later, guess what? They changed it. So I said I'd forget these people. The fact is I wrote my own definition of The Zachman Framework, so that’s on the back there, with the little audio.

Now, you understand what I am telling you. This is Enterprise Architecture. It’s architecture for every other object known to humankind. It’s architecture for airplanes, buildings, locomotives, computers, for XYZ. It doesn't make any difference. I just put enterprise names on it.

By the way, for those of you technical aficionados, the meta-entity names are at the bottom of every cell, and there are only two meta-entities on every cell. But the names are very carefully selected to make sure they are precisely unique and single variable. You only have one and only one thing in each one of these -- one type effect in any one of these cells. So in any case, this is Enterprise Architecture.

Friends of mine wanted me to change the name of this to Zachman Ontology, because if you recognize this, this is not a methodology; this is an ontology. This does not say anything about how you do Enterprise Architecture -- top-down, bottom-up, left to right, right to left, where it starts. It says nothing about how you create it. This just says this is a total set of descriptive representations that are relevant for describing a complex object. d I happen to have enterprise names on them, but it doesn't tell you anything about how to do this.

Not either/or

For a lot of years, people didn't know what to do with this. They were saying, "I don’t know what to do with it. How do you do Enterprise Architecture?" Now you understand where I am going to take you with this. This is an ontology, and you need a methodology. It is neither a methodology or an ontology. It’s an ontology and a methodology. It’s not either/or.

However, this is an ontology. It’s classifying. It has unique categories of every set of facts that are relevant for describing a complex object basically.

Now, by the way, there is another graphic in this and the reason I put this is that my name is on a number of websites, but I am excluded from those websites, I have nothing to do with those websites, even though they have my name on them. There is only one website that I have any access to, and that’s zachman.com. That’s why I put that slide in there and there’s some other stuff in there.

Now, you understand what I basically am saying here. Architecture is architecture is architecture. I simply put enterprise names on the same descriptive representations relevant for describing everything. Why would anyone think that the descriptions of an enterprise are going to be any different from the descriptions of anything else your manager has ever described? I don’t believe it.
I don't think Enterprise Architecture is arbitrary… and it is not negotiable.

Now, you could argue enterprises are different. Hey, airplanes are different than buildings too, and buildings are different than computers, and computers are different than tables, and tables are different than chairs, and everything is different, they are all different, but they all have Bills of Material, Functional Specs, Geometry. They all have Concepts, Logic, Physics, so this is basically architecture is architecture is architecture. That’s my observation.

I am trying to do this in a very short period of time and I haven’t had half a day or a day to soften all you guys up, but get ready, here you go. I don't think Enterprise Architecture is arbitrary… and it is not negotiable. My opinion is, we ought to accept the definitions of architecture that the older disciplines of architecture, construction, engineering, and manufacturing have already established and focus our energy on learning how to use them to actually engineer enterprises. I think that’s what we ought to be doing.

So I don’t think it’s debatable. Architecture is architecture is architecture.

I have to tell you another thing, Depth and Width. For every cell, you could have a cell that’s enterprise wide and it's an excruciating level of detail. That would be the whole cell basically.

Or you could have a model that is enterprise wide and only a medium level of detail. That would be half of it. You could have a model that’s enterprise wide at a high level of detail. So there is nothing that says that you have to have an excruciating level of detail. You can just say that’s another variable.

By the way, you could have a model that’s less enterprise wide. It’s an excruciating level of detail. It’s half of the enterprise excruciating or it could be the whole enterprise excruciating level of detail. So you have those two other variables. You have to be able to represent them in some fashion.

The implication is that anything that is white space here, if you don’t make it explicit, it’s implicit, which basically says that you're allowing anybody and everybody to make way.

Risk of defects

It may be fine. You may be willing to accept the risk of making erroneous assumptions. You're going to accept the risk of defects. In fact, in manufacturing airplanes they will accept some degree of risk of defects. When the parts don’t start to fit together in the scrap, the work cost starts to go up. Now, then they will say, wait a minute, you can’t complete the implementations until you have a complete engineering design release.

So that other variable you have to read into this as well. There are two different things here in ontology. I didn't even know what an ontology was till fairly recently.

I'm going to give you my John Zachman layman's definition of ontology. Some of you guys may be ontological wizards. I don’t know, but the probability in a group this big is that somebody really is familiar with ontology.

The Zachman Framework scheme technically is an ontology. Ontologies they are a theory of existence. Ontologies have to do with what exists, a theory of the existence of a structured set. That says a classification, a schema, that is rational, logical, and structured --  it’s not arbitrary -- of essential components of an object. Those essential components that says the end object is dependent for its existence on the components and the components exist as well.
A structure is not a process, and a process is not a structure. You have two different things going on here.

So you have a kind of existence of the object -- it just isn’t the components -- for which explicit expression is necessary. Probably it’s mandatory for designing, operating, and changing the object -- the object being an enterprise, a department of an enterprise, a value chain, many enterprises, a sliver, a solution, a project, an airplane, a building, a bathtub or whatever -- it doesn’t make too much difference what it is. It’s whatever that object is.

A framework is a structure. A structure defines something. In contrast, a methodology is a process, a process to transform something. And a structure is not a process, and a process is not a structure. You have two different things going on here.

Now, this is really an important idea too. Here is a comparison between ontology and methodology. An ontology is the classification of the total set of primitive elemental components that exist and are relevant to the existence of an object. A methodology produces composite compound implementations of the primitives.

All the implementation, the instantiations, are derivative of the methodology. The methodology produces the implementation. The implementations are compounds, and primitives, elements, are timeless and the compounds are temporal.

Now, that’s an important point, and I'll try to give you an illustration of that.

Here is an ontology. I learned a lot from this metaphor by the way. This is a classification of all the elements in the universe actually. It’s a two-dimensional schema. It’s normalized; one factor in one place. You are classifying the elements of the universe in terms of neutrons and protons -- the number of neutrons and protons by the number of electron. That is not a process.

This tells you nothing about how do you do this: top-down, bottom-up, left to right, right to left, or what compound that you might want to create out of this thing. This just says here is the total set of elements from which you can create whatever you want to create.

And once again, I didn’t say this yet, but until an ontology exists, nothing is repeatable and nothing is predictable. There is no discipline.

Best practices

Before Mendeleev published the periodic table, there were chemists. They weren’t chemists actually; they were alchemists, and they were very clever by the way, really competent, very clever. They could produce implementation, produce compounds, but it was based upon their life experience. It was a best practice kind of a thing, not based upon a theoretical construct.

And elements -- these elements are timeless. If you have an element that has six neutrons and protons and two electrons, that’s carbon. The rest of the world calls it carbon. Do yourself a favor and call it carbon. You can call it whatever you want to, but if you want to communicate with anybody else, just call it by the name that is recognizable by the rest of the universe.

Now, in any case, those are the elements and they are timeless. They are just forever.

Here are compounds. This is a process. A process transforms, creates something. This is a process. Take a bowl of bleach and add it to a bowl of alkali. It has to get transformed into saltwater. This is not an ontology; this is a process. Take this, add it to that, and it’s going to produce whatever you want to produce.
We could not have written this down like that until Mendeleev published the periodic table. We didn’t have any notation to produce that.

Now, the compounds are temporal. You produce saltwater for some reason, something good for some whatever, whatever it happens to be that you are trying to create.

Here are some examples of other compounds. This is an acid and a base, or a base or an alkali, and again, sodium chloride on water. It’s a balanced compound. Here is hydrogen, there is hydrogen, there are the two hydrogen. Here is chlorine, there is chlorine, here is the sodium, there is a sodium, here is the oxygen, there is oxygen.

We could not have written this down like that until Mendeleev published the periodic table. We didn’t have any notation to produce that.

So here are some other compounds: here is salt, that’s sodium chloride, here is aspirin. C9H8O4, Vicodin is C18H21NO3, Naproxen is C14H14O3, Ibuprofen, Viagra, sulphuric acid and water and so on and so on.

How many of these can you create out of the periodic table? The answer is infinite. It would go infinite. I don’t want to take the time to elaborate, but it’s infinite. And these are temporal. These are specifically defined to do specific things.

Here is an ontology. How many different enterprises could you create out of this ontology? And the answer again is going to be infinite. Until an ontology exists, nothing is repeatable, nothing is predictable. There is no discipline. Everything is basically best practice. The perimeters are timeless.

Now, here are some compounds. The elements are what I would call a primitive component. The compounds are implementations, instantiations. COBOL Programs, you can read Java 2 or Smalltalk or whatever you want to read; Objects, BPMN, Swimlanes, Business Architecture, Capabilities, Mobility, Applications, Data Models, Security Architecture, Services, COTS, Technology Architecture, Big Data, Missions/Visions, Agile Code, Business Processes, DoDAF Models, Balanced Scorecard, Clouds, I.B. Watson, TOFAF Artifacts, and so on. How many of these are there? It’s infinite. 

Specific reasons

How long will it be until we can add one to the list? What time is it? People get really creative. They create a lot of these things. And these are temporal. They are for specific reasons at a specific point in time.

Here is alchemy. It’s a practice. It’s a mythology without an ontology. Process is down in the basement with a chemistry set, trying things out. If it works and it doesn’t blow the house up, write that one down; that’s a good one. If it blows up, you probably have to write that one down too; don’t do that one again.

So a process with no ontological structure is ad hoc, fixed, and dependent on practitioner skills. It’s not a science; it is alchemy; it’s a practice.

I've got to tell you, the alchemists were really clever. Man, they figured out how to create gunpowder long before they ever had the periodic table. So these people were really creative. However, few hundred years later, Mendeleev published the periodic table.

I don’t know whether you guys realize this or not, but we tend to think the periodic table has been around forever, because the elements have been around forever. Basically we learn that in chemistry or whatever. Periodic table was only published in the 1880-1890 time frame.
If you just built them to get the code to run, they're not going to be integrated, not flexible, not interoperable, not reusable. They are not aligned; they are not meeting expectations.

If you think about this, within 50 years of the publication of the periodic table, the physicists and chemists basically were splitting atoms. Think about this. Once you have order, now research actually works. Things become predictable and repeatable. We don’t have to learn everything by experience. We can hypothetically define other possibilities and get really creative.

Like I say, in a very short period of time, friction goes to zero, and you can get really creative and really sophisticated in very short periods of time, so I just throw that one away.

So ontology versus process, engineering versus manufacturing, architecture versus implementation. It's not "either/or;" it is "and." And the question is, how did you get your composite manufacturing implementation? Did you reuse components of primitive, ontological, engineering constructs, or did you just manufacture the composite ad hoc to some problem or some system requirement?

Actually the enterprise is the total aggregate sum of composite implementations.

Now, the question is, how did you get your composite? Were you just building systems or did you have the periodic table of primitive components from which you assembled the implementation?

If you just built them to get the code to run, they're not going to be integrated, not flexible, not interoperable, not reusable. They are not aligned; they are not meeting expectations.

So the question is, how did you get the composite, the compounds? Did you have the periodic table? Now, obviously I am taking it to a point where I am saying, it’s not an "or;" it’s an "and."

Allen and I were talking about this yesterday. I don’t want to take a lot of time to develop this, but this came from Roger Greer, who was the Dean of the School of Library and Information Management USC years ago, and I just happened to run across some notes I had taken at an IBM GUIDE Conference in 1991.

Professional vs. trade

Roger was talking about the difference between a profession and a trade. He basically didn’t make any differentiation. This is the Professional Service Cycle. The professional starts with a diagnosis, analysis of need, and diagnoses the problem. Then you prescribe the solution. Then the technician applies the solution. He evaluates the application and, depending upon the evaluation, enters into the cycle again.

So what differentiates the professional from the trade or labor is the diagnosis and a prescription, where the trade or labor is involved with the implementation and any evaluation.

My observation is that this is where the engineering has taken place. That’s where you need the ontology to do the diagnosis and the prescription. And then, you need the methodology to do the implementation basically -- the manufacturing. The engineering work is going on over here; the manufacturing work is going on over there.

So what differentiates the professional from the trade? Well, if you start with the diagnosis of the problem and the prescription, that’s what the doctor does. The x-ray technician shoots the x-ray, takes the picture, and then evaluates whatever the result is.
Those of us who come from the architecture domain, need to begin to develop the characteristics of a profession. This is a profession.

Leon Kappelman is a friend of mine. He's an academic guy. He traces the CEO surveys for years and years -- 20, 30 years. In 20 or 30 years, one of the top ten issues that the CEOs of the world say those of us who come from the information community need to deal with turns out to be alignment.

They're basically saying, "I don’t know what you guys are doing. You're spending a lot of money down there in IT. Whatever you're doing with it does not align with what I think the enterprise is about."

So there's an alignment problem. I would submit to you, if you are starting over here, you are going to always be a solution in search of a problem.

So we want to change it. Allen and I really feel strongly about this. Those of us who come from the architecture domain, need to begin to develop the characteristics of a profession. This is a profession. Well, that presumes a discipline, and the implication is that we need to change our whole concept to diagnose the enterprise problem. In fact, that’s the one last slide I would use.

The end object is not to build the system. The end object is to diagnose the enterprise problem. Then, you can prescribe. The enterprise really complicates it. You can probably prescribe three, four, or a dozen different possible solutions that they could pursue. Okay chief, here are a set of things that you can do.

Somebody, I think it was Steve Jobs in his book, said that you had to go in with two recommendations to Steve Jobs, but you have a third one in your pocket, because he would tear them up. So, you have to go in and have a third one.

How many do you want chief? We can construct however many you want to, and you can evaluate them or analyze them for whatever the implications are. What are the capital expense implications, or cultural? You can analyze them and let them understand what the alternatives are, what the implications are, or the alternatives. and you can pick one and you can do the implementation, then you evaluate and so on.

Lessons to be learned

This is what differentiates the profession from the trade. This is important. The more I think about it, there is really lessons to be learned here.

Here are the research lessons that we've learned. It is possible to solve general management problems very quickly with a small subset of primary components -- simply lists and their interdependencies short of the complete primitive models.

You don’t have to do a lot of architecture to begin. You have enough that you can do the diagnosis of the problem. Then, different complex, composite constructs can be created dynamically, virtually cost-free, from the inventory of primitive lists for addressing subsequent general management problems.

Once you begin to populate the inventory of the primitive components, they are reusable to analyze or diagnose other problems. This is not just limited to whatever precipitated the first population of the primitives.
There is a TOGAF development strategy, and I would evolve TOGAF to become an engineering methodology as well as a manufacturing methodology.

And many scenarios can be evaluated to test strategy alternatives before making commitments. You can analyze the implications and make recommendations around those implications before you actually spend money or actually create infrastructure kinds of things.

These are really important issues. So here are my conclusions.

Here is what I would propose to TOGAF. There is a TOGAF development strategy, and I would evolve TOGAF to become an engineering methodology as well as a manufacturing methodology.

Those of us who come from the IT community, for the last 75 years we've been building it to run a system. Technically that’s what people say. If you ask somebody from IT what they do for a living, we would say we're building to run systems.

So all of us are very manufacturing dominant. That's the way we tend to think. We tend to think in terms of composite contracts. Every artifact, if it has more than one variable in it, is a manufacturing artifact; it's not an engineering artifact. I can tell pretty quickly by looking at the descriptive representation, looking at the model.

If you have more than one variable in that model, I know you're using it for manufacturing, for implementation purposes, probably manufacturing, I would say the implementation purposes.

I would just broaden TOGAF to dig in to deal with the engineering issues. The way I would do that in Phase I. I'll tell you what I think phase two and three might be as well. I'm just getting creative here. Allen may say, "That's interesting, Zachman, but you're out of here. We don't need a lot of help. We already got enough things to do here."

Existing process

But, first of all, I would use the existing data-gathering process to populate the inventory of single-variable, primitive podels. We're already doing the gathering, and I would just factor out what are the primitive components and begin to populate the inventory.

We have a little workshop this afternoon to show you how to do that. It is not that hard. Anybody who goes to the workshop will see. The workshop is created in order to just show you that it's not too complicated. You can do this.

I would just use the existing data-gathering, purchase methodology, begin to populate. Then I would reuse the primitive components in the creations of present TOGAF artifacts. You’ve got to create the artifacts anyway, you might as well just reuse the primitive components.

Now that presumes another thing for those of you who are into the tooling domain, but you’d have to map the primitive metamodel against the TOGAF metamodel. So there is a metamodel issue here.
You have to look at the metamodel, the TOGAF artifacts, and see if there is a composite construct in the Metamodel and just factor out what the primitive components are.

But what that would tell you is that you have to look at the metamodel, the TOGAF artifacts, and see if there is a composite construct in the Metamodel and just factor out what the primitive components are.

That's the way you would map the composite you're trying to create from the primitive implementations. That's what I would do. That would be just looking at right where we are today. So, here is the set of primitives and here is the methodology. Let's just use the methodology to do engineering work, and it will still end up creating the same implementation composites.

I was getting creative, here is what I would do. Here is what we were doing. I probably have to say it that way, and I encourage you to do this by the way. I would extend the methodology for enterprise problem diagnosis and solution prescriptions --single-variable, primitive components, binary relationships, and impact analysis.

What you need in order to do the diagnosis is the single-variable, primitive construct, and only binary models, because what you're going to do with this is to impact analysis. You touch one thing, and here are all the other things that are impacted by it.

That application has been around for a long time. I'm not telling you something that nobody knows how to do. But there are single-variable models and binary models. Building a binary model, is this related to that, is this related to this, there are two things at a time. The models are pretty simple. I'm not going to take more time to do that but -- then I would segue to the current TOGAF methodology.

I would come out of here and go into the current methodology, making enhancements incrementally as practical. You have been improving TOGAF forever, from the very beginning. I would just start to begin to improve it based upon what we've learned with the diagnostic and the prescription enhancement.

The transformation

Then, in Phase III, I would orchestrate the transformation from the TOGAF artifacts to implementation, lower rows. I would orchestrate that transformation. So you have the transformation from the strategy up here and to the concept, the logic and physics, totally a configuration.

I would orchestrate that and I would extend the TOGAF governance process. With a governance process, and TOGAF's is really strong, I would just take a hard look at that and elaborate that to manage the entire reification set of transformation. That's where I would take it to.

Some of you may say, "We will not take so long or cost too much, whatever." The argument might be a guess, but I think that’s where I would go. Principally, I think that's where I go because of the implication of changing the fundamental concept of the Enterprise Architecture. The profound significance is this. It alters the concept of Enterprise Architecture from one of building models to one of solving general management problems.

Man, that would be really interesting. It buys the time for the experts to build out the complete Enterprise Architecture -- thing-relationship-thing -- primitive models iteratively and incrementally. You don’t have to do it all at once, but general management problem, my problem, my problem, my problem, iteratively and incrementally start building out little more adding to the primitives over time.
If we could change the perception of Enterprise Architecture to be one of solving general management problems, we would have no problem getting the resources and the time to do it.

Then, it builds significant creditability for the information-technology community, and I would submit, we need all help we can get. If we begin to be perceived to be the enterprise doctors, we would be perceived to be direct not indirect. It wouldn’t be an optional, but mandatory, kind of a responsibility. Most importantly, it would position Enterprise Architecture to become a general management operational process, not simply an IT exercise. I think that's where you have to go.

If we could change the perception of Enterprise Architecture to be one of solving general management problems, we would have no problem getting the resources and the time to do whatever Enterprise Architecture we want to do. That valuation issue will tend to go away. I saw a presentation yesterday about the valuation. It was talking about the Internet of Things, and it was really a creative presentation. I really appreciated it a lot.

But if we can solve general management problems, you don’t have to worry about valuation. I will say one more thing about valuation. The fundamental problem with architecture is that it doesn't save money in the current accounting period. It’s not an expense. You don't make money or save money in the current accounting period. You are building an inventory of assets. What make an asset different than an expense? How many times you use it. Use it more than once, it's as an asset. So you build the inventory of assets.

The assets don’t save you money in the current accounting period, but they enable you to make money or save money in many accounting periods in the future. The problem with asset valuation is the accounting people in the US are precluded from putting values on assets. That's probably because there's not an absolute way to value the asset, because the value of it is derived from how many times you use it, or from what the market will pay for at some point in time.

It’s difficult to value assets and it’s really difficult to value intellectual assets. I would submit Enterprise Architecture as intellectual asset, and we’re just beginning to learn some things about that. But that issue turns out to go away. If you just solve general management, you don’t have to worry about valuing the value proposition.

I think I made it in an hour, but actually an hour and three minutes. I owe Allen three minutes now and that’s not too bad on my part, but  there will be a panel. We will have some discussion, answer any questions, and then also there is a workshop of anybody who cares about trying to work with some of these things. I would be glad to do it. Thank you, Allen, and thank you guys for taking the time to listen, I appreciate it a lot.

Questions

Brown: We have some questions. We talked about professionalizing Enterprise Architecture. We both feel passionately about it, and having these professionals as the enterprise doctors, as you say. The person to ask the questions is actually the CEO of the AEA, Steve Nunn. The AEA is now 44,000 members. And they are actually active, as well, which is great. So, Steve, what have you got?

Steve Nunn: Unsurprisingly no shortage of questions and compliments on the presentation, John.

Here’s a long question, but bear with it. Given a composite of an enterprise, a methodology existed for its construction. Today, I have a million assets with individual change records associated with them. The EA methodology did not govern the maintenance after construction. What do you suggest to correct the EA to the Ontology?

Zachman: This not atypical, by the way. I think that's where most of us live. I normally make the case that those of us who come from IT have been manufacturing the enterprise for last 60 to 70 years. The enterprise was never engineering. We are manufacturers. So we are manufacturing parts. We don't manufacture the enterprise, and the parts don't fit together.

So what do you do if you manufacture parts that don't fit together? We will actually scrap and rework. There is no way to fix that problem after the fact. If you want the parts to fit together, you have to engineer them to fit together before you manufacture them. After you get them manufactured and then try to fit them together, you can't get them to fit together.
You don't have to have them all, but you have to begin. So you have to have solve whatever you can out of the TOGAF activity that you have at your disposal.

We're all sitting in kind of the same position. Somebody has to break the pattern. If you just keep on writing code, you're just going to get more parts. You're not going to change anything. You have to have a different paradigm. I was describing for you a different paradigm, and I was describing for you an engineering paradigm.

I would do just exactly what I said. I’ll start taking TOGAF. We already have this methodology -- very, very widely respected, very widely used. I would take the methodology, the data gathering methodology portion of it, and I would begin to populate the inventory of primitive assets. You don't have to have them all, but you have to begin. So you have to have solve whatever you can out of the TOGAF activity that you have at your disposal.

Once you do that, then you're going to populate them with the primitives that are required to create the TOGAF composites right now, so we can produce whatever we are producing out of TOGAF right now. I would just start with something I know, something I have my hands on. I can start with TOGAF. I would start to populate the primitive artifacts and then create by reusing the primitives to create the composites.

So I would start with there. And then I begin to enhance that over time. I have to begin to enhance the methodology to elaborate. I gave you some thoughts about how I would enhance it, but in the meantime what you can do is, once you start creating the architectural constructs, you have to orchestrate your way out of where you're at.

Migrating what exists

We don't have a blank sheet of paper when we start with our Enterprise Architecture. We already have an enterprise out there. You have to figure out a way to migrate out of the existing environment into the architectural environment. am not just going to tell you what I think the solution is without elaborating how I would use it, but I would use it, the data warehouse kind of a concept. I create the architecture. I extract and transform the database out of the existing application to populate the architectural environment.

I didn't learn this. I didn't figure out this all myself. People from Codec and I were sitting in the Codec Theater one time. They were saying, once we have the architected data, we know what the data is, and now we're going to rebuild all the transaction processing systems to update the data warehouse. Then, after we update the data warehouse, we're going to turn off the legacy system.

Over some period of time, however long you want to take, you just move it out little by little, transaction by transaction by transaction to the architectural environment. If you're going to rebuild the transaction processing systems to populate the data warehouse, then I would add the process specification. I would add the distribution. I would add the other characteristics of architecture. That's the way it would orchestrate my migration out of the existing environment.

Brown: There is also a sense that came out of that question. The architecture, once it was done, it was done. And then things changed afterwards. So there was no concept that the architecture should be referenced any time you make a change to the instantiation and to update the architecture accordingly.

Zachman: Allen, that’s really an important point. I’ll tell you where I learned how to manage change. I set that up at Boeing. It took me about 10 years to figure this out. How do you manage the changes in the Boeing 747’s? Very carefully, right?
If you really want to change the Enterprise Architecture before you change the enterprise, if you have a general management responsibility for Enterprise Architecture, this a piece of cake.

Brown: Yes.

Zachman: You don’t walk around a Boeing 747 with a hammer and screwdriver making the changes. Forget that. You have an engineering administration. Who are they? They're managing the drawings, the functional space, the building materials, and so on. You pull out the one you want to change. You change the artifact and then you figure out the impact on the adjoining artifacts.

When you figure out the impact, you make changes to all those other artifacts. You don’t throw away the old version, but you keep the old version. It is regulated in the airplanes. You have to trace the artifact back to the last time we had an airplane that flew successfully.

But once you change the artifact, then you go to the shop for a particular change kit. You take the change kit out of the Boeing 747 and you put the change in the Boeing 747. If you manage change in that fashion, you will minimize the time, disruption, and cost of change in the Boeing 747.

Every artifact precisely represents the Boeing 747 as it existed in this moment. And one thing that people would not tend to know, every Boeing 747 is unique. They are all different and they have a set of these artifacts for every Boeing 747. You can trace it back to the origin, whatever they changed and flew since the last time. The Boeing 747 is not complicated enough. These artifact were on paper.

The first electronically designed airplane was the 777. Now you understand the reason I'm telling you this. If you really want to change the Enterprise Architecture before you change the enterprise, if you have a general management responsibility for Enterprise Architecture, this a piece of cake.

Making changes

So by the way Ms. Vice President of Marketing, before you change your allocation responsibility, your organization responsibility, come-up and see me. We’ll change the repository first, and then you can change the allocation responsibility.

Oh, by the way Ms. Programmer, before you change a line of code in that program, you come up and see what changes are in the repository first. Then, you can change the line of code on the program. Before you change anything, you change the architecture first, and then you change it.

And, by the way, it’s dynamic because you can continuously solve problems, you can then populate, put more primitive components into the architecture. That's why this becomes really important. It becomes an operating responsibility for the general management.

If they really understood what they are, that’s a knowledge base, everything they can possibly know about the enterprise. They can change anything or have a great deal of creativity to do lots of things they haven’t even dreamed about doing before. That’s really important idea that that becomes a general management who is integrated into the enterprise operation.
The problem we have now is that when you try to make things common or federal, when they are not common, that’s where you get the hate and discontent.

Nunn: Next question. Have you considered what changes might be required to your framework to accommodate an ecosystem of multiple enterprises?

Zachman: That’s what I would call federated architecture. You know some things you want in common with more than one enterprise and some things you want to be provincial, if you will. Some things you want to make federal, some things you want to leave provincial. The problem we have now is that when you try to make things common or federal, when they are not common, that’s where you get the hate and discontent. The framework is really helpful for thinking through what you what to make common or you want to leave a provincial artifact.

That’s the way you need to deal with that and that would be any complex environment. In  most every enterprise these days, there is more than one framework that you might possibly want to populate, but then you have to understand what you want to be the same and what you want to leave. That’s the way we would handle that.

Brown: So if you are an architect, you pull out the drawing for the entire urban landscape. You pull out the drawing for specific buildings. You pull out the drawing for functions within that building, a different framework.

Zachman: Actually this was implemented. I learned it from the Province of Ontario. There was a premier about 20 years ago who was pretty creative. We sorted all of the departments in the Province of Ontario into categories. You can call them clusters.

You had the social services cluster, the land cluster, the finance cluster. Then, he put a super minister in-charge of each cluster, and their role was to integrate -- get rid of all the redundancy and make it as integrated as possible.

That was the process we were using. You have a federation at each cluster, but then you have the federation, a second level up at the province level as well.

Common connector

Nunn: Do you envision a common connector from a given architecture development method, like TOGAF, DoDAF, FEA to the Zachman Framework?

Zachman: When we talk in the workshop, we'll get into that a little bit. If you have the primitive components and say which set you want to change, if you want the TOGAF components, click, there is TOGAF. Oh. You want DoDAF, oh no problem; click, there is the DoDAF. Oh you want balanced scorecard, no problem; click, there is a balanced scorecard. Oh, you want your profit and loss statement; click, there is a profit and loss.

What you're doing is you are creating a composite dynamically. Whatever the composite is, you want to take a look at. I was really impressed with Don's presentation yesterday. I was at Raytheon last week and there was a presentation I had seen recently about the hardware -- the price performance improvements and the capabilities in hardware. What it basically was saying is that you'll be able to put big data, all the structured data, all this data on a chip. And that chip will go into your processor. The guys at Raytheon said that it's not when you can do it; you can do it now.
So if you have big data on a chip, you get dynamically identified threats and opportunities. What do you think that's going to do to the decision cycle? It's going to make that decision cycle very short.

Because you have big data on a chip, and you can analyze that big data and find a threat, an opportunity, or something external or even internal, the immediate question is going to become what are you going to change in the enterprise. Are you going to increase or decrease the inventory, increase or decrease the process transformation, going to increase or decrease the storage capacity of the node? What are you going to do to your enterprise?

So if you have big data on a chip, you get dynamically identified threats and opportunities. What do you think that's going to do to the decision cycle? It's going to make that decision cycle very short.

So you have to make up your mind. What you are going to do real quickly. So you like several alternatives? Okay, chief, here are the three or four alternatives. Which one do you want to pick?

It's going to shorten the decision cycle dramatically. Dawn was frightening me yesterday. We're not talking about the sweet by and by. She was talking about stuff that is here now, and that's what the guys at Raytheon were telling me. This is here now.

I talked about big data before, and the fundamental question is, once you figure out something external or even internal, what are you going to do to your enterprise? Where is your Enterprise Architecture? What are you going to change?

The next question is, who is working on your architecture? Somebody might be working on this. I'll tell you about that. I don't think too many people have an idea of the sense of urgency we have here. You're not going to do this today. You have to start working on this, and you’ve got to eat this elephant -- bite, bite, bite. It's not going to happen overnight.

Nunn: How can the Zachman Framework be applied to create an architecture description that can be implemented later on, without falling into a complex design that could be difficult to construct and maintain. Following on from that, how do you avoid those descriptions becoming out of date, since organizations change so quickly?

Manufacturing perspective

Zachman: Whoever posed that question is thinking about this from a manufacturing perspective. They're thinking about it as a composite construct. And if you separate the independent variables and populate the primitive components, don't bind anything together until you click the mouse, and you can change any primitive component anytime you want to, okay.

You're not locked into anything. You can change with minimum time. You can change one variable without changing everything. The question is couched in terms of our classic understanding of Enterprise Architecture, the big monolithic, static, it takes long time and costs lot of money. That's a wrong idea. Basically, you build this iteratively and incrementally, primitive, by primitive, by primitive, and then you can create the composites on-the-fly.

Basically, that would be the approach that I would take. You're not creating fixed implementation, extreme complex implementations. That’s probably not the way that you want to do it.
Basically, you build this iteratively and incrementally, primitive, by primitive, by primitive, and then you can create the composites on-the-fly.

Nunn: Short question. Is business architecture part of Enterprise Architecture or something different?

Zachman: Well, out of the context in my framework, I started to suggest that some people say, well, the business processes are as architecture, it would be column 2, row 2. Some people say, well, no, it’s actually column 6, row 1. Some people will say, well it’s actually the composite of column 1, and column 2 at row 2.

The Chief Operating Officer of a utility I work with, this is years ago now, he basically said, "My DP people know this. My DP people want to talk to me about data, process, and network, and I don't care about data, process and network. I want to talk about allocation, responsibility, business cycles, and strategy. So I don’t want to talk about column 1, 2, and 3. I only care about column 4, 5, and 6."

I couldn't believe this guy said that. I knew the guy. You don’t care about the inventories, the process transformations, and the distribution structure? Are you kidding me -- in a utility? Come on. It is just unfathomable.

At some point of time you probably are going to wish, you’d have more and more and more of these primitives. Build them up iteratively and incrementally for some long period of time. There's not one way to do it, there are n different ways to do it, and some work better than others. The fact that you’ve got a tested methodology, you had to use that, why not.

Brown: WI think it depends on which one of the 176 different definitions of business architecture you use.

Zachman: Yes.

Business architects

Brown: In my definition, the people I spoke to in Australia and New Zealand, had the title of business architects and they quite clearly felt that they were part of Enterprise Architect. But the other side of things is that some of the greatest business architects would be Bill Gates, Michael Dell, Steve Jobs, Jack Roush.

Zachman: I was pontificating around the architectural idea and I lost sight of the business architecture question. The question turns out to be, which primitives you want to consider. If you want to say you want to open up new markets, then we’ve got to figure out whatever choice you are going to need, what process, what location, and that would create the composite that you need for addressing whatever the issues you have --

Brown: And it is too tough, Enterprise Architecture.

Zachman: Yeah, right exactly.

Nunn: TOGAF’s primary, short- and long-term guidance is achieved through principles or achieved with principles. How would you propose to reconcile that with the idea of extending TOGAF’s framework and method with the Zachman Framework?
The principles don’t go away. One thing is that when you define principles, they have a lifetime.

Zachman: The principles don’t go away. One thing is that when you define principles, they have a lifetime. Somebody was making that case yesterday at the presentation. He defined the principle, and I think there is a set of architectural principles. One thing is, if you want flexibility to separate the independent variables, that’s a good principle, you have a single point of control, you have single sort of truth. Those tend to be principles that people would establish, and then take whatever the principles are that anybody has in mind, and you could figure out how that manifests itself in the architectural structure, in the framework structure and, in fact, the ontological construct, and you manage that. I mean the governance system has to enforce the principle.

There is another principle. I would not change the enterprise without changing the artifact first. I would change the architecture before I change the enterprise. Here is another one. I wouldn't allow any programmer to just spuriously create any line or code or data element or any kind of technology aggregation. You reuse what's in the primitive. And if it’s not, if you need something that’s not in that primitive, then fix the primitive. Don’t create something spurious just to get the code to run. That’s another principle.

There was probably an array thing, and off the top of my head there is a couple that I would be interested in, but those tend to be deriving from these ideas about architecture. I learned it all of this. I didn't invent any of this. I learned it by looking to other people and, I saw the patterns. All I do is I put enterprise names and the same architectural constructs in any other object and then I learned about migration. I learned about federation. I learned about all these other things by managing change and by looking at what other people did.

This has been a special BriefingsDirect presentation and panel discussion from The Open Group San Diego 2015. Download a copy of the transcript. This follows an earlier discussion on cybersecurity standards for safer supply chains. And another earlier discussion from the event focused on synergies among major Enterprise Architecture frameworks.

Transcript of a live panel discussion at February's The Open Group San Diego 2015. Copyright The Open Group and Interarbor Solutions, LLC, 2005-2015. All rights reserved.

You may also be interested in: