Neal Ford  |
  • Author, Thoughtworker, & Meme Wrangler

SOA and the Tar Pit of Irrelevancy

During my work at ThoughtWorks, I sometimes end up on enterprise integration gigs, mostly because I didn’t dodge fast enough. I tend to dislike those gigs because frequently they devolve into politics and other foolishness and away from actually solving problems. My frustrations with these projects led to a series of blog posts, aggregated here under the series title, containing generalized observations of disfunction.

Tactics vs. Strategy

No company starts out as an Enterprise, they all start as a company, with just a few employees. As such, their IT needs are small, handled by a small group of developers who can chat with each other over lunch about what’s going on in the IT “department”. As the business needs software, they have some process to get the requirements to the developers, and they write code. Thus, the accounting department comes to the developers and says “Here are our requirements for our new accounting application”, and the developers build some version of it. Inside that application are some small parts of something that the entire company cares about, for example, some of the aspects of customers. Meanwhile, the marketing department comes to the developers and says “We need an application, and here are the requirements”, and the developers build it. Of course, this application also encapsulates some aspects of a Customer (not the same as accounting, but possibly with some overlap). It is rare indeed that anyone looks around and tries to come up with a comprehensive strategy for application interoperability: you don’t have time for that – you’re a small company, and if you don’t get your software, you’ll go out of business. This goes on for a while.

Then, one day, you wake up, and you’re an Enterprise. The CIO looks around with dismay at all the actual entities the corporation cares about because they are scattered in bits and pieces in all these different siloed and packaged applications. And the database stuff is a mess, with shared tables and views and ETL jobs running on cron jobs. And the CIO throws up in his mouth, just a little. The CIO looks at the landscape, and realizes that the technical debt incurred over the last few years can only get worse from here, so he calls out “Help.” Big software vendors are highly attuned to people in big companies (sorry, Enterprises) who can write checks saying “Help”. They ride in with a solution wrapped around the SOA moniker. More about our friends the Enterprise vendors in a future part of the series.

What I’m concerned about in this post is the overall landscape, which is another way of asking “How did you get where you are now?” You got here because of 2 reasons: first, you took the path of least resistance when you were a company (before you became an Enterprise) because, if you had taken the time to build a comprehensive strategy, you’d have never survived as a company. Second, and more important, what is strategic to the business is always tactical to IT. Business people can go into a conference room and change the entire direction of the company in 2 hours. Or, the business may decide to merge with another company that has truly dysfunctional IT, but other business concerns override that. IT can never move as fast as the business, with means that IT always has to respond tactically to the decisions and initiatives brought forth from the business. No matter how much effort you put into a comprehensive, beautiful, well-designed enterprise architecture, it’ll be blown out of the water the first time the business makes a decision unlike the ones that came before. The myth of SOA sold by the big vendors is that you can create this massively strategic cathedral of enterprise architecture, but it always falls down in the real world because the COO (and CEO) can override the CIO (and his sidekick, the CTO). If you can convince your organization to allow IT to set the strategy for what capabilities the business will have long-term, you should. However, your more agile competitors are going to eat your lunch while you build your cathedral.

Any enterprise strategy you implement must realize that you will always be in tactical mode in IT because the business strategy doesn’t require physical labor. Any enterprise architecture you develop must allow the business to evolve according to it’s wants (and needs). This is what my colleague Jim Webber calls “Guerrilla SOA” and what I call “Evolutionary SOA”. More about the details of evolutionary SOA in upcoming installments.

Standards-based vs. Standardized

Back in the very early days of Java web development, interacting with the tools sucked. Every servlet engine vendor had their own deployment scenario, and they varied wildly between vendors. I worked on several projects where we moved a web application from one dysfunctional servlet engine to another (this was in the early days, and they all sucked in one way or another). Then, the J2EE standard came along, including the WAR file format. Suddenly, you had a fighting chance (well, after a few iterations of the standard) of deploying to multiple application servers (notice how “servlet engine” became “application server”). And the vendors hated that.

They hated it because the J2EE standard turned their application servers into commodities. And the price of commodity software quickly approaches zero. That’s why the application server vendors immediately started building more elaborate stuff on top of J2EE: portlet engines, content management, elaborate load balancers, etc. They knew that something like JBoss would come along and support the J2EE standard, making it impossible to charge $17,000 per CPU to deploy their application server (this number sticks out because one of my clients in the early 2000’s paid WebLogic exactly that…for 8 CPUs). The J2EE standard essentially ran the server vendors out of the business of supplying just an application server.

Contrast that with database vendors. They are still able to charge big bucks for their servers, and keep companies (sorry, Enterprises) on the upgrade treadmill. An ANSI standard for SQL exists but (and this is the key part) it is so weak that it’s useless. When last I checked, the ANSI standard for SQL didn’t even include indexes, and no one would reasonably deploy a database server without indexing. The database vendors dodged the commoditization bullet by ensuring that the SQL standard would remain toothless in perpetuity.

Now, let’s apply this to Enterprise Service Buses. One of the bits of marketing mantra touted by all the vendors of these Rube Goldberg machines is “standards”. And they’re selling these things into Java shops accustomed to the J2EE standard, so companies equate “standards” to “works the same across all players”. All these shops are thinking of standards under the J2EE light, not the database server light. But here’s the rub:

ESBs are standards-based but not standardized.

This distinction is important. All the extant ESBs use standards in every nook and cranny, but it’s all held together by highly proprietary glue. The glue shows up in the administration tools, the way their BPEL designer works (along with their custom BPEL meta-data), how you configure the thing, how you handle routing, etc. The list goes on and on. These vendors will never allow the kind of standardization imposed by Sun in J2EE. The last thing the vendors want is to see their (crazy money making) babies turned into commodity software again. They’ll make noise about creating a true standard, but it won’t happen. They want to be more like the database vendors, not the application server vendors.

Even the open source offerings in the ESB space suffer a little from this because they are giving the bits away and selling consulting and training. This is a good business model (look what it’s done for JBoss), but they have the same motivation to keep you locked into their version of proprietary glue.

Having proprietary glue is not necessarily a bad thing. It’s one of the factors you have to consider anytime you are thinking about turning part of your infrastructure over to a externally developed piece of software. Obviously, no one is going to build their own database server – it makes good sense to buy an existing one, and fight the nasty battle if and when it comes time to move to another one. BUT, you need to understand the distinction between standards-based and standardized so that you don’t buy yourself into a real long term nightmare.

Tools & Anti-behavior

While rank and file developers go to conferences to soak in deep technical content, their peripherally technical managers (the ones who wrote some rockin’ good Cobol code back in the day, but now they make decisions about modern enterprise architecture) go to different conferences in Palm Springs. At those conferences, they have a 2-hour morning session, run by a big tool vendor, then play golf for the balance of the afternoon. And what the vendors show them is poison.

Mostly what they see these days are tools that support SOA and ESBs. And in particular, the favorite demo-ware application is their BPEL (Business Process Execution Language) designer. This designer allows you to wire together services by drawing lines between boxes. The lines can include transformations and other sexiness. And it demos great. “Look, just draw a couple of lines here and here, click on the Run button and voila! Instant SOA”.

Then, the manager brings it back home and notifies the developers that this is the new tool of choice. When developers start using it, they realize the awful truth: they’ve been sold a hairball generator. Tools like this work great on really small scales, when it’s easy to see all the connections between things. But, as things get complicated, they start suffering from the hairball effect: all the lines start running together, and you can’t create a diagram that makes sense to anyone anymore. Perhaps maybe you can fight through this, by creating workflows in chunks, and zooming in and out.

Then reality arrives. Because you create workflows using these tools, you are coding, in the worst possible language (a graphical representation). Thus, you are defining behavior, just like you do when you write source code. But the behavior you define lacks all the benefits you get from writing it in code.

Tools like this fall into the category one of my colleagues identified as doodleware tools. They let you create pretty pictures but collapse under scale. And they don’t support all the common facilities offered by good old fashioned code. Is it really worth giving up robust reuse, refactoring, testing, programmability, and versioning/diffing just to see the pretty picture? Ironically, it’s pretty easy to generate a similar picture from code, using tools like GraphViz.

I am a strong believer in the mantra

Keep behavior in code.

We have great tools for code (including ways to generate doodles) – why would you want to abandon what works for something new and shiny? Except, of course, that code won’t take you out for a golf outing in Scotland if you choose it.

Rubick’s Cubicle

Developers love to solve puzzles. One project manager with which I used to work kept a jar full of little twisted nail puzzles, where the goal is to pull the entwined nails apart, on his desk.

Any time he was having a conversation that he didn’t want developers to listen in on, he’d grab one of the puzzles and toss it to them. Inevitably, the developer would grab the toy and immediately become totally absorbed in solving the puzzle. After about 10 minutes, the puzzle would yield up it’s secret, and the developer would look up and ask “Did anything important just happen?”

Developers tend to be problem solvers – it’s one of the appealing things about fiddling with computers. But what happens when you take a natural problem solver and present them with dull work, with no interesting challenges? What happens frequently is what I’ve deemed the Rubick’s Cubicle anti-pattern.

If the presented problem isn’t complex enough, developers will figure out ways to make it complicated and therefore challenging.

Writing the same dull CRUD application over and over is boring. But what if you could figure out a way to get all the simple CRUD applications to talk to one another? That’s a nice and juicy puzzle. This perhaps explains the complexity fetish I see in so many “Enterprise” architectures and applications. Some of it is accidental complexity, accrued from years of piecing together parts that were never meant to work with one another. But I don’t think accidental complexity covers the entirety of why things are so convoluted.

I remember back in the mid-90s, I was the CTO of a small training and consulting company. We were absolutely delighted when we first saw EJB: here was a technology no one could understand without extensive training. The same thing happened with all the variations of COM, DCOM, and CORBA. Those were flush times for training companies because we knew that developers would be drawn like moths to a flame, frequently with the same result.

Building the simplest thing that can work is sometimes duller than crafting some devilishly complex Rube Goldberg machine, but keeping it simple is a worthy challenge in it’s own right. If you find yourself in Rubick’s Cubicle, stop and ask yourself: “is there a simpler way to do this? Perhaps dismantling something that no longer serves it’s purpose? What is the simplest thing that could possibly work?”

The Triumph of Hope over Reason

A very funny site shows that this Internet thing might not be a fad. The Chuck Norris Facts web site has lots of great hyperbolic claims about Chuck Norris, American actor and legendary bad-ass. Some of the “facts”:

Some of this may be just a tad exaggerated. I’m pretty sure that when Chuck Norris does pushups, he is not in fact pushing the earth down instead of pushing himself up. The site is here, if you want to go read more of them. I’ll wait.

OK, now that you understand more about Chuck Norris, here’s another site of over-the-top exaggeration about an over-hyped subject: SOA Facts, modeled after Chuck Norris Facts:

I used a bunch of these in one of my SOA talks as bumper slides between the various topics, which provided a nice fun icebreaker. But I reserved two of them for the last part of the talk because I think they aren’t exaggerations at all, merely deep truths:

SOA has gotten so complex, with so many moving parts, that getting it right is extraordinarily difficult. Once you’ve lived through one of these projects (especially if you’ve fallen into the other tarpits I discuss in the previous installments), you understand the first quote at a deep level. That you would try it again truly is the triumph of hope over reason.

Guerrilla SOA (SOA & The Tarpit of Irrelevancy)

This is the sixth in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it’s being sold by vendors. For those who are behind, you can read the previous installments here:

In the previous sections, I’ve been basically elucidating all the reasons why I think that most SOA projects are a quagmire: mis-understanding the goals of why you are involved in this approach, the way most vendors are poisoning the water by selling ridiculously complex tools and services, why it’s so seductive for developers who fetish complexity to get knee deep in these projects, and hype. If you read all these posts back to back, you’ll surely have the impression that I think all hope is lost for enterprise integration.

But it’s not.

Just like any software project, it is possible to do SOA right. My colleague Jim Webber has done lots of outstanding work in this area, under a rubric called Guerilla SOA. The basic ideas are:

It’s best said from Jim’s own mouth. Jim’s British, so when he curses in technical presentations, people just think it’s quaint (whereas when I do it, it’s crass).

This is the way we generally approach SOA pursuits: like other pursuits. SOA doesn’t have to be a huge scary thing. It’s just software, and we aren’t going to throw our playbook out the window just because it sounds scary.

The term SOA has been so co-opted by vendors trying to sell stuff that I think it will die off as a term. We’ll still be doing soa (note the lower case letters), but we’ll have develop another three-letter acronym because the old one has slipped into the tarpit of irrelevancy.



   Follow Neal on Twitter music image opera image reading image

Neal Ford  |
  • Author, Thoughtworker, & Meme Wrangler