Delivering SOA Across Projects: Dealing with Shared Dependencies & Risks

I posted not long ago about professionalising the planning and delivery of SOA initiatives, making realising them more than just a gamble based on theory and good intentions.

One of the biggest pieces of this puzzle often seems to be dealing with the project delivery issues. The industrialisation of SOA if you like. The key issue is that many are not ready to deal with the shared and cross-project dependencies, the additional technology complexities, and the cross-project risk mitigation demands that enterprise-level SOA creates. Such issues create havoc with the conventional project delivery mechanisms and mindsets of the moment, but without project delivery being on-board, an enterprise-level SOA is extremely difficult to scale up to the level required for it to start paying off.

IT Project Delivery
Modern IT delivery (pre-SOA) practices that combine both robust project management and proper diligent delivery management have improved significantly over the incarnations of the 90s and before. They're particularly stronger in areas like governance, visibility and indeed success rates, learning lessons from mistakes of the past. However they've been shaped by a diet of scope definition, issue identification, risk mitigation, and industrialisation of production in traditional systems-centric world. Where each system and project has its own direct business case. Of course its not perfect, for example the delivery models of many are not as mature as they should be in how they effect global sourcing and production in order to take advantage of off-shore production whilst maintaining quality and efficiency. However, ignoring globalisation, many of the basics of today's delivery get turned on their heads by cross-project SOA initiatives.

Real cross-project SOA introduces dependencies between projects who are sharing services, coexisting in environments, or reusing central assets. It also generally means that issue resolution and risk mitigation within a project is no longer within that project's own hands. And in seeking to optimise the technology across projects, it mostly means that each project is sub-optimised, increasing the 'number of moving parts' of technology (and therefore generally the cost) beyond what is needed to realise the project's business case. A situation which often isn't exactly welcomed by the leadership of the projects concerned. And of course there are many more such changes, but you get the idea.

Some organisations have laudable ambitions for the use of architecture in reversing years of IT duplication, proliferation, conflicts and silos that business-unit-centric or project-centric technology deployment has caused, but they embark on it without a proven or complete plan as to how to deal with the road-blocks, or handle the stakeholders who's incentivisation or behaviours lie elsewhere.

1. Project Effort and Cost
For example, you have the additional factors in estimating the effort required in a project and allocating its cost. Even where a project has agreed to take on the additional technologies and development costs of the new middleware (e.g. ESBs) needed to effect SOA it will need more extra budget provision just for the dialogue needed with the central team doing the architecture and technology-usage governance (be they an EA team or called something else), and participate in the joint service designs and testing. Let alone the change control needed if a project has to comply with a late emerging technology standard, or extra regression testing required because of a change request in a parallel project which shares services. And this is on top of the actual effort taken to build the services that they need to build, and effectively feels to the project concerned like time that is not value adding and is not being spent getting on with the solution.

As often is the case, a commercial contract between two parties focuses attention on things that would otherwise be hidden, and if a 3rd party implementer is involved in delivering a particular project, these additional provisions will need to be surfaced up front, and potentially put into the contracting. That's not a nice conversation to have, but infinitely preferable to a much later conversation justifying changes in budgets when it's too late to do anything about it.

2. Design, Configuration and Release Management
There are also the additional complexities of service design, configuration, and release management in cross-project delivery. Firstly, the environments. On one hand there are more technologies that require environments (to develop->test->cut-over), and then all the technologies (new or old) require demarcation between the (develop->test) environments within their own project's control, versus the (testing->cut-over) environments in common shared environments.

Then secondly there is the new design governance process required to collectively agree a shared design that multiple projects work towards, to manage requests for changes to it during development and when its live, to coordinate integration of parallel work-streams, and to sign-off completion of delivery to the pre-agreed design. That last one is particularly troublesome as many 'design committees' that may exist around an SOA initiative may take on the design sign-off responsibility, but are often not resourced or mandated to sign-off delivery to the design. And yet without that ability the projects are at the mercy of factors outside of their control (and timeline) for sign-off, which may be incompatible with the obligations they have to the business, and their business cases.

3. Risk Management
Even with an appropriate configuration management process, and the appropriate bodies with the appropriate mandate to play the roles in it, it still is more than a little problematic to deliver federated cross-project SOA. Particularly with parallel delivery (as opposed to sharing only assets that have already been deployed) which is probably the most ambitious SOA delivery model.

Management of the risk profiles involved causes significant uncertainties for the commercial models each project is using to manage its cost. Fundamentally the budget that any project uses will be based on a certain scope, certain assumptions, and certain known risks for which it will allow contingencies. Whether there is a 3rd party prime contractor involved, 3rd party subcontractors, or only internal heads involved will affect what type commercial model is used, and therefore how the cost and contingency are estimated. But whatever the case, the project will be carrying a certain risk profile that it will look to manage so as to keep its commitment to deliver.

Of course the dependencies of the project on central activities, on existing seemingly vague assets, and on other projects activities, creates additional risks that existing techniques find it difficult to identify, and which are not within their control. This means that projects are potentially exposed in carrying risks that they find it difficult to estimate contingencies for, and that they are unable to mitigate.

Shared Services (...for Delivering Shared Services!)
But these are of course not really new problems. And they're not really peculiar to SOA. They are an aspect of executing on any shared initiative above the level of a project and business unit (BU), which is dependent on the project or BU to realise it, and therefore applies across those projects and BUs. Ultimately this boils down to being an aspect of achieving objectives across reporting lines, in a situation where those objectives include indirect (federated) business cases, in addition to the (projects' own) direct business cases.

The techniques that has most successfully been used to solve similar problems before in previous generation of IT are simple operating model concepts widely used in other areas of business, taught in most b-schools, and written about many times. They're shared services (, although I'll use the old IT term 'centre of excellence' here to save confusion between the shared services in SOA delivery from the the shared services of service oriented architecture itself).

A centre of excellence (CoE) or competency centre for SOA delivery allows the projects to all sub-contract the production of the (SOA) services to that CoE and therefore to exchange the risk that they can't manage themselves for a service-level agreement (SLA) that they can manage to. It also allows the CoE to manage the internal dependencies between the different requests being made of it, formalising and managing th change on contracts with its customers (the projects) so that it can coordinate the cost and SLAs to the risk it carries at any time. It also of course allows the CoE to drive centralised knowledge management, developing its own capabilities and professionalising its own working practices using its own revenue stream, and gaining from the economies of scale on the supply side that no one project could have experienced on its own.

Its great that its becoming acknowledged that an enterprise-level architecture capability (although often not a traditional EA group with a traditional EA framework) is key to being able to drive cross-project technology and architecture strategies and improvements. In such scenarios that EA capability is in some ways acting like a shared service for the design phase (and to a more limited degree, for the preceding demand management phase). But that is only a part of the lifecycle. A shared design is much harder to make it into reality without some level of shared delivery (construction) and shared operations (run and support). Sounds simple when it's said like that.

Technorati Tags: Enterprise IT Project Management IT Delivery Shared Services Center of Excellence Competency Center Enterprise Architecture SOA Services Architecture Service-Oriented Architecture

Survival of the loudest?: 'Social' evolution in Enterprise IT

Last year (after a recommendation from Nigel Green) I finally got round to reading Lila by Pirsig (probably better known for his previous book 'Zen and the Art of Motorcycle Maintenance'). It contains some great thoughts and insights that, whilst written for a totally different purpose and audience, I found very interesting and relevant to enterprise IT. Particularly to enterprise processes, systems, data and architecture.

Bear with me as we go a bit abstract here ... One of the models that he proposes is that the world as we know it is made up of series of 'systems' constructed one on top of another.

-He identifies the inorganic systems (how atoms, molecules, materials etc work) to be the first.
-Then he identifies the biological systems (how cells, organisms, life etc work) to be next, existing on top of the inorganic systems.
-Then he identifies the social systems (how individuals, groups, cities, cultures etc work) to be next, existing on top of the biological systems.
-Finally he identifies intellectual systems (how ideals, concepts, intellectual values etc work) to be next, existing on top of the social systems.

One of the several things he does with the model above is to point out that whilst evolution is well accepted in the biological level, it is rarely seriously considered at the social and intellectual levels. He proposes that the concept of evolution is just as appropriate in describing the change in the patterns seen in the social systems and the intellectual systems of the world, as it is to the biological ones.

He suggests that the patterns in each level have evolved on top of the patterns in the level below. That they have not been designed to serve the systems in the level above. Therefore he proposes that the social systems (e.g. cultures, cities) have evolved on top of biological patterns (e.g. man). They have not been predetermined to underpin the intellectual patterns (e.g. intellectual ideals). He proposes that social patterns may have been retrospectively formalised to support intellectual values (such as ideals of a culture), but actually the intellectual values themselves evolved from the platform provided by social patterns. In turn, he proposes that intellectual systems have evolved on top of social systems, not vice versa. Clearly, although this is not unintuitive, it challenges certain conventional wisdoms that cultures are the result of the intellectual ideals and cities are the creation of man.

You may wonder why I am describing something so metaphysical on a blog about Enterprise IT, well there is method in my madness. The example he gives about how people often consider cities to be the creation of man, that have been planned out and created as some kind of master plan in the heads of the individuals involved is a very interesting one. Pirsig rejects the idea, describing how his model suggests that actually the city at any point has evolved on top of the biological patterns involved, governed by the value systems involved, and the ways they have interacted. He rejects the idea of grand predetermined design of a complex social system such as a city (although these are my words not his).

Given how often Enterprise Architecture and IT planning are compared to city planning (inc. by me, Todd, & Villas to mention a few recent blog postings alone) it should be interesting to anyone involved in IT strategy or architecture. The equivalent of the concept in the Enterprise IT world could be that an Enterprise IT estate is not the creation of the technologies or the people involved, it does not exist as a result of any master plan, but is a complex system that has evolved as a result of the value systems that have interacted.

The obvious activity for a rational IT chap is to consider what would be the actualisation of Pirsig's models in the enterprise IT world, using an Enterprise Architect, IT planning or Portfolio Management viewpoint. Maybe the inorganic level could be the technology infrastructure, both software and hardware. Maybe the biological level could this be related to systems and data. Maybe the social level could this be the work-practices, policies, and networks/communities. Maybe the intellectual level could be the business objectives, models, and processes.

However rather than get too caught up in debating the potentially academic details of the mappings, I'm more initially interested in some of the dynamics of this model. It's not like the applicability of top-down 'intelligent design' in Enterprise IT has never been questioned before after all, but rather than just practitioner's scepticism, this gives alternative hypotheses to consider.

Of course there's enough material in such a consideration for a book in itself, but one of the dynamics that jumped out at me right away that I thought I'd mention here was that concept of social systems evolving on top of biological ones. If there is an enterprise IT equivalent of the biological level, then I suspect it's comparatively well understood and catered for compared to the social level. I have often been of the opinion that social dynamics and value systems are very badly understood in Enterprise IT, much to its detriment. Management science is better (although far from perfect) at considering the social systems that make up the organisations we work in or work with, but such matters often seem to be almost a complete blind-spot to IT.

The biological evolution of IT systems we can appreciate (although many organisations struggle with managing it), and the intellectual evolution of business models and objectives we can also appreciate (although again many struggle), but the conventional IT concept of 'the users' and the tools of functional requirements, flow-charts, use cases etc always feel like vaguely prehistoric blunt instruments for considering the social systems of an organisation. Extrapolating the concept that social systems 'evolve' on top of biological systems, based on the interaction of the value systems (and which proliferates most effectively) seems to suggest to us that we should have a far better way of describing the different communities, networks, ownerships, motivations, behaviours etc in an organisation around its use of, and opportunity for IT. One might even over-simplistically describe it as the politics around the use of IT.

Of course the re-popularisation of the internet ideals as part of the trend often referred to as Web 2.0 has brought a lot of focus to the ideas of community in consumer-focused IT, including blogs, tagging, wikis, inclusive economics etc, and many varieties of social software. The Enterprise 2.0 initiative of Andrew McAfee and others has brought an interesting perspective to how these technologies could and can be applied into enterprise IT scenarios. But even though these new technologies will very likely be important going forward, the dynamics of the social systems around IT of course apply to all system-types and all technology generations, not just social software. It can't be the preserve of a new generation of enterprise technology alone.

Sidebar: Pirsig also uses some excellent metaphors about academics that are scarily applicable to the IT industry. For example he talks about 'restaurants with 300,000 page menus and no food'. That reminds me of far too much of IT than is healthy. Another that stuck in my mind was his 'highways full of drivers too busy telling each other how to drive to actually get anywhere'. If there is an activity where that applies more than in IT then I'd love to know which? Other than politics itself of course.

BTW: I'd be interested to see what some real Pirsig experts think of this misapplication of MoQ. But please be gentle, I'm conscious that it's not exactly a pure representation...

Technorati Tags: Enterprise IT Web 2.0 Web_2.0 web-20 Enterprise 2.0 Enterprise Architecture Systems Architecture Information Systems Pirsig Robert Pirsig Lila MOQ