Ashes to ashes.

This is for those who have expressed concerns to me about Project Phoenix.

I joined Emirates Group on the day Project Phoenix was announced, 1 July 2006.  Patrick Naef had been in post for five months and it was clear to him (indeed everyone) that some serious work had to be done to improve the service being delivered to the business.  The number of operational outages was mind blowing and the project delivery record appalling.  I do not know when Patrick had engaged the external consultants (who were heavily involved on the day of the announcement) but he clearly had decided at an early stage to take drastic action, deciding to dismantle the organisation and starting again.

It did not take long in my new post (VP Technology Services) to appreciate the scale of the problems, nor to identify the main areas to work on.  Most of the infrastructure was not fit for purpose (a lot of it was so old it no longer enjoyed support from its supplier), effective standard IT support processes (such as change, risk and incident management) did not exist, management competence was scarce and the IT departmental culture was swamped with cronyism and blame.  Quite a list!  But there were some positives, including an incredibly dedicated and hard working team, all of whom agreed (and desired) that things had to change.

Virtually all the IT infrastructure, including the entire Data Centre, needed replacing and this would take time.  But there were some serious issues which required immediate attention.  As an example, I discovered that the Data Centre had no resilience in the air conditioning system.  The failure of just one small unit would have put the entire Data Centre out of action.  The reason given to me for this state of affairs was “no budget”!  Also, there was an auto start generator in place to provide backup power in the event of a main supply failure.  The generator was perfectly capable of supplying all the IT equipment, but it was not connected to the full air conditioning system!  So, all it gave us was time to power everything down in a controlled manner, leaving our passengers waiting until the power came back.  It is a credit to the Data Centre staff and also our colleagues in the Facilities department, that the necessary remedial work was completed before the peak (in terms of both passenger numbers and temperature) of the summer of 2007.  We did not have to spend a huge amount of money (if my memory serves me well, the air conditioning units were around  AED10K each), but carrying out the work (particularly power) without impacting a 24/365 operation was a major challenge for the teams involved.  The work was completed without any unplanned service disruptions.  There is absolutely no doubt that, without this work, the summer of 2007 would have seen outages unheard of in the industry, with very serious consequences for Emirates Airline.

The good news was that Patrick had already gained approval for what was called the Resilience Project.  Although the benefits had been oversold to the business (“seamless changeovers during server outages” which was not actually achievable on a lot of systems), it was good that procurement of new hardware was underway.  I can only assume that such exaggerations of benefits were necessary in a political environment which I had never encountered before.  The company clearly had money, but had chosen to spend virtually none of it on its internal IT systems.  I believe that the Resilience Project had been scoped before Patrick had arrived but he had given it the necessary impetus to get it started.  But there was a long list of other work that was required, such as on the email system which was out of support and about to collapse.

High severity incidents, i.e. those that had a direct operational impact on the business, were running at over 300 per quarter, more than 3 per day.  I found this difficult to believe.  My previous experience had been in environments where the target was always zero, with just a single such incident in a quarter generating huge complaints from our customers.  But, at Emirates, we often had two, sometimes three, such outages at the same time!  The problem was not solely due to overloaded and ‘not fit for purpose’ infrastructure, there was no effective change management process.  The resolution for many outages was often stated as “resolved by itself” which was IT speak for ‘someone carried out an unauthorised change, it went wrong, so it was backed out quickly’.  Activities were not logged so no-one, other than the individual who had carried it out, would ever know about the change.  Not all changes were done in secret, others were planned and notified, but there was rarely any effective risk management applied.  The answer to my standard question of “what can go wrong” was normally “nothing, we’ve done it before”.

A huge factor was the general level of management competence.  There were exceptions, but some managers were struggling.  They were very dedicated people with sound technical skills but I am afraid that, across the globe, IT people do not always make good managers.  Understandably, a lack of management responsibilities will always dilute a job evaluation in any organisation but Emirates HR had apparently insisted that the ceiling for any ‘non-management’ IT operational job was G8.  This was absurd.  A company the size of Emirates which provided its IT support internally should be able to accommodate technical subject matter experts as far as G10, as long as the individual’s depth and width of knowledge can justify the grade.  Technical staff should certainly have been able to aspire to a G9 technical role without getting bogged down with management chores, something that many did not want.  But the HR rule meant that the only way a good technical person could be adequately rewarded, was by making them a manager.  The result was inevitable.

But the culture was probably going to be the biggest challenge.  The name of the game was to please the boss and drop your colleagues into as much trouble as possible.  And this applied at all levels in the organisation, it was a standard survival technique.  Subservience dominated – most managers in my team would not get close to disagreeing with me, so management debate was initially almost non-existent.  In a meeting with my managers, at the end of my first week, I asked them what they thought of me.  They looked at the floor.  Eventually, one of them said that he thought that “I was a reasonable sort of chap”.  That was the first time anyone had told me that!

So there was a lot to do, but it was achievable and I was more than happy to get on with it.  One immediate focus had to be on the physical infrastructure replacements, so funds had to be secured.  No elaborate business cases were needed, everything was about to go pear shaped so the company had no choice.  For some reason Emirates had made the decision to run the whole place on a shoestring and now the consequences were staring us in the face.  The change/risk/incident elements were easy, they could be (and were) implemented immediately with a total hands on approach initially, and then slowly delegating to those who showed the ability and inclination to take it on.  You cannot effectively (and therefore should not) change management teams overnight.  Working with everyone allows time to understand strengths, career aspirations, learning needs, etc. and, if it proves necessary to move an individual into a non-management role in due course, they themselves will appreciate it.  By then, the G9 Technical role issue would have been resolved – HR had to change their minds, they were simply wrong.  As for the culture, that was going to take a bit longer!  But everyone contributes to the culture, so a push in the right direction with the right level of resolve, often works wonders.

So, had Phoenix not been thought of, personally I would not have engaged external consultants to effect the required changes.  I would have considered using outside help with accelerating some hardware replacements (indeed we did engage an external organisation when the mail system finally collapsed) and I am sure at some stage help with the culture may have been appropriate.  But I believe in continued management ownership of change;  not only directing it, but being part of it and then living with the failures as well as the successes with everyone else.  Only then can you genuinely claim to be part of the team.

But Patrick Naef had a wider portfolio than I did, so he probably would have seen more problems than I had, and possibly seen fewer potential fixes.  And, until I had arrived, he had no-one to lead the assault on the appalling operational service (my post had been vacant).  Had I joined earlier, I am sure I would not have been championing his cause of bringing in a team of consultants, I would have encouraged him to take it on ourselves.  But I would not have ‘died in a ditch’ on the issue.  I am not the world’s biggest advocate of consultants and am aware that affects my approach to them at times.  So, had Patrick felt that this was the only solution, I would have given it my full support.  As I had arrived on ‘announcement day’, my opinions were not only irrelevant but far from crystallised at that time, so I naturally gave the Phoenix Project my total support without any debate.  Patrick Naef often accused me of “not supporting Phoenix because it wasn’t your idea”, but he had no grounds to do that.  I do not know if this was a result of one of his misunderstandings, or if it was one of his usual ‘pitch rolling’ antics so that, if his Phoenix Project failed to deliver, he had an excuse of “Tom undermined it” lined up.  I only ever made one serious point about Phoenix and this was right at the start and I never mentioned it again.  The entire IT department was effectively dissolved and a new organisation would evolve.  All managers would have to apply for whatever jobs emerged.  I pointed out that we ought to put the managers ‘at risk’ because it was possible that some could be without jobs at the end of the process.  I was patronisingly brushed off along the lines of ‘you don’t need to introduce UK HR processes here’ but it had nothing to do with processes and legal definitions, just common sense and reality.  If someone ends up without a job, they are redundant.  It does not matter if you use a capital R or a small r, the impact on the individual and his/her family is the same and it is a major issue which will need managing.  (In fact, it did happen and it was indeed difficult to manage.)  The final response on the subject (not by Patrick, I must add) was, and it would not be the last time I would hear it, “Tom, this is Dubai.  We can do what we like.”  The Phoenix Project was always given my full backing, both during and after it.  Views I had on various matters were voiced with my usual passion within the project as it progressed and then, whatever decision was made, was given my full and continued support.  For Patrick Naef to say otherwise is disingenuous.

It was clear that there would be some serious challenges thrown up by Phoenix.  Not only were managers being told that they would have to apply for jobs, they were being excluded from all day to day aspects of the project.  A team of individuals, around 20, had been selected to assist the external consultants.  This was a major opportunity for the individuals and they threw themselves into the task with great enthusiasm and commitment.  But it left the senior managers feeling very exposed and alienated.  The name of the project did not go down well either, the term Phoenix was supposed to represent something ‘rising out of the ashes’, in other words the so called ‘ashes’ of a department that many hundreds of loyal staff had dedicated themselves to over many years.  If you wanted to find someone responsible for that fire, looking in the direction of senior management would have been a good start.  It had been the loyal staff who had been perpetually fighting the fire.  Ironically, Patrick Naef (the newly recruited ‘fireman’) had arrived with an unlimited supply of kerosene and matches, which he would use liberally over the coming years.  But no-one knew that, as Patrick was innocently waving a fire extinguisher at the time.

Specific approval for the project had to be secured, so a presentation was drafted.  The cost took us all (including Patrick Naef) by surprise.  It would be inappropriate to reveal the amount, but it was far bigger than I had expected.  And no account was taken for internal labour costs, we were just looking at the bill for the consultants.  I was surprised that it was the only option being proposed.  There were no quotations from other suppliers (there had not even been a tender) and no other options (such as using in-house resources, smaller scope, etc.).  I suggested that we ought to have other options but Patrick insisted that there was no plan b.  The proposal was going to Nigel Hopkins and Gary Chapman.  In my experience, you normally need plans down to at least ‘f’ in such circumstances.  But it sailed through and was then ratified by the IT Steering Board (EVP’s and above).  Here I have to take my hat off to Patrick Naef and I do not know how long it took Gary Chapman to spot the handy work.  Patrick Naef had introduced a project approval process where the senior business managers across the Group (Portfolio Review Board – PRB) had to approve all IT expenditure so, theoretically, Phoenix should have been approved by the PRB.  But version one of the PRB process not only had the obvious lower threshold of expenditure requiring its approval, strangely it also had an upper financial limit.  This limit was a very big number, but it was well short of the cost of Phoenix.  So the business leaders in the PRB did not have any sight of the amount or of the project.  Had they done so, I think it would have been a pretty short discussion.  At each PRB meeting there were always a chunk of ‘non-negotiable’ projects (for example those meeting regulatory needs) which had to be rubber stamped and this then led into the more challenging part of the meeting which was, in effect, a bidding war for about 80 projects, with the remaining money enough to pay for just about a quarter of them.  Phoenix would have consumed over a quarter of that money in one go!  Not many months later, someone came up with the bright idea of all projects being presented to the PRB first, with the very big ones requiring further ratification by the IT Steering Board.  Patrick Naef had no issues with that proposal.  The horse, Phoenix, had already bolted!

Interestingly, the entire external sales operation and organisation (Mercator) was left out of scope of Phoenix.  Patrick Naef said that Mercator would be subject to a separate review which would be carried out by Gary Chapman, Nigel Hopkins and him.  It’s a pity, because the Phoenix team was far too competent to have come up with the selected solution, i.e. setting totally unrealistic growth targets, continually blaming everyone who failed to meet those targets, significantly diluting the ability to deliver solutions to our own businesses, totally mismanaging the acquisition, ongoing operation and disposal of a third party company, taking seven years to wake up to the fact that the model was never going to work and then ignominiously abandoning the entire doomed operation.            

Then there was the Phoenix purchase order to consider, so the Procurement department was asked to place an order with the supplier.  Prior to this, they had not been involved.  The process had bypassed all formal company procedures so, understandably, Procurement refused to handle the work.  At this stage I not only began to question Patrick Naef’s judgement but also wondered how long he would survive in this environment.  Patrick Naef had engaged a company (with which he had previous contact) without a tender, without any alternative bids, without a plan b and had placed a contract of enormous value with them.  He brushed off any criticism with his usual aggression and seemed not to care.  I still do not know how he managed to get away with it.

Once the project was underway, there were a few major milestones and decision points, some needing approval from Gary Chapman.  At a presentation to Gary, the process for filling posts at the completion of Phoenix was included.  A thorny issue was one of grades of the new jobs.  It was going to be a bit difficult to get them graded before advertising them without putting an unacceptable delay in the whole process.  A simple solution was to advertise the jobs with ‘grade to be advised’ and we discussed this the evening before the meeting with Gary.  Of course, HR said this was not possible but Patrick’s argument, which I agreed with, was that people should apply for jobs because they wanted the job, not the grade.  That was all very well but meanwhile, back in the real world, especially the real world of the most hierarchical and grade conscious company I had ever come across, this was probably not the right time.  HR did not want to do it this way, I did but not yet and Nigel Hopkins did not want to do it yet either.  (This was the topic mentioned in an earlier update when I had Patrick Naef telling me the following day never to disagree with him in front of Nigel Hopkins again.)  Those of us ‘against’ just felt it was too much too soon for everyone, particularly as views of management were, at best, highly suspicious.  But Patrick won the argument and from then on I fully supported him on the matter.  When the proposal was put to Gary Chapman, he seemed to like it.  He turned to the HR ensemble, which included Sophia Panayiotou, and said “And HR fully support this”  I deliberately leave out a punctuation here as I genuinely do not know whether he was using a full stop or a question mark.  The HR team obediently nodded.  Forgive me for remembering that exchange when Gary Chapman assured me four years later that “HR fully supported” my sacking.

The idea of Project Phoenix was supposed to be ‘by the people, for the people’.  Managers were excluded but I do not think anyone believed for one moment that Patrick Naef was not pulling the important strings behind the scenes.  As part of the approved structure of the project, Patrick Naef had ‘full and final veto’ and had even gained assurances right from the top of the company that no decisions made by Phoenix (in effect, by him) would be overturned.  I have to say that I was impressed that, within six months or so, Patrick Naef seemed to have wrapped the entire Emirates Group around his little finger.

Although not involved in day to day matters of the project, there were many opportunities for me to join the team during the regular progress reports and feedback sessions.  I must say I was very impressed with all aspects of the project.  Out team members were relishing the opportunity and had grasped it with both hands.  The consultants were easily the best I had ever worked with and had created a very relaxed and open working atmosphere which added to the liberation of my colleagues.  For me, this aspect alone suggested that the project would be a success – all we had to do was extend this working philosophy into day to day working and we would have cracked one of the biggest cultural issues we faced.  Maybe the consultants were expensive, but they were good.  They also did not do what almost every company does, that is to use their A team for the sale only to withdraw it once the project gets underway.  We had the full attention of their A team, but that should not undermine the other members of their team, as they had good strength in depth.  I thoroughly enjoyed, and learnt from, the time I spent with the project team.

The project duly came up with the necessary processes and an organisation to support it.  Not all of it suited everyone but it was sound enough for me.  I never get too excited about organisation charts, as long as everyone knows what the customer needs and how they, and everyone else, can meet those needs, it is fine with me.  Filling the vacant management roles was the obvious challenge.  On one hand we were in a good position with a clean sheet of paper and therefore the ability to select people with the right management capabilities, but we still had the fundamental problem – a lack of management skills.  There was a lot of emotion expended on the process, particularly the limit of two applications per person (otherwise the whole procedure would have taken months) and the need to apply for jobs before they had been evaluated (a lot of people assumed this was a trap and the result of the evaluation would depend on who got the job!).  Cronyism was hardly wiped out in this single exercise.  After all, the initial project team itself had been hand chosen so many people felt those individuals had a head start in securing the new jobs.  And I was told much later that the selection process for some management roles was polluted by secret pre-selection meetings to decide on so called ‘self’ nominations and therefore to generate contrived candidate lists.  But, for the first time, there was at least an open process of interviews, with several managers and HR involved in each interview.  Yes, the whole process was more rushed than we would have liked.  Yes, we would have preferred to have given more people more choice.  Yes, we made a few appointments that turned out to be less successful than we would have hoped.   But the biggest problem was a lack of suitable candidates and inevitably we had to relax our criteria to some degree. In my area, I was left with far too many vacancies but I drew the line and refused to make the same mistakes that had been made before.  The brutal truth was that we had to look outside of EG-IT to find the necessary talent.  It was not a comfortable position for a number of people but I stand by my decisions as strongly now as I did then.  I can look everyone in the eye now, as I did then, and explain my reasoning which was based on my clear understanding of what the organisation needed.  With so many managerial vacancies, I had to adjust my organisation a bit to avoid too many empty roles in one management line – when you find that your boss’s boss is, in fact, yourself, you know you have a problem!

Was the project a success?  Yes, it met its objectives and, for every critic of the outcome, there were many supporters.

Was Phoenix value for money?  That is a difficult one as we had no comparisons to make.  It was very expensive but, if it had fixed IT for the Emirates Group forever, then it may well have been a good price to pay.  And, in the Patrick Naef land of wasting inordinate amounts of company money, it was by no means a leader.  The direct losses associated with the Mercator Asia debacle were over double the cost of Phoenix, possibly treble, maybe more.   

Was Phoenix a waste of money?  Eventually, most of it was.  Much of the process work added and, I assume, continues to add value but the ground work on the culture was progressively undone by the fear subsequently generated by Patrick Naef.  That fear grew and grew and has spread like a cancer across EG-IT.  Culture cannot be measured and, even if it were, I am not there now to measure it.  But I am consistently informed that the culture in EG-IT is now significantly worse than it was in 2006.  In 2006 I would have said that was not possible and I cannot stress how upset I am about this situation.  Say what you will about Phoenix, at its completion we had the perfect springboard to turn EG-IT into something that everyone could have been forever proud of.

And, of course, five years after Phoenix was completed, the concept was dumped and Patrick started yet another change project.  Presumably, the company was happy to write the Phoenix ‘investment’ off over the five year period.  But that does beg the question why it had previously expected to squeeze seven, even ten, or more, years’ life out of infrastructure equipment which other companies would write off over three years.

Was Patrick right to do it?  If I had been in his shoes I would have waited, focussing on some immediate tangibles first and then doing something less revolutionary, mainly from in house resources.  But then Patrick Naef always was impatient, too drastic and a big spender.  And he would accuse me of being too slow and too cautious!  The fact is that IT in Emirates was in an absolutely perilous state and, to Patrick’s credit, he actually did something.  From what I could see, no-one else had done anything for a long time.

But I certainly would not have dismantled the organisation and made managers re-apply for what were effectively their jobs.  I do not think this approach has a place in any company that purports to care for its people.  Yes, changes had to be made but it must be remembered that it was the company that put managers into their jobs.  And it’s pretty weak, in my opinion, to hide behind the fact that “it was before my time”.  We had a problem of the company’s making and, as senior managers of that company, now had to deal with it, in a professional manner. ‘Professional’ means taking care of the individuals, as well as the company, and that takes time.  Time to improve if possible, time to find a suitable alternative if not.  But Patrick Naef’s only answer to any individual not performing to his required level is to remove them from their jobs.  And Phoenix did this for him perfectly and he did not even have to get his hands dirty.

And then Patrick Naef did it again, this time it was under the Shaheen banner.  In 2006 we could perhaps have been justified in distancing ourselves from the organisation that we had inherited.  But once the Phoenix organisation was in place, we had no excuses as the organisation was ours.  From when I joined, right up until I was kicked out in September 2010, I take responsibility for all appointments made right across EG-IT.  Clearly I was not directly involved in every selection, but I endorsed the processes and I personally drove them forward and monitored them as actively as I could.  I ensured that as much openness and fairness was applied at all levels. And Patrick Naef fully supported and endorsed those processes and appointments (some, as always in the real world, being more successful than others), as did every member of the IT Executive.  If anyone has ever tried to paint a different picture, they have not been truthful. 

And we must remember that there was a large redundancy exercise in EG-IT in 2009.  I managed that exercise myself and made sure that it was as fair and as comprehensive as I believed possible.  We had a target, but we did not have a limit.  Everyone in the EG-IT organisation when Project Shaheen was initiated had either emerged from Phoenix or had been recruited/appointed using robust processes which had been totally supported by every member of the IT Executive.  Those in post during 2009 had also survived a rigorous redundancy selection process which, again, was signed off by the full IT Executive. It is therefore clear that Project Shaheen (which resulted in staff being removed by yet another round of assessments) was applied to an organisation for which Patrick Naef (as leader of the IT Executive) had total ownership and responsibility. Nobody had been inherited, everyone was the responsibility of Patrick Naef.  Do not ever allow him to present the situation differently.

Like everyone, once Phoenix was completed I was keen on getting on with matters.  Apart from making the organisation work, we had many other issues to deal with.  In my area there were massive problems to address, but we had already made good inroads on the service front.  Outages had been reduced by a massive two thirds in the first six months, but I was annoyed when this, along with further subsequent improvements which were made, was claimed by the consultants to be the result of the change project.  Phoenix achieved a number of things, but it had no hand whatsoever in the drastic improvement of operational service in 2006-7.  All that was achieved by a lot of hard work by the network and data centre teams replacing hardware (some of which Patrick had already initiated) and embracing a much more disciplined approach to change, risk and incident management.  Some benefits were realised before Phoenix even started, most before the project finished.  I complained to Patrick, but he just shrugged it off which was hardly surprising as it was part of a European road show to enhance his, as well as the consultants’, reputation.

After the project was completed, Patrick Naef regularly came up with initiatives that he wanted to implement and these were a constant drain and undermined the Phoenix work.  Most of them were theoretically good ideas, many impractical and some wholly inappropriate.  For instance, he wanted to introduce a ‘high performance culture’.  Well, we all supported the idea of an ordered meritocracy, I even had a tool that I had used elsewhere which we could (and did) implement to support the idea.  But, to give the whole thing a bit more edge, Patrick Naef wanted to “fire the bottom 5%” every year and replace them with external recruits!  Here we were, with a vacancy level of around 20%, struggling to find suitable candidates, beginning to recover from the huge disruption of Phoenix, with everyone’s trust in management still virtually non-existent and Patrick wanted us to pick off the weakest 5% in an annual cull.  One of Patrick Naef’s problems is that he reads about initiatives that have worked elsewhere and then tries to implement them immediately, regardless of whether they are practical, ethical or even needed.  Fortunately, it was not only me on the IT Executive who had major problems with the 5% idea.  This was just one example (of many) of how Patrick Naef wanted to manage EG-IT, with an approach which could only (and did) systematically undermine all the benefits that came out of Phoenix.  

Of course, internally, Patrick Naef never formally put his name to Project Phoenix.  Everyone knew it was his baby, but he actually delegated the task of its Project Management.  I was told that this was recognised at the time in some business circles as “firing his gun from someone else’s shoulder”.  After the project’s completion, there was no doubt it was Patrick Naef’s project during presentations at external industry gatherings.  However, when a government audit report concluded that the project had been poor value and had been commissioned outside of company procurement procedures, Patrick Naef took a very swift backward step.  I do not know exactly when that audit was published but I do know that just a few days after Patrick Naef told me about it, he suddenly gave the project a mention (not related to the audit) in his weekly staff update, taking the opportunity to remind everyone who the project manager was (i.e. not him!).  Maybe it was just a coincidence, but the timing was interesting as the project had not been mentioned for a long time, nor was it mentioned again for a long time.  I do not know how many of you have seen the illusionist and trickster Derren Brown in action, but most audiences find his ability to fool people quite extraordinary.  Personally, having worked with Patrick Naef, I cannot say that I am ever particularly overwhelmed by Mr Brown.   

I will finish this off with a short quiz to test readers’ understanding of what a high performance culture really means in EG-IT.

1.  How many people who expressed concern about, and/or refused to paint a false picture of the benefits of, the ‘Mercator Asia’ acquisition kept their jobs?

2.  How many people who fully supported and commended that acquisition lost their jobs?

3.  How much money was wasted by the Emirates Group on the acquisition, operation and disposal of Mercator Asia?

I will provide a clue – the answers contain a heck of a lot of zeros!

But, for me, the most intriguing question is – what has Gary Chapman been told the answers are?