Category: Metrics and Process Improvement

When to Benchmark and How to Leverage the Results

    Jim D

Benchmarking is an important tool for management and yet frequently I have found most organizations do not take advantage of it. There seem to be three camps when it comes to benchmarking:

  • those who either don’t have the time or through some rationale, avoid tapping into benchmarking
  • those that try to use it infrequently but for a variety of reasons, are never able to leverage and progress from its use
  • those who use benchmarking regularly and make material improvement

I find it surprising that as so many IT shops don’t benchmark. I have always felt that there were only two possible outcomes from doing benchmarking:

  1. You benchmark and you find out that you compare very well with the rest of the benchmark population and you can now use this data as part of your report to your boss and the business to let them know the value your team in delivering
  2. You benchmark and you find out a bunch of things that you can improve. And you now have the specifics as to where and likely how to improve.

So, with the possibility of good results with either outcome, when and how should you benchmark your IT shop?

I recommend making sure that your team and the areas that you want to benchmark have an adequate maturity level in terms of defined processes, operational metrics, and cost accounting.  Usually, to take advantage of a benchmark, you should be at a minimum a Level 2 shop and preferably a Level 2+ shop where you have well understood unit costs and unit productivity. If you are not at this level, then in order to compare your organization to others, you will need to first gather an accurate inventory of assets, staff, time spent by activity (e.g., run versus change). This data should supplemented with defined processes and standards for the activity to be compared. And then for a thorough benchmark you will need data on the quality of the activity and preferably 6 month trending of all data. In essence, these are prerequisite activities that must take place before you benchmark.

I do think that many of teams that try to benchmark but then are not able to do much with the results are unable to progress because:

  • they do not have the consistent processes on which improvements and changes can be implemented
  • they do not routinely collect the base data and thus once the benchmark is over, no further data is collected to understand if any of the improvements had effect or not
  • the lack of data and standards results in so much estimation for the benchmark that you cannot then use it to pinpoint the issues

So, rather than benchmark when you are a Level 1 or 2- shop, instead just work on the basics of improving your maturity of your activities. For example, collect and maintain accurate asset data – this foundational to any benchmarking. Similarly collect how your resources spend their time — this is required anyway to estimate or allocate costs to whoever drives it, so do it accurately. And implement process definitions, base operational metrics, and have the team review and publish monthly.

For example, let’s take the Unix server area. If we are looking to benchmark we would want to check various attributes against the industry including:

  • number of servers (versus similar size firms in our industry)
  • percent virtualized
  • unit cost (per server)
  • unit productivity (servers per admin)
  • cost by server by category (e.g. staff, hardware, software, power/cooling/data center, maintenance)

By having this information you can quickly identify where you have inefficiencies or you are paying too much (e.g., the cost of your software per server) or you are unproductive (your servers per admin is low versus the industry). This then allows you to draw up effective action plans because you are addressing problems that can likely be solved (you are looking to bring your capabilities up to what is already best practice in the industry).

I recall a benchmark of the Unix server area where our staff costs per server were out of line with the industry though we had relatively strong productivity. Upon further investigation we realized we had mostly a very senior workforce (thus being paid the top end of the scale and very capable) that had held on to even the most junior tasks. So we set about improving this in several ways:

  • we packaged up and moved the routine administrative work to IT operations (who did it for far less and in a much more automated manner)
  • we put in place a college graduate program and shifted nearly all of our hires in this group from a previous focus of mid to senior only new hires to one where it was mostly graduates, some junior and only a very few mid-level engineers.
  • we also put in place better tools for the work to be done so staff could be more productive (more servers per admin)

The end result after about 12 months was a staff cost per server that was significantly below the industry median, approaching best in class (and thus we could reduce our unit allocations to the business). Even better, with a more balanced workforce (i.e., not all senior staff) we ended up with a happier team because the senior engineers were now looked up to and could mentor the new junior staff. Moreover, the team now could spend more of their time on complex change and project support rather than just run activities. And with the improved tools making everyone more productive, it resulted in a very engaged team that regularly delivered outstanding results.

I am certain that some of this would have been evident with just a review. But by benchmarking, not only were we able to precisely identify where we had opportunity, we were better able to diagnosis and prescribe the right course of action with targets that we knew were doable.

Positive outcomes like this are the rule when you benchmark. I recommend that you conduct a yearly external benchmark for each major component area of infrastructure and IT operations (e.g. storage, mainframe, server, network, etc). And at least every two years, assess IT overall, assess your Information Security function, and if possible, benchmark your development areas and systems in business terms (e.g., cost per account, cost per transaction).

One possibility in the development area is that since most IT shops have multiple sub-teams within development, you can use the operational development metrics to compare them against each other (e.g. defect rates, cost per man hour or cost per function, etc). Then the sub-team with the best metrics can share their approach so all can improve.

If you conduct regular exercises like this, and combine it with a culture of relentless improvement, you will find you achieve a flywheel effect, where each gain of knowledge and each improvement becomes more additive and more synergistic for your team. You will reduce costs faster and improve productivity faster than taking a less open and less scientific approach.

Have you seen such accelerated results? What were the elements that contributed to that progress? I look forward to hearing from you.

Best, Jim

 

 

Delivering More by Leveraging the ‘Flow of Work’

    Jim D

As you start out this year as an IT leader and you are trying to meet both project demand on one hand and savings goals on the other, remember to leverage what I term the IT ‘Flow of Work‘. Too often, once work comes into an organization, either through a new project going into production or through the original structure of the work, it is static. The work, whether it is server administration, batch job execution, or routine fixes, continues to be done by the same team that often developed it in the project cycle. Or at best, the system and its support are handed off to a dedicated run team, that continues to treat it as ‘custom’ work. In other words, once the system has been crafted by the project team, the regular work to run, maintain, and fix the system continues to be custom-crafted work.

This situation, where the work is static and continues to be executed by the initial, higher cost resources, would be analogous to a car company crafting a prototype or concept car, and then using that same team to produce every single subsequent car of that model with the same ‘craft’ approach. This of course does not happen as the car company moves the prototype to a production factory where the work is standardized and automated and leaned, and far lower cost resources execute the work. Yet in the IT industry we often fail to leverage this ‘flow of work’. We use senior server engineers to do basic server administration tasks (thus making it custom work). We fail to ‘package’ or productionalize or automate the tasks thus requiring exorbitant amounts of manual work because the project focused on delivering the feature and there was not optimization step to get it into the IT ‘factory’.

Below is a diagram that represents the flow of work that should occur in your IT organization.

Moving work to where it is most efficiently executed

The custom work, work done for new design, or complex analysis or maintenance, is the work done by your most capable and expensive resources. Yet, many IT organization waste these resources by doing custom work where it doesn’t matter. A case in point would be anywhere that you have IT projects leveraging custom designed and built servers/storage/middleware (custom work) instead of standardized templates (common work). And rarely do you see those tasks documented and automated such that they can be executed by the IT Operations team (routine work). And not only do you then waste your best resources on design that adds minimal business value, you then do not have those best resources available to do the new projects and initiatives the business needs to have done. Or similarly, your senior and high cost engineers are doing routine administrative work because that is how the work was implemented in production. Later, no investment has been made to document or package up this routine work so it can easily be done by your junior or mid-level engineering resources.

Further, I would suggest that you often find the common or routine engineering work stays in that domain. Investment is infrequently made to further shrinkwrap and automate and document the routine administrative tasks your mid-level engineers so that you can hand it off to the IT Operations staff to execute as part of their regular routine (and by the way, the Ops team typically executes these tasks with much greater precision than the engineering teams).

So, rather than fall into the traps of having static pools of work within your organization, drive and investment so that the work can be continually packaged and executed at a more optimal level and free up your specialty resources to tackle more business problems.  Set a bar for each of your teams for productivity improvements. Enable them the time and investment to package the work and send it to the optimal pool. Encourage your teams to partner with the group that will receive their work on the next step of the flow. And make sure they understand that for every bit of more routine work that they can transition to their partner team, they will receive more rewarding custom work.

After a few cycles of executing the flow of work within your organization, you will find you have gained significant capacity and reduced your cost to execute routine work substantially. This enables you to achieve a much improved ratio of run cost versus build costs by continually compressing the run costs.

I have seen this approach executed with great effect in both large and mid-sized organizations. Have you been part of or lead similar exercises? What results did you see?

Best, Jim

 

Ensuring Project Success: Best practices in Project Delivery

    Jim D

Project delivery is one of the critical services that IT provides. Unfortunately, in the industry, the track record of project delivery for IT is at best a mixed bag. A number of different studies over the past five years put the industry success rate below 50%. A good reference in fact is the Dr. Dobbs site where a 2010 survey found project success rates to be:

  • Ad-hoc projects: 49% are successful, 37% are challenged, and 14% are failures.
  • Iterative projects: 61% are successful, 28% are challenged, and 11% are failures.
  • Agile projects: 60% are successful, 28% are challenged, and 12% are failures.
  • Traditional projects: 47% are successful, 36% are challenged, and 17% are failures.

So, obviously not a stellar track record in the industry. And while you may feel that you a doing a good job of project delivery, typically this means the most visible projects are doing okay or even good, but there are issues elsewhere. In this post and in the next few I will provide a quick primer to check and ensure you are leveraging the key project delivery best practices that enable more successful track record.

Key project delivery practice areas: These areas are project initiation, project communications and reporting, project management, release management, and program management. There are also a few critical techniques to overcome obstacles including how to get the right resource mix, doing the tough stuff first, and when to use a waterfall approach versus incremental or agile. I will cover the first two areas today and the rest over the few weeks.

Project Initiation – I have rarely, and I suspect similarly for you, been part of an organization where the demand for IT to deliver projects is less than the supply of IT resources to execute those projects. In fact, this over-demand is often chronic and our IT response sometimes exacerbates the situation. Because of our desire to meet the business’s wishes, and to show progress, we make the mistake of initiating projects before they or our teams are ready to start. It is critical to ensure that you have effective sponsorship, a good bead on requirements and adequate resources before you initiate the project. There is no issue with doing concept or project exploration, initial requirements and designs, high level planning or estimation, but these must all be divided from the formal project effort with a strong entry gate that ensures you have the sponsorship, understanding of the deliverable, and adequate resources. Most projects that fail (and remember, this is probably half of the projects you do), started to fail at the beginning. They were started without clear business ownership or decision-makers, or with a very broad or ambiguous description of delivery. or they were started when you were already over-committed, without senior business analysts or technology designers.  This is just a waste of money, resource time, and business opportunity.

Address this by setting a robust entry gate and be disciplined about when to start a project. If the business sponsors aren’t there, this is a discussion for you to have with that business leader. If the project is defined as overly broad or is ambiguous, don’t fall into that trap. Take the two or three most well-defined needs and split off or chunk the project into starting with just that piece. The rest will fall into place as the work is done and everyone understands better the situation. You and the business may decide not to progress further, If you decide to proceed then start up the next chunk as a separate project. And, as much as you want to get more done, don’t start the project before you have resources. As you operate your project ‘factory’ at extremely high utilization levels, each additional piece of work you add makes you less efficient and more costly. You must use higher priced contractors rather than internal resources. And your key experts that must do the work, now have to juggle one more thing and will be less effective. It would be far better for your 500 resource team to do say, 80 projects effectively, rather than 110 projects ineffectively. It is like a server that has not enough memory for the applications active on the system. More and more time is spent on overhead, paging things in and out rather than on real work. The end result is less work done in total even though more projects are active. This is a common trap that is overcome by knowing you effectiveness range and then simply prioritizing with the business. If something critical has come up then for it to be started you must ice and take off the table other projects. Make these choices with your business partners. If you maintain this balance, then you will get more projects done in the year — and that should be the true measure of delivery (not how man projects you started).

Project reporting and communication – Doing a project is a team effort. You have many staff with different backgrounds and skills from different organizations that must come together to deliver on a single blueprint to a common goal. With such a diverse team, effective communications are paramount. And yet, often, the only formal communications are those that a directed towards senior management. There are three primary audiences that a project manager should communicate formally: senior management, the business customer, and the project team. Further, the project manager should leverage the same detailed reporting and advanced analysis that they are doing to manage the project to quickly reformat into a digestible report for their audience.

Oftentimes, you find the project managers are burdened by multiple reporting systems where they are manually entering the same information two or three times. And then middle management for both IT and the business demand additional reports on metrics that are not useful and forms for approval that are bureaucratic and repetitive. Meanwhile, the critical data (e.g. risk items, resource utilization, critical path delays) are not reported broadly if at all. And the project manager is overwhelmed with the busy work versus the real task at hand.

So streamline the reporting process. Ensure your team a single, effective project reporting tool and invest in one if not. I recommend that all the but the smallest projects produce a weekly ‘4-Box’ report. This one pager can be used for all three primary audiences and ensures the project manager, the sponsor and the key stakeholders are paying attention to the important aspects of the project.

I will be placing a 4-Box sample on the reference page for your use later this month. But they key components are simple:

  • a landscape page with 4 quadrants, a left margin column and a header section
  • the header consists of the Project Title centered and Project Status in color boxes on the far right upper corner, and the project mission in small font (it always amazing how many people work on a project and do not know what it is intended to do — thus the mission on every communication)
  • the left margin contains the names and phone numbers of the project manager, sponsor, and all key participants and stakeholders. Thus everyone knows who to call if there is an issue or question
  • The upper left quadrant contains a brief description and the key milestones for the project with dates and a status indicator (e.g. completed, underway, etc)
  • The lower left quadrant box contains accomplishments and progress for the past week (or time period). There should be a brief description of progress and a listing of the key milestones or tasks completed.
  • The upper right quadrant is a listing of the key risks and issues the project faces. They should be catalogued and a status indicated (e.g. Mitigated, Underway, Open) with a color status as well.
  • The bottom right quadrant should provide what will will get done this next week or time period by milestone or significant task with dates and with owners.

Additional information can be used to augment the report, but they key is now you can use the same one page to communicate effectively with all your audiences. This ensures everyone is one the same page (literally) at a minimum of effort. Note also that we avoid the ‘ creative writing’ of project status reports that some many organizations waste time and use to put an optimistic spin on the project progress. Instead, just the facts.

By aligning your project process and teams to these two best practice approaches, you will find:

  • you are not starting projects before they are ready to be started
  • you will run your project factory at optimal output and effectiveness
  • you will lighten the overhead load on your project managers, so they can do more real work
  • your project teams will be on the same page (and thus more effective)
  • you and your businesses will know what is going on and can identify issues much earlier and solve them more quickly

In essence, you will deliver projects more successfully.

What are the variations on these approaches that you have used with success? What would you do differently? What other areas of project delivery are problematic that you have solutions for? Have a wonderful holiday and I look forward to your perspectives.

Best, Jim

 

IT Transparency: A key approach to delivering value and ensuring focus

    Jim D

Frequently, IT shops have a difficult time convincing their business users of the value of IT. It is straightforward for a businessperson to look at the cost of IT, which is readily available but not have a good reference for the benefits and the overall value of IT. This lack of a good reference then leads to further concerns on IT budgets and investment. As CIO, it is imperative that you provide transparency on the benefits and impacts of IT so your customer and the business team can understand the value of IT and properly allocate budget and investment to make the business more successful. Yet frequently, IT does a very poor job of measuring IT impact and benefits, and when it is measured it is often done only in technical metrics which are not easily translated into understandable business metrics.

A good area to start to provide transparency to your business partners is with your production services. Typical IT metrics here are done only in technical terms not in business terms. For example, you often find availability measured in outage time or percentage or as the number of incidents per month. And while these are useful metrics to do some technical analysis, they are poor metrics to communicate business impact. Instead you should be providing your production metrics to your customer in business form. For example, you should report on the key service channels that are delivered to the end customer (e.g., ATMs, retail points of sale, call centers, etc) and the underlying metric should be customer impact availability. You derive this metric by counting the number of customer interaction or transactions that were successful divided by the total number of possible customer interactions or transactions for that time period.

Specifically, take an ATM channel as the example. Let’s assume there are normally 100,000 customer transactions in that month. If the ATMs were down for 1 hour that month and there would have been 1000 customer transactions that normally would have been completed in that hour then the customer impact availability was 99% (= (100,000 -1,000)/100,000). Just as importantly, your business also knows 1,000 customers were impacted by the systems issue. And if the outage occurred at peak usage — say 1 hour on a Friday evening rather than 3 AM on a Sunday, you may have 2,000 customers impacted versus 100. And your business-oriented metric, customer impact availability, would show this rather than the constant view that a system time availability metric would show. But this is critical knowledge, for you and for your business partners. You will have to collect information on how your customers use your service channels and understand daily, weekly, and seasonal variations, but you should know this anyway so you can better plan capacity upgrades and implement changes and releases.

You should apply the same criteria to your project delivery. Typically, IT shops only provide on-time and on-budget metrics. And these are often flawed because they either accept major changes through the project lifecycle (what I call ‘drawing the bullseye after the arrow is shot’) and they take no account of whether the project is ‘on-function’ or on-quality. Even worse, few shops track if the promised benefits were delivered or the expected revenue materialized. Thus making it very hard to assess if you are getting the expected return on investment. To remedy this, partner with the CFO office and tackle the large projects and big investments first. Insist on tracking with the CFO’s office the original cost, timeframe, benefits and and function and quality targets for this subset of the projects. If there is a major change to scope or requirements, note the change and track the additional revised cost, time, quality, etc. Then at project closeout, the project is not completed without a full assessment on how well you met the original estimates and any lessons learned on why not. It may also require a revisit of the benefits and revenues 6 months down the road to properly assess them. This information is extremely valuable to then have for assessing future investments. Is the business consistently over-estimating the revenue that will result? Is your team often underestimating the time to get projects completed? At the end of the year, you will be able to communicate not just these trends back to business but also you can outline that for X spend on large projects you delivered Y in benefits (with Finance’s backing). That should provide a compelling value statement if the projects were well-chosen. And if not, you now have the rationale to adjust the company’s investment process.

Similarly, for the ‘routine services’ of IT (delivering small systems, help desk services, desktop support, etc) you should provide metrics that are business relevant.  Provide the cost of the routine services as a cost per user (e.g. cost for desktop service per user per year) or cost per service (cost of a help desk call).  Go further and enable the business team to understand the impact of investment in the services by defining the number of staff hours of work saved. If you are implementing self service (e.g. password resets as an example) you may have a cost save (reduced help desk calls and thus fewer help desk staff, but you should really have reduced staff time to reset and less staff downtime as a result (improving staff productivity and engagement). Similarly, a new departmental workflow system should reduce both staff hours to process say an HR transaction but also substantially reduce the wall time of that transaction (again driving improved productivity and engagement). These are often neglected areas within a corporation that by translating into business metrics (staff hours saved, wall time reductions) will make visible the business benefits.

By routinely expressing what IT does in terms of business metrics, you will enable you business partners to understand the impact and value of IT on their customers and services. It will drive better decisions on how much and where to invest in technology. And it will also raise the level of business awareness of your team. When they know an outage impacted 32,000 customers, that comes across as much more material and important than a 40 minute router outage. You can adjust or denote the metrics as appropriate for your business. For example, if you are handling large financial sums then system outages may be expressed in terms of the sum of the financial amounts delayed or not processed. Further, several large Financial Services firms with robust shops report not just the customer impacts but also correlate if there was an industry cause and impact as well as if a portion of the functionality was available (e.g. you could access the ATM and withdraw cash but not see your balance).  The key is to put your primary measure in business relevant metrics.

What business relevant metrics have you used with success? What hurdles have you encountered?

I look forward to your thoughts and success with more transparent metrics.

Best, Jim