Category: Efficiency and Cost Reduction

Best practices in how to achieve Efficiency and Cost Reduction goals

Outsourcing and Out-tasking Best Practices

    Jim D

I recently published this post first at InformationWeek and it generated quite a few comments, both published and several sent directly via e-mail.  I would note that a strong theme is the frustration of talented staff dealing with senior leadership that does not understand how IT works well or do not appear to be focused on the long term interests of the company. It is a key responsibility of leadership to ensure they keep these interests at the core of their approach, especially when executing complex efforts like outsourcing or offshoring so that they do achieve benefits and do not harm their company. I think the national debate that is occurring at this time as well with Romney and Obama only serves to show how complex executing these efforts are. As part of a team, we were able to adjust and resolve effectively many different situations and I have extracted much of that knowledge here. If you are looking to outsource or are dealing with an inherited situation, this post should assist you in improving your approach and execution.

While the general trend of more IT outsourcing but via smaller, more focused deals continues, it remains an area that is difficult for IT management to navigate successfully.  In my experience, every large shop that I have turned around had significant problems caused or made worse by the outsourcing arrangement, particularly large deals. While understanding that these shops performed poorly for primarily other reasons (leadership, process failures, talent issues), achieving better performance in these situations required substantial revamp or reversal of the outsourcing arrangements. And various industries continue to be littered with examples of failed outsourcing, many with leading outsource firms (IBM, Accenture, etc) and reputable clients. While formal statistics are hard to come by (in part because companies are loathe to report failure publicly), my estimate is that at least 25% and possibly more than 50% fail or perform very poorly. Why do the failures occur? And what should you do when engaging in outsourcing to improve the probability of success?

Much of the success – or failure – depends on what you choose to outsource followed by effectively managing the vendor and service. You should be highly selective on both the extent and the activities you chose for outsourcing. A frequent mistake is the assumption that any activity that is not ‘core’ to a company can and should be outsourced to enable focus on the ‘core’ competencies. I think this perspective originates from principles first proposed in The Discipline of Market Leaders by Michael Treacy and Fred Wisrsema. In essence, Treacy and Wisrsema state that companies that are market leaders do not try to be all things to all customers. Instead, market leaders recognize their competency either in product and innovation leadership, customer service and intimacy, or operational excellence. Good corporate examples of each would be 3M for product, Nordstrom for service, and FedEx for operational excellence. Thus business strategy should not attempt to excel at all three areas but instead to leverage an area of strength and extend it further while maintaining acceptable performance elsewhere. And by focusing on corporate competency, the company can improve market position and success. But generally IT is absolutely critical to improving customer knowledge intimacy and thus customer service. Similarly, achieving outstanding operational competency requires highly reliable and effective IT systems backing your operational processes.  And even in product innovation, IT plays a larger and large role as products become more digital and smarter.

Because of this intrinsic linkage to company products and services, IT is not like a security guard force, nor like legal staff — two areas that are commonly fully or highly outsourced (and generally, quite successfully). And by outsourcing intrinsic capabilities, companies put their core competency at risk. In a recent University of Utah business school article, the authors found significantly higher rates of failure of firms who had outsourced. They concluded that  “companies need to retain adequate control over specialized components that differentiate their products or have unique interdependencies, or they are more likely to fail to survive.” My IT best practice rule is ‘ You must control your critical IP (intellectual property)’. If you use an outsourcer to develop and deliver the key features or services that differentiate your products and define your company’s success, then you likely have someone doing the work with different goals and interests than you, that can typically easily turn around and sell advances to your competitors. Why would you turn over your company’s fate to someone else? Be wary of approaches that recommend outsourcing because IT is not a ‘core’ competency when with every year that passes, there is greater IT content in products in nearly every industry. Chose instead to outsource those activities where you do not have scale (or cost advantage), or capacity or competence, but ensure that you either retain or build the key design, integration, and management capabilities in-house.

Another frequent reason for outsourcing is to achieve cost savings. And while most small and mid-sized companies do not have the scale to achieve cost parity with a large outsourcer, nearly all large companies, and many mid-sized do have the scale.  Further, nearly every outsourcing deal that I have reversed in the past 20 years yielded savings of at least 30% and often much more. Cost savings can only be accomplished by an outsourcer for a large firm for a broad set of services if the current shop is a mediocre shop. If you have a well-run shop, your all-in costs will be similar to the better outsource firms’ costs. If you are world-class, you can beat the outsourcer by 20-40%.

Even more, the outsourcer’s cost difference typically degrades over time. Note that the goals of the outsourcer are to increase revenue and margin (or increase your costs and spend less resources doing your work). Invariably, the outsourcer will find ways to charge you more, usually for changes to services and minimize work being done. And previously, when you had used your ‘run’ resources to complete minor fixes and upgrades, you could find you are charged for those very same resources for such efforts once outsourced. I have often seen that ‘run’ functions will be hollowed out and minimized and the customer will pay a premium for every change or increase in volume. And while the usual response to such a situation is that the customer can put terms in the contract to avoid this, I have yet to see such terms that ensure the outsourcer works in your best interest to do the ‘right’ thing throughout the life of the contract. One interesting example that I reversed a few years back was an outsourced desktop provisioning and field support function for a major bank (a $55M/year contract). When an initial (surprise) review of the function was done, there were warehouses full of both obsolete equipment that should have been disposed and new equipment that should have been deployed. Why? Because the outsourcer was paid to maintain all equipment whether in use in the offices or in a warehouse, and they had full control of the logisitics function (here, the critical IP). So, they had ordered up their own revenue in effect. Further, the service had degraded over the years as the initial workforce had been hollowed out and replaced with less qualified individuals. The solution? We immediately in-sourced back the logistics function to a rebuilt in-house team with cost and quality goals established. Then we split the field support geography and conducted a competitive auction to select two firms to handle the work. Every six months each firm’s performance would be evaluated for quality, timeliness and cost and the higher performing firm would gain further territory. The lower performing firm would lose territory or be at risk of replacement. And we maintained a small but important pool of field support experts to ensure training and capabilities were kept up to par and service routines were updated and chronic issues resolved. The end result was far better quality and service, and the cost of the services were slashed by over 40% (from $55M/year to less than $30M/year). And these results — better quality at lower costs — from effective management of the functions and having key IP and staff in-house are the typical results achieved with similar actions across a wide range of services, organizations and locales.

When I was at BankOne, working under Jamie Dimon and his COO Austin Adams, they provided the support for us to tackle bringing back in what had been the largest outsourcing deal ever consummated at its time in 1998. Three years after the outsource had started, it had become a millstone around BankOne’s neck. Costs had been going up every year, quality continued to erode to where systems availability and customer complaints became worst in the industry. In sum, it was a burning platform. In 2001 we cut the deal short (it was scheduled to run another 4 years). In the next 18 months, after hiring 2200 infrastructure staff (via best practice talent acquisition), revamping the processes and infrastructure, we reduced defects (and downtime) to 1/20th of the levels in 2001 and reduced our ongoing expenses by over $200M per year. This supported significantly the bank’s turnaround and enabled the merger with JP Morgan a few years later.  As for having in-house staff do critical work, Jamie Dimon said it best with ‘Who do you want doing your key work? Patriots or mercenaries?’

Delivering comparable cost to an outsourcer is not that difficult for mid to large IT shops. Note that the outsourcer must include a 20% margin in their long term costs (though they may opt to reduce profits in the first year or two of the contract) as well as an account team’s costs. And, if in Europe, they must add 15 to 20% VAT. Further, they will typically avoid making the small investments required for continuous improvement over time. Thus, three to five years out, nearly all outsourcing arrangements cost 25% to 50% more than a well-run in-house service (that will have the further benefit of higher quality). You should set the bar that your in-house services can deliver comparable or better value than typical out-sourced alternatives. But ensure you have the leadership in place and provide the support for them to reach such a capability.

But like any tool or management approach, used properly and in the right circumstances, outsourcing is a benefit to the company. As a leader you cannot focus on all company priorities at once, nor would you have the staff even if you could, to deliver. And in some areas such as field support there are natural economies of scale that benefit a third party doing the same work for many companies. So consider outsourcing in these areas but the extent of the outsource carefully. Ensure that you still retain critical IP and control. Or use it to augment and increase your capacity, or where you can leverage best-in-class specialized services to your company’s benefit. Then, once selected and effectively negotiated, manage the outsourcing vendor effectively. Since effective management of large deals is complex and nearly impossible, it is far better to do small outsourcing deals or selective out-tasking. The management of the outsourcing should be handled like any significant in-house function, where SLAs are established and proper operational metrics are gathered, performance is regularly reviewed with management and actions are noted and tracked to address issues or improve service. Properly constructed contracts that accommodate potential failure are key if things do not go well. Senior management should jointly review the service every 3 to 6 months, and consequences must be in place for performance (good or bad).

Well-selected and managed outsourcing will then complement your in-house team with more traditional approaches that leverage contractors for peak workloads or projects or the modern alternative to use cloud services and out-task some functions and applications. With these best practices in place and with a selective hand, your IT shop and company can benefit from outsourcing and avoid the failures.

What experiences have you had with outsourcing? Do you see improvement in how companies leverage such services? I look forward to your comments.

Best, Jim Ditmore

 

 

 

Optimizing Technology Infrastructure with Master Craftsmen

    Jim D

One of my former colleagues, Ralph Bertrum, has provided the primary material for today’s post on how to optimize a technology infrastructure with master craftsmen. Ralph is one of these master craftsmen in the mainframe infrastructure space. If you are a CIO or senior IT leader looking to improve your shop’s cost or performance, I recommend optimizing your infrastructure and systems through high payback platform efficiency reviews.

In today’s shops, often with development and coding partially or fully outsourced, and not enough experienced and capable resources on staff, many applications are built for functionality without much regard for efficiency.  And nearly every shop has legacy applications where few engineers, if any, actually understand how they work. These applications have often been patched and extended that just to have them run is viewed as the goal, rather than run effectively and efficiently. The result is that for most shops, there is a  build up of 10, 20, or even 30% of their compute and storage capacity that is wasted on inefficient systems. This happens partly because it is easiest to just throw hardware at a problem and partly because they do not have the superior engineering resources — or master craftsmen — required to locate, identify, and resolve such inefficiencies. Yet, it is a tremendous waste and a major recurring cost for the IT shop. It is a significant opportunity for IT leaders to attack these inefficiencies.  In my experience, every one of these master craftsmen, if given the framework and support, can easily return 4 to 10 times their annual compensation in savings each quarter!

So, how do you go about building and then leveraging such an efficiency engineering capability? First, to build the capability, you must be willing to invest in select engineers that are heavily dedicated to this work. I recommend focusing on mainframe efficiency and server efficiency (Unix and Windows) as the primary areas of opportunity. Given the different skill sets, you should establish two separate efficiency teams for these two areas. Storage usage should be reviewed as a part of each team’s mission. A small team of two to four individuals is a good starting point. You can either acquire experienced talent or build up by leveraging promising engineers on staff and augmenting with experienced contractors until your staff have attained full capability. Ensure you invest in the more sophisticated tools needed to instrument the systems and diagnose the issues.  And importantly, ensure their recommend application and systems changes are treated with priority and implemented so the savings can be achieved. A monthly report on the recommendations and results completes the building the team and framework.

Now for the approach, which Ralph Bertrum, an experienced (perhaps even an old school) efficiency engineer has provided for the mainframe systems:

Having spent 50 years in Information Technology working on Mainframe Computers, I have seen a great many misunderstandings.  The greatest single misunderstanding is the value and impact of system engineering training and experience and it’s use in performing maintenance on a very costly investment. Many CIOs prefer to purchase a computer engine upgrade and continue to run a wasteful collection of jobs on a new faster machine.  It is the easiest way out but definitely not the most cost effective.  It is the equivalent to trading in your car every time the air filter, spark plugs or hoses need changing or the tires need air and then moving the old air filter, old spark plugs, old hoses and old tires to the new car.

Would you drive around with a thousand pound bag of sand in the trunk of your car?  Would you pull a thousand pound anchor down the street behind your car?  That is exactly what you are doing when you don’t regularly review  and improve the Job Control Language (JCL), Programs, and Files that run on your Mainframe Computer.  And would you transfer that thousand pound bag of sand and that anchor to your new car every time you purchased a new one?  Most IT shops are doing that with every new mainframe upgrade they make.  Every time they upgrade their computer they simply port all the inefficiencies to the upgrade.

Platform efficiency reviews will reduce waste impacting all kinds of resources: CPU, storage, memory, I/O, networks. And the results will make the data center greener and reduce electricity bills and footprints of equipment, speed online and batch processing, eliminate or delay the need for  upgrades, reduce processing wall times and cycle time, and ultimately improve employee efficiency, customer satisfaction and company profitability.

You can apply platform efficiency reviews to any server but let’s use the mainframe as a primary example. And, we will extend the analogy to a car because there are many relevant similarities.

Both automobiles and computers have a common need for maintenance.  An automobile needs to have the oil, the air filter, and spark plugs changed, tires rotated and the tire air pressure checked.  All of these are performed regularly and save a large amount of gas over the useful life of the automobile and extend the life of the car.  Reasonable maintenance on a car can improve mileage by three to four miles per gallon or about a 20% improvement. When maintenance is not performed the gas mileage begins to degrade and the automobile becomes sluggish, loses its reliability and soon will be traded in for a newer model.  The sand is growing in weight every day and the anchor is getting heavier.

For a mainframe, the maintenance is not just upgrading the systems software to the most recent version. Additional maintenance work must be done to the application software and its databases. The transactions, files, programs, and JCL must be reviewed, adjusted and tuned in order to identify hot spots, inefficient code and poor configurations that have been introduced with application changes or additional volume or different mixes. Over the last twenty five years I have analyzed and tuned millions of Mainframe Computer Jobs, Files, JCL, and Programs for more than one thousand  major data centers and all of them were improved.  I have never seen a Mainframe computer that couldn’t have its costs reduced by at least 10% to 15% and more likely 20% through a platform efficiency review.

Often, there are concerns that doing such tuning can introduce issues. Adjusting JCL or file definitions is just as safe as changing a spark plug or putting air in a tire.  It is simple and easy and does not change program logic.   The typical effect is that everything runs better and costs less.  The best thing about maintenance in a data center is that almost all the maintenance lasts much longer than it does in an automobile and stays in effect with continued savings in upgrade after upgrade, year after year.

Think of maintenance of a Mainframe Computer as a special kind of motor vehicle with thousands of under-inflated tires.  By making simple adjustments you can get improvements in performance from every one of these under-inflated tires. And even though each improvement is small in total, because there so many, you multiply the improvement to get significant effect.   You get this cost reduction every time the file is used or transaction executed and when all the savings from all the little improvements are added together you will get a 15% to 20% reduction in processing costs. The maintenance is a onetime cost that will pay for itself over and over upgrade after upgrade.

Here are some areas to focus for performance improvement with examples:

Avoid excessive data retention:  Many IT shops leave data in a file long after its useful life is over with or process data that is not meaningful.  An example would be Payroll records for an employee no longer with the company, General Ledger transactions from previous years, or inventory parts that are no longer sold.  By removing these records from the active file, and saving them in separate archive storage, you are saving CPU every time the file is used and work may complete much faster.  For example, an IT shop had an Accounts Receivables file that had 14 million records.  Every day they would run the file through a billing program that produced and mailed invoices.  At that time the cost of a stamp was $0.32 cents for a first class postage stamp.  A recommendation was made to the CFO that they purge all billing amounts of $0.32 cents or less from the billing file.  It was silly to pay $ 0.32 to collect $0.32 or less.  Two million records were removed from the file, the daily job ran four hours faster and they saved $35,000.00 a month on CPU and DASD space to say nothing about employee time and postage costs.  After a trial test period the minimum billing amount was raised to $1.00 and another set of very large savings was accomplished.

Optimize databases for their use.  An IT shop was looking to reduce the run time of a mailing list label system.  After looking at the data it was found that 90% of the labels were located in California and that a table looked up the city and state from a zip code table.  Each time the program needed a California city name, the program had to do ninety thousand zip code table compares before finding the correct city and state for the address.  The table was rearranged to optimize searching for California zip codes and the job went from running twenty hours to running only one hour.  CPU dropped by over 90%.  This has also worked with online transaction tables and file placement in Local Shared Resource (LSR) buffer pools. Optimizing databases is a key improvement technique.

Optimize the infrastructure configuration for your system.   One shop had jobs that would run very quickly one day and very slowly the next day.  After analyzing the jobs and the file locations it was determined that the public storage pools contained two different types of disk drives.  The Temporary work files would be placed on different disk drives every day.  What was the very best setting for one type of disk was the very worst for the other and this was causing the erratic behavior.  The storage pool was changed to contain only one type of disk device. The problem went away and the jobs ran fast every day.

Tune your systems to match your applications:   A mainframe comes with a great many abilities and features, but if your team are not adept with them your systems will not be optimized to run well.  I have analyzed over 1,000 data centers and applications and never once failed to discover significant tuning that could be accomplished within the existing system configuration. This occurs because of a lack of training or experience or focus. Ensure your team places that as a priority and if needed, bring in experts to adopt best practices. As an example of system tuning, I worked with a shop that had an online file accessing a data base and was having a major response time problem. They were afraid they were going to need a very costly upgrade. Every time they entered a transaction the system would go into a 110% utilization mode with paging.  An efficiency analysis was conducted and a system file was discovered that was doing sixteen million Input Output (I/O) instructions a day. After working with IBM to optimize the configuration, we achieved a 50% drop in I/O to eight million per day and response time improved to less than one second.  Apparently the shop had installed the system file as it was delivered and never modified it for their environment.

Tune your configurations to match your hardware.  When you make a hardware change be sure to make all the necessary other software changes as well.  Last year I worked with a very large bank that upgraded their disk drives but forgot to change a System Managed Storage (SMS) storage pool definition and continued to run forty five thousand monthly jobs using the worst blocksize possible in two thousand five hundred files.  When found and corrected, the forty five thousand jobs ran 68% faster with significant CPU savings as well as a 20% disk space savings.

Ensuring you are optimizing the most costly resource.  Remember that the efficient use of disk space is important, but not nearly as important as CPU consumption.  Analysis at another company discovered that in order to save disk space many files were using a Compression option.  The storage group had implemented this to save DASD space. In doing, the increased CPU usage unwittingly caused a multi-million dollar CPU upgrade. The Compression was removed on some of the files and CPU dropped by 20% across the board for both batch and online processing and delayed another upgrade for two more years. Optimizing disk usage at the expense of CPU resources may not be a good strategy.

Tune vendor software for your configuration.    Remember that the vendors sell their product to thousands of customers and not just you. Each vendor’s product must run in all IBM compatible environments and many of those environments will be older or smaller than your environment.  When you install the vendor software it should always be adjusted to fit properly in your environment.  Last year I did an analysis for a company that was beginning to have a run time problem.  They had an online viewing product from a vendor. They had set it up to create an online file for each customer.  They had created over three million online files and were adding one hundred thousand new ones every day.  They had run into serious performance issues because they did not understand the vendor software and the setup had been done incorrectly. So don’t add more sand to your computer by just not understanding how to best use a vendor product and configuring it correctly for your system.

Understand your systems and avoid duplication whenever possible.  Duplication of data and work is a common issue. We reviewed one IT shop that had the three million online files backed up to one hundred volumes of DASD every night taking five to six hours to run each night (and missing their SLA every morning).  Analysis showed that the files were viewing files and were never updated or changed in any way.  Except for a small number of new files, they were exactly the same unchanged report files backed up over and over.  It would have been much better to just backup the newly created files each night.  After all how many copies of a report file do you need?

If it’s not used remove it.  Remember that every file that you are not using is typically being backed up and stored every night.  If a file is not used it should be backed up, saved on archive and removed.  This space will be released and can be used for other purposes.

So the next time you think you need a computer upgrade don’t move the thousand pound bag of sand or connect the anchor to your new computer. Remember that maintenance is easy, simple, safe, and green.  Maintenance has a much greater return on investment than an upgrade.  Conduct a thorough platform efficiency review, it will save you a great deal more than you think over and over, year after year, upgrade after upgrade from now on.

Best, Ralph Bertrum and Jim Ditmore

About Ralph: Since 1986, Ralph is the co-founder and principle of Critical Path Software. He is an inventor, designer and software developer of The TURBO suite of mainframe analysis tools and expert performance tuning database. He has provided performance tuning services and analyses for over 1,100 major Fortune 5,000 corporations worldwide.  He is a former MVS, VSE, VM, CICS, and EDOS/VSE systems programmer, and an IDMS and IMS DBA.

Using Performance Metric Trajectories to Achieve 1st Quartile Performance

    Jim D

I hope you enjoyed the Easter weekend. I have teamed up today with Chris Collins, a senior IT Finance manager and former colleague. Our final post on metrics is on unit costing — on which Chris has been invaluable with his expertise. For those just joining our discussion on IT metrics, we have had 6 previous posts on various aspects of metrics. I recommend reading the Metrics Roundup and A Scientific Approach to Metrics to catch you up in our discussion.

As I outlined previously, unit costing is one of the critical performance metrics (as opposed to operational or verification metrics) that a mature IT shop should leverage particularly for its utility functions like infrastructure (please see the Hybrid model for more information on IT utilities). With proper leverage, you can use unit cost and the other performance metrics to map a trajectory that will enable your teams to drive to world-class performance as well as provide greater transparency to your users.

For those just starting the metrics journey, realize that in order to develop reliable sustainable unit cost metrics, significant foundational work must be done first including:

  • IT service definition should be completed and in place for those areas to be unit costed
  • an accurate and ongoing asset inventory must be in place
  • a clean and understandable set of financials must be available organized by account so that the business service cost can be easily derived

 If you have these foundation elements in place then you can quickly derive the unit costing for your function. I recommend partnering with your Finance team to accomplish unit costing. And this should be an effort that you and your infrastructure function leaders champion. You should look to apply a unit cost approach to the 20 to 30 functions within the utility space (from storage to mainframes to security to middleware, etc). It usually works best to start with one or two of the most mature component functions and develop the practices and templates. For the IT finance team, they should progress the effort as follows:

  • Ensure they can easily segregate cost based on service listing for that function
  • Refine and segregate costs further if needed (e.g., are there tiers of services that should be created because of substantial cost differences?)
  • Identify a volume driver to use as the basis of the unit cost (for example, for storage it could be terabytes of allocated storage)
  • Parallel to the service identification/cost segregation work, begin development of unit cost database that allows you to easily manipulate and report on unit cost.  Specifically, the database should contain:
    • Ability to accept RC and account level assignments
    • Ability to capture expense/plan from the general ledger
    • Ability to capture monthly volume feeds from source systems including detail volume data (like user name for an email account or application name tied to a server)

For the function team, they should support the IT Finance team in ensuring the costs are properly segregated into the services they have defined. Reasonable precision of the cost segregation is required since later analysis will be for naught if the segregations are inaccurate. Once the initial unit costs are reported, the function technology can now begin their analysis and work. First and foremost should be an industry benchmark exercise. This will enable you to understand quickly how your performance ranks against competitors and similar firms. Please reference the Leveraging Benchmarkspage for best practices in this step. In addition to this step, you should further leverage performance metrics like unit cost to develop a projected trajectory for for your function’s performance. For example, if your unit cost for storage is currently $4,100/TB for tier 1 storage, then the storage team should map out what their unit cost will be 12, 24, and even 36 months out given their current plans, initiatives and storage demand. And if your target is for them to achieve top quartile cost, or cost median, then they can now understand if their actions and efforts will enable them to deliver to that future target. And if they will not achieve it, they can add measures to address their gaps.

Further, you can now measure and hold them accountable on a regular basis to achieve the proper progress towards their projected target. This can be done not just for unit cost but for all of your critical performance measures (e.g., productivity, time to market, etc).  Setting goals and performance targets in this manner will achieve far better results because a clear mechanism for understanding cause and effect between their work and initiatives and the target metrics has been established.

A broad approach to also potentially utilize is to establish a unit cost progress chart for all of your utility functions. On this chart, where the y axis is cost as a percentage of current cost and the x axis is future years, you should establish a minimum improvement line of 5% per year. The rationale behind this is that improving hardware (e.g., servers, storage, etc) and improving productivity, yield an improving unit cost tide of at least 5% a year. Thus, to truly progress and improve, your utility functions should well exceed a 5% per year improvement if they are below 1st quartile. This approach also conveys the necessity and urgency of not sitting on our laurels in the technology space. Often, with this set of performance metrics practices employed along with CPI and other best practices, you can then achieve 1st quartile performance within 18 to 24 months for your utility function.

What has been your experience with unit cost or other performance measures? Where you able to achieve sustained advantage with these metrics?

Best,

Jim Ditmore and Chris Collins

 

Tying Consumption to Cost: Allocation Best Practices

    Jim D

In 1968, Garrett Hardin wrote about the over-exploitation of common resources in an essay titled the “The Tragedy of the Commons“. While Garrett wrote about the overexploitation of common pastureland where individual herders overused and diminished common pasture, there can be a very similar effect with IT resources within a large corporation. If there is no cost associated with the usage of IT resources by different business unit, than each unit will utilize the the IT resources to maximize its potential benefit to the detriment of the corporate as a whole. Thus, to ensure effective use of the IT resources there must be some association of cost or allocation between the internal demand and consumption by each business unit. A best practice allocation approach enables business transparency of IT cost and business drivers of IT usage so that thoughtful business decisions for the company as a whole can be made with the minimum of allocation overhead and effort.

A well-designed allocations framework will ensure this effective association as well as:

  • provide transparency to IT costs and the particular business unit costs and profitability,
  • avoid wasteful demand and alter overconsumption behaviors
  • minimize pet projects and technology ‘hobbies’

To implement an effective allocations framework there are several foundation steps. First, you must ensure you have the corporate and business unit CFOs’ support and the finance team resources to implement and run the allocations process. Generally, CFOs look for greater clarity on what drives costs within the corporation.  Allocations allow significant clarity on IT costs which are usually a good-sized chunk of the corporation’s costs.  CFOs are usually highly supportive of a well-thought out allocations approach. So, first garner CFO support along with adequate finance resources.

Second, you must have a reasonably well-defined set of services and an adequately accurate IT asset inventory. If these are not in place, you must first set about defining your services (e.g. and end user laptop service that includes laptop, OS, productivity software, and remote access or a storage service of high performance Tier 1 storage by Terabyte) and ensuring your inventory of IT assets is minimally accurate (70 to 80 %). If there are some gaps, they can be addressed by leveraging a trial allocation period where numbers and assets are published, no monies are actually charged, but every business unit reviews its allocated assets with IT and ensures it is correctly aligned. Once you have the service defined and the assets inventoried, your finance team must then set about to identify which costs are associated with which services. They should work closely with your management team to identify a ‘cost pool’ for each service or asset component. Again, these costs pools should be at least reasonably accurate but do not need to be perfect to begin a successful allocation process.

The IT services defined should be as readily understandable as possible. The descriptions and missions should not be esoteric except where absolutely necessary. They should be easily associated with business drivers and volumes (such as number of employees, or branches, etc) wherever possible.  In essence, all major categories of IT expenditure should have an associated service or set of services and the services should be granular enough so that each service or component can be easily understood and each one’s drivers should be easily distinguished and identified. The targets should should be somewhere between 50 and 150 services for the typical large corporation.  More services than 150 will likely lead to more effort being spent on very small services and result in too much overhead. Significantly, less than 50 services could result in clumping of services that are hard to distinguish or enable control. Remember the goal is to provide adequate allocations data at the minimum effort for effectiveness.

The allocations framework must have an overall IT owner and a senior Finance sponsor (preferably the CFO). CFOs want to implement systems that encourage effective corporate use of resources so they are a natural advocate for a sensible allocation framework. There should also be a council to oversee the allocation effort and provide feedback and direction where majors users and the CFO or designate are on the council. This will ensure both adequate feedback as well as buy-in and support for successful implementation and appropriate methodology revisions as the program grows. As the allocations process and systems mature, ensure that any significant methodology changes are reviewed and approved by the allocation council with sufficient advance notice to the Business Unit CFOs. My experience has been that everyone agrees to a methodology change if it is in their favor and reduces their bill, but everyone is resistant if it impacts their business unit’s finances regardless of how logical the change may be. Further, the allocation process will bring out intra business unit tensions toward each other, especially for those that have an increase versus those that have a decrease, if the process is not done with plenty of communication and clear rationale.

Once you start the allocations, even if during a pilot or trial period, make sure you are doing transparent reporting. You or your leads should have a monthly meeting with each business area with good clear reports. Include your finance lead and the business unit finance lead in the meeting to ensure everyone is on the same financial page.  Remember, a key outcome is to enable your users to understand their overall costs, what the cost is for each services and, what business drivers impact which services and thus what costs they will bear. By establishing this linkage clearly the business users will then look to modify business demand so as to optimize their costs. Further, most business leaders will also use this allocations data and new found linkage to correct poor over-consumption behavior (such as users with two or three PCs or phones) within their organizations. But for them to do this you must provide usable reporting with accurate inventories. The best option is to enable managers to peruse their costs through an intranet interface for such
end-user services such as mobile phones, PCs, etc . There should be readily accessible usage and cost reports to enable them to understand their team’s demand and how much each unit costs.  They should have the option right on the same screens to discontinue, update or start services. In my experience, it is always amazing that once leaders understand their costs, they will want to manage them down, and if they have the right tools and reports, managing down poor consumption happens faster than a snowman melting in July — exactly the effect you were seeking.

There are a few additional caveats and guides to keep in mind:

  • In your reporting, don’t just show this month’s costs, show the cost trend over time and provide a projection of future unit costs and business demand
  • Ensure you include budget overheads in the cost allocation, otherwise you will have a budget shortfall and neglect key investment in the infrastructure to maintain it.
  • Similarly, make sure you account for full lifecycle costs of a service in the allocation — and be conservative in your initial allocation pricing, revisions later that are upward due to missed costs will be painful
  • For ‘build’ or ‘project’ costs, do not use exact resource pricing. Instead use an average price to avoid the situation where every business unit demands only the lowest cost IT  resources for their project resulting in a race to the bottom for lowest cost resources and no ability to expand capacity to meet demand since these would be high cost resources on the margin.
  • Use allocations to also avoid First-In issues to new technologies (set the rate at the project volume rate not the initial low volume rate) and to encourage transition off of expensive legacy technologies (Last out increases)
  • And lastly, and ensure your team knows and understands their services and their allocations and can articulate why what costs what they cost

With this framework and approach, you should be able to build and deliver an effective allocation mechanism that enables the corporation to avoid the overconsumption of free, common resources and properly direct the IT resources to where the best return for the corporation will be. Remember though that in the end this is an internal finance mechanism so the CFO should dictate the depth, level and allocation approach and you should ensure that the allocations mechanism does not become burdensome beyond its value. remember that allocations framework.

What have been your experiences with allocations frameworks? What changes or additions to these best practices would you add?

Best, Jim Ditmore

 

When to Benchmark and How to Leverage the Results

    Jim D

Benchmarking is an important tool for management and yet frequently I have found most organizations do not take advantage of it. There seem to be three camps when it comes to benchmarking:

  • those who either don’t have the time or through some rationale, avoid tapping into benchmarking
  • those that try to use it infrequently but for a variety of reasons, are never able to leverage and progress from its use
  • those who use benchmarking regularly and make material improvement

I find it surprising that as so many IT shops don’t benchmark. I have always felt that there were only two possible outcomes from doing benchmarking:

  1. You benchmark and you find out that you compare very well with the rest of the benchmark population and you can now use this data as part of your report to your boss and the business to let them know the value your team in delivering
  2. You benchmark and you find out a bunch of things that you can improve. And you now have the specifics as to where and likely how to improve.

So, with the possibility of good results with either outcome, when and how should you benchmark your IT shop?

I recommend making sure that your team and the areas that you want to benchmark have an adequate maturity level in terms of defined processes, operational metrics, and cost accounting.  Usually, to take advantage of a benchmark, you should be at a minimum a Level 2 shop and preferably a Level 2+ shop where you have well understood unit costs and unit productivity. If you are not at this level, then in order to compare your organization to others, you will need to first gather an accurate inventory of assets, staff, time spent by activity (e.g., run versus change). This data should supplemented with defined processes and standards for the activity to be compared. And then for a thorough benchmark you will need data on the quality of the activity and preferably 6 month trending of all data. In essence, these are prerequisite activities that must take place before you benchmark.

I do think that many of teams that try to benchmark but then are not able to do much with the results are unable to progress because:

  • they do not have the consistent processes on which improvements and changes can be implemented
  • they do not routinely collect the base data and thus once the benchmark is over, no further data is collected to understand if any of the improvements had effect or not
  • the lack of data and standards results in so much estimation for the benchmark that you cannot then use it to pinpoint the issues

So, rather than benchmark when you are a Level 1 or 2- shop, instead just work on the basics of improving your maturity of your activities. For example, collect and maintain accurate asset data – this foundational to any benchmarking. Similarly collect how your resources spend their time — this is required anyway to estimate or allocate costs to whoever drives it, so do it accurately. And implement process definitions, base operational metrics, and have the team review and publish monthly.

For example, let’s take the Unix server area. If we are looking to benchmark we would want to check various attributes against the industry including:

  • number of servers (versus similar size firms in our industry)
  • percent virtualized
  • unit cost (per server)
  • unit productivity (servers per admin)
  • cost by server by category (e.g. staff, hardware, software, power/cooling/data center, maintenance)

By having this information you can quickly identify where you have inefficiencies or you are paying too much (e.g., the cost of your software per server) or you are unproductive (your servers per admin is low versus the industry). This then allows you to draw up effective action plans because you are addressing problems that can likely be solved (you are looking to bring your capabilities up to what is already best practice in the industry).

I recall a benchmark of the Unix server area where our staff costs per server were out of line with the industry though we had relatively strong productivity. Upon further investigation we realized we had mostly a very senior workforce (thus being paid the top end of the scale and very capable) that had held on to even the most junior tasks. So we set about improving this in several ways:

  • we packaged up and moved the routine administrative work to IT operations (who did it for far less and in a much more automated manner)
  • we put in place a college graduate program and shifted nearly all of our hires in this group from a previous focus of mid to senior only new hires to one where it was mostly graduates, some junior and only a very few mid-level engineers.
  • we also put in place better tools for the work to be done so staff could be more productive (more servers per admin)

The end result after about 12 months was a staff cost per server that was significantly below the industry median, approaching best in class (and thus we could reduce our unit allocations to the business). Even better, with a more balanced workforce (i.e., not all senior staff) we ended up with a happier team because the senior engineers were now looked up to and could mentor the new junior staff. Moreover, the team now could spend more of their time on complex change and project support rather than just run activities. And with the improved tools making everyone more productive, it resulted in a very engaged team that regularly delivered outstanding results.

I am certain that some of this would have been evident with just a review. But by benchmarking, not only were we able to precisely identify where we had opportunity, we were better able to diagnosis and prescribe the right course of action with targets that we knew were doable.

Positive outcomes like this are the rule when you benchmark. I recommend that you conduct a yearly external benchmark for each major component area of infrastructure and IT operations (e.g. storage, mainframe, server, network, etc). And at least every two years, assess IT overall, assess your Information Security function, and if possible, benchmark your development areas and systems in business terms (e.g., cost per account, cost per transaction).

One possibility in the development area is that since most IT shops have multiple sub-teams within development, you can use the operational development metrics to compare them against each other (e.g. defect rates, cost per man hour or cost per function, etc). Then the sub-team with the best metrics can share their approach so all can improve.

If you conduct regular exercises like this, and combine it with a culture of relentless improvement, you will find you achieve a flywheel effect, where each gain of knowledge and each improvement becomes more additive and more synergistic for your team. You will reduce costs faster and improve productivity faster than taking a less open and less scientific approach.

Have you seen such accelerated results? What were the elements that contributed to that progress? I look forward to hearing from you.

Best, Jim

 

 

A Continued Surge in IT Investment

    Jim D

In recent posts, I have noted previous articles on how the recovery in the US has been a ‘jobless’ recovery yet one with stronger investment in equipment and IT. This peculiar effect in the latest reporting appears to be even more pronounced, perhaps even running in ‘overdrive’. According to yesterday’s Wall Street Journal article, the investment in labor savings in the US stepped up at the beginning of the decade, but with the recession, companies found bigger opportunities to automate even more with machinery and software.  Timothy Aeppel, the author notes, this investment and spending level has continued broadly through the fourth quarter. And while certainly investment in technology will cause employment subsequently to rise, I think it appears that CEOs are investing in technology far more than adding staff as in previous recoveries. And they are doing this for a reason — they can get better returns from the machinery and robotics and software than before. Part of this is due to low interest rates and unique tax breaks, but I believe that technology is enabling greater returns on reducing costs and improving productivity than previously. Further, I think fundamental changes in IT capabilities and robotics are fueling improved returns from automation even more than has occurred in the past, spurring even greater investment in IT.

Historically, IT applications and systems were applied to domains with large amounts of the routine work, often done by hundreds if not thousands of staff. These were the areas that justified the cost of IT and provided the greatest return. Improvements in application and database technology enabled technology to tackle tougher and more complex problems. It also enabled leverage of technology on medium scale processes. Basic toolsets like email and Sharepoint tackled the least complex and departmental processes. This progress is represented in the diagram below.

The introduction of client server platforms allowed solutions to be applied to routine work on a smaller scale, to large departments and medium sized companies rather than just divisions and large corporations. This accelerated with the internet and the advent of virtualization.

But the cumulative and accelerating effect of new development technologies and methods, new toolsets, new client devices, cloud infrastructure, and advancing data and analytics capabilities has enabled a far broader range of solutions to be applied easily and effectively across a wide range of institutions and problem sets. Small and medium sized companies through cloud services can now leverage similar infrastructure capabilities that previously could only be implemented and afforded by the largest corporations. This step change in progression is represented by the chart below.

The increase of automation scope with new Tech toolsets

The new lightweight workflow tools like IBM’s Lombardi toolset and many others open up almost any departmental process to be easily and rapidly automated and achieve a decent ROI.  The proliferation of client interfaces through the internet and mobile allow customers to self serve intelligently for almost any product and service, enabling the elimination of large amounts of front office and back office work. This cumulative and compounding effect is truly a step change in what can be done by IT to automate the work within medium and large companies. And yet, despite this sea change in capabilities, you still find many IT departments focused almost wholely on their traditional scope. The projection initiation and selection process is laborious and even arduous, oriented towards doing large (and fewer) projects. The amount of overhead required to execute almost any typical project would overwhelm lightweight automation for departmental-sized efforts. And yet, there are huge new areas of scope and automation that are possible now in almost every company. And so how do you start to prove out these new areas and adapt your processes to enable them to get done?

I think there are several ways to get started here, some of which I mentioned briefly in my December post on  ‘A few things to add to your technology plans’.

1. Improve customer signup or account opening: Unless your company has redone these functions within the past 24 months, it is doubtful that this area is up to the latest expectations of customers.  Enable account opening from the web and mobile devices, leverage the app stores to provide mobile clients that have additional, useful and cool functions (nearest store, nearest ATM, or if you are a climbing gear company, the current temperature and weather at base camp on Mount Everest). Make them easy to navigate and progress through the application with  progress bar and associated menu. Ensure the physical process at the store or branch is as easy as the internet version (i.e., not twenty pages of forms). And tighten up the security with strong passwords (many sites today have a strength indicator as you type in the new password) and two factor security on critical transactions (e.g. wire transfers or bill payments). Remember you can now deliver the two factor security through the customer’s mobile device and not a separate token.

2. Fix two or three process issues that are basic transactions for customers: Just as you need to continually cycle back through your customer interfaces to keep them fresh and take advantage of the latest consumer technology, you should also need to revisit some of the basic transactions that business typically fail to put on their list to invest and yet become problematic service areas for customers. These would be areas like change of address or statement reprints or getting a replacement card. Because they are never on the project list, these services remain backwaters of process and automation with predictable frustration for the customers, high error rates, and disproportionate manual effort to complete. Work with a strong business partner (perhaps the COO or someone close to the customer experience) to tee these up. Use the latest workflow tools to tackle the process piece. Leverage the latest data warehouse and ETL capabilities to integrate the customer data across business units and applications so that the process can be once and done. If you are not sure on what basic customer process to start, then talk to the unit handling customer complaints and look for those processes that have the highest number of issues yet the customer is trying to do something quite basic. Remember, every customer complaint requires expensive responses that by eliminating, you drive material improvements in productivity and cost.

3. Implement more self-service: An oft-neglected area is the improvement of corporate support functions and their productivity and service. In a typical corporation almost every employee is touched by the HR and Fiance processes, which can be heavily manual and archaic (again they rarely show up on the project list) By working with your Finance and HR functions you can reduce their costs and improve their delivery to their users through implementation of automation and self-service. The advanced workflow toolsets (IBM’s BPM) mean you can do far more with incremental, small tiger team efforts than ever before. Your scope to automate and move to self service on your intranet is much greater. More and more minor business processes than ever before can be automated at far less cost and effort. The end results are higher productivity for your business, lower operations costs in HR and Finance, a more satisfied user base, and a better perception of IT.

4. Get to a modern production and service toolset for IT: For the past twenty years, there have been two traditional toolsets that most companies leveraged for production processes and service requests (Remedy and Peregrine). And most of us have implemented (with some struggle) reasonable implementations that met the bulk of our needs. But the latest generation of these toolsets (e.g., ServiceNow) make our previous implementations look like dinosaurs. And when you consider the 60 or 70% of your staff and service desk are using these tools every day, and you can make them far more productive with a new toolset, it is worth taking a look at. Further, your business users, will love the new IT ordering facilities on the intranet that are better than ordering a book from Amazon. By the way, the all-in operating cost of the new tools should be substantially less than your current costs for Remedy or Peregrine. And your team will be operating at a step level improvement in efficiency and productivity.

5. Get Going in Business intelligence: One last item to make sure your company is capitalizing on is leveraging the data you already have to know your customer better, improve your products or services, and reduce your costs. Why have advertising for customers who never click through or buy? Why do customers call your call center when they should be able to do it easier online? Wade through all the unstructured data being generated on social about your company to figure out how to improve your brand. Knowing the mood of the market, understanding your customers and the perspectives on your products and services requires IT to partner with the business to leverage the data you have to obtain intelligence. Investing in this area can now be tackled with the new big data tools on the market. If you are not doing much here, then I recommend finding out what your competitors are doing and sitting down with your business partners to sort through what you must do.

So, there’s 5 things that five years ago, would have never made any list. Yet if you make real progress in 3 of the 5, you can hit home runs in customer satisfaction, service quality, and a much better view of IT. And most important, you can ensure your company stays ahead of the game to achieve greater productivity and lower costs.

Any views or alternate perspectives on the progression of IT tools and solutions? Do you see the same sea change that I am calling out here for us to take advantage of?

Let me know. Best, Jim

Delivering More by Leveraging the ‘Flow of Work’

    Jim D

As you start out this year as an IT leader and you are trying to meet both project demand on one hand and savings goals on the other, remember to leverage what I term the IT ‘Flow of Work‘. Too often, once work comes into an organization, either through a new project going into production or through the original structure of the work, it is static. The work, whether it is server administration, batch job execution, or routine fixes, continues to be done by the same team that often developed it in the project cycle. Or at best, the system and its support are handed off to a dedicated run team, that continues to treat it as ‘custom’ work. In other words, once the system has been crafted by the project team, the regular work to run, maintain, and fix the system continues to be custom-crafted work.

This situation, where the work is static and continues to be executed by the initial, higher cost resources, would be analogous to a car company crafting a prototype or concept car, and then using that same team to produce every single subsequent car of that model with the same ‘craft’ approach. This of course does not happen as the car company moves the prototype to a production factory where the work is standardized and automated and leaned, and far lower cost resources execute the work. Yet in the IT industry we often fail to leverage this ‘flow of work’. We use senior server engineers to do basic server administration tasks (thus making it custom work). We fail to ‘package’ or productionalize or automate the tasks thus requiring exorbitant amounts of manual work because the project focused on delivering the feature and there was not optimization step to get it into the IT ‘factory’.

Below is a diagram that represents the flow of work that should occur in your IT organization.

Moving work to where it is most efficiently executed

The custom work, work done for new design, or complex analysis or maintenance, is the work done by your most capable and expensive resources. Yet, many IT organization waste these resources by doing custom work where it doesn’t matter. A case in point would be anywhere that you have IT projects leveraging custom designed and built servers/storage/middleware (custom work) instead of standardized templates (common work). And rarely do you see those tasks documented and automated such that they can be executed by the IT Operations team (routine work). And not only do you then waste your best resources on design that adds minimal business value, you then do not have those best resources available to do the new projects and initiatives the business needs to have done. Or similarly, your senior and high cost engineers are doing routine administrative work because that is how the work was implemented in production. Later, no investment has been made to document or package up this routine work so it can easily be done by your junior or mid-level engineering resources.

Further, I would suggest that you often find the common or routine engineering work stays in that domain. Investment is infrequently made to further shrinkwrap and automate and document the routine administrative tasks your mid-level engineers so that you can hand it off to the IT Operations staff to execute as part of their regular routine (and by the way, the Ops team typically executes these tasks with much greater precision than the engineering teams).

So, rather than fall into the traps of having static pools of work within your organization, drive and investment so that the work can be continually packaged and executed at a more optimal level and free up your specialty resources to tackle more business problems.  Set a bar for each of your teams for productivity improvements. Enable them the time and investment to package the work and send it to the optimal pool. Encourage your teams to partner with the group that will receive their work on the next step of the flow. And make sure they understand that for every bit of more routine work that they can transition to their partner team, they will receive more rewarding custom work.

After a few cycles of executing the flow of work within your organization, you will find you have gained significant capacity and reduced your cost to execute routine work substantially. This enables you to achieve a much improved ratio of run cost versus build costs by continually compressing the run costs.

I have seen this approach executed with great effect in both large and mid-sized organizations. Have you been part of or lead similar exercises? What results did you see?

Best, Jim

 

Delivering Efficiency and Cost Reduction: Long Terms Tactics Wrapup

    Jim D

In the last several posts we have covered how you should deliver IT cost reductions with long term tactics that also enable you to build capability and capacity.  If executed well, you will also yield some longer term benefits such as a better workforce balance or elimination of redundant or low value systems. These tactics require relentless leadership and focus by you. You must keep them in the forefront and ensure you are progressing on them everyday. Your patience and influence are required as well. You must obtain business support for efforts that often have a longer cycle than the business team is used to supporting.  If you are both consistent and persistent in your approach, and employ these tactics, you will make a material difference, in some cases a massive difference, that will make your business far more competitive and is simply unattainable through other approaches.

We discussed that these long term tactics are executed by first laying a groundwork for sustainable improved cost through quality. Second, you build a highly productive and well-balanced team in order to meet a world class cost profile. Throughout you leverage metrics-based, transparent framework with continuous process improvement that enables you to progress and achieve world class performance. A concurrent effort with building a high performing team is to migrate to a modern, consolidated infrastructure and well-architected core systems. We will discuss this effort today and wrap up our efficiency discussion.

Nearly every IT shop that has quality or cost problems has a proliferation of archaic systems and fragmented, legacy infrastructure. The causes of this are myriad but generally result from: poor or shifting business vision and sponsorship (e.g., when you have 3 different sales executives in 4 years, IT delivers 2 and 1/2 application systems trying to match the business variances and results in multiple systems that never do what was needed); an inability of the business to invest in a steadfast manner (e.g., IT projects run out of funding, or funding shifts, thus every year projects are half completed resulting in the old systems staying in place with the new systems going in); a stagnant IT team with poor best practices and leadership that allow multiple mediocre solutions where one good solution would suffice.

As the IT leader, you must tackle the business causes first before setting out to fix this area. Draw up a compelling vision of what the systems should provide. Research and gather the facts on the costs, time to market impacts, business productivity, feature loss and quality issues due to the current multiple mediocre systems. Map out a new process and authorization where you get the multi-year business support and sponsorship to execute the vision. Going from three mediocre overlapping business systems to one well architected systems with better quality will enable efficiency and productivity gains across the board — in your shop and in theirs. This is a vision that you should get everyone to rally around. Ensure there is a investment process that enable multiyear funding and review of major projects and conversions that also tracks to ensure the benefits are delivered. The CFO should be your partner in this. It is in his interest to see that everyone meets the costs reductions and other benefits promised. And it is in yours as well, because often you accumulate the layers of barnacles by only doing 80 or 90% of a project and not doing the final steps of decommissioning the old systems. By having a multi-year project process with benefit review or scorecarding in place, there will be far more impetus to ensure projects are completed and old systems removed.

Assuming you address the business sponsorship and project process issues, you must also address poor leadership and implementation on your side. First and foremost, IT cannot be a place for ‘hobbies’. That is, you need to ensure that all those pet technology projects and explorations that are not absolutely critical to a business capability or technology implementation that has real, near term benefits, are killed. Otherwise, they are a distraction and a multiplication of your technologies and thus costs.

Second, require every major application area and infrastructure component to map out current world and a future world where they are consolidated and at best practice. Then sit down with each team and your architectures and lay out the trajectory to get from today to best practice. Estimate the benefits from the transition. Ensure you capture the gains from your productivity and quality improvements. Then pick the top 3 to 5 areas and target them as major transformation initiatives for investment and turnaround. Garner the business support, get the approvals, make it part of your IT goals and communicate the importance to your team. And of course execute them. But also, in the other areas that mapped out their trajectories, work with them to come up with ways to make progress without major funding. Identify and execute the quick ROI projects that enable you to save within the year or almost within the year. Work with vendors to come up with creative ways to overcome the investment hurdle to move your technology to a best in class platform. And every time you can come up with efficiencies or savings elsewhere, turn around, pull the next best consolidation and replatforming project off the queue and get it going. As these projects execute and land, it will drive a virtuous cycle of additional savings and quality that will enable you to more rapidly transform as well as to shift, over time, more and more resources to the point of the spear where you are attacking the business problems and needs.

The long term tactics will take 9 to 12 months to achieve some results and 18 to 24 months to yield impacts, but once you really start executing and implementing, the compound gains can be enormous. Given IT can be a very large cost center within a corporation, IT can often one of the top contributors to cost reductions. And given IT systems availability and quality drive customer service, IT improvements can be the biggest push behind improved customer satisfaction. These are huge wins for you company. So, as you are faced with cost efficiencies demands of todays environment, leverage the near term and long term tactics described to put you and your company in a winning position.

So, we have spent nearly two months covering the very important topic of cost efficiency in IT. Every IT shop today is faced with these demands. You now have the tools to drive this far more effectively. Let me know where things have gone awry or where more detail is need to handle the complexity of your situation.

Best,

Jim

 

Delivering Efficiency and Cost Reduction: Long term tactics II

    Jim D

We have spend the past several weeks covering how to deliver efficiency and cost reduction in IT, and particularly how to do it while maintaining or improving your capabilities. In my last post we discussed some two long terms tactics to drive efficiency but also at the same time to achieve improved capability. In this post we will cover the another key long term tactics that you can leverage to achieve world class cost and performance: leveraging metrics and benchmarking.

Let’s assume you have begun to tackle the quality and team issues in your shop. One of the key weapons to enable you to make better decisions as a manager and to identify those areas that you need to focus on is having a level of transparency around the operational performance of your shop. This doesn’t mean your team must produce reams of reports for senior management or for you. If fact, that is typically wasted effort as it is as much a show as real data. What you require to obtain effective operational transparency is the regular reporting of key operational metrics as they are used by the teams themselves. You and your senior team should be reviewing the Key Process Metrics(KPMs) that are used by the operational teams to drive the processes day-to-day and week-to-week. These would include things such as change success rates by team or a summary of project statuses with regular project reporting behind each status.

The ability to produce and manage using metrics depends substantially on the maturity of your teams. You should match the metrics to be reported and leveraged to your team’s maturity with an eye for the to master their current level and move to the next level.

For example, for production metrics you should start, if a Level 1 or 2 organization, with having basic asset information and totals by type and by business along with basic change success volumes and statistics and incident volumes and statistics. A level 3 organization would have additional information on causal factors. So in addition to change or incident statistics there would be data on why (or root cause) for a failed change or a production incident. A level 4 organization would have information on managing the process improvement itself. Thus you would have information on how long root cause takes to complete, how many are getting resolved, what areas have chronic issues, etc. Consider the analogy of having information on a car. At the lowest level, we have information on the car’s position, then we add information on the car’s velocity, and then on its acceleration at the highest level.

These three levels of information (basic position and volumes, then causes and verification metrics, then patterns and improvement metrics and effects) can be applied across any IT service or component. And in fact, if applied, your management team can move from guessing what to do, to knowing what the performance and capabilities are to understanding what is causing problems to scientifically fixing the problems. I would note that many IT managers would view putting the metrics in place as a laborious time-consuming step that may not yield results. They would rather fly and direct by the seat of their pants. This is why so many shops have failed improvement programs and a lack of effective results. It is far better to take a scientific, process-oriented approach and garner repeatable results that enable your improvement programs to accelerate and redouble their impact over time.

Another advantage of this level of transparency is that both your team and your business partners can now see measurable results rather than promises or rosy descriptions. You should always seek to make key data a commodity in your organization. You should be publish the key process metrics broadly. This will reinforce your goals and the importance of quality throughout your team and you will be surprised and the additional benefits that will be gathered by employees able to now to a better job because they have visibility of the impact of their efforts.

And when you do develop key process metrics, make sure you include metrics that have business meaning. For example, why publish metrics on number of incidents or outage time (or conversely system time availability) to your business partners? While there is some merit in these data for the technical team, you should publish for the business in more appropriate terms. For availability, publish the customer impact availability (which would be the total number of transaction failed or impacted to a customer divided by the total number of transaction that typically occur or would have occurred that month) plus the number of customers impacted. If you tell the business they had 50,000 customers impacted by IT last month versus you had 98.7% system time availability, it now becomes real for them. And they will then better understand the importance of investing in quality production.

In essence, by taking a metrics-based approach, we in IT are then following in the footsteps of the manufacturing revolutions on process that has been underway for the past 60 years. Your team should know about the basics on Continuous Process Improvement and Lean. Perhaps give them some books on Deming or others. We should run our shops as a factory for all of our routine or volume services. Our businesses must compete in a lean manufacturing world, and our work is not handcraft, it should be formalized and improved to yield astounding gains that compound over 12, 24 or 36 months.

On a final note, once you have the base metrics in place, you should benchmark. You will have one of two good outcomes: either you will find out you have great performance and a leader in the industry (and you can then market this to your business sponsors) or you will find out lots of areas you must improve and you now have what to work on for the next 12 months.

 

Have you seen success by applying these approaches? How has your team embraced or resisted taking such a disciplined approach? What would you add or change?

 

Best, Jim

Delivering Efficiency and Cost Reduction: Long term tactics

    Jim D

We have spend the past several weeks covering how to deliver efficiency and cost reduction in IT, and particularly how to do it while maintaining or improving your capabilities. In my last post we discussed what are some of the recent technology industry trends that enable reduced costs at the same or improved capabilities. In this post we will cover the first two of the long term tactics that you can leverage to achieve world class cost and performance: quality and a high performance team.

The first and really most important area to tackle is quality. If a factory had a quarter of the output that it produced was defective and became scrap, it could no long compete in the lean manufacturing regime of today’s industry. Yet frequently, IT shops have defect rates of 10%, 20%, or even north of 50%. And much of the time and effort of the IT team is actually spent fixing things as opposed to new work or proper maintenance. You need to regard every defect in your shop as waste and as a cost to you and your business. You should tackle this waste wherever it occurs in your shop. It is not uncommon for more than 50% of large projects or programs to be over schedule and over budget. It is not uncommon to see shops where more than 10 or 20% of the changes result in some other problem, typically impacting production and service to the customers. These areas must be addressed with rigor.
For projects and programs, are you following a robust project methodology? Do you have proper sponsorship and governance? Are you leveraging a strong analysis and requirements management methodology and toolset? Are you taking advantage of modern methodologies that solve the tough parts of a problem first, get good prototypes out early for user review and avoid the big bang and timeliness issues of a waterfall approach? Are you giving your project managers a full toolset along with the industry training and empowerment to make the right calls? If not, then you are likely experiencing issues and delays on more than 25% and perhaps more than 50% of your projects. And that is where your money is being wasted. When a fully staffed project team is waiting for requirements signoff, or when requirements are being changed again, too late in the project cycle, you are burning project monies while the resources idle or have to do rework. By introducing a rigorous process and robust tools and metrics, you will be able to avoid most issues before they start and for those that do occur, you will know precisely why and be able to correct it for the next project.
For your production services, you should insist on at least a 98% change success rate and if you wish to be a 1st quartile shop you need to drive to a 99% or 99.5% change success rate. This means that for every 200 changes only 1 or 2 fail. These success rates can be achieved by ensuring you have an effective but not burdensome change process and you have rigorous change testing and planning (including a backout plan). Have your operations and service management experts participate in and provide guidance to those making the changes (either the application areas or the infrastructure teams). And ensure full root cause on any change that fails with the requisite actions to prevent re-occurance being completed. Publish simple and straightforward reports on change and project success and quality. By measuring quality your team will get the message and place much more emphasis on getting it right the first time. And remarkably, by focusing on quality first, you will get a strong reduction in cost (whereas if you first tried to reduce costs then to improve quality you would make little progress on either). Accompanying this thrust with simple but bedrock true messages such as ‘Do it right the first time’ and ‘Spend it like it is your own money’ go a long ways to get the spirit of what you are trying to get accomplished across. You must mean it though and you must back up doing it right, even if it costs more initially. Remember you are looking to establish a quality culture and achieve longer term returns here.

The other long term tactic we will discuss today is achieving a highly productive and well-balanced team. First, understand that while you are implementing longer term team plans you should leverage some or all of the near term tactics for staffing that I identified in the October 31st post. The long term tactics are straightforward and are based on you attaining a staffing mix that is composed of a balanced mix of top performing individuals who, for your company, are in the right geography and at critical mass sites.

Some key truths that should be recognized and understood before setting out to build such a highly productive team:

– top performing engineers, typically paid similar as their mediocre peers are not 10% better but 2x to 10x better

– having primarily only senior engineers and not a good mix of interns, graduates, junior and mid and senior level engineers will result in stagnation and overpaid senior engineers doing low level work

– having a dozen small sites with little interaction is far less synergistic and productive than having a few strategic sites with critical mass

– relying on contractors to do most of the critical or transformational work is a huge penalty to retain or grow top engineers

– line and mid-level managers must be very good people managers, not great engineers, otherwise you are likely have difficulty retaining good talent

– engineers do not want to work in an expensive in-city location like the financial district of London (that is for investment bankers)

– enabling an environment where mistakes can be made, lessons learned, and quality and innovation and initiative are prized means you will get a staff that behaves performs like that.

With these truths in mind, set about building the team by addressing your workforce strategy (sites and mix); upgrading your recruiting and performance management; and revising your goals and rewards. Then execute these relentlessly while you up the training and coaching. As this begins to bear results you will then need to prune the poor performing managers and filter out the low performing staff.

So build a workforce strategy that matches the size and scale of your company. If your company is global, you will need a global workforce. If it is domestically focused, be domestic but look at near shore engineering locations as well. Establish the proper contractor staff mix based on function (again see the near term efficiencies staff post). Ensure your locations match up to where you can draw talent. For example, minimize expensive in-city locations. Choose instead locations with good commuting, very good nearby engineering universities and vibrant nearby communities. You will be rewarded with better quality engineers and lower attrition. Do you have 12 locations each with 50 to 150 engineers? Consider consolidating to 3 or 4 sites each with 300 to 500 engineers. And one or two should be global or near shore sites. And ensure you set it up so any significant IT function is done in two locations (thus eliminating the cost for BCM and establishing opportunities for work handoff between the sites yielding faster time to market). Establish strong intern and graduate programs with the universities near your key sites. If you are overweight with senior engineers, ensure that your new hires are graduates or junior engineers (even if by mandate). As you compose the plan, engage you senior business leaders to ensure you have support for the changes and potential synergies with other business sites or locations. Ensure they understand you will always locate client-facing personnel with the client, but IT ‘back and middle office’ staff will be where it is best to recruit and retain IT talent.

Make sure your recruiting practices are up to snuff. You should be doing a thorough filtering of technical talent while also ensuring that you are getting someone who can work well with others and has the same key values of quality, initiative and responsibility that you are seeking. Leverage team interview or top-grading practices as appropriate to ensure you weed out those that do not interact well (especially management recruits or team leads). Invariably, management in a IT organization can improve how they handle performance management. Because most of the managers are engineers, their ability to interact firmly with another person in a highly constructive manner is typically under-developed. Provide classes and interactive session on how to do coaching and provide feedback to employees. Even better, insist that performance reviews must be read and signed off by the manager’s manager before being given to improve the quality of the reviews. This a key element to focus on because the line manager’s interaction with an employee is the largest factor in undesired attrition and employee engagement.

Even if you execute your workforce mix and your recruiting and performance management flawlessly and you do not align your goals and rewards (and incentives) you will not get the change you desire. Quite simply, your smartest team members, will observe what you reward, and if you do not reinforce the values you are looking to achieve (productivity, quality, initiative, etc), you will not get the changes desired. So, gather your senior team. Work together to revise and update the vision and goals of your organization. Ensure they are big enough goals (e.g., Not ‘ Save $40M in costs’ instead ‘ Become top quartile in quality and efficiency in IT for our industry and company size by 201x’). And then line up quarterly awards that are very public that reward those who exemplify the values and results you are looking to achieve. The organization will then align their efforts much more keenly to your vision and goals.

Now that you have the foundation in place for long term results, execute and improve every quarter. You will need to begin pruning. And as you add new and better capability to your organization through improved recruiting techniques, you can afford to prune those marginal managers that you couldn’t lose before. This will provide another round of lift for your team productivity as they are replaced with better internal candidates who have grasped the new values and effort or better filtered external talent.

In the next post on long term tactics, we will talk about the metrics and benchmarking you will use to ensure you stay on track an enable you to identify additional and continual improvements. But you are on your way now.

What areas would you change? What pitfalls do you see or have you encountered? Post a comment and let me know. I look forward to your feedback.

Best, Jim