Tag: IT Metrics

Using Performance Metric Trajectories to Achieve 1st Quartile Performance

    Jim D

I hope you enjoyed the Easter weekend. I have teamed up today with Chris Collins, a senior IT Finance manager and former colleague. Our final post on metrics is on unit costing — on which Chris has been invaluable with his expertise. For those just joining our discussion on IT metrics, we have had 6 previous posts on various aspects of metrics. I recommend reading the Metrics Roundup and A Scientific Approach to Metrics to catch you up in our discussion.

As I outlined previously, unit costing is one of the critical performance metrics (as opposed to operational or verification metrics) that a mature IT shop should leverage particularly for its utility functions like infrastructure (please see the Hybrid model for more information on IT utilities). With proper leverage, you can use unit cost and the other performance metrics to map a trajectory that will enable your teams to drive to world-class performance as well as provide greater transparency to your users.

For those just starting the metrics journey, realize that in order to develop reliable sustainable unit cost metrics, significant foundational work must be done first including:

  • IT service definition should be completed and in place for those areas to be unit costed
  • an accurate and ongoing asset inventory must be in place
  • a clean and understandable set of financials must be available organized by account so that the business service cost can be easily derived

 If you have these foundation elements in place then you can quickly derive the unit costing for your function. I recommend partnering with your Finance team to accomplish unit costing. And this should be an effort that you and your infrastructure function leaders champion. You should look to apply a unit cost approach to the 20 to 30 functions within the utility space (from storage to mainframes to security to middleware, etc). It usually works best to start with one or two of the most mature component functions and develop the practices and templates. For the IT finance team, they should progress the effort as follows:

  • Ensure they can easily segregate cost based on service listing for that function
  • Refine and segregate costs further if needed (e.g., are there tiers of services that should be created because of substantial cost differences?)
  • Identify a volume driver to use as the basis of the unit cost (for example, for storage it could be terabytes of allocated storage)
  • Parallel to the service identification/cost segregation work, begin development of unit cost database that allows you to easily manipulate and report on unit cost.  Specifically, the database should contain:
    • Ability to accept RC and account level assignments
    • Ability to capture expense/plan from the general ledger
    • Ability to capture monthly volume feeds from source systems including detail volume data (like user name for an email account or application name tied to a server)

For the function team, they should support the IT Finance team in ensuring the costs are properly segregated into the services they have defined. Reasonable precision of the cost segregation is required since later analysis will be for naught if the segregations are inaccurate. Once the initial unit costs are reported, the function technology can now begin their analysis and work. First and foremost should be an industry benchmark exercise. This will enable you to understand quickly how your performance ranks against competitors and similar firms. Please reference the Leveraging Benchmarkspage for best practices in this step. In addition to this step, you should further leverage performance metrics like unit cost to develop a projected trajectory for for your function’s performance. For example, if your unit cost for storage is currently $4,100/TB for tier 1 storage, then the storage team should map out what their unit cost will be 12, 24, and even 36 months out given their current plans, initiatives and storage demand. And if your target is for them to achieve top quartile cost, or cost median, then they can now understand if their actions and efforts will enable them to deliver to that future target. And if they will not achieve it, they can add measures to address their gaps.

Further, you can now measure and hold them accountable on a regular basis to achieve the proper progress towards their projected target. This can be done not just for unit cost but for all of your critical performance measures (e.g., productivity, time to market, etc).  Setting goals and performance targets in this manner will achieve far better results because a clear mechanism for understanding cause and effect between their work and initiatives and the target metrics has been established.

A broad approach to also potentially utilize is to establish a unit cost progress chart for all of your utility functions. On this chart, where the y axis is cost as a percentage of current cost and the x axis is future years, you should establish a minimum improvement line of 5% per year. The rationale behind this is that improving hardware (e.g., servers, storage, etc) and improving productivity, yield an improving unit cost tide of at least 5% a year. Thus, to truly progress and improve, your utility functions should well exceed a 5% per year improvement if they are below 1st quartile. This approach also conveys the necessity and urgency of not sitting on our laurels in the technology space. Often, with this set of performance metrics practices employed along with CPI and other best practices, you can then achieve 1st quartile performance within 18 to 24 months for your utility function.

What has been your experience with unit cost or other performance measures? Where you able to achieve sustained advantage with these metrics?

Best,

Jim Ditmore and Chris Collins

 

Tying Consumption to Cost: Allocation Best Practices

    Jim D

In 1968, Garrett Hardin wrote about the over-exploitation of common resources in an essay titled the “The Tragedy of the Commons“. While Garrett wrote about the overexploitation of common pastureland where individual herders overused and diminished common pasture, there can be a very similar effect with IT resources within a large corporation. If there is no cost associated with the usage of IT resources by different business unit, than each unit will utilize the the IT resources to maximize its potential benefit to the detriment of the corporate as a whole. Thus, to ensure effective use of the IT resources there must be some association of cost or allocation between the internal demand and consumption by each business unit. A best practice allocation approach enables business transparency of IT cost and business drivers of IT usage so that thoughtful business decisions for the company as a whole can be made with the minimum of allocation overhead and effort.

A well-designed allocations framework will ensure this effective association as well as:

  • provide transparency to IT costs and the particular business unit costs and profitability,
  • avoid wasteful demand and alter overconsumption behaviors
  • minimize pet projects and technology ‘hobbies’

To implement an effective allocations framework there are several foundation steps. First, you must ensure you have the corporate and business unit CFOs’ support and the finance team resources to implement and run the allocations process. Generally, CFOs look for greater clarity on what drives costs within the corporation.  Allocations allow significant clarity on IT costs which are usually a good-sized chunk of the corporation’s costs.  CFOs are usually highly supportive of a well-thought out allocations approach. So, first garner CFO support along with adequate finance resources.

Second, you must have a reasonably well-defined set of services and an adequately accurate IT asset inventory. If these are not in place, you must first set about defining your services (e.g. and end user laptop service that includes laptop, OS, productivity software, and remote access or a storage service of high performance Tier 1 storage by Terabyte) and ensuring your inventory of IT assets is minimally accurate (70 to 80 %). If there are some gaps, they can be addressed by leveraging a trial allocation period where numbers and assets are published, no monies are actually charged, but every business unit reviews its allocated assets with IT and ensures it is correctly aligned. Once you have the service defined and the assets inventoried, your finance team must then set about to identify which costs are associated with which services. They should work closely with your management team to identify a ‘cost pool’ for each service or asset component. Again, these costs pools should be at least reasonably accurate but do not need to be perfect to begin a successful allocation process.

The IT services defined should be as readily understandable as possible. The descriptions and missions should not be esoteric except where absolutely necessary. They should be easily associated with business drivers and volumes (such as number of employees, or branches, etc) wherever possible.  In essence, all major categories of IT expenditure should have an associated service or set of services and the services should be granular enough so that each service or component can be easily understood and each one’s drivers should be easily distinguished and identified. The targets should should be somewhere between 50 and 150 services for the typical large corporation.  More services than 150 will likely lead to more effort being spent on very small services and result in too much overhead. Significantly, less than 50 services could result in clumping of services that are hard to distinguish or enable control. Remember the goal is to provide adequate allocations data at the minimum effort for effectiveness.

The allocations framework must have an overall IT owner and a senior Finance sponsor (preferably the CFO). CFOs want to implement systems that encourage effective corporate use of resources so they are a natural advocate for a sensible allocation framework. There should also be a council to oversee the allocation effort and provide feedback and direction where majors users and the CFO or designate are on the council. This will ensure both adequate feedback as well as buy-in and support for successful implementation and appropriate methodology revisions as the program grows. As the allocations process and systems mature, ensure that any significant methodology changes are reviewed and approved by the allocation council with sufficient advance notice to the Business Unit CFOs. My experience has been that everyone agrees to a methodology change if it is in their favor and reduces their bill, but everyone is resistant if it impacts their business unit’s finances regardless of how logical the change may be. Further, the allocation process will bring out intra business unit tensions toward each other, especially for those that have an increase versus those that have a decrease, if the process is not done with plenty of communication and clear rationale.

Once you start the allocations, even if during a pilot or trial period, make sure you are doing transparent reporting. You or your leads should have a monthly meeting with each business area with good clear reports. Include your finance lead and the business unit finance lead in the meeting to ensure everyone is on the same financial page.  Remember, a key outcome is to enable your users to understand their overall costs, what the cost is for each services and, what business drivers impact which services and thus what costs they will bear. By establishing this linkage clearly the business users will then look to modify business demand so as to optimize their costs. Further, most business leaders will also use this allocations data and new found linkage to correct poor over-consumption behavior (such as users with two or three PCs or phones) within their organizations. But for them to do this you must provide usable reporting with accurate inventories. The best option is to enable managers to peruse their costs through an intranet interface for such
end-user services such as mobile phones, PCs, etc . There should be readily accessible usage and cost reports to enable them to understand their team’s demand and how much each unit costs.  They should have the option right on the same screens to discontinue, update or start services. In my experience, it is always amazing that once leaders understand their costs, they will want to manage them down, and if they have the right tools and reports, managing down poor consumption happens faster than a snowman melting in July — exactly the effect you were seeking.

There are a few additional caveats and guides to keep in mind:

  • In your reporting, don’t just show this month’s costs, show the cost trend over time and provide a projection of future unit costs and business demand
  • Ensure you include budget overheads in the cost allocation, otherwise you will have a budget shortfall and neglect key investment in the infrastructure to maintain it.
  • Similarly, make sure you account for full lifecycle costs of a service in the allocation — and be conservative in your initial allocation pricing, revisions later that are upward due to missed costs will be painful
  • For ‘build’ or ‘project’ costs, do not use exact resource pricing. Instead use an average price to avoid the situation where every business unit demands only the lowest cost IT  resources for their project resulting in a race to the bottom for lowest cost resources and no ability to expand capacity to meet demand since these would be high cost resources on the margin.
  • Use allocations to also avoid First-In issues to new technologies (set the rate at the project volume rate not the initial low volume rate) and to encourage transition off of expensive legacy technologies (Last out increases)
  • And lastly, and ensure your team knows and understands their services and their allocations and can articulate why what costs what they cost

With this framework and approach, you should be able to build and deliver an effective allocation mechanism that enables the corporation to avoid the overconsumption of free, common resources and properly direct the IT resources to where the best return for the corporation will be. Remember though that in the end this is an internal finance mechanism so the CFO should dictate the depth, level and allocation approach and you should ensure that the allocations mechanism does not become burdensome beyond its value. remember that allocations framework.

What have been your experiences with allocations frameworks? What changes or additions to these best practices would you add?

Best, Jim Ditmore

 

Evolving Metrics to Match Your Team’s Maturity

    Jim D

We have covered quite a bit of ground with previous posts on IT metrics but we have some important additions to the topic. The first, that we will cover today, is how to evolve your metrics to match your team’s maturity. (Next week, we will cover unit costs and allocations).

To ground our discussion, let’s first cover quickly the maturity levels of the team. Basing them heavily on the CMM, there are 5 levels:

  1. Ad hoc: A chaotic state with no established processes. Few measures are in place or accurate.
  2. Repeatable: The process is documented sufficiently and frequently used. Some measures are in place.
  3. Defined: Processes are defined and standard and highly adhered. Measures are routinely collected and analyzed.
  4. Managed: Processes are effectively controlled through the use of process metrics with some continuous process improvement (CPI).
  5. Optimized: Processes are optimized with statistical and CPI prevalent across all work.

It is important to match your IT metrics to the maturity of your team for several reasons:

  • capturing metrics which are beyond the team’s maturity level will be difficult to gather and likely lack accuracy
  • once gathered, there is potential for unreliable analysis and conclusions
  • and it will be unlikely that actions taken can result in sustained changes by the team
  • the difficulty and likely lack of progress and results can cause the team to negatively view any metrics or process improvement approach

Thus, before you start your team or organization on a metrics journey, ensure you understand their maturity so you can start the journey at the right place. If we take the primary activities of IT (production, projects, and routine services), you can map out the evolution of metrics by maturity as follows:

Production metrics – In moving from a low maturity environment to a high maturity, production metrics should evolve from typical inward-facing, component view measures of individual instances to customer view, service-oriented measures with both trend and pattern view as well and incident detail. Here is a detailed view:

Production Metrics Evolution

 

Project metrics – Measures in low maturity environments are project-centric usually focus on date and milestone with poor linkage to real work or quality. As the environment matures, more effective measures can be implemented that map actual work and quality as the work is being completed and provide accurate forecasts of project results. further, portfolio and program views and trends are available and leveraged.

Project Metrics Evolution

 

Routine Services – Low maturity measures are typically component or product-oriented at best within a strict SLA definition and lack a service view and customer satisfaction perspective. Higher maturity environments address these gaps and leverage unit costs, productivity, and unit quality within a context of business impact.

Routine Services Metrics Evolution

The general pattern is that as you move from low to medium and then to high maturity: you introduce process and service definition and accompanying metrics; you move from task or single project views to portfolio views; quality and value metrics are introduced and then exploited; and a customer or business value perspective becomes the prominent measure as to delivery success. Note that you cannot just jump to a high maturity approach as the level of discipline and understanding must be built over time with accumulating experience for the organization. To a degree, it is just like getting fit, you must go to the gym regularly and work hard – there is nothing in a bottle that will do it for you instead.

By matching the right level of metrics and proper next steps to your organization’s maturity, you will be rewarded with better delivery and higher quality, and your team will be able to progress and learn and leverage the next set of metrics. You will avoid a ‘bridge too far’ issue that often occurs when new leaders come into an organization that is less mature than their previous one, yet they impose the exact same regimen as they were familiar with previously. And then they fail to see why there are resultant problems and the blame either falls on the metrics framework imposed or the organization, when it is neither… it is the mismatch between the two.

And you will know your team has successfully completed their journey when they go from:

  • Production incidents to customer impact to ability to accurately forecast service quality
  • Production incidents to test defects to forecasted test defects to forecasted defects introduced to production
  • Unit counts of devices to package offerings to customer satisfaction
  • Unit counts or tasks to unit cost and performance measures to unit cost trajectories and performance trajectories

What has been your experience applying a metrics framework to a new organization? How have you adjusted it to ensure success with the new team?

Best, Jim Ditmore

Metrics Roundup: Key Techniques and References

    Jim D

As an IT leader either of a function or of a large IT team with multiple functions, what is the emphasis you should place on metrics and how are you able to leverage them to attain improvement and achieve more predictable delivery? What other foundational elements must be in place for you to effectively leverage the metrics? Which metrics are key measures or leading indicators and which ones are lagging or less relevant?

For those of you just joining, this is the fourth post on metrics and in our previous posts we focused on key aspects of IT metrics (transparency, benchmarking, and a scientific approach). You can find these posts in the archives or, better yet, in the pages linked under the home page menu of Best Practices. The intent of the RecipeforIT blog is to provide IT leaders with useful, actionable practices, techniques and guidance to be more successful IT leaders and enable their shops to achieve much higher performance. I plan to cover most of the key practices for IT leaders in my posts, and as a topic is covered, I try to migrate the material into Best Practices pages. So, now back to metrics.

In essence there are three types of relevant metrics:

  • operational or primary metrics – those metrics used to monitor, track and decision the daily or core work. Operational metrics are the base of effective management and are the fundamental measures of the activity being done. It is important that these metrics are collected inherently as part of the activity and best if the operational team  collects, understands and changes their direction based on these metrics.
  • verification or secondary metrics – those metrics used to verify that the work completed meets standards or is functioning as designed. Verification metrics should also be collected and reviewed by the same operational team though potentially by different members of the team or as participation as part of a broader activity (e.g. DR test). Verification metrics provide an additional measure of either overall quality or critical activity effectiveness.
  • performance or tertiary metrics – those metrics that provide insight as to the performance of the function or activity. Performance metrics enable insight as to the team’s efficiency, timeliness, and effectiveness.

Of course, your metrics for a particular function should consist of those measures needed to successfully execute and manage the function as well as those measures that demonstrate progress towards the goals of your organization. For example, let’s take am infrastructure function: server management. What operational metrics should be in place? What should be verified on a regular basis? And what performance metrics should we have? While this will vary based on the maturity, scale, and complexity of the server team and environment, here is a good subset:

Operational Metrics:

  • Server asset counts (by type, by OS, by age, location, business unit, etc) and server configurations by version (n, n-1, n-2, etc) or virtualized/non-virtual and if EOL or obsolete
  • Individual, grouped and overall server utilization, performance, etc by component (CPU, memory, etc)
  • Server incidents, availability, customer impacts by time period, trended with root cause and chronic or repeat issues areas identified
  • Server delivery time, server upgrade cycle time
  • Server cost overall and by type of server, by cost area (admin, maintenance, HW, etc) and cost by vendor
  • Server backup attempt and completion, server failover in place, etc

Verification metrics:

  • Monthly sample of the configuration management database server records for accuracy and completeness, Ongoing scan of network for servers not in the configuration management database, Regular reporting of all obsolete server configs with callouts on those exceeding planned service or refresh dates
  • Customer transaction times, Regular (every six months) capacity planning and performance reviews of critical business service stacks including servers
  • Root cause review of all significant impacting customer events, auto-detection of server issues versus manual or user detection ratios
  • DR Tests, server privileged access and log reviews, regular monthly server recovery or failover tests (for a sample)

Performance metrics:

  • Level of standardization or virtualization, level of currency/obsolescence
  • Level of customer impact availability, customer satisfaction with performance, amount of headroom to handle business growth
  • Administrators per server, Cost per server, Cost per business transaction
  • Server delivery time, man hours required to deliver a server

Obviously, if you are just setting out, you will collect on some of these metrics first. As you incorporate their collection and automate the work and reporting associated with them you can then tackle the additional metrics. And you will vary them according to the importance of different elements in your shop. If cost is critical, then reporting on cost and efficiency plays such as virtualization will naturally be more important. If time to market or availability are critical, than those elements should receive greater focus. Below is a diagram that reflects the construct of the three types of metrics and their relationship to the different metrics areas and score cards:

Metrics Framework

So, you have your metrics framework, what else is required to be successful leveraging the metrics?

First and foremost, the culture of your team must be open to alternate views and support healthy debate. Otherwise, no amount of data (metrics) or facts will enable the team to change directions from the party line. If you and your management team do not lead regular, fact-based discussions where course can be altered and different alternatives considered based on the facts and the results, you likely do not have the openness needed for this approach to be successful. Consider leading by example here and emphasize fact based discussions and decisions.

Also you must have defined processes that are generally adhered. If your group’s work is heavily ad hoc and different each time, measuring what happened the last time will not yield any benefits. If this is the case, you need to first focus on defining even at a high level, the major IT processes and help your team’s adopt them. Then you can proceed to metrics and the benefits they will accrue.

Accountability, sponsorship and the willingness to invest in the improvement activities are also key factors in the speed and scope of the improvements that can occur. As a leader you need to maintain a personal engagement in the metrics reviews and score card results. They should into your team’s goals and you should monitor the progress in key areas. Your sponsorship and senior business sponsorship where appropriate will be major accelerators to progress. And hold teams accountable for their results and improvements within their domain.

How does this correlate with your experience with metrics? Any server managers out there that would have suggestions on the server metrics? I expect we will have two further posts on metrics:

  • a post on how to evolve the metrics you measure as you increase the maturity and capability of your team,
  • and one on unit costing and allocations

I look forward to your suggestions.

Best, Jim

 

A Scientific Approach to IT Metrics

    Jim D

In order to achieve a world class or first quartile performance, it is critical to take a ‘scientific’ approach to IT metrics. Many shops remain rooted in ‘craft’ approaches to IT where techniques and processes are applied in an ad hoc manner to the work at hand and little is measured. Or, a smattering of process improvement methodologies (such as Six Sigma or Lean) or development approaches (e.g., Agile) are applied indiscriminately across the organization. Frequently then, due to a lack of success, the process methods or metrics focus are then tarred as being ineffective by managers.

Most organizations that I have seen that were mediocre performers typically have such craft or ad hoc approaches to their metrics and processes. And this includes not just the approach at the senior management level but at each of the 20 to 35 distinct functions that make up an IT shop (e.g., Networking, mainframes, servers, desktops, service desk, middleware , etc, and each business-focused area of development and integration). In fact, you must address the process and metrics at each distinct function level in order to then build a strong CIO level process, governance and metrics. And if you want to achieve 1st quartile or world-class performance, a scientific approach to metrics will make a major contribution. So let’s map out how to get to such an approach.

1) Evaluate your current metrics: You can pick several of the current functions you are responsible for and evaluate them to see where you are in your metrics approach and how to adjust to apply best practices. Take the following steps:

  • For each distinct function, identify the current metrics that are routinely used by the team to execute their work or make decisions.
  • Categorize these metrics as either operational metrics or reporting numbers. If they are not used by the team to do their daily work or they are not used routinely to make decisions on the work being done by the team, then these are reporting numbers. For example, they may be summary numbers reported to middle management or reported for audit or risk requirements or even for a legacy report that no one remembers why it is being produced.
  • Is a scorecard being produced for the function? An effective scorecard would have quantitative measures for the deliverables of the functions as well as objective scores for function goals that have been properly cascaded for the overall IT goals

2) Identify gaps with the current metrics: For each of IT functions there should be regular operational metrics for all key dimensions of delivery (quality, availability, cost, delivery against SLAs, schedule). Further, each area should have unit measures to enable an understanding of performance (e.g., unit cost, defects per unit, productivity, etc). As an example, the server team should have the following operational metrics:

    • all server asset inventory and demand volumes maintained and updated
    • operational metrics such as server availability, server configuration currency, server backups, server utilization should all be tracked
    • also time to deliver a server, total server costs, and delivery against performance and availability SLAs should be tracked
    • further secondary or verifying metrics such as server change success, server obsolescense, servers with multiple backup failures, chronic SLA or availability misses, etc should be tracked as well
    • function performance metrics should include cost per server (by type of server), administrators per server, administrator hours to build a server, percent virtualized servers, percent standardized servers, etc should also be derived

3) Establish full coverage: By comparing the existing metrics against the full set of delivery goals, you can quickly establish the appropriate operational metrics along with appropriate verifying metrics. Where there are metrics missing that should be gathered, work with the function to incorporate the additional metrics into their daily operational work and processes. Take care to work from the base metrics up to more advanced:

    • start with base metrics such as asset inventories and staff numbers and overall costs before you move to unit costs and productivity and other derived metrics
    • ensure the metrics are gathered in as automated a fashion as possible and as an inherent part of the overall work (they should not be gathered by a separate team or subsequent to the work being done

Ensure that verifying metrics are established for critical performance areas for the function as well. An example of this for the server function could be for the key activity of backups for a server:

    • the operational metrics would be perhaps backups completed against backups scheduled
    • the verifying metric would be twofold:
      • any backups for a single server that fail twice in a row get an alert and an engineering review as to why they failed (typically, for a variety of reasons 1% or fewer of your backups will fail, this is reasonable operational performance. But is one server does not get a successful backup for many days, you are likely putting the firm at risk if there is a database or disk failure, thus the critical alert)
      • every month or quarter, 3 or more backups are selected at random, and the team ensures they can successfully recover from the backup files. This will verify everything associated with the backup is actually working.

4) Collect the metrics only once: Often, teams collect similar metrics for different audiences. The metrics that they use to monitor for example configuration currency or configuration to standards, can be mostly duplicated by risk data collected against security parameter settings or executive management data on a percent server virtualization. This is a waste of the operational team’s time and can lead to confusing reports where one view doesn’t match another view. I recommend that you establish and overall metrics framework that includes risk and quality metrics as well as management and operational metrics so that all groups agree to the proper metrics. The metrics are then collected once, distilled and analyzed once, and congruent decisions can then be made by all groups. Later this week I will post a recommended metrics framework for a typical IT shop.

5) Drop the non-value numbers activity: For all those numbers that were identified as being gathered for middle management reports or for legacy reports with an uncertain audience; if there is no tie to a corporate or group goal, and the numbers are not being used by the function for operational purposes, I recommend to stop collecting the numbers and stop publishing any associated reports. It is non-value activity.

6) Use the metrics in regular review: At both the function team level and function management level the metrics should be trended, analyzed and discussed. These should be regular activities: monthly, weekly, and even daily depending on the metrics. The focus should be on how to improve, and based on the trends are current actions, staffing, processes, etc, enabling the team to improve and be successful on all goals or not. A clear feedback loop should be in place to enable the team and management to identify actions to take place to correct issues apparent through the metrics as quickly and as locally as possible. This gives control of the line to the team and the end result is better solutions, better work and better quality. This is what has been found in manufacturing time and again and is widely practiced by companies such as Toyota in their factories.

7) Summarize the metrics from across your functions into a scorecard: Ensure you identify the key metrics within each function and properly summarize and aggregate the metrics into an overall group score card. Obviously the score card should match you goals and key services that you deliver. It may be appropriate to rotate in key metrics from a function based on visibility or significant change. For example, if you are looking to improve overall time to market(TTM) of your projects, it may be appropriate to report on server delivery time as a key subcomponent and hopefully leading indicator of your improving TTM.  Including on your score card, even at a summarized level, key metrics from the various functions, will result in greater attention and pride being taken in the work since there is a very visible and direct consequences. I also recommend that on a quarterly basis, that you provide an assessment as to the progress and perhaps highlights of the team’s work as reflected in the score card.

8 ) Drive better results through proactive planning: The team and function management, once the metrics and feedback loop are in place, will be able to drive better performance through ongoing improvement as part of their regular activity. Greater increases in performance may require broader analysis and senior management support. Senior management should do proactive planning sessions with the function team to enable greater improvement to occur. The assignment for the team should be how take key metrics and what would be required to set them on a trajectory to a first quartile level in a certain time frame. For example, you may have both a cost reduction goal overall and within the server function there is a subgoal to achieve greater productivity (at a first quartile level)  and reduce the need for additional staff. By asking the team to map out what is required and by holding a proactive planning session on some of the key metrics (e.g. productivity) you will often identify the path to meet both local objectives that also contribute to the global objectives. Here, in the server example, you may find that with a moderate investment in automation, productivity can be greatly improved and staff costs reduced substantially. Thus both objectives could be obtained by the investment.  By holding such proactive sessions, where you ask the team to try and identify what would needs to be done to achieve a trajectory on their key metrics as well as considering what are the key goals and focus at the corporate or group level, you can often identify such doubly beneficial actions.

By taking these steps, you will employ a scientific approach to your metrics. If you add a degree of process definition and maturity, you will make significant strides to controlling and improving your environment in a sustainable way. This will build momentum and enable your team to enter a virtuous cycle of improvement and better performance. And then if add to the mix process improvement techniques (in moderation and with the right technique for each process and group), you will accelerate your improvement and results.

But start with your metrics and take a scientific approach. In the next week, I will be providing metrics frameworks that have stood well in large, complex shops along with templates that should help the understanding and application of the approach.

What metrics approaches have worked well for you? What keys would you add to this approach? What would you change?

Best, Jim

IT Transparency: A key approach to delivering value and ensuring focus

    Jim D

Frequently, IT shops have a difficult time convincing their business users of the value of IT. It is straightforward for a businessperson to look at the cost of IT, which is readily available but not have a good reference for the benefits and the overall value of IT. This lack of a good reference then leads to further concerns on IT budgets and investment. As CIO, it is imperative that you provide transparency on the benefits and impacts of IT so your customer and the business team can understand the value of IT and properly allocate budget and investment to make the business more successful. Yet frequently, IT does a very poor job of measuring IT impact and benefits, and when it is measured it is often done only in technical metrics which are not easily translated into understandable business metrics.

A good area to start to provide transparency to your business partners is with your production services. Typical IT metrics here are done only in technical terms not in business terms. For example, you often find availability measured in outage time or percentage or as the number of incidents per month. And while these are useful metrics to do some technical analysis, they are poor metrics to communicate business impact. Instead you should be providing your production metrics to your customer in business form. For example, you should report on the key service channels that are delivered to the end customer (e.g., ATMs, retail points of sale, call centers, etc) and the underlying metric should be customer impact availability. You derive this metric by counting the number of customer interaction or transactions that were successful divided by the total number of possible customer interactions or transactions for that time period.

Specifically, take an ATM channel as the example. Let’s assume there are normally 100,000 customer transactions in that month. If the ATMs were down for 1 hour that month and there would have been 1000 customer transactions that normally would have been completed in that hour then the customer impact availability was 99% (= (100,000 -1,000)/100,000). Just as importantly, your business also knows 1,000 customers were impacted by the systems issue. And if the outage occurred at peak usage — say 1 hour on a Friday evening rather than 3 AM on a Sunday, you may have 2,000 customers impacted versus 100. And your business-oriented metric, customer impact availability, would show this rather than the constant view that a system time availability metric would show. But this is critical knowledge, for you and for your business partners. You will have to collect information on how your customers use your service channels and understand daily, weekly, and seasonal variations, but you should know this anyway so you can better plan capacity upgrades and implement changes and releases.

You should apply the same criteria to your project delivery. Typically, IT shops only provide on-time and on-budget metrics. And these are often flawed because they either accept major changes through the project lifecycle (what I call ‘drawing the bullseye after the arrow is shot’) and they take no account of whether the project is ‘on-function’ or on-quality. Even worse, few shops track if the promised benefits were delivered or the expected revenue materialized. Thus making it very hard to assess if you are getting the expected return on investment. To remedy this, partner with the CFO office and tackle the large projects and big investments first. Insist on tracking with the CFO’s office the original cost, timeframe, benefits and and function and quality targets for this subset of the projects. If there is a major change to scope or requirements, note the change and track the additional revised cost, time, quality, etc. Then at project closeout, the project is not completed without a full assessment on how well you met the original estimates and any lessons learned on why not. It may also require a revisit of the benefits and revenues 6 months down the road to properly assess them. This information is extremely valuable to then have for assessing future investments. Is the business consistently over-estimating the revenue that will result? Is your team often underestimating the time to get projects completed? At the end of the year, you will be able to communicate not just these trends back to business but also you can outline that for X spend on large projects you delivered Y in benefits (with Finance’s backing). That should provide a compelling value statement if the projects were well-chosen. And if not, you now have the rationale to adjust the company’s investment process.

Similarly, for the ‘routine services’ of IT (delivering small systems, help desk services, desktop support, etc) you should provide metrics that are business relevant.  Provide the cost of the routine services as a cost per user (e.g. cost for desktop service per user per year) or cost per service (cost of a help desk call).  Go further and enable the business team to understand the impact of investment in the services by defining the number of staff hours of work saved. If you are implementing self service (e.g. password resets as an example) you may have a cost save (reduced help desk calls and thus fewer help desk staff, but you should really have reduced staff time to reset and less staff downtime as a result (improving staff productivity and engagement). Similarly, a new departmental workflow system should reduce both staff hours to process say an HR transaction but also substantially reduce the wall time of that transaction (again driving improved productivity and engagement). These are often neglected areas within a corporation that by translating into business metrics (staff hours saved, wall time reductions) will make visible the business benefits.

By routinely expressing what IT does in terms of business metrics, you will enable you business partners to understand the impact and value of IT on their customers and services. It will drive better decisions on how much and where to invest in technology. And it will also raise the level of business awareness of your team. When they know an outage impacted 32,000 customers, that comes across as much more material and important than a 40 minute router outage. You can adjust or denote the metrics as appropriate for your business. For example, if you are handling large financial sums then system outages may be expressed in terms of the sum of the financial amounts delayed or not processed. Further, several large Financial Services firms with robust shops report not just the customer impacts but also correlate if there was an industry cause and impact as well as if a portion of the functionality was available (e.g. you could access the ATM and withdraw cash but not see your balance).  The key is to put your primary measure in business relevant metrics.

What business relevant metrics have you used with success? What hurdles have you encountered?

I look forward to your thoughts and success with more transparent metrics.

Best, Jim