Optimizing Technology Infrastructure with Master Craftsmen

July 3, 2012 Jim D

One of my former colleagues, Ralph Bertrum, has provided the primary material for today’s post on how to optimize a technology infrastructure with master craftsmen. Ralph is one of these master craftsmen in the mainframe infrastructure space. If you are a CIO or senior IT leader looking to improve your shop’s cost or performance, I recommend optimizing your infrastructure and systems through high payback platform efficiency reviews.

In today’s shops, often with development and coding partially or fully outsourced, and not enough experienced and capable resources on staff, many applications are built for functionality without much regard for efficiency. And nearly every shop has legacy applications where few engineers, if any, actually understand how they work. These applications have often been patched and extended that just to have them run is viewed as the goal, rather than run effectively and efficiently. The result is that for most shops, there is a build up of 10, 20, or even 30% of their compute and storage capacity that is wasted on inefficient systems. This happens partly because it is easiest to just throw hardware at a problem and partly because they do not have the superior engineering resources — or master craftsmen — required to locate, identify, and resolve such inefficiencies. Yet, it is a tremendous waste and a major recurring cost for the IT shop. It is a significant opportunity for IT leaders to attack these inefficiencies. In my experience, every one of these master craftsmen, if given the framework and support, can easily return 4 to 10 times their annual compensation in savings each quarter!

So, how do you go about building and then leveraging such an efficiency engineering capability? First, to build the capability, you must be willing to invest in select engineers that are heavily dedicated to this work. I recommend focusing on mainframe efficiency and server efficiency (Unix and Windows) as the primary areas of opportunity. Given the different skill sets, you should establish two separate efficiency teams for these two areas. Storage usage should be reviewed as a part of each team’s mission. A small team of two to four individuals is a good starting point. You can either acquire experienced talent or build up by leveraging promising engineers on staff and augmenting with experienced contractors until your staff have attained full capability. Ensure you invest in the more sophisticated tools needed to instrument the systems and diagnose the issues. And importantly, ensure their recommend application and systems changes are treated with priority and implemented so the savings can be achieved. A monthly report on the recommendations and results completes the building the team and framework.

Now for the approach, which Ralph Bertrum, an experienced (perhaps even an old school) efficiency engineer has provided for the mainframe systems:

Having spent 50 years in Information Technology working on Mainframe Computers, I have seen a great many misunderstandings. The greatest single misunderstanding is the value and impact of system engineering training and experience and it’s use in performing maintenance on a very costly investment. Many CIOs prefer to purchase a computer engine upgrade and continue to run a wasteful collection of jobs on a new faster machine. It is the easiest way out but definitely not the most cost effective. It is the equivalent to trading in your car every time the air filter, spark plugs or hoses need changing or the tires need air and then moving the old air filter, old spark plugs, old hoses and old tires to the new car.

Would you drive around with a thousand pound bag of sand in the trunk of your car? Would you pull a thousand pound anchor down the street behind your car? That is exactly what you are doing when you don’t regularly review and improve the Job Control Language (JCL), Programs, and Files that run on your Mainframe Computer. And would you transfer that thousand pound bag of sand and that anchor to your new car every time you purchased a new one? Most IT shops are doing that with every new mainframe upgrade they make. Every time they upgrade their computer they simply port all the inefficiencies to the upgrade.

Platform efficiency reviews will reduce waste impacting all kinds of resources: CPU, storage, memory, I/O, networks. And the results will make the data center greener and reduce electricity bills and footprints of equipment, speed online and batch processing, eliminate or delay the need for upgrades, reduce processing wall times and cycle time, and ultimately improve employee efficiency, customer satisfaction and company profitability.

You can apply platform efficiency reviews to any server but let’s use the mainframe as a primary example. And, we will extend the analogy to a car because there are many relevant similarities.

Both automobiles and computers have a common need for maintenance. An automobile needs to have the oil, the air filter, and spark plugs changed, tires rotated and the tire air pressure checked. All of these are performed regularly and save a large amount of gas over the useful life of the automobile and extend the life of the car. Reasonable maintenance on a car can improve mileage by three to four miles per gallon or about a 20% improvement. When maintenance is not performed the gas mileage begins to degrade and the automobile becomes sluggish, loses its reliability and soon will be traded in for a newer model. The sand is growing in weight every day and the anchor is getting heavier.

For a mainframe, the maintenance is not just upgrading the systems software to the most recent version. Additional maintenance work must be done to the application software and its databases. The transactions, files, programs, and JCL must be reviewed, adjusted and tuned in order to identify hot spots, inefficient code and poor configurations that have been introduced with application changes or additional volume or different mixes. Over the last twenty five years I have analyzed and tuned millions of Mainframe Computer Jobs, Files, JCL, and Programs for more than one thousand major data centers and all of them were improved. I have never seen a Mainframe computer that couldn’t have its costs reduced by at least 10% to 15% and more likely 20% through a platform efficiency review.

Often, there are concerns that doing such tuning can introduce issues. Adjusting JCL or file definitions is just as safe as changing a spark plug or putting air in a tire. It is simple and easy and does not change program logic. The typical effect is that everything runs better and costs less. The best thing about maintenance in a data center is that almost all the maintenance lasts much longer than it does in an automobile and stays in effect with continued savings in upgrade after upgrade, year after year.

Think of maintenance of a Mainframe Computer as a special kind of motor vehicle with thousands of under-inflated tires. By making simple adjustments you can get improvements in performance from every one of these under-inflated tires. And even though each improvement is small in total, because there so many, you multiply the improvement to get significant effect. You get this cost reduction every time the file is used or transaction executed and when all the savings from all the little improvements are added together you will get a 15% to 20% reduction in processing costs. The maintenance is a onetime cost that will pay for itself over and over upgrade after upgrade.

Here are some areas to focus for performance improvement with examples:

Avoid excessive data retention: Many IT shops leave data in a file long after its useful life is over with or process data that is not meaningful. An example would be Payroll records for an employee no longer with the company, General Ledger transactions from previous years, or inventory parts that are no longer sold. By removing these records from the active file, and saving them in separate archive storage, you are saving CPU every time the file is used and work may complete much faster. For example, an IT shop had an Accounts Receivables file that had 14 million records. Every day they would run the file through a billing program that produced and mailed invoices. At that time the cost of a stamp was $0.32 cents for a first class postage stamp. A recommendation was made to the CFO that they purge all billing amounts of $0.32 cents or less from the billing file. It was silly to pay $ 0.32 to collect $0.32 or less. Two million records were removed from the file, the daily job ran four hours faster and they saved $35,000.00 a month on CPU and DASD space to say nothing about employee time and postage costs. After a trial test period the minimum billing amount was raised to $1.00 and another set of very large savings was accomplished.

Optimize databases for their use. An IT shop was looking to reduce the run time of a mailing list label system. After looking at the data it was found that 90% of the labels were located in California and that a table looked up the city and state from a zip code table. Each time the program needed a California city name, the program had to do ninety thousand zip code table compares before finding the correct city and state for the address. The table was rearranged to optimize searching for California zip codes and the job went from running twenty hours to running only one hour. CPU dropped by over 90%. This has also worked with online transaction tables and file placement in Local Shared Resource (LSR) buffer pools. Optimizing databases is a key improvement technique.

Optimize the infrastructure configuration for your system. One shop had jobs that would run very quickly one day and very slowly the next day. After analyzing the jobs and the file locations it was determined that the public storage pools contained two different types of disk drives. The Temporary work files would be placed on different disk drives every day. What was the very best setting for one type of disk was the very worst for the other and this was causing the erratic behavior. The storage pool was changed to contain only one type of disk device. The problem went away and the jobs ran fast every day.

Tune your systems to match your applications: A mainframe comes with a great many abilities and features, but if your team are not adept with them your systems will not be optimized to run well. I have analyzed over 1,000 data centers and applications and never once failed to discover significant tuning that could be accomplished within the existing system configuration. This occurs because of a lack of training or experience or focus. Ensure your team places that as a priority and if needed, bring in experts to adopt best practices. As an example of system tuning, I worked with a shop that had an online file accessing a data base and was having a major response time problem. They were afraid they were going to need a very costly upgrade. Every time they entered a transaction the system would go into a 110% utilization mode with paging. An efficiency analysis was conducted and a system file was discovered that was doing sixteen million Input Output (I/O) instructions a day. After working with IBM to optimize the configuration, we achieved a 50% drop in I/O to eight million per day and response time improved to less than one second. Apparently the shop had installed the system file as it was delivered and never modified it for their environment.

Tune your configurations to match your hardware. When you make a hardware change be sure to make all the necessary other software changes as well. Last year I worked with a very large bank that upgraded their disk drives but forgot to change a System Managed Storage (SMS) storage pool definition and continued to run forty five thousand monthly jobs using the worst blocksize possible in two thousand five hundred files. When found and corrected, the forty five thousand jobs ran 68% faster with significant CPU savings as well as a 20% disk space savings.

Ensuring you are optimizing the most costly resource. Remember that the efficient use of disk space is important, but not nearly as important as CPU consumption. Analysis at another company discovered that in order to save disk space many files were using a Compression option. The storage group had implemented this to save DASD space. In doing, the increased CPU usage unwittingly caused a multi-million dollar CPU upgrade. The Compression was removed on some of the files and CPU dropped by 20% across the board for both batch and online processing and delayed another upgrade for two more years. Optimizing disk usage at the expense of CPU resources may not be a good strategy.

Tune vendor software for your configuration. Remember that the vendors sell their product to thousands of customers and not just you. Each vendor’s product must run in all IBM compatible environments and many of those environments will be older or smaller than your environment. When you install the vendor software it should always be adjusted to fit properly in your environment. Last year I did an analysis for a company that was beginning to have a run time problem. They had an online viewing product from a vendor. They had set it up to create an online file for each customer. They had created over three million online files and were adding one hundred thousand new ones every day. They had run into serious performance issues because they did not understand the vendor software and the setup had been done incorrectly. So don’t add more sand to your computer by just not understanding how to best use a vendor product and configuring it correctly for your system.

Understand your systems and avoid duplication whenever possible. Duplication of data and work is a common issue. We reviewed one IT shop that had the three million online files backed up to one hundred volumes of DASD every night taking five to six hours to run each night (and missing their SLA every morning). Analysis showed that the files were viewing files and were never updated or changed in any way. Except for a small number of new files, they were exactly the same unchanged report files backed up over and over. It would have been much better to just backup the newly created files each night. After all how many copies of a report file do you need?

If it’s not used remove it. Remember that every file that you are not using is typically being backed up and stored every night. If a file is not used it should be backed up, saved on archive and removed. This space will be released and can be used for other purposes.

So the next time you think you need a computer upgrade don’t move the thousand pound bag of sand or connect the anchor to your new computer. Remember that maintenance is easy, simple, safe, and green. Maintenance has a much greater return on investment than an upgrade. Conduct a thorough platform efficiency review, it will save you a great deal more than you think over and over, year after year, upgrade after upgrade from now on.

Best, Ralph Bertrum and Jim Ditmore

About Ralph: Since 1986, Ralph is the co-founder and principle of Critical Path Software. He is an inventor, designer and software developer of The TURBO suite of mainframe analysis tools and expert performance tuning database. He has provided performance tuning services and analyses for over 1,100 major Fortune 5,000 corporations worldwide. He is a former MVS, VSE, VM, CICS, and EDOS/VSE systems programmer, and an IDMS and IMS DBA.

Achieving Outstanding IT Strategy

June 20, 2012 Jim D

Developing your IT strategy should be based on a thoughtful, ongoing process. Too often, strategy is developed as a one time event (typically with consultants) or is a hurried episode following a corporate vision statement that has been handed down. A considered approach, where there is robust industry and technology trend analysis coupled with a two way dialogue on business strategy can yield much better results. I have mapped out below a best practice strategy process that I have leveraged in previous organizations that will ensure a strong connection with the business strategy, leverage of technology trends and clear cascade into effective goals and plans. With such a process in hand, the senior technology leader should be able to both drive a better IT strategy, and importantly, an improved business strategy.

The IT strategy process should start with two sets of research and analysis that interplay: a full review of the business strategy and a comprehensive survey of the key technology trends, opportunities and constraints. It is critical that the business strategy should drive the technology strategy but aspects of the business strategy can and should be driven by the technology. Utilize the technology trend analysis as well as the understanding of the key strengths and weaknesses of the current technology platform to as a feedback loop into the business strategy.

When working with the business, to help them hone their strategy, I recommend leveraging a corporate competency approach from The Discipline of Market Leaders by Michael Treacy and Fred Wisrsema. In essence, Treacy and Wisrsema state that companies that are market leaders do not try to be all things to all customers. Instead, market leaders recognize their competency either in product and innovation leadership, customer service and intimacy, or operational excellence. Good corporate examples of each would be 3M for product, Nordstrom for service, and FedEx for operational excellence. Thus your business strategy should not attempt to excel at all three areas but instead to leverage your area of strength and extend it further while maintaining acceptable performance elsewhere. This focus is particularly valuable when working to prioritize an overly broad and ambitious business strategy.

Below is a diagram that maps out this strategy process or cascade:

The process anticipates that the corporate strategy will drive multiple business unit strategies that IT will then support. It is appropriate to develop the business unit technology strategies that will operate in concert with both the business unit strategy and the corporate technology strategy. Once the strategies are established, it is then critical to define the technology roadmao for each business unit. The roadmap can be viewed as a snapshot of the critical technology capabilities and systems every 3 or 6 months for the next two years that provides a definitive plan of how the business unit’s technology will evolve and be delivered to meet the business requirements. These roadmaps should be tied into and should support an overall technology reference architecture for the corporation. This ensure that the technology roadmaps will work in concert with each other and enable critical corporate capabilities such as understanding the entire relationship with a customer across products and business units.

I recommend executing the full process on an annual basis, synchronous with the corporate planning cycle with quarterly updates to the roadmaps. It is also reasonable to update the the technology trends and business unit strategies on a six month basis with additional data and results.

What would you add to this strategic planning approach? Have you leveraged different approaches that worked well?

Best, Jim Ditmore

Key Steps to a High Performance Team: Coaching and Development

June 3, 2012 Jim D

Today I revisit a core topic of Recipes for IT: High Performance IT Teams. This post is the last of six on how to build and sustain High Performance Teams. I think the aspiration of building a high performing team is a lofty, worthwhile, and achievable vision. If you have ever participated in a high performance team at the top of their game, in other words a championship team, then you know the level of professional reward and sense of accomplishment that accompanies such membership. I have been fortunate enough to have been part of several such teams and it was a remarkable experience, especially in terms of what was accomplished. And for most companies that rely significantly on IT, if their IT team is a high performing team, it can make a very large difference in their products, their customer experience, and their bottom line. I hope you find the material and this last post on the topic to be both enlightening and actionable. Best, Jim

Coaching and Developing High Performance Teams: As I have mentioned previously, I have a positive outlook on the competence of today’s managers and leaders. I see more material and approaches available for managers than ever before and more effort and study applied by the managers as well. Much of the material though is either a very narrow spectrum or a single technique which does not address the full spectrum of practices and knowledge that must be brought to bear to build and sustain a high performance IT team. So, I have assembled a set of practices that I have leveraged or I have seen peers or other senior IT leaders use to build high performance IT teams in this series of posts to enable managers to have a broad source of practice at their disposal. Senior IT leaders, with his or her senior management team, can use these practices to build a high performing team, in the following steps:

define a compelling vision and set the right goals, expectations and behaviors
establish high performance team principles as the norm for their group
evaluate and select the right talent
actually build the team and the culture as well as move to the right site and staff mix
or prune and improve as required
and employ ongoing coaching and development to sustain a high performance team

Today’s post covers how to coach and develop to sustain your high performance team. The previous steps are prior posts and I have further constructed reference pages with links above.

If you do exceedingly well, then your organization will become a net exporter of talent. In fact, you should set this as a personal goal where your organization earns a reputation for having talent that can make a difference elsewhere in the corporation or entity.

Sustaining such a team at its peak performance requires the following ongoing ingredients:

a compelling organizational vision
a positive results-oriented culture that leverages data-based decisioning and rewards quality results
a corresponding set of individual expectations and goals
a thoughtful and well-matched development plan for each of your staff
and challenging assignments and experiences coupled with thoughtful immediate coaching

We have already covered previously how to define a compelling vision but understand that it will be necessary to evolve your vision as you reach the initial goals. Your goals should not move beyond reach or beyond reason, instead they should become more multidimensional in terms of your contributions to the corporate vision. For example, if you have reached your initial service and availability goals, then you should look to improve your transaction performance or the batch cycle time. Or you should move to top quartile or better in online rankings for your industry (such as those produced by Gomez). If you have met your overall budget and cost goals, you should look to improve your unit costs and map out a trajectory to achieve first quartile unit cost in 12 to 24 months. If you have achieved your project delivery goals you should look to improve your time to market and enhance the business capability through greater partnering with the business and delivering innovation. These are all appropriate evolutions that will enable you to contribute more to the corporate goals and enable your team to drive to a higher level of performance.

It is also important that you maintain the open culture based on merit and quality results as well. Assuming you have been able to attract the right talent on your team, you should now look for them to grow through increased responsibility. Part of that comes not just from them performing their roles but also by them not being overburdened with your direction. As your team matures and improves in capability, your profile as a leader should become less directional and more of a coach. For insight into your own profile, I recommend you read this post on PDI as well as the material by Jim Collins on a Level 5 leader. Encourage your team to do the underlying analysis and map out recommended directions. This will enable them to take on more responsibility and become better leaders. Look to guide and correct as needed, but they will become better leaders through the experience. Remember that enabling an environment where mistakes can be made, lessons learned, and quality and innovation and initiative are prized means you will get a staff that behaves and performs in that manner.

To enable each individual to achieve their potential does require a specific and well-thought development plan. Fundamentally, this development plan must be based on an robust job profiles or descriptions set within and overall career framework. It is important for an IT organization that these job profiles map out the competencies, experience and credentials that staff should attain in order to progress. Further, each job profile should be set out in logical steps or a ladder so that your staff can understand how their career can progress. And there should be the opportunity to progress I will be providing a sample template of such a framework later this week. Once you have a robust job profile coupled with effectively cascaded goals, you can now map out the proper development areas for your staff. I recommend that you invest in the time to do it well. Focus on the key areas of development and provide constructive, specific examples from which one can understand and learn. It would be even better if these examples were also situations where you have provided immediate feedback or observation. Unfortunately, many managers fail to assess appropriately the importance of writing good performance reviews and development plans. They either fail set out thoughtful goals, or they provide only general feedback or criticism. Remember to leverage the talent assessment work done in your ‘prune and improve stage. Recall there are generally three types of staff that need different coaching and guidance: those that are top performers that you will need to further develop and challenge; the ‘well-placed experts’ and solid performers that will need support and attention and will execute reliably; and those whose performance and potential is lacking and who must step up to continue in their role. With these three groupings identified, ensure you lay out crisp plans for all three groups and execute against them.

If this is an area where you would like to improve, I do recommend leveraging the book FYI: For Your Improvement. It is a seminal work on the 65 professional competencies and provides good descriptions and examples of each competency (or a weakness) as well as thoughtful suggestions and coaching on how to improve that particular competency. It can be a very good assist to writing effective reviews and improving your coaching.

The most critical part to any development is enabling your team to take on new assignments and responsibilities. These can occur based on new corporate initiatives, additional scope or responsibilities or through rotational assignments. One tendency, particularly prevalent in large organizations, is to ‘pigeonhole’ current talent into the work they have done previously. If you have someone that you have ranked as having potential and who is a good performer, I would suggest that with the right coaching and investment in training, they are likely to be successful in different roles. As a good performer, they would have already mastered the culture, relationships and potentially the business knowledge, and thus, adding technical or different skills may not be that large. Obviously, there will be situations where someone who excels in operational roles does not do well in planning or strategy roles, but I recommend pressing boundary here more than not.

So, look to provide new experiences and opportunities so your staff can grow, and be close by to provide coaching, correction and support so you can increase their likelihood of success. But if it is not a fit, again, does not force them to endure, adjust their role back to a sweet spot of their capabilities quickly if needed.

With these measures in place, you will generate a strong team and begin to export your talent (as they are sought out and also encouraged to take on new opportunities outside your organization). With this release of talent, along with normal attrition, you will need to build a bench that extends through the lowest levels in your organization so you can fill the vacancies that then occur. By bringing in graduates and junior staff, you can train and develop them to take on the mid-level positions who can then be trained and developed for the senior level positions. This natural flow, if done well, will then minimize the amount of senior staff you must recruit and bring onboard. Understand though that until you have this bench built, when you are first starting out and have inadequate talent, you can only work your way out of the issue by recruiting the right new talent at all levels. If done well, this should become a reinforcing, virtuous cycle and you can reduce external senior recruitment. Then you will know have built and are able to sustain a championship team.

What techniques would you add or change in the development or coaching process? How do you see this fitting into building championship teams?

Best, Jim Ditmore

Getting Things Done: A Key Leadership Skill

May 22, 2012 Jim D

It is a bit ironic that this post has taken me twice as long to do as my average post. But while it is an important topic, it is difficult to pinpoint, of all the practices you can leverage, which ones really help you or your team or organization get the right things done. So, just before the Memorial Day holiday, here is a post to help you execute better for the rest of the year and meet those goals.

Have a great holiday weekend. Jim

Getting things done is a hallmark of effective teams. Unfortunately, the focus and flow of large business organizations combined with influences of the modern world erode our ability to get the right things done. To raise the productivity to a high performance team, as a senior leader, you should impart an ability to get the right things done at the divisional and team level within your organization. And while there are myriad reasons that conspire to reduce our focus or effectiveness, there are a number of techniques and practices that can greatly improve the selection and capacity at all levels: at the overall organization or division, at the working team level, and for the individual.

Realize that the same positive forces that ensure a focus on business goals, drive consensus within an organization, or require risk and control to be addressed, can also be mis- or over-applied and result in organizational imbalance or gridlock. Coupled with too much waterfall or ‘big bang’ approaches and you can get not just ineffectiveness but spectacular failures of large efforts. At the organizational level, you should set the right agenda and framework so the productivity and capacity of your IT shop can be improved at the same time you are delivering to the business agenda. To set the right agenda look to the following practices:

provide a clear vision with robust goals that include clear delivery milestones and that are aligned to the business objectives. The vision should also be compelling — your team will only outperform for a worthwhile aspiration.
avoid too many big bets (an unbalanced portfolio) – your portfolio should be a mix of large, medium and small deliveries. This enables you to deliver a regular stream of benefits across a broader set of functions and constituents with less risk. Often a nice balancing investment area is drive several small efforts in HR and Finance that streamline and automate common processes in these areas used by much of the corporation (thus a good, broad positive impact on the corporate productivity).
aggregate your delivery – often IT efforts can be so tightly tied to immediate delivery for the business that the IT processes are substantially penalized including:

where a continuous stream of applications and updates are introduced into production without a release schedule (causing large amount of duplicative or indequate design, testing and implementation)
where a highly siloed delivery approach where every minor business unit has its own set of business systems resulting in redundant feature build and maintain work.

address poor quality standards and ineffective build capability including:

correct defects as early in the build process as possible. Defects correct at their source (design or implementation) are far less costly to fix than those corrected once in production
lower build productivity due to a lack of investment in the underlying ‘build factory’ including tools, training and processes or the teams do not leverage modern incremental or agile methods
delivery by the internal team of the full stack, where packaged software is not leveraged (recently I have encountered shops trying to do their own software distribution tools or data bases

So, in sum, at the organizational level, provide clarity of vision, review your portfolio for balance, make room for investments in your factory and look to simplify and consolidate.

At the team level, employ clarity, accountability, and simplicity to get the right things done. Whether it is a project or an ongoing function:

are the goals or deliverables clear?
are the efforts broken into incremental tasks or steps?
are the roles clear?
are the tasks assigned?
are there due dates? or good operational metrics?
is the solution or approach straightforward?
is there follow up to ensure that the important work takes priority and the work is done?

And then, most important, are you recognizing and rewarding those who gets things done with quality? There are many other factors that you may need to address or supplement to enable the team to be achieve results from providing specific direction to coaching to adding resources or removing poor performers. But frequently well-resourced teams can spin their wheels working on the wrong things, or delivering with poor quality or just not focusing on getting results. This is where clarity, accountability and simplicity make the difference and enable your team to get the right things done.

Most importantly, getting the right things done as an individual is a critical skill that enables outperformance. Look to hone your abilities with some of following suggestions:

recognize we tend to do what is urgent rather than what is important. Shed the unimportant but urgent tasks and spend more time on important tasks. In particular, use the time to be prepared, improve your skills, or do the planning work that is often neglected.
hold yourself accountable, make your commitments. As a leader you must demonstrate holding yourself to the same (or higher) standards as those for your team.
Make clear, fact-based decisions and don’t over-analyze. But seek inputs where possible from your team and experts. And leverage a low PDI style so you can avoid major mistakes.
and finally, a positive approach can make a world of a difference. Do your job with high purpose and in high spirit. Your team will see it and it will lighten their step as well.

So, those are the practices from my experience that have been enablers to getting things done. What would you add? or change? Do let me know.

Best, Jim Ditmore

Key Steps to Building a High Performance Team: Prune and Improve

May 8, 2012 Jim D

Today I revisit a core topic of Recipes for IT: High Performance IT Teams. Before I provide background on this series of posts, I thought it was about time for a quick blog update. Recipes for IT continues to attract new readers and has a substantial ongoing readership. It is quite heartening to see the level of interest and I really appreciate your visits and comment. I will strive to regularly add thoughtful and relevant material for IT leaders and hope that you continue to find the site useful. I do recommend for new readers that you check out the introduction page and the various topic areas as you should find useful material of strong depth and actionability that can help you be more successful. This site also continues to do well in Google page rankings on a number of topic areas, particularly service desk queries and IT metrics and reporting. If there are topics you would like me to tackle, please do not hesitate to send me a comment.

Now back to some background on Building High Performance Teams. This post is now the fifth on this topic and there will be one further post to complete the steps of building a high performance team. I hope you find the material to be both enlightening and actionable. One key for IT leaders is that you consider the tasks required to build a HP team as some of your most important activities. At nearly every poor performing organization that I have been responsible for turning around, I have found that many times, the primary reason for inadequate talent and poor performing teams is inadequate manager attention and focus on these activities. So, work hard to make the time, even though you would much rather be doing other activities. And now for the post. Best, Jim

Building High Performance Teams: As I have mentioned previously, I have a positive outlook on the competence of today’s managers and leaders. I see more material and approaches available for managers than ever before and more effort and study applied by the managers as well. Much of the material though is either a very narrow spectrum or a single technique which does not address the full spectrum of practices and knowledge that must be brought to bear to build and sustain a high performance IT team. So, I have assembled a set of practices that I have leveraged or I have seen peers or other senior IT leaders use to build high performance IT teams in this series of posts to enable managers to have a broad source of practice at their disposal.

Senior IT leaders, with his or her senior management team, can use these practices to build a high performing team, in the following steps:

define a compelling vision and set the right goals, expectations and behaviors
establish high performance team principles as the norm for their group
evaluate and select the right talent
actually build the team and the culture as well as move to the right site and staff mix
or prune and improve as required
and employ ongoing coaching and development to sustain a high performance team

Today’s post covers how to prune and improve as required. The previous steps are prior posts and I have further constructed reference pages with links above on the first four steps. Subsequent posts will cover the last steps as well as a summary.

I think the aspiration of building a high performing team is a lofty, worthwhile, and achievable vision. If you have ever participated in a high performance team at the top of their game, in other words: a championship team, then you know the level of professional reward and sense of accomplishment that accompanies such membership. And for most companies that rely significantly on IT, if their IT team is a high performing team, it can make a very large difference in their products, their customer experience, and their bottom line. Building such a championship team is not only about attracting or retaining top talent, it is also necessarily about identifying those team members who do not have the capabilities, behaviors, or performance to remain part of the team and addressing their future role constructively but firmly.

Let’s first revisit some key truths that underly how to build a high performance team:

– top performing engineers, typically paid similar to their mediocre peers are not 10% better but 2x to 10x better

– having primarily senior engineers and not a good mix of interns, graduates, junior and mid and senior level engineers will result in stagnation and overpaid senior engineers doing low level work

– having a dozen small sites with little interaction is far less synergistic and productive than having a few strategic sites with critical mass

– relying on contractors to do most of the critical or transformational work is a huge penalty to retain or grow top engineers

– line and mid-level managers must be very good people managers, not great engineers, otherwise you are likely have difficulty retaining good talent and you will not develop your talent

– engineers do not want to work in an expensive in-city location like the financial district of London (that is for investment bankers)

– enabling an environment where mistakes can be made, lessons learned, and quality and innovation and initiative are prized means you will get a staff that behaves and performs like that.

With these truths in mind, (and these are the same ones you used to set about building the team), having executed the first four steps, you should have adequate capacity to begin thoughtful pruning and improvement of your organization. While there are circumstances when a poor performing manager or senior engineer causes so many issues that it is a benefit to remove them, in many cases you must have adequate resource capacity to meet demands so that once you begin pruning your team is not overtaxed and penalized as a result.

Pruning should begin at the top and work down from there. Start with your directs and the next level below. Consider the span of control of your organization and the number of levels. High performing organizations are generally flatter with greater spans of control. In considering your team, I recommend leveraging a talent calibration approach of either the typical 9 box or a top-grading variant. The key to calibration is to essentially formulate three sets of results: those on your staff that are top performers that you will need to further develop and challenge; the ‘well-placed experts’ and solid performers that will need support and attention but will execute reliably; and those whose performance and potential is lacking and who must step up to continue in their role. With these three groupings of your management team identified, ensure you lay out crisp plans for all three groups and execute against them. (Remember, it will be very difficult for you to subsequently demand of your line managers that they address their staff issues if you have not shown a capability to execute such accountability with your team.)

One area to particularly focus on is time-boxing the development plans for poor performers. As these are senior managers the time to address performance issues should be shorter not longer. I recommend you start the development plan with a succinct, clear conversation on high expectations and shortcoming of their performance with examples where possible. You should provide a writeup covering this discussion at the end of the discussion. Jointly layout key deliverables, milestones, expected behavior changes and results with the affected leader. Be open to the possibility that the employee may know they are in over their head and may be looking for an alternative. While not advocating moving problem performers around, there may be a role within the company or elsewhere outside the company that is a much better fit. Look to assist with such a transition if beneficial for the company and the employee. If the employee insists this is the role they want and they are willing to step up and adjust, then you should provide support under a tight timeline for them to achieve it. Monitor the plan regularly with HR. If you follow up diligently it will become evident quite quickly that the employee can muster to the new level or not. Generally, in my experience, a surprisingly large percentage of poor performing employees will drop out of their own accord once you have provided clear expectations and no escape routes other than the hard work to get there — assuming of course that there is a modest but respectable exit plan for them. It is also key to treat the employee with respect and fairness throughout the process and focus on the results and outcomes.

Equally though, I have more than a handful of senior leaders and managers who have expressed surprise when confronted with poor performance as no one had communicated clearly and firmly their performance issues previously. Once understood and once the higher goals and expectations were known, many of these individuals (and others as well), definitively stepped up and improved significantly. Thus, until you communicate the higher goals and expectations clearly AND communicate where they must improve (constructively, with specifics) the likelihood of improvement is minimal. So, allocate the time to hold the tough but fair conversations and provide this information. Once the conversations are held, over the next 2 to 3 months you should take action based on the results. Either poor performing managers will be exited (or moved to a role much more befitting) or poor performers will become good performers. One of the interesting results from such actions is that the remaining team, upon seeing poor performers exited, will view the results positively. In fact, I have experienced some very strong reactions from other team members who now felt a dead weight was off of their shoulders as they no longer had to make up for the defects and negative performance of the just exited team member. Further, I have received multiple (back-handed) compliments along the lines of ‘Wow, we are glad management finally figured out what to do and took action!’ . So do not be persuaded that the team will view performance actions solely in a negative light.

Once you have initiated the performance management process and you are well in the process of pruning your team, you can work with your managers and HR department to address areas lower in the organization. Remember it is key to first set expectations and goals that cascade and match your overall goals. Then ensure you hold managers and senior engineers to a higher bar than the mid and junior staff. For senior staff, you are not looking just for technical competence but also they must meet the standard for such behaviors as problem solving/solution orientation, teamwork, initiative and drive, and quality and focus on doing things right. And they should exhibit the right leadership and communication skills.

Driving such pruning and development work through your organization is important but also a delicate task. Generally, with little exception, management in a IT organization can improve how they handle performance management. Because most of the managers are engineers, their ability to interact firmly with another person in a highly constructive manner is typically under-developed. Thus, some managers may not be up to this pruning task or their calibration of talent could be well off the mark. So, leverage your HR resources to guide management and personally check in to ensure proper calibration of talent by your lower level managers. Provide classes and interactive session on how to do coaching and provide feedback to employees. Even better, insist that performance reviews and development plans must be read and signed off by the manager’s manager before being given to improve their quality. This a key element to focus on because a poorly executed resource improvement plan could backfire. Remember that the line manager’s interaction with an employee is the largest factor in undesired attrition and employee engagement. Of course, these is all the more reason to replace poor performing managers with good leaders, but do so effectively and firmly. Use the workforce plans that you developed in the Build step to ensure your pruning and development also helps you move toward your strategic site goals, contractor/staff mix targets, and junior/mid/senior profiles.

Pruning and improvement is the tough but necessary step in building a high performance team. If done well, pruning and improvement will provide additional substantial lift to the team and more importantly, enable ongoing sustainment. It requires discipline and focus to execute the steps we would all prefer to avoid, but are necessary for reaching the final high performance stages.

What has been your experience either as a leader or participant in such efforts? What have you seen go very well? or terribly wrong? I look forward to your perspective.

Best, Jim Ditmore

Another Wave of Security Breaches: Meeting It with Security Best Practices

April 22, 2012 Jim D

With the latest breaches in the news, I felt it was important to map out base practices and well as some of the best practices in Information Security. In the age of LulzSec, industrial espionage, and everyday breaches, it’s more important than ever to be proactive about security. I consulted with several top security engineers that I have worked with in the past to construct these practices. Much of this post was first published in early April in Information Week and I have updated it further. Unfortunately, this area should be a top priority for IT leaders to protect their firms, customers and information. If it’s not at your firm, you need to change that. Best, Jim.

PS. Here is a good reference on the biggest data breaches the past 15 years to help you get the investment required to properly implement IT Security.

Mark Twain observed 150 years ago: “A lie can travel halfway round the world while the truth is putting on its shoes.” With the advent of social media, these days that lie has likely made it all the way around the world and back while the truth is still in bed.

And today it is not just the false information it’s the confidential information, your customer’s information or your company intellectual property that is spirited away. The pace and sophistication of attacks by hackers and others who expose confidential data and emails has increased dramatically. For their latest exploit, a group calling itself LulzSec Reborn recently hacked a military dating website releasing the usernames and passwords of more than 170,000 of the site’s subscribers.

Then there are the for-profit attacks by nation states and companies seeking intellectual property, and fraud by organized crime outfits. Consider the blatant industrial espionage conducted against Nortel and more recently, AMSC, or the recent fraud attack against Global Payments. These are sobering stories of how company’s falter or fail in part due to such espionage.

One of a CIO’s most critical responsibilities is to protect his or her company’s information assets. Such protection often focuses on preventing others from entering company systems and networks, but it must also identify and prevent data from leaving. The following recommendations can help you do this. They are listed in two sections: conventional measures that focus on system access, and best practices given the profiles of today’s attacks.

Conventional Measures:

Establish a thoughtful password policy. Sure, this is pretty basic, but it’s worth revisiting. Definitely require that users change their passwords regularly, but set a reasonable frequency–any less than three months and users will write their passwords down, compromising security. As for password complexity, require at least six or seven characters, with one capital letter and one number or other special character.

Publicize best security and confidentiality practices. Do a bit of marketing to raise user awareness and improve security and confidentiality practices. No security tool can be everywhere. Remind your employees that security threats can follow them home from work or to work from home. Help your employees take part of your company’s security practices — there is a good post on this at How To Make Information Security Everyone’s Problem.

Install and update robust antivirus software on your network and client devices. Enough said, but keep it up-to-date and make it comprehensive (all devices)

Review access regularly. Also, ensure that all access is provided on a “need-to-know” or “need-to- do” basis. This is an integral part of any Sarbanes-Oxley review, and it’s a good security practice as well. Educate your users at the same time you ask them to do the review. This will reduce the possibility of a single employee being able to commit fraud resulting from retained access from a previous position.

Put in place laptop bootup hard drive encryption. This encryption will make it very difficult to expose confidential company information via lost or stolen laptops, which is still a big problem. Meanwhile, educate employees to avoid leaving laptops in their vehicles or other insecure places.

Require secure access for “superuser” administrators. Given their system privileges, any compromise to their access can open up your systems completely. Ensure that they don’t use generic user IDs, that their generic passwords are changed to a robust strength, and that all their commands are logged (and subsequently reviewed by another engineering team and management). Implement two-factor authentication for any remote superuser ID access.

Maintain up-to-date patching. Enough said.

Encrypt critical data only. Any customer or other confidential information transmitted from your organization should be encrypted. The same precautions apply to any login transactions that transmit credentials across public networks.

Perform regular penetration testing. Have a reputable firm test your perimeter defenses regularly.

A Thoughtful Set of Additional Current Best Practices: With the pace of change of technology and the rise of additional threats from hackers and state-sposored espionage, your company’s security posture must adopt the latest best techniques and be updated regularly. Here are the current best practices that I would highly recommend.

Provide two-factor authentication for customers. Some of your customers’ personal devices are likely to be compromised, so requiring two-factor authentication for access to accounts prevents easy exploitation. Also, notify customers when certain transactions have occurred on their accounts (for example, changes in payment destination, email address, physical address, etc.).

Secure all mobile devices. Equip all mobile devices with passcodes, encryption, and wipe clean. Encrypt your USD flash memory devices. On secured internal networks, minimize encryption to enable detection of unauthorized activity as well as diagnosis and resolution of production and performance problems.

Further strengthen access controls. Permit certain commands or functions (e.g., superuser) to be executed only from specific network segments (not remotely). Permit contractor network access via a partitioned secure network or secured client device.

Secure your sites from inadvertent outside channels.Implement your own secured wireless network, one that can detect unauthorized access, at all corporate sites. Regularly scan for rogue network devices, such as DSL modems set up by employees, that let outgoing traffic bypass your controls.

Prevent data from leaving. Continuously monitor for transmission of customer and confidential corporate data, with the automated ability to shut down illicit flows using tools such as NetWitness. Establish permissions whereby sensitive data can be accessed only from certain IP ranges and sent only to another limited set. Continuously monitor traffic destinations in conjunction with a top-tier carrier in order to identify traffic going to fraudulent sites or unfriendly nations.

Keep your eyes and ears open. Continually monitor underground forums (“Dark Web”) for mentions of your company’s name and/or your customers’ data for sale. Help your marketing and PR teams by monitoring social networks and other media for corporate mentions, providing a twice-daily report to summarize activity.

Raise the bar on suppliers. Audit and assess how your company’s suppliers handle critical corporate data. Don’t hesitate to prune suppliers with inadequate security practices. Be careful about having a fully open door between their networks and yours.

Put in place critical transaction process checks. Ensure that crucial transactions (i.e., large transfers) require two personnel to execute, and that regular reporting and management review of such transactions occurs.

Best, Jim D.

In some ways you can view it as no longer a matter of if you get hacked, but when. Information Week has a special retrospective of news coverage, Monitoring Tools And Logs Make All The Difference, where they take a look at ways to measure your security posture and the challenges that lie ahead with the emerging threat landscape. (Free registration required.)

Key Steps to Building High Performance Teams

April 19, 2012 Jim D

Today I have returned to a topic that is at the core of Recipes for IT: High Performance IT Teams. While tax day did take a bit of time and I am slightly delayed in posting this, I have actually laid out three accompanying posts or pages for today’s post. I think it is a good start on the complex topic of how to build or energize your team and create a high performing team. I look forward to your comments! Best, Jim

Building High Performance Teams: The essence of being a leader is defining a vision and compelling others to pursue and achieve that vision. Recently a good colleague relayed an article in Harvard Business Review describing how it is more difficult today to be a outstanding leader due to a number of factors including the wider availability of knowledge and easier access to each other as well as a reduced perception of glory of institutions that leaders represent.

And while I would agree these factors may make things more difficult to be a great leader, I tend to believe that we have just as many, if not more, good and decent men and women who are effective and even outstanding leaders today as ever in history. But because the circumstances are less dire (e.g., there is not a world war to require a Churchill) and because the competence has risen (yes, management is a far more analyzed and practiced field than ever before), there are not the towering gaps between the best and the average that might have previously been. So with a positive outlook on the competence of today’s managers and leaders, I have assembled a set of practices that I have leveraged or I have seen peers or other senior IT leaders use to build high performance IT teams.

For the emerging senior IT leader with his or her senior management team, can use these practices to build a high performing team, in the following steps:

define a compelling vision and set the right goals, expectations and behaviors
establish high performance team principles as the norm for their group
evaluate and select the right talent
actually build the team and the culture as well as move to the right site and staff mix
or prune and improve as required
and refine ongoing coaching and development to sustain a high performance team

Today’s post covers how to set such a vision, then define and cascade the goals to match your vision, align the incentives, and set the proper expectations and behaviors. And I have constructed pages with links above on the next three steps. Subsequent posts will cover the remaining steps.

I think the aspiration of building a high performing team is a lofty, worthwhile, and achievable vision. If you have ever participated in a high performance team at the top of their game, in other words: a championship team, then you know the level of professional reward and sense of accomplishment that accompanies such membership. And for most companies that rely significantly on IT, if their IT team is a high performing team, it can make a very large difference in their products, their customer experience, and their bottom line. But if you are to set out to build such a team it must be for a vision that is more than just the team, it must be to enable your company to achieve achieve outsized goals of appropriate scale and aspiration. You will not attract or retain top talent and inspire others if you and your company have only modest goals.

So, first, consider your company’s goals and then outline what IT must become and must accomplish to enable corporate success of major significance. And the draw out the IT vision and goals that will enable that success. Do not get trapped by cloaking your vision in uninspired definitions (e.g., don’t state your vision as ‘ Save $40M in costs’ instead ‘ Become top quartile in efficiency in IT for our industry and company size by 201x’). You can only state your vision in this manner, of course, if you mean it. So, I will assume you have true aspirations for your team to become a world class IT organization and you will meld those goals with your company’s goals for a compelling vision. Further, consider IT goals to match both your company’s service and operations goals as well as product and innovation. Make sure the vision you define for IT drives both areas as well looking to a two to three year horizon for the target. (Rebuilding or energizing a team usually takes such a time period to truly reach high performance and you must lift the sight line to the horizon to ensure your team does not get trapped in just extending the every day steps.)

Once you have defined a compelling vision, the next step is to set the right goals to achieve the vision. The right goals will logically cascade as mileposts on the journey to high performance as well as be inevitable products of achieving corporate excellence. I recommend framing such goals as the primary measures by year that you will determine if you are to achieve the required progress to reach your vision. For the upcoming year, it is often worthwhile to set quarterly milestones as well. These measures should be relevant, well-defined, as quantifiable as possible and they should be set at stretch but achievable levels. If, for example, your vision is for your company to become an industry leader in service quality then you would want to set cascaded goals where the IT team dramatically improves its quality (so the systems now enable much better customer service for the company) as well as delivers key workflow improvements or feature enhancements to enable the company to lift its service directly (e.g., such as the package tracking capability that Federal Express uses to ensure extremely high quality service). Ensure that your measures are not uni-dimensional, that is, they only cover one aspect of what your company and thus your team must achieve. There should be clear focus in one area (e.g. quality and operational excellence, or product feature, or speed, or innovation) but it should not be a the full neglect of the other areas. Further, you should set at least modest goals for both cost and risk, otherwise these could become risk areas as your team pursues only one facet.

Once you have defined the right, cascading goals you will need to reinforce the goals with a set of behaviors and expectations as well as aligned incentives. And the approach to achieving the goals should reflect the strengths of the teamFor example, if one of your goals is to achieve outstanding quality then measures for the goals may include process definition work and metrics implementation if your team has low maturity or jump right to leveraging already reported metrics and driving improved feedback cycles if of high maturity. Further though, if your team has an engineering bias you may approach the solution through robust root cause and better design processes whereas if the team has a strong collaboration approach you may reach the same quality goal through better peer reviews and additional coordination and validation of changes.

More importantly though is to reinforce your goals through aligned behaviors and expectations and most importantly, incentives. For example, if you are looking to drive more predictable project delivery for the business than having incentives that reward firefighting for some of your staff when they contributed to the potential issues in the first place will tremendously undermine how much the rest of the staff support your goals. Similarly, if you reward those who while delivering a particular set of results cause significant damage to other team members or ignore other standards or principles, than you will minimize the likelihood that such principles or standards will be followed in the future. It is important, as a leader to reward not just factor in the results but more critically how the effort was achieved. Often in organizations, those who quietly and effectively carry out significant projects with excellent team behaviors are neglected by management when good leaders would call out the very same individuals for exemplary performance. Quite simply, your smartest team members, will observe what you reward, and if you do not reinforce the values you are looking to achieve (productivity, quality, initiative, etc), you will not get the changes desired.

Most importantly though, your behaviors as leaders must reflect the very same expectations you have outlined for your team. And you must demonstrate a tenacious focus on the goals and vision you have defined. Your behaviors must reinforce your expectations. Every day there are conflicts and setbacks that strong leaders would turn these into episodes that strengthen rather than weaken your team. Understand, that when you lead by example, you will make a daily difference and demonstrate to your team what should be done. Your focus, discipline, thoughtfulness, and sacrifice for the team and goals will not be lost and will result in better effort all around.

In sum, it is important to define a compelling vision and establish the right goals and incentives, but at each stage of the journey, there will be key moments where you as a leader will reinforce the vision, goals and principles you have set or you will undermine them. As a leader, perhaps we can easily step into the role of setting a direction, sponsoring a program or making a major decision, but for all the visibility and importance of these actions, the time we spend interacting and communicating and coaching your team will determine the effectiveness and reach of our goals.

Now with the steps above you will have the foundation to build a high performance team and deliver sustainable and outstanding results.

What steps or approaches have you used to successfully define a vision and goals for a team? What would you change to this approach?

Best, Jim Ditmore

Using Performance Metric Trajectories to Achieve 1st Quartile Performance

April 10, 2012 Jim D

I hope you enjoyed the Easter weekend. I have teamed up today with Chris Collins, a senior IT Finance manager and former colleague. Our final post on metrics is on unit costing — on which Chris has been invaluable with his expertise. For those just joining our discussion on IT metrics, we have had 6 previous posts on various aspects of metrics. I recommend reading the Metrics Roundup and A Scientific Approach to Metrics to catch you up in our discussion.

As I outlined previously, unit costing is one of the critical performance metrics (as opposed to operational or verification metrics) that a mature IT shop should leverage particularly for its utility functions like infrastructure (please see the Hybrid model for more information on IT utilities). With proper leverage, you can use unit cost and the other performance metrics to map a trajectory that will enable your teams to drive to world-class performance as well as provide greater transparency to your users.

For those just starting the metrics journey, realize that in order to develop reliable sustainable unit cost metrics, significant foundational work must be done first including:

IT service definition should be completed and in place for those areas to be unit costed
an accurate and ongoing asset inventory must be in place
a clean and understandable set of financials must be available organized by account so that the business service cost can be easily derived

If you have these foundation elements in place then you can quickly derive the unit costing for your function. I recommend partnering with your Finance team to accomplish unit costing. And this should be an effort that you and your infrastructure function leaders champion. You should look to apply a unit cost approach to the 20 to 30 functions within the utility space (from storage to mainframes to security to middleware, etc). It usually works best to start with one or two of the most mature component functions and develop the practices and templates. For the IT finance team, they should progress the effort as follows:

Ensure they can easily segregate cost based on service listing for that function
Refine and segregate costs further if needed (e.g., are there tiers of services that should be created because of substantial cost differences?)
Identify a volume driver to use as the basis of the unit cost (for example, for storage it could be terabytes of allocated storage)
Parallel to the service identification/cost segregation work, begin development of unit cost database that allows you to easily manipulate and report on unit cost. Specifically, the database should contain:

Ability to accept RC and account level assignments
Ability to capture expense/plan from the general ledger
Ability to capture monthly volume feeds from source systems including detail volume data (like user name for an email account or application name tied to a server)

For the function team, they should support the IT Finance team in ensuring the costs are properly segregated into the services they have defined. Reasonable precision of the cost segregation is required since later analysis will be for naught if the segregations are inaccurate. Once the initial unit costs are reported, the function technology can now begin their analysis and work. First and foremost should be an industry benchmark exercise. This will enable you to understand quickly how your performance ranks against competitors and similar firms. Please reference the Leveraging Benchmarkspage for best practices in this step. In addition to this step, you should further leverage performance metrics like unit cost to develop a projected trajectory for for your function’s performance. For example, if your unit cost for storage is currently $4,100/TB for tier 1 storage, then the storage team should map out what their unit cost will be 12, 24, and even 36 months out given their current plans, initiatives and storage demand. And if your target is for them to achieve top quartile cost, or cost median, then they can now understand if their actions and efforts will enable them to deliver to that future target. And if they will not achieve it, they can add measures to address their gaps.

Further, you can now measure and hold them accountable on a regular basis to achieve the proper progress towards their projected target. This can be done not just for unit cost but for all of your critical performance measures (e.g., productivity, time to market, etc). Setting goals and performance targets in this manner will achieve far better results because a clear mechanism for understanding cause and effect between their work and initiatives and the target metrics has been established.

A broad approach to also potentially utilize is to establish a unit cost progress chart for all of your utility functions. On this chart, where the y axis is cost as a percentage of current cost and the x axis is future years, you should establish a minimum improvement line of 5% per year. The rationale behind this is that improving hardware (e.g., servers, storage, etc) and improving productivity, yield an improving unit cost tide of at least 5% a year. Thus, to truly progress and improve, your utility functions should well exceed a 5% per year improvement if they are below 1st quartile. This approach also conveys the necessity and urgency of not sitting on our laurels in the technology space. Often, with this set of performance metrics practices employed along with CPI and other best practices, you can then achieve 1st quartile performance within 18 to 24 months for your utility function.

What has been your experience with unit cost or other performance measures? Where you able to achieve sustained advantage with these metrics?

Best,

Jim Ditmore and Chris Collins

Tying Consumption to Cost: Allocation Best Practices

March 25, 2012 Jim D

In 1968, Garrett Hardin wrote about the over-exploitation of common resources in an essay titled the “The Tragedy of the Commons“. While Garrett wrote about the overexploitation of common pastureland where individual herders overused and diminished common pasture, there can be a very similar effect with IT resources within a large corporation. If there is no cost associated with the usage of IT resources by different business unit, than each unit will utilize the the IT resources to maximize its potential benefit to the detriment of the corporate as a whole. Thus, to ensure effective use of the IT resources there must be some association of cost or allocation between the internal demand and consumption by each business unit. A best practice allocation approach enables business transparency of IT cost and business drivers of IT usage so that thoughtful business decisions for the company as a whole can be made with the minimum of allocation overhead and effort.

A well-designed allocations framework will ensure this effective association as well as:

provide transparency to IT costs and the particular business unit costs and profitability,
avoid wasteful demand and alter overconsumption behaviors
minimize pet projects and technology ‘hobbies’

To implement an effective allocations framework there are several foundation steps. First, you must ensure you have the corporate and business unit CFOs’ support and the finance team resources to implement and run the allocations process. Generally, CFOs look for greater clarity on what drives costs within the corporation. Allocations allow significant clarity on IT costs which are usually a good-sized chunk of the corporation’s costs. CFOs are usually highly supportive of a well-thought out allocations approach. So, first garner CFO support along with adequate finance resources.

Second, you must have a reasonably well-defined set of services and an adequately accurate IT asset inventory. If these are not in place, you must first set about defining your services (e.g. and end user laptop service that includes laptop, OS, productivity software, and remote access or a storage service of high performance Tier 1 storage by Terabyte) and ensuring your inventory of IT assets is minimally accurate (70 to 80 %). If there are some gaps, they can be addressed by leveraging a trial allocation period where numbers and assets are published, no monies are actually charged, but every business unit reviews its allocated assets with IT and ensures it is correctly aligned. Once you have the service defined and the assets inventoried, your finance team must then set about to identify which costs are associated with which services. They should work closely with your management team to identify a ‘cost pool’ for each service or asset component. Again, these costs pools should be at least reasonably accurate but do not need to be perfect to begin a successful allocation process.

The IT services defined should be as readily understandable as possible. The descriptions and missions should not be esoteric except where absolutely necessary. They should be easily associated with business drivers and volumes (such as number of employees, or branches, etc) wherever possible. In essence, all major categories of IT expenditure should have an associated service or set of services and the services should be granular enough so that each service or component can be easily understood and each one’s drivers should be easily distinguished and identified. The targets should should be somewhere between 50 and 150 services for the typical large corporation. More services than 150 will likely lead to more effort being spent on very small services and result in too much overhead. Significantly, less than 50 services could result in clumping of services that are hard to distinguish or enable control. Remember the goal is to provide adequate allocations data at the minimum effort for effectiveness.

The allocations framework must have an overall IT owner and a senior Finance sponsor (preferably the CFO). CFOs want to implement systems that encourage effective corporate use of resources so they are a natural advocate for a sensible allocation framework. There should also be a council to oversee the allocation effort and provide feedback and direction where majors users and the CFO or designate are on the council. This will ensure both adequate feedback as well as buy-in and support for successful implementation and appropriate methodology revisions as the program grows. As the allocations process and systems mature, ensure that any significant methodology changes are reviewed and approved by the allocation council with sufficient advance notice to the Business Unit CFOs. My experience has been that everyone agrees to a methodology change if it is in their favor and reduces their bill, but everyone is resistant if it impacts their business unit’s finances regardless of how logical the change may be. Further, the allocation process will bring out intra business unit tensions toward each other, especially for those that have an increase versus those that have a decrease, if the process is not done with plenty of communication and clear rationale.

Once you start the allocations, even if during a pilot or trial period, make sure you are doing transparent reporting. You or your leads should have a monthly meeting with each business area with good clear reports. Include your finance lead and the business unit finance lead in the meeting to ensure everyone is on the same financial page. Remember, a key outcome is to enable your users to understand their overall costs, what the cost is for each services and, what business drivers impact which services and thus what costs they will bear. By establishing this linkage clearly the business users will then look to modify business demand so as to optimize their costs. Further, most business leaders will also use this allocations data and new found linkage to correct poor over-consumption behavior (such as users with two or three PCs or phones) within their organizations. But for them to do this you must provide usable reporting with accurate inventories. The best option is to enable managers to peruse their costs through an intranet interface for such
end-user services such as mobile phones, PCs, etc . There should be readily accessible usage and cost reports to enable them to understand their team’s demand and how much each unit costs. They should have the option right on the same screens to discontinue, update or start services. In my experience, it is always amazing that once leaders understand their costs, they will want to manage them down, and if they have the right tools and reports, managing down poor consumption happens faster than a snowman melting in July — exactly the effect you were seeking.

There are a few additional caveats and guides to keep in mind:

In your reporting, don’t just show this month’s costs, show the cost trend over time and provide a projection of future unit costs and business demand
Ensure you include budget overheads in the cost allocation, otherwise you will have a budget shortfall and neglect key investment in the infrastructure to maintain it.
Similarly, make sure you account for full lifecycle costs of a service in the allocation — and be conservative in your initial allocation pricing, revisions later that are upward due to missed costs will be painful
For ‘build’ or ‘project’ costs, do not use exact resource pricing. Instead use an average price to avoid the situation where every business unit demands only the lowest cost IT resources for their project resulting in a race to the bottom for lowest cost resources and no ability to expand capacity to meet demand since these would be high cost resources on the margin.
Use allocations to also avoid First-In issues to new technologies (set the rate at the project volume rate not the initial low volume rate) and to encourage transition off of expensive legacy technologies (Last out increases)
And lastly, and ensure your team knows and understands their services and their allocations and can articulate why what costs what they cost

With this framework and approach, you should be able to build and deliver an effective allocation mechanism that enables the corporation to avoid the overconsumption of free, common resources and properly direct the IT resources to where the best return for the corporation will be. Remember though that in the end this is an internal finance mechanism so the CFO should dictate the depth, level and allocation approach and you should ensure that the allocations mechanism does not become burdensome beyond its value. remember that allocations framework.

What have been your experiences with allocations frameworks? What changes or additions to these best practices would you add?

Best, Jim Ditmore

Evolving Metrics to Match Your Team’s Maturity

March 19, 2012 Jim D

We have covered quite a bit of ground with previous posts on IT metrics but we have some important additions to the topic. The first, that we will cover today, is how to evolve your metrics to match your team’s maturity. (Next week, we will cover unit costs and allocations).

To ground our discussion, let’s first cover quickly the maturity levels of the team. Basing them heavily on the CMM, there are 5 levels:

Ad hoc: A chaotic state with no established processes. Few measures are in place or accurate.
Repeatable: The process is documented sufficiently and frequently used. Some measures are in place.
Defined: Processes are defined and standard and highly adhered. Measures are routinely collected and analyzed.
Managed: Processes are effectively controlled through the use of process metrics with some continuous process improvement (CPI).
Optimized: Processes are optimized with statistical and CPI prevalent across all work.

It is important to match your IT metrics to the maturity of your team for several reasons:

capturing metrics which are beyond the team’s maturity level will be difficult to gather and likely lack accuracy
once gathered, there is potential for unreliable analysis and conclusions
and it will be unlikely that actions taken can result in sustained changes by the team
the difficulty and likely lack of progress and results can cause the team to negatively view any metrics or process improvement approach

Thus, before you start your team or organization on a metrics journey, ensure you understand their maturity so you can start the journey at the right place. If we take the primary activities of IT (production, projects, and routine services), you can map out the evolution of metrics by maturity as follows:

Production metrics – In moving from a low maturity environment to a high maturity, production metrics should evolve from typical inward-facing, component view measures of individual instances to customer view, service-oriented measures with both trend and pattern view as well and incident detail. Here is a detailed view:

Evolving Production Metrics — Production Metrics Evolution

Project metrics – Measures in low maturity environments are project-centric usually focus on date and milestone with poor linkage to real work or quality. As the environment matures, more effective measures can be implemented that map actual work and quality as the work is being completed and provide accurate forecasts of project results. further, portfolio and program views and trends are available and leveraged.

Routine Services – Low maturity measures are typically component or product-oriented at best within a strict SLA definition and lack a service view and customer satisfaction perspective. Higher maturity environments address these gaps and leverage unit costs, productivity, and unit quality within a context of business impact.

The general pattern is that as you move from low to medium and then to high maturity: you introduce process and service definition and accompanying metrics; you move from task or single project views to portfolio views; quality and value metrics are introduced and then exploited; and a customer or business value perspective becomes the prominent measure as to delivery success. Note that you cannot just jump to a high maturity approach as the level of discipline and understanding must be built over time with accumulating experience for the organization. To a degree, it is just like getting fit, you must go to the gym regularly and work hard – there is nothing in a bottle that will do it for you instead.

By matching the right level of metrics and proper next steps to your organization’s maturity, you will be rewarded with better delivery and higher quality, and your team will be able to progress and learn and leverage the next set of metrics. You will avoid a ‘bridge too far’ issue that often occurs when new leaders come into an organization that is less mature than their previous one, yet they impose the exact same regimen as they were familiar with previously. And then they fail to see why there are resultant problems and the blame either falls on the metrics framework imposed or the organization, when it is neither… it is the mismatch between the two.

And you will know your team has successfully completed their journey when they go from:

Production incidents to customer impact to ability to accurately forecast service quality
Production incidents to test defects to forecasted test defects to forecasted defects introduced to production
Unit counts of devices to package offerings to customer satisfaction
Unit counts or tasks to unit cost and performance measures to unit cost trajectories and performance trajectories

What has been your experience applying a metrics framework to a new organization? How have you adjusted it to ensure success with the new team?

Best, Jim Ditmore

Recipes for IT

Best practices for achieving high performance IT

Author: Jim D

Optimizing Technology Infrastructure with Master Craftsmen

Achieving Outstanding IT Strategy

Key Steps to a High Performance Team: Coaching and Development

Getting Things Done: A Key Leadership Skill

Have a great holiday weekend. Jim

Key Steps to Building a High Performance Team: Prune and Improve

Another Wave of Security Breaches: Meeting It with Security Best Practices

Key Steps to Building High Performance Teams

Using Performance Metric Trajectories to Achieve 1st Quartile Performance

Tying Consumption to Cost: Allocation Best Practices

Evolving Metrics to Match Your Team’s Maturity