J9 Technologies

Can we virtualize the virtual servers?

2009-10-04T16:21:00.002-07:00

I'm actually surprised that I haven't heard this question yet. The last few months it seems everyone has been budget conscious to the point of the ridiculous. It's been difficult to convince customers that somewhere out there, real hardware still exists. BAC is supported on VMWare but due to our field experiences we cannot recommend anything less than separate dedicated DPS and Gateway hardware. Even getting virtual servers allocated has been a struggle. It seems like many organizations, in their zeal to reap the benefits of virtualization, have forgotten that under the hood there has to be sufficient hardware to actually support the number of virtual machines, especially if all the vm's are going to operate at fully capacity.

What's perhaps really surprising to me is that companies continue to put full PCs on the desks of all of their employees rather than equipping them with ($400) netbooks or even just Blackberries or iPhones. So a $5,000 server for a core business process is out, but a $1,000 PC belongs on everyone's desk, even those employees who work remotely or who only need a machine for basic word processing and email tasks.

I have to admit, I don't understand the penny-wise pound-foolish thinking about physical hardware. Four weeks ago my laptop, a two-year old Mac Book Pro, suffered a hardware failure at 5:00pm on a Friday night. I knew that it would likely be at least a week before I could get it back from Apple - and a week's worth of lost productivity would easily be more than the cost of a new machine. I barely hesitated to drop the three grand for a new laptop (out of my own pocket) yet companies with a national footprint, who need that hardware to run mission critical systems, aren't able or willing to do the same. Certainly there is a difference between the hard drive in my laptop and RAID 5 network attached storage, but nonetheless disk is cheap by all accounts. Utility computing via "cloud" providers are driving costs down as well, and really creates a situation where the argument that "we can't get hardware for this project because we don't have budget" into a non-starter. There's really no good excuse for the view that "hardware" has to be treated as a precious resource.

Who Owns Monitoring?

2009-10-04T16:21:00.001-07:00

Large companies are often able to support a 24x7 operations center, which make a natural place for monitoring to occur though perhaps not for the administration of those tools. Many of our smaller customers have struggled with the question of where their monitoring and its administration belongs organizationally. By default monitoring tasks generally end up the responsibility of either a few developers or with the administrators responsible for the systems in the production environment. Naturally just because something is the default doesn't make it right.

Monitoring tools generally come into the organization via three sources: operations, support, or development. How tools come into an organization can say a lot about organizational culture. For example, operations and support will generally bring in a tool because they feel constrained by the visibility they have into the production applications and environments -- they have angry customers, but no way to appease them. Developers tend to focus on monitoring tools that enable them to build better applications, but often overlook the importance of tools in the production environment. The team that brings in the tools often ends up as the administrators of the tools, regardless of whether they have the appropriate skills or resources to provide adequate support.

For our customers who are trying to make the most of restricted budgets and headcount, we've found some common trends that lead to success.

- Appoint a champion: Find someone within the organization who has adequate time, skills, and organizational knowledge to become the dedicated subject matter expert for service management.
- Define a process for monitoring and triage: Make it clear who is responsible for operational monitoring and the process for escalation.
- Administration of monitoring tools and actual the monitoring may be separate teams or personnel.
- Don't overlook support: Adding a centralized support engineering team within a support organization can streamline problem resolution.
- Defining a virtual triage team made from experts from database, networking, systems, and application teams that can be immediately activated in crisis situations is a better approach than waiting for the crisis and then taking action.

Ultimately who owns monitoring in an organization is driven by organizational needs and culture, but should be driven by a concerted decision rather than indecision.

Duct Tape Programming

2009-10-02T11:21:00.000-07:00

Joel Spolsky has hit the mark again with his recent blog posting "The Duct Tape Programmer" - http://joelonsoftware.com/items/2009/09/23.html. There are two key quotes "you’re not here to write code; you’re here to ship products" and “Overengineering seems to be a pet peeve of yours.” That's the premise in a nutshell -- quit over-engineering, start shipping. The only thing I fear, and some of the commentary across the net expresses this, that too many will miss his message and instead read Spolsky as justifying sloppy decisions. That's not it at all. Instead, he's taking the experienced, pragmatic approach in which could be best summarized as "just simple enough." A good read, as are his other postings.

Management by Flying Around

2009-10-02T11:17:00.000-07:00

Being up at 4:30 am and a long drive to Portland gave me a chance to reflect on a problem I've been mulling for a while now.

For decades, thinking about business and management has been driven by sports and military analogies and experiences. The post-war generation that built the United States into the world's largest economy brought practices and organizational structures from their military experiences. Even within technology we are not immune to this. When I first saw Scrum, an "agile" method for developing software, my immediate reaction was "This is exactly like the Romans structured their military command, 2000 years ago!" We intuitively understand command-control management, work in "teams," "quarterback" meetings, and of course what executive doesn't play golf?

J9's consultants are located all across the United States - I've never met in person some of the people I work closely with, and others I see in person only rarely. The tactics commonly deployed and many of the management techniques of the past quickly fall apart when you don't have the proverbial water-cooler. The inter-personal issues -- health, relationships, personal interests -- become difficult to track and yet plenty of research has shown management empathy to personal needs as a significant factor in employee retention and job satisfaction. Career planning and reviews, especially when criticism needs to be levied, are lost when timezones and thousands of miles separate your staff.

It isn't simply a problem in personnel management either. I recall vividly the first time I saw a Gantt chart, at age 13. Those colorful bars and perfectly placed diamond milestones sparkled with their organizational efficiency. Perfection, yet completely useless if your project consists of loosely related tasks without strict dependencies, especially one where the personnel ebb and flow in and out of the project. Installing a piece of software -- there's something you can put on a gantt chart. Whether the customer has successfully developed the skills to support the software? A less well-defined task.

So here comes the summary: Companies are ever more virtualized, global, and 24x7, and it isn't just the largest companies and in the executive office that these demands appear. The management practices of the past, with their roots in industrialism, simply aren't working. I don't yet know what the answer is, but change is imminent.

Happy Days are Here Again

2009-09-30T10:05:00.000-07:00

Recession? What recession? J9 is actively seeking Solution Architects. Are you an experienced consultant who understands the benefits of life at a smaller firm, where you can direct your career? Check out our posting here: http://www.j9tech.com/careers.html and apply today.

But did you do the phosphorus test?

2009-07-06T08:57:00.000-07:00

I heard the phone clang down and my colleague Steve distraughtly mumble "She's going to kill the fish." His wife called to tell him about a phosphorus problem in their fish tank at home. She's a medical researcher, a biologist by training. Steve's first reaction when she told him there was a phosphorus problem was to ask if she had in fact done a phosphorus test. No, she said, but she'd run through all of the other chemical and algae tests, so of course it had to be the phosphorus and thus she'd started adding more phosphorus to the tank -- they'd know in a few days if that was the problem. Steve, imagining coming home to a tank of dead fish, was not impressed that his scientist wife had failed to use the scientific method at home.

It's so often like that in technology as well. Despite years of rigorous training to use the scientific method to guide our actions (it is called "computer science" for a reason), it's easy to throw all that away when faced with a challenge. A customer came to me the other day asking about monitoring tools to help with a production triage situation for a failing web service. A developer assigned to the task interrupted us saying that a fix had been deployed ten minutes prior and it looked like it was working. Let's reflect upon that:

a) No load or performance testing scripts existed for this web service.
b) No monitoring or profiling tools had been deployed with this service in either a pre-production or production setting.
c) A hopeful fix had been hot-deployed to production and left to run for a mere ten minutes before victory was declared.
d) No permanent monitoring was put in place to prevent the next occurrence of the problem.
e) Apart from a few manual executions of the service and a face-value assessment by one individual, no further validation to correlate the fix with the perceived problem occurred.

Chances are good that Steve's fish will be fine, but can the same be said for those cases where we play roulette with mission critical IT systems? Just as in the case of Steve's fish, there is no legitimate reason for a lack of objective, quantitative analysis except basic human apathy. Anyone who has ever taken a statistics course or been face-to-face with a serious production issue knows that just because many other tests have ruled out many options does not mean its safe to jump ahead and make assumptions just because of gut feeling -- why abandon a working method for one that brings doubt, risk, and exposure to criticism? Run the phosphorus test and let the results be your guide.

A video speaks a thousand words

2009-07-03T08:22:00.000-07:00

It is nothing new for us to be constantly developing new educational tools. Demos and lab materials for trainings on site, or content for our evolving KnowledgeBase that augments the HP software support we provide to our customers. But the videos are the biggest hits so far. They pack a three minute punch of information without leaning on those lazy powerpoint icons. Check 'em out.

Business Transaction Management in palatable terms (no yawning required):
http://www.youtube.com/watch?v=49tQ9BpnrT0

In case you missed the first one, here it is:
Why J9? Well, since you asked...
http://www.youtube.com/watch?v=FjPlvO01SmA

Please rate them! We'd love to get some feedback on how well these videos connect with you and for god sakes, if they are still boring, please let us know.

How would you test a 4000 user community?

2009-07-02T22:54:00.000-07:00

That question was the lead in to a discussion I had with a colleague this week. He had been interviewing someone for a performance testing role and that was the key question that could make or break a candidate. The typical response goes something like "I'd start with one user, then move on to five, then ten, then 50 then 100, then... all the way up to 4000." While the most common answer, this is entirely wrong. This kind of common yet broken testing process explains why the group of us that joined the conversation could each retell case studies of customers who had spent multiple years (and millions of dollars) on failed testing efforts.

The right answer goes like this:

a) Ask the hard questions
How many of the 4000 users are concurrent users and what is their use pattern? For example, many batch billing systems do nothing for 29 days per month, but then run through a massive number of transactions on the last day. Other systems have limited daily use until 5pm when their user community arrives home from work and then signs in. Are the users spread across multiple timezones?
If the data to discern the number of real concurrent users isn't available, that actually means two things to our project:
1) A separate project is needed to put in place tools to capture user behavior. The lack of such information can cause poor decisions in the areas of testing, capacity planning, security, and product usability design and functionality.
2) If no such data exists and the 4000 number simply means we have 4000 users in our database, we can now back into a more realistic upward bound through some basic calculations.

b) Functional performance test
Start with one user as a means of functional performance test. This enables you to validate your test cases and test scripts and flush out any immediate functional problems with the application(s).

c) Longevity testing, peak testing, failover testing
There are a variety of other tests with greater pertinence and validity in understanding the application's serviceability than simply running through the same script with a randomly increasing number of virtual users.

d) Load and Performance testing
If we've determined that simply starting with one user and continuing to double isn't the right process for load testing our application, then what is the right heuristic for getting to the Nth user? The answer is that it doesn't really matter, as we've determined, in effect, all of the above through the answers to our questions about the user community. If we have 4000 users in our database but don't know how and when they use the application, a test of 200 users as a top number is just as valid as a test of 2000 users. Using these numbers though, one can arrive at some guidelines by looking at the length of a user day. For example, if our application is used by an internal business customer that only works standard business hours in the eastern time zone, then we can surmise a roughly 8 hour work day, 5 days per week. Take 4000 users, divided by 8 hours, we can take an educated guess that there are 500 users per hour. Take an 8 hour day, multiply by 60 to get 480 minutes, divide the 4000 users by 480 and we can surmise that at any one minute interval there are likely to be 8 users on the system. In the absence of further information about our user community, we now have real, actionable numbers to test against. Rather than the dozens and dozens of incremental tests we were potentially facing, we can now break our cases into one user, 10 users, 500 users, and anything above that is essentially to discover the upward bound of our capacity.

These steps are a productive tool to improve the quality of your testing, as well as a great way to gain new insight into the candidates you interview.

OVIS to BAC migration, anyone?

2009-06-29T08:14:00.000-07:00

J9 Technologies, Inc. Announces Migration Solution for OpenView Internet Services Users

J9 Technologies, Inc. announces a limited-time migration solution for Hewlett Packard customers currently using OpenView Internet Services (OVIS). As of December 31, 2009, current OVIS customers must migrate from OVIS to HP's BAC and SiteScope solutions in order to receive licenses, customer support, patches, and updates from HP. As a result of the Mercury acquisition, HP announced that it is dropping support for the OVIS solution in favor of the Business Availability Center (BAC) and SiteScope solutions. HP is offering, free-of-charge, a license exchange from OVIS to BAC and SiteScope.

J9 offers expert services to streamline the customer's license acquisition process, pre-migration planning, end-to-end migration from OVIS to BAC, and identifying best practices for ongoing utilization of the new BAC solution. With J9, the migration process is not just one of conversion, but evolution. J9's solution reduces a customer's time to value risk during the migration process.

J9 works with each customer individually to determine their current state, identify gaps and provide a plan for migration. Throughout the process, J9's experts partner with the customer to ensure the migration not only replaces their current state, with no gaps, but allows the customer to be positioned to expand their capabilities utilizing the enhanced BAC platform. J9's services expand beyond OVIS migration solutions to offer a complete range of basic and advanced BAC implementation services and training programs designed to enable the customer to take full advantage of the power of the BAC platform.

Business Availability Center brings a robust platform to address a broad range of application environments. BAC's rich feature-set offers customers an enhanced platform addressing their Business Service Management needs and initiatives, including application diagnostics, service level management, business transaction management, and superior dashboarding capabilities. The BAC platform is capable of supporting a wide range of application environments beyond standard web-based applications, resulting in enterprise-wide coverage for the customer's most business critical systems, such as ERP/CRM, E-Mail, SOA, Web 2.0, and client-based systems.

About J9 Technologies
J9 Technologies is a certified Hewlett Packard Gold Partner, specializing in Business Service Management and Application Performance Lifecycle management.

Offering strategic consulting, training, installation and ongoing support for HP software products such as SOA Systinet, Service Test, Business Availability Center, Diagnostics, LoadRunner / Performance Center, Real User Monitoring (RUM), TransactionVision. With a focus on application diagnostics, composite application management, and business transaction management, J9 Technologies strives to ensure that customers can quickly identify the root-cause of issues and minimize mean time to resolution.

For more information:
J9 Technologies, Inc.
24 Roy St., Box 211
Seattle, WA 98109
Tel: (866) 221-8109
Fax: (206) 374-2901
www.j9tech.com

J9 @ HP Software Universe 2009

2009-06-24T10:28:00.000-07:00

We are just catching our breath after few a long and fruitful days in Las Vegas, where HP held it's annual Software Universe conference. We exhibited, we rolled out our OVIS to BAC migration offering, we chatted, we happy hour-ed, we presented, and after it all, we slept. For a long time.

And now here we are, in all of our glory, having emerged from the gambling flames of software sales and services just a little bit tougher, a little bit wiser, and a feeling lot more connected. Thanks to everyone who came by the booth (and picked up one of our lovely one-handed bottle openers pictured above) and who attended our happy hour event. It really was a blast getting to be face to face with all of these people we work with everyday. See you next year!

What are you testing?

2009-06-08T10:54:00.000-07:00

Yet another Monday after a red-eye flight, at another new customer in some city far from home. I'm ushered into a conference rooms with a team randomly checking their watches and Blackberries, each tense, nervous, and wishing they were somewhere other than this. A project manager begins the briefing, which everyone has heard a dozen times before: the project is past its scheduled delivery, application performance is terrible, the users are angry, the sponsors are talking about pulling funding, and the order for the past month has been forced 15 hour days and weekends. This afternoon they want to begin another series of load tests and they ask me if our team of consultants will be able to help them. My only question is: What are you testing?

It's a simple question. I find myself asking it several times per day with customers and sadly the usual answer is a blank and distant stare. The reality is they don't know, but it's not their fault. To be honest, I'm not entirely clear myself on how the state of the industry reached this point. The trend though goes like this:

1) Testing happening just prior to deployment, often not even starting until the same week.
2) Tests being run by an operations team or perhaps by a lone consultant, without the involvement of a business owner who understands the functionality of the application.
3) A random mishmash of tools, some properly licensed, some not, downloaded and used without any training or guidance.
4) Test cases being made up on the fly - "Hey, what if we try this?" without any rhyme or reason as to the validity of the test.
5) Little or no written record of the test cases executed or their results, leading to duplicate efforts, errors, and a lack of objectivity.

It all really boils down to that same question -- if you can't answer, objectively, the reason for a particular test run, then your efforts are for nought. Once you are on that treadmill of test run after test run, it's impossible to get off without a hard reset; the only objective test at that point will come when the application is introduced to the real world.

Once an organization has reached this point, where production firefighting is a daily routine, the only way to bring order back is to reverse the list above. Testing must be an integrated part of any development or implementation project. Developing a written, comprehensive, repeatable test plan requires input not only from technical teams but also from end users and business analysts who understand the use cases -- both common and edge. Only then will you know what you are testing.

This is a plea, continued...

2009-06-04T10:53:00.000-07:00

Shortly after I wrote the last entry, about tools that get purchased and not used, I realized that I hadn't really described the reasons why this happens. Sure, there are reasons like insufficient training, inadequate tools for the problems at hand, or organizational ownership and politics. But the most common reason is far more human and harder to mitigate: apathy.

A friend of mine is a Six-Sigma and Toyota Process expert. He knows and understands quality and process control methods inside and out. I asked him about the barriers to organizational adoption of these practices. His answer was very telling. It wasn't money, time, tools, training, or anything measurable. Instead his answer was very simple -- because people don't like being told what to do.

There's a parallel here. Unless a tool, process, practice, or even idea fits within someone's world view, they simply aren't going to adopt it. It's even worse if they feel like something is being forced upon them, rather than being an engaged participant. Even if they feel invested in the tools, good old fashioned apathy can take hold.

Why? Incentives and consequences. Its just tremendously easy to become apathetic if there are no incentives or consequences to our actions. Many times after J9 consultants find a problem with a customer's application, we hear rumblings of "oh yeah, I wondered if that was a problem." Easier to turn a blind eye then to invest ones self in learning a tool, exploring a problem, and coming up with a solution. Especially since in so many organizations that live by firefighting, volunteering is akin to accepting blame for the problem.

So, the guiding points here appear to be:

1) It's imperative that all types of users are included in the selection and implementation process so that they feel invested.
2) Management has to continually communicate intentions and uphold the standard.

On this second point, here's a really example of how this can be done in a way that emphasizes the point yet stays away from simply giving orders. Say for example it's a Monday morning triage meeting and an outage that happened over the weekend is the topic. The questions isn't simply "What happened?" but also "What did the tools tell you happened?" This is a means of continuous communication about the expected tools and the expected method, plus a means of continual assessment of the value proposition -- if your tools are proving less than useful, then perhaps there is a legitimate reason why they aren't being used.

This Is A Plea

2009-04-25T08:00:00.000-07:00

This is a plea: please, please use the tools you've invested in. Nothing makes a J9 consultant sadder than to see customers throw their money away on tools and training. Second to that is to see customers try to build tools on their own when there are other options available.

Two cases in point here: HP Diagnostics and HP TransactionVision.

The Diagnostics product has been around for many years, coming into the HP fold in January 2007 as part of the acquisition of Mercury Interactive. Many customers are resistant to using this product, believing either that its overhead is extreme or that free/cheap tools can provide the same information. The truth is, Diagnostics adds very minimal overhead to any production system if it is configured correctly. It has been safely deployed in high-volume production settings to dozens and dozens of J9's customers. Many free or inexpensive tools offer point solutions, but none are capable of providing the complete environment monitoriing and historical data collection capabilities of Diagnostics.

Many customers have recently shared with us that they are so desperate for business transaction reporting and monitoring that they are considering building their own solutions. TransactionVision came to HP via an acquisition of Bristol Technologies, which had been a partner of Mercury Interactive. When we demonstrate it, the feedback is almost always a sense of relief "Thank goodness I don't have to build this for myself!" Sensors are deployed for Java, .Net, WebSphereMQ, CICS, and Tuxedo, with reporting and monitoring integrated into HP Business Availability Center.

So, please, don't build what you can buy, and if you are going to buy, please don't let it become shelfware.

When Java Memory Issues aren't.

2009-04-20T08:50:00.000-07:00

What I'm going to write is likely an unpopular viewpoint, but one that needs to be expressed. One of the most frequent sentiments expressed by customers when we are doing an application analysis is that their Java application is suffering from crippling memory issues. I'm not going to go on record saying that memory issues don't happen in Java -- I'm quite certain that there are situations where legitimate memory management issues within applications and the virtual machine happen. But, a more common scenario is something like customers placing very large objects into user http sessions, then being surprised when their application server slows to a crawl and crashes out of memory. Guess what? That's not a memory issue, that's a poor architecture and design issue.

Normally about this time I hear a chorus of "but, but, but! Why isn't the VM managing my memory correctly?!?!?!" It's a virtual machine, not a miracle worker. In the case above, the http session is being abused and overused. There are several options to that particular problem, depending on the business requirements:

a) Reexamine necessity of all data being stored in the session and reduce.
b) Move much of the data to a separate cache with more granular control over cache clear/timeout strategy.
c) Externalize session data entirely to a separate user session server with purpose-built hardware and cache timeout strategy.

All of the above are overkill you say? Sure beats running out of memory it seems to me. About this time I often hear someone pipe up about "Why don't we just force garbage collection?" or some similar pearl of wisdom, such as explicitly finalizing objects. Both are flawed -- you'll recall that the vm specification states that marking an object to be finalized does not explicitly force it to be finalized on any particular schedule, and any suggestion of forced garbage collection is specifically not recommended. Generally the response I hear to this is some grumbling about a lack of control over memory management. Therein lines the rub: If you need explicit memory management due to the nature of your requirements, then Java and .Net are not for you. And, as demonstrated above many issues that at first appears to be memory related are in fact not.

Don't be held hostage by the BOfH

2009-04-17T14:14:00.000-07:00

The Bastard Operator from Hell stories (http://www.theregister.co.uk/odds/bofh/) have delighted IT personnel since the early 1990's with their comic view of the power of systems administrators and their place in corporate America. For those not familiar, the basic premise is that of a renegade system administrator who makes mischief on systems and users as he pleases. The real world version is unfortunately not as amusing as the comic.

Years ago I worked at a company held hostage by such an individual. As systems crashed at random intervals, enough peculiarities came to light to make it apparent that this individual took some strange delight at causing disasters that he could then "rescue" as a way of self-promotion. Sadly in that organization it worked for a long time, mostly out of fear. After all, this BOfH was of course the only one who understood how much of the network was configured, the only one who knew various passwords, and other assorted bits of knowledge. Past attempts to involve others had only increased the number of "surprises" which further increased the dependence.

This issue is particularly relevant right now as tough economic times cause companies to scale back. Cutbacks, in the form of business activities, training, or personnel, can cause key knowledge to consolidate into a few essential personnel. This introduces tremendous organizational risk can be harmful to both the staff and the company as a whole in the following ways:

1) Makes it impossible for those individuals to take sick days or vacation, increasing the potential for burnout.
2) Puts the company at the mercy of those individuals in accomplishing work.
3) Breeds animosity through the creation of a "two-tier" team where some are "in the know" and others are not.
4) Creates a priority mismatch where the agenda of a few can trump the priorities of the team and company.
5) Contributes to organizational churn as resources are directed toward firefighting.

The best way to manage this is simple: take strides to counter it and build a culture that encourages openness rather than firefighting and heroics. Build in knowledge sharing as personnel evaluation criteria. Assess the resource needs of your teams in order to ensure adequate staffing and tools. And don't be a victim of the BOfH.

Why J9?

2009-04-10T11:41:00.001-07:00

Check out our new video! It'll take 3 minutes or less, we promise.

http://j9tech.com/whyJ9.html

Or check us out on youtube:

http://www.youtube.com/watch?v=FjPlvO01SmA

To keep up to date on new our videos and other new developments here at J9, shoot us an email at info@j9tech.com.

Reducing the Operational Costs of SOA

2009-04-10T10:34:00.000-07:00

Introduction

Software evolution has taken us a long way from the mainframe to distributed computing in the cloud. What hasn’t changed is the need for operations teams to effectively manage the applications and infrastructure that businesses and organizations rely on. In this article, we will discuss how to effectively manage these composite/distributed/SOA applications by leveraging the existing expertise of operations teams along with tools and methodologies that ensure high availability, reduce the impact of outages, and cut out the reliance on costly all-hands bridge calls. In addition, we’ll give a few examples of how other organizations have adapted to these new software architectures and steps taken to prepare for them before they landed in production. Lastly, we’ll cover the cost of not doing anything and how that price is paid elsewhere.

Case Study: Cable Nation

The challenges for today’s operations teams are increasing exponentially, given the number of physical and virtual hardware layers that are used to provide a single piece of functionality. One fictional company that I’ll refer to throughout this article is a cable company called Cable Nation. Cable Nation used to be a small, regional cable provider with a single billing system that was hosted on a single mainframe system. At it’s infancy, the health of the billing system could be determined by the sole operations engineer who could log into the mainframe and check the vital system stats – CPU, memory, disk, IO rates and a few other key metrics that were hardly ever out of whack. Today, the billing system for Cable Nation stretches across multiple physical machines, virtualized devices, across many internal data centers and integrating with multiple external vendors. This rise in complexity solves many business-related issues involving disparate accounting methods across subdivisions, access to customer account information across organizations and ensuring that some of this data is available to external business partners. The architecture chosen to meet these new challenges is a SOA architecture.

Nowadays, tracking down a particular issue with the billing system involves a lot more servers, databases, applications, services and multiple mainframes. As a functional unit, the billing system continues to provide many of the same functions that it always has, but now it is much more difficult to find the root cause of a problem due to the distributed and complicated nature of this system. In many cases, the application architects are brought in to help resolve critical production issues, primarily because the operations teams are ill-equipped with their system monitoring tools to address what are now issues that can only be identified and isolated using application monitoring tools. In this paper, we will discuss some of the fundamental differences between the systems versus application management approaches and some of the tools and techniques that enable operations teams to more effectively manage these new systems with the same level of adeptness as they were able to when these systems were less complex.

In the broadest sense, the problems encountered in these distributed systems are tractable, and identification of the root cause of an issue is largely deterministic, based off of the capture of a few key metrics or key performance indicators (“KPIs”). In many cases, finding the source of a software problem may be tracked down to a single web service, operating on a specific machine, hosted in one physical data center, administered by a person who is responsible for the overall uptime of that service. However, companies like Cable Nation pay a huge price in terms of the number of hours spent by engineers on bridge calls, development tasks being delayed, or lost customers due to the inability to service them in a timely manner. This doesn’t even count the additional applications that depended on that single service that failed during the down time or the cost of missed external SLA’s tied to real dollars. Usually, these types of scenarios can be reduced with the appropriate amount of testing, but in many cases, the cause of downtime can be as trivial as a hardware upgrades or the application of a patch that causes unintended side effects. At the end of the day, there are always going to be problems in production and reasonable measures must be taken to alleviate the cost of those errors.

In essence, the primary cost of decreased availability and performance is due to the time and # of resources needed to resolve a problem. Consider this relatively conservative equation:

In this case, we’re looking at ¾ of a million dollars just for 125 issues!!! Imagine how quickly this number increases when you double or triple the # of incidents. Note that this was also a very conservative estimate. In many companies, the # of engineers involved in severity 1 & 2 issues is much higher – I’m sure many of you can relate to hanging out on bridge calls until a problem is resolved. Imagine how much money would be saved if the # of engineers needed was cut in half or the mean time to repair (“MTTR”) was reduced even by 25%? This is the magnitude of the problem and in this particular scenario, we’re considering that the # of incidents stays the same. In other words, there are some processes that we can put in place to reduce the potential # of incidents, but we’ll talk about those separately. The key to this is to recognize that the cost impact just in terms of people and their time is a large part of what we’d like to address as a part of this discussion.

In the case of Cable Nation, executive management had already recognized this cost was eating away at their ability to do business, in terms of both time spent resolving issues, but also in diverting time away from new development projects. In order to keep these costs under control and adhere to their hiring freeze, they began looking for tools that could address the main 2 factors they had control of:

* Reducing the time involved in resolving issues
* Reducing the # of people involved in issue/problem resolution

Taking a top-down approach, the best means of reducing the time to resolve problems isn’t simply getting more people involved, but rather arming a few key individuals with the appropriate quality and quantity of data that allows them to quickly pinpoint, triage, escalate, or resolve the problems in a matter of minutes. In the case of distributed applications, the sheer # of metrics that must be tracked is daunting in and of itself, but when those metrics and their relevance changes due to a few simple software modifications, then the ability to triage issues is compromised greatly. In the case of Cable Nation, the # of unique web services rolled out on a single release was measured in the 100’s initially. Only the developers and a few of the QA engineers knew how and when these services communicated with both internal and external systems. This dependence on the knowledge of developers and QA teams kept them constantly in the mode of fighting fires and taking on the role of operations staff due to the inability to track down issues using the same old methods. While the first release of the SOA re-architecture was deemed success, the second release was delayed multiple times, primarily because development was constantly brought in to resolve production outages and performance issues. The frustrating part for the developers was that they produced almost exactly the same # of bugs that they had when developing standalone (non-SOA) applications. The main difference was that the operations teams were ill-equipped to handle the complexities and issues associated with this new distributed architecture and so they were often unable to help solve problems.

So how do you prevent the scenario that Cable Nation faced? In other words, how do you arm operations with the effective tools and expertise to manage a SOA-based architecture? We will now begin to answer these questions and address the costs associated with not taking the appropriate steps…

Understanding MTTR

I had mentioned earlier the concept of Mean Time to Resolution, sometimes called Mean Time to Repair – luckily both can use the MTTR acronym. In the problem resolution process, there are actually 2 phases – problem identification and then the problem repair. In many cases, the identification phase is the most difficult. Usually, the symptoms of a particular issue may be similar to another that had been encountered, but often it turns out that the root cause is vastly different. Take the following example: “The database appears to be slow” is the issue identified, but the root cause may be due to a number of factors: network I/O, disk I/O, missing indexes, tablespace locations, SAN storage, etc. The key to effectively reducing the mean time to identification (“MTTI”) consists of having the best data and a person equipped with minimal training to interpret the data. For instance, in the case above, the MTTI could be reduced if the tier 2 support engineer was armed with an alert or report that tracked the performance of a single database query over time and was able to spot an increase in time. The escalation then would be handed off to the DBA who would tweak the execution plan for the query. All of this proactive identification and resolution may have very well taken place without having to raise a severity 1 or severity 2 incident.

Unfortunately, in complex software systems, humans are often the weakest link in determining the root cause of issues. In the best case scenario, all relevant system health and metric data would flow to a single engine that was able to rank, sort, categorize and analyze the data until the problem was detected. Once the issue was found, all the operator would have to do is press a big red button that made the problem go away once and for all. In reality, the big red button doesn’t exist, but there are many tools that provide the operations engineer the data that wil allow him to resolve the issue on his own and with in a fraction of the time it would normally take.

So What Makes SOA Management Different?

You’ll note that we’ve steered clear of the SOA terminology mostly through the previous paragraphs. From the perspective of the operations engineer, whether the application uses web services or adheres to a SOA architecture is not nearly as relevant as understanding the differences between system and application management. In other words, while it’s advantageous for operations teams to understand some fundamentals of SOA, it is not necessary for them to be experts in the lowest level details other than how problems should be identified and resolved. One example of this is the use of web services security – the operations team may need to know that there are web services that help govern the security of the SOA system and that there are key metrics that indicate whether or not the authentication system is operating at it’s optimal capacity (which is a combination of CPU utilization on the authentication web service, the # of concurrent authentication requests and an optimal connection pool to the authentication database). Operations engineers do not need to understand the specific authentication algorithms used or the fact that XML is the primary on the wire protocol. It is these differences that separate operations from development, QA and the business. At some point, the well-formedness of the XML or a buggy authentication algorithm may be identified as the root cause of an issue, but ideally the operations teams have ruled out all other problems before handing it off to the R&D engineers to investigate. Hopefully, the tools have also captured the information necessary to resolve these lower-level problems as well.

The differences between SOA/distributed systems management can be described primarily as a function of the # of unique services and the distributions of those services across the network. In addition to the distributed nature of the applications, the traditional application teams boundaries are a challenge as well. This can complicate the resolution of an issue because there are often more than one distinct development groups that develop a single “application”. This makes SOA application development quite different from more traditional development, and this has a downstream effect on how these applications are managed. The differences in SOA management versus more traditional non-distributed applications is summarized in the following table:

In order for operations teams to deal with these issues effectively, there are a few strategies that we’ll cover in the next section.

Developing a SOA Management Strategy

At the heart of any SOA management strategy, is the recognition that loose coupling brings along with it an increased amount of management. The traditional “application” may consist of a single web page along with two web services, or it may stretch across multiple service layers, consisting of both synchronous and asynchronous requests. The view for operations teams into these varied application topologies should be simple and consistent, so that when a performance or availability issue occurs, the exact location (physical, virtual, and relative to other services) can be identified along as the impact it has on other applications. The key elements of a cost-effective management strategy for SOA consists of the following elements:

Problem Prevention – Performance Testing

Obviously, the best way to resolve issues is to prevent them from occurring in the first place. The 1-10-100 rule should always be considered – it costs $1 to solve a problem in development, $10 to solve the same problem in QA, and $100 to solve it in production. To ensure that operations teams are ahead of the curve, the application and QA teams need to determine the Key Performance Indicators that are relevant to the operations teams. In many cases, this determination is a collaborative effort between QA, operations and development teams.

The primary need to identify KPI’s is to reduce the total # of potential metrics that an operations engineer is responsible for tracking and understanding. In some cases, this is a best guess based on performance data ascertained during performance testing, and in other cases this KPI data can simply be extracted by thumbing through help desk tickets, incident and problem logs, or simply by interviewing individual developers to identify components and/or functionality that may be cause for concern. At the end of the day, teams that conduct extensive testing not only root out problems before they creep into production, but as a side benefit – it helps to narrow down a core set of metrics that operations teams can utilize to track the health of the application in production.

In addition to manually identifying KPI’s, many application monitoring tools have out-of-the-box configurations that vastly reduce the time and effort needed to identify these KPI’s. For instance, HP Diagnostics monitors JVM heap memory, and web service request latencies without requiring developers to provide hooks or logging code to identify issues.

Here’s a few steps that can be used to start you down the road towards preventing issues before they crop up in production:

1. Establish baseline for production application KPI’s
2. Utilize the baseline and KPI’s for conducting realistic performance tests
3. Feed KPI’s back to development and architecture teams for use in planning additional system enhancements

Problem Prevention – SOA Governance

One of the most effective way of reducing production headaches is through the implementation of an effective SOA governance strategy. In the more traditional sense, SOA governance is largely concerned with providing a contract between the business and the IT organizations. The implementation of a governance strategy ensures that the business remains in control of it’s own destiny, instead of handing the reigns over to IT, crossing their fingers and hoping that the solutions provided meet the ever-changing demands of the business.

So what does an operations team care about governance in the first place? Well, the impact of a good governance strategy insures that the SOA architecture is designed to reduce the number of applications and services needed to meet the business demands. Reduction in the number of services needed to provide more or less the same functionality may halve the number of potential points of failure in a SOA-based architecture. Reuse of services is a key component to effective governance, but this reduces the overall number of potential failure points for the applications – in a sense more bang for the buck.

In addition to reducing the number of potential services deployed across an organization is also the insurance of setting policies on those services so that they behave as expected. For instance, setting policies that restrict or limit access to certain services ensures that web services aren’t utilized outside their defined scope. This level of problem prevention and runtime governance of services can be accomplished using HP’s SOA Policy enforcer. The importance of policies can’t be more relevant when operations engineers are faced not only with the uptime management of services, but of the legal and business requirements that they remain compliant.

Problem Prevention – Capacity Planning

Another effect a SOA governance strategy has on operations is by informing of additional services that will be deployed in subsequent releases so that operations teams can make the best use of existing resources or gain enough lead time to plan for additional capacity. Services definitions include the anticipated quantity of requests expected and combining this with the service monitoring tools will ensure that hardware can be allocated in sufficient quantity to serve additional service demands.

Additional runtime metrics obtained by SOA management tools like HP’s BAC for SOA gives operations and developers realtime performance data that can be used to identify bottlenecks, thereby providing capacity where it was previously constrained.

Reduce Problem Impact – Reduce MTTR/MTTI

As we saw earlier, the cost of reducing MTTR is measured in the millions of dollars for even a modest number of issues. Any steps that can be taken to reduce the time to identify and resolve a problem will drastically reduce the cost to the business. In the case of SOA applications, the need to proactively monitor and alert on problems gains additional significance due to the greater potential for a single service to effect a larger number of applications. The more difficult challenge for the operations teams is simply to understand and be able to visualize where to begin tracking down problems.

The first step to reducing MTTR is to do everything possible to identify problems before they become critical. Reducing a problem from a severity 1 issue to a severity 2 issue is one of the key practices employed by successful operations teams. There are 2 means of helping to identify problems before they raise in severity:

* Active monitoring – to help identify issues before end users experience them
* Passive monitoring - with trend-spotting capabilities

Active monitoring is accomplished primarily through the use of simulating end users, or in the case of web services – service consumers. These business process monitors (“BPM”s) regularly perform requests that occur on a daily basis. These monitors may be tied into a deep-dive monitoring tool that provides end-to-end visibility of where bottlenecks occur or where there are service interruptions by tracking every network byte sent and all the way down to the line of code executed by the web service. HP’s BAC for SOA software provide this top-down monitoring capability, allowing operations to be notified as soon as a service begins misbehaving. Once the service is identified and metric snapshot is taken (via HP Diagnostics for SOA), the operations staff has the information necessary to pinpoint the root cause of the issue or provide high quality metrics to a tier 2 or tier 3 engineer for additional troubleshooting.

Passive monitoring along with trend analysis can help to identify problems that emerge over time. For instance, the auto-baselining feature of HP BAC along with the Problem Isolation module informs the operations team when there is an aberration in the metrics. For example, a metric that captures the response time for a web service could be auto-baselined and as the service response time starts to degrade (due to higher request volume or latency in it’s database requests), then an alert would be sent to the operations team for investigation.

The shift for operations teams in managing SOA applications is that many of the existing tools used to chart performance and availability simply don’t work as well in a distributed/SOA-based architecture. In even the simplest of SOA architectures, the path a transaction follows is largely dependent on the data contained in the request. Services route requests to the appropriate recipients and so when errors occur or transactions fail to complete, the determination of where the failure occurred often requires knowing what individual user sessions initiated the request and what the request contained. Because the data is split across multiple service layers, the ability to trace a specific request is reduced unless tools that provide end-to-end transaction tracing are employed.

Adding to the complexity, many SOA-systems consist of multiple vendor’s SOA application servers and messaging solutions. Any tool used to monitor these systems must take into account the heterogeneous nature of these systems and still present a common view to the operations teams. No matter how good an engineer may feel their skills at problem solving may be, the challenges presented by a vast array of web services infrastructure is daunting, even to the seasoned pro. The worst case scenario is for every system to be monitored by a different vendor-dependent tool and so problem identification is hampered because they are constrained to their own environment and don’t capture the interactions between system boundaries, which is the place where issues most often occur.

In this case, the operations teams need a single pane of glass into this new layer of application architecture. Tools such as HP’s Diagnostics for J2EE/.NET/SAP can provide service and application-level metrics along with automatically discovering application topologies by tracing requests across system boundaries. Tools like HP’s TransactionVision provide even deeper insight into the data-dependent routing of requests through heterogeneous SOA-based architectures by capturing events spanning J2EE containers, .NET CLR’s, MQ queues, Tuxedo systems, or even CICS or IMS calls. Knowing exactly where a fault occurs is one of the biggest issues with SOA applications. If operations teams are provided the appropriate tools, then the tried and true operations procedures begin to kick in. Otherwise, developers and QA staff will continue to be pulled into every issue and instead of empowering operations with the tools and processes to manage these new SOA applications, operations will be handled by developers and no one will be left minding the store.

Reduce Problem Impact – Issue Impact and Awareness

Due to the inherent interconnectedness of SOA-based architectures, the need to reduce the impact of outages has risen in importance. The additional dependencies between applications and services require operations teams to understand the effects configuration changes have on dependent services. Though the use of application topology discovery and service dependency mapping, operations teams can isolate the impact that changes have across all applications dependent on a single service.

In terms of cost, operator configuration changes account for upwards of 60% of unplanned application downtime. In the case of SOA applications, this impact can stretch higher when operators and configuration managers are unaware of service dependencies. Though great pains my be taken by the SOA governance teams, the ability to quickly detect service changes and additional consumers can result in additional downtime due to the unexpected consequences of change. In order to restore service to a malfunctioning web service, operations teams need to have the full picture of which services upstream or downstream will be effected by an upgrade or change so that adequate measures can be implemented to reduce service disruption.

Once again, tools are essential to map dependencies between systems, especially those that keep the models up to date on a minute-by-minute basis. HP’s Dynamic Discovery and Dependency Mapping (“DDM”) and TranscationVision software tools empower operations teams with a clear picture of service dependency and allow them to determine the moment a transaction is failing to complete whether due to a bug in the code or due to a malformed request. Visualization allows operations teams to proactively notify the businesses when an outage occurs so that they can take the appropriate steps to work around the problem. This level of interaction with the business ensures a positive dialog with the IT organization.

Reduce Problem Impact – Reduce # and Cost of Resources

In the same vein as reducing MTTR, the reduction in the number of resources required to solve an issue is a critical step in developing an effective SOA management strategy. Best case scenario is that a tier 1 or tier 2 operations engineer utilize a single dashboard that notifies them of events and aids in the correlation so that a root cause is identified more quickly. The worst case scenario is one that many of you are familiar with – bridge calls lasting multiple hours or even days. By far, the best way to reduce the number of people involved in a bridge call is to provide a smaller subset of engineers with KPI data to quickly pinpoint issues.

After the problem identification step, the quickest route to a resolution is to escalate the problem to the appropriate person or team that has the experience to solve the issue the quickest. In many cases, the process of triage in and of itself is complicated with SOA applications due to the sheer number of groups supporting a single piece of functionality or transaction type. In these cases, having the tools that pinpoint the team and owner of a service or application can greatly reduce the unnecessary calls and time wasted pinpointing the expert who can resolve the problem the quickest and with the smallest impact on other resources.

Another important aspect of resource reduction is the empowerment of tier 1 and tier 2 engineers to resolve issues on their own. Even if they are unable to pinpoint the exact root cause, the engineers can be supplied with run books or standard procedures that ensure they collect the most pertinent data before escalating to other tiers. Additional tools can enable these tier 1 and tier 2 engineers with automated processes for narrowing down problems. Software tools like HP’s Operations Orchestration enable these operations teams to perform a series of steps to help troubleshoot problems before requiring the assistance of a tier 3 resource. Automatically checking log file contents, pinging servers, or capturing performance metrics can reduce the time spent by involving more advanced techniques and more expensive resources. In other words, the more that can be done to automate the problem resolution process, the less time is needed by engineers downstream.

Conclusion

As we’ve seen, the operational management needs surrounding SOA are in many ways the same as those for more traditional web-based applications. The fundamental differences come down to a few key areas and require operations teams to employ new tools that provide them the insight they need to effectively manage these new SOA-based architectures. The key to an effective strategy is to recognize that operations teams need to be aligned in much the same way that architecture teams have met this new distributed application paradigm. The alternative is for these same development teams to spend more time doing the work of operations teams – which is a very cost inefficient way to manage applications. The most critical component to enable operations is to empower them with the tools and expertise that ensure that they continue to support the business’s most important IT assets.

Why so slow around the J9 Blog?

2009-03-03T07:52:00.000-08:00

We're busy! We'll be back on track with updating the blog soon. We've had questions off and on from our customers about whether the economic conditions are hurting us. In a time when bad news seems to roll in daily, we're almost embarrassed to say things are full steam ahead at J9. We've actually added more personnel to serve you. Stay turned for more updates.

Register now for our free webinar on February 27th!

2009-02-09T11:56:00.000-08:00

Reducing the Operational Costs of SOA

Software evolution has taken us a long way from the mainframe to distributed computing in the cloud. What hasn’t changed is the need for operations teams to effectively manage the applications and infrastructure that businesses and organizations rely on.

Join us on Friday, February 27th to learn how you can more effectively manage these composite/distributed/SOA applications by leveraging the existing expertise of operations teams along with tools and methodologies that:

-- Ensure high availability
-- Reduce the impact of outages
-- Remove the need for issue escalations or bridge calls

You’ll also hear real-world examples of how other organizations have adapted to these new software architectures and steps taken to prepare for them before they landed in production. Lastly, we’ll cover the cost of not doing anything and how that price is paid elsewhere.

EVENT
Reducing the Operational Costs of SOA

DATE
Friday, February 27, 2009

TIME
10am-11am PST

Register now!

Cloud Computing Expo / SOAWorld 2008 in San Jose

2008-12-01T10:07:00.000-08:00

J9 just returned from the Cloud Computing Expo and SOAWorld convention, held last week in San Jose, CA. The most common question we heard from convention participants who came by our booth was "What *is* SOA anyway?" Pretty telling that while there is a need and a place for SOA, it's not for everyone and it's not a trivial adoption.

Perhaps equally telling was the shadow the topic of cloud computing cast. On one hand, there was an expectation that everyone needs a cloud computing strategy, while at the same time murmurs of "it's just a fad." Like most technologies, it will turn out to be somewhere in the middle.

The trends of distributed computing, cheaper and more plentiful hardware, escalating power, cooling, and administration all lead to a situation where cloud computing makes sense for many applications. On the other hand, those pundits who notice the intellectual property, vendor lock-in, security, and reliability concerns of the cloud have valid points. Many companies will find that the potential fiscal savings cannot be outweighed by these business risks.

What's the J9 take on all of this? Pragmatism. For certain customers and applications, SOA and cloud computing make a lot of sense. We've developed deep organizational understanding of how to make customers successful with SOA. We are active consumers of cloud computing and intend to be active proponents where it makes sense for our customer's needs. As vendor offerings mature, we are confident that many of the current concerns, especially around lock-in, will be resolved such that cloud computing will make sense for more and more types of applications.

Sometimes it's not about the technology

2008-10-29T12:20:00.000-07:00

In a recent conversation, I was asked about tool support for enforcing application architecture. The original team for this project had been increased from under 5 to more than 20, and many of the newcomers failed to grasp the overall architecture, choosing instead to forge their own way ahead for each new functional component. A reflection of this was the copy-paste coding, duplicating blocks of functionality with minor variations throughout the code base. What had started as a an application rewrite aimed at creating a model for future application projects throughout the department was quickly turning into a very sloppy project.

The question I was asked is a common, but unfortunate one. Things have been lean in many IT departments for a long time, even before the current economic downturn. I can't remember the last time I met a development team that was using anything other than cost-free tools and it's becoming very rare to meet any development team that has more than a skeleton crew as so much work has been sent offshore in the name of cost cutting. This is not to complain about either of those factors, but rather to recognize the state of affairs; this is reality, and it isn't going to change. The real frustration is the lack of skilled managers who recognize the difference between management and technology problems, which is directly correlated with the emphasis on cost over quality.

In the particular case I mentioned, it is impractical to try to enforce architecture through technology and even if it were, a poor excuse for good management. Don't misunderstand - the right architecture is addicting and leads developers to want to follow the pattern, so can act as a natural enforcer. My argument is that expecting technology to separate the wheat from the chaff when it comes to talent or productivity is asking for trouble and letting managers act irresponsibly. If a team has "renegade" developers who are creating sloppy, unmaintainable code outside the constraints of a selected architecture, then that's a reflection on ineffective technical management as much as a reflection on those developers. A tool might tell you who broke the build or how many times the same code block has been copied and pasted, but it is the responsibility of technical managers to notice problems, quantify, respond, guide, and otherwise act as stewards. The root cause is that due an endemic lack of training, rampant age discrimination, short sighted yet ineffective cost cutting, and the outright gutting of departments, the IT managers of today are grossly ill-equipped to both understand and handle these problems.

Here's another story. I once worked for a company with a publicly stated policy of never hiring consultants or contractors of any kind in its IT division. The history behind this was that many years before, a large well-known technology consultancy had come in and spent several years and several million dollars evaluating the technology operations. The final deliverable was two binders worth of "best practices." Needless to say, the CIO of the time was less than pleased with the way his money had been spent. The irony of course is that he should have been pointing the finger at himself, not the consultants.

The problem itself in this case wasn't with the vendor, or the technology and methodology being recommended, but with the way the project was mismanaged by that CIO. Anyone who has been in this industry for even a few years has dozens of these stories to tell and a pattern becomes clear. It really doesn't matter what tools you are using or how modern your methodologies. The differentiating factor between successfully delivering working technology solutions and utter disasters boils down to one thing: people. If your managers are simply filling out a space on the organizational chart and not actually being effective, or even worse expecting software itself to act as panacea for personnel matters, then you've got problems regardless of how SOA, Agile, and Cloud you may wish things to be.

Tips and tricks for discovering performance issues before they become production issues.

2008-10-17T11:47:00.001-07:00

Nobody wants to get the phone call. The one that comes at 6 am, informing you that it is going to be a very long day due to a crisis in the software you built. It could be an outright crash, data corruption, or users fleeing your application due to its lagging performance. It doesn't have to be like this; there are reliable ways to put your application under the microscope within the bounds of reasonable effort.

First and foremost, you need tools. There are a range of tools available for all budgets and as in most things, you get what you pay for. Start simple -- you need a way to generate load against your application and you need a code profiler. Freely available open source tools like JMeter are a good place to start -- you can become productive in under an hour. There are countless inexpensive code profilers available, just a web search away. What these lack in features, they make up for in simplicity and price. If you've never examined the performance of your application, then anything is a step up at this point.

Once we start generating consistent load against our applications and run a code profiler, what should we be looking for? The approach we use at J9 is one of pragmatism: focus on the areas with the largest potential return. For example, many books on Java application performance start with a discussion of String versus StringBuffer or StringBuilder. Perhaps they include information on choosing the appropriate Collection types to reduce synchronization overhead. These are fantastic suggestions, but there are plenty of other, more pragmatic improvements to be made before we get to this level of granularity. Take initial jvm heap sizes as an example. This is now one of the first questions we ask customers when evaluating their application performance -- have you explicitly set an appropriate size? We've seen countless customers pulling their hair out due to server crashes have their concerns dissipate by making this trivial change.

We've seen a common list of problems over the years that yield significant improvements with a few simple changes. Besides the initial jvm heap setting:

-- Object pooling to external dependencies. Are you using it? Are your pools sized correctly to expected demand?
-- XML serialization: This one normally shows up as high-CPU use and causes your application to spend all its effort in processing wrappers rather than the business problem at hand.
-- Poor database interaction: There are a number of basic issues here, like cumbersome sql statements and failure to properly index tables.
-- Lack of caching: Dynamically fetching otherwise static data (like jndi entries) or a weak caching strategy leading to poor hit ratio. This assumes that any caching at all has been implemented.
-- Slow report or page rendering: This is common, especially on pdf generation with large data sets. Typically this is an architectural problem stemming from a monolithic approach.
-- Slow network, inadequate hardware: Need to have the basics in place before we can expect performance.

Under even a consistent, moderate amount of load a rudimentary code profiler should offer hints at the above common problems.

Once we've got our tools selected and set up and we start looking for trouble, what should our approach be? Optimally, we're striving to employ the scientific method, beginning with simple divide-and-conquer. First, I take a look across the entire application, noticing what transactions stand out in terms of latency. My search begins there because that tends to be the low-hanging fruit: many problems, regardless of their root cause, manifest as latency issues. It also allows me to potentially focus on getting an early win -- if I can reduce an annoying 10 second response time to 5 seconds, it's a noticeable difference to an end user versus saving an extra 256 megabytes of ram which only a purist would notice. With the information about the slowest performing transactions, I can then take a tier or layer perspective. This step involves examining the performance of my application at each step -- where are the slow downs in the web tier? Are there bottlenecks in the database? What about web services or message oriented middleware? Maybe there are problems within the business logic itself. The key to this examination is keeping concerns separated: just look at the performance within one layer at a time, ignoring the performance within other layers.

Once you've identified the layer or tier where a problem is occurring, begin looking for data that enables you to build a testable hypothesis. Be careful to not assume the first problem you find is the root cause -- many problems are side-effects of the real issue. For example, is the sql statement slow running because it has not been optimized, because the invoked tables have not been indexed, or because of poor data validation and legitimacy? A typical work flow might be as follows:

-- Slowness is identified at the database.
-- Slowest running sql statements are identified.
-- Statement execution is divided into connection and execution latency. Which one is worse?
-- Assuming connection latency is significant, look for obvious issues:
-- Are we using connection pooling?
-- Are we actually pulling our connections from the pools we have configured?
-- Are the pools sized adequate with the expected transaction throughput?
-- Are there network or connectivity issues that would cause connections to expire or be slow to create?
-- Create a test or collect supporting data to eliminate each of these potential concerns.
-- For each issue identified, devise a solution and test the effectiveness of that solution.
-- Repeat process until application achieves acceptable performance levels.
-- Implement monitoring, thresholds, and alerts to proactively catch future issues.

As you work through problems, be aware of what you can and cannot control. There are physical limits to computing that are outside your control, just like code from a vendor will seldom quickly be repaired.

We can summarize the approach recommended by J9 as follows:

-- Look for solutions with a high probability of success
-- Be aware of the basic limits and issues with your environment.
-- Use a scientific, quantitative approach.
-- Put in place tools that make finding and testing for issues easier

SOA does not mean agnostic

2008-10-14T10:55:00.000-07:00

As a follow up to our last post about the nature of SOA, I wanted to speak to a common misconception. Here are two statements I hear with all-too-common frequency:

1) SOA is about masking the route to a service from its implementation.

2) The underlying transport protocols used in SOA should always be hidden.

It seems in technology there is an ever-pervasive interest in adding more and more layers of indirection. Its a situation where if we don't understand the reason for such layers, we can quickly add unnecessary overhead and complexity to otherwise simple problems. In the case of the above statements, the underlying goal -- to create flexible implementations while masking details from a service user -- is a good one, but without understanding the nuances involve, the commonly heard refrains noted above are in gross error.

Let's address point one. It's true that things like UDDI and other kinds of service discovery help us to change the location or even implementation of our services, but let's be clear that implementing UDDI is not a primary goal of SOA, rather a by-product of an implementation approach. You can do SOA without having an intermediary routing requests between service providers and service consumers; you will simply have a more tightly coupled implementation. To be clear, there have been ways of creating this kind of indirection since long before the word SOA entered our lexicon.

Next we should address the idea of protocol agnosticism. If I had a nickel for every time I heard the statement "SOA masks the underlying protocol" I'd be retired. Let's reflect upon reality for a minute -- even if we ask a service directory for the means of reaching a service, we've made some kind of conscious effort to ask that that service directory, which innately means we've chosen a protocol. On a related example, if we offered access to a service over both message-oriented middleware and via http, we'd be forced to supply an sdk/api to service consumers that would mask the underlying protocol used. Whoops -- sdks? apis? Isn't that part of the very draw to Web Services in the first place, that is not having to support binary apis in multiple languages for all of our consumers? In the end, it's a fallacy and in some cases even undesirable to declare protocol agnosticism realistic or even a desired goal of SOA.

Systems are not Services

2008-10-13T10:57:00.000-07:00

In our ongoing discussion about what SOA is and is not, it's time to address another popular misconception -- that everything is a service. Nothing could be further from the truth, which is why we also have to follow up with this reality check: SOA is hard.

Too often when working with customers, I hear the common misconception that they are already doing SOA. After all, they have a claims service, a billing service, and an order management service that have all been "web-enabled" therefore they are completely buzzword compliant! This is where we need to be able to address some terminology:

-- Service: A concrete, decomposed, independent functional offering.
-- Application: An aggregation of related services
-- System: An aggregation of related applications.

Now time for an example:

-- Service: User login, Change customer address.
-- Application: User account management
-- System: Revenue Management (encompassing applications like user account management, bill pay, and product ordering)

Thus, having two or more monolithic systems or applications now speaking http does not constitute SOA.

Now, on that note about SOA being hard. The reality is, while SOA makes a lot of sense on greenfield development or in times of major IT overhauls, it's a challenge to decompose a system or application into independent, reusable services and to carry through on their implementation. It will take aggressive sheparding by technical architects to both lay out such a vision and to keep services in their appropriate scope. For many organizations, SOA is enticing but will ultimately prove too overwhelming in the face of declining budgets and more pressing priorities. And, as long as vendors benefit from system lock-in, the likelihood of SOA adoption through vendor selection is unclear.

Why J9 is building protocols for HP LoadRunner?

2008-10-10T11:00:00.000-07:00

Our experience last week at StarWest was filled with eureka moments. That's really the greatest benefit we get from attending conference like StarWest -- the opportunity to get hands on with IT practitioners and get a dose of honest feedback.

One of the questions we found ourselves answering over and over related to our jms and jdbc protocols for HP LoadRunner offerings, phrased typically as "Why do I need that?" In fact, the most surprising yet common question we answered was "Doesn't HP already offer this?"

Here's how this situation plays out. When you license HP LoadRunner, whether you realize it or not, you are paying for specific protocols that plug into it. The most common scenario involves purchasing the web protocol, which enables you to record and replay a series of actions against a browser-delivered application. This is a fantastic start down the road of performance and scalability testing, but it isn't the complete story. What if in the course of your performance testing, you want to understand just the scalability of your database separate from the overall scalability of your application? With just the web protocol, you can't do it. And, HP doesn't offer anything that will target your databases and message-oriented middleware in their current protocol lineup. This is where HP Partners like J9 are stepping in to fill a gap, with HP's blessing. With J9's JDBC protocol, you record and replay all of the sql statements -- the interactions between your application and the database -- which allows you to measure just the database performance. The same concept applies to your messaging providers. With J9's JMS protocol, JMS messages are captured and both the consumer and provider perspective can be simulated under load conditions.

Using feedback like we received at StarWest, J9 is continuing our road map of building protocols to enable customers to maximize their investment in HP LoadRunner. We encourage you to download our free, full-featured trial version and to send us feedback on your performance testing experiences.

Let's try to define SOA, shall we?

2008-09-30T13:00:00.000-07:00

I've been holding a kind of informal survey lately, asking everyone I meet to define SOA for me. Not that it stands for Service-Oriented Architecture, but rather what it actually means within their organization and what value it holds. The reality is, while there are many viewpoints on what SOA might entail and a general sense that there's something worthwhile to it, it's a sadly misused term. The reason for that is that SOA is being sold to two different groups for two very different purposes. With this in mind, I'd like to propose two completely distinct, yet equally valid viewpoints:

1) SOA is the abstraction of corporate intellectual property and processes from their method of delivery and provider source.

2) SOA is a software design architecture where applications are decomposed into fine-grained, decoupled, abstracted, multi-purpose services. A hallmark of SOA is an emphasis on loose-coupling through message-format based protocols, rather than the function signature based agreement of earlier distributed architectures.

This second definition involves some delicate thinking about how distributed applications are built from an application performance perspective. At J9, we've been working with this type of application and the performance problems common in distributed applications for a number of years now. SOA is hard to do right, but solves a major, common bottleneck if undertaken with care. This problem is one of course-grained applications, which invariably end up with sizable data sets to be serialized / deserialized on both ends of the connection. Put simply, J9's consultants have seen countless applications where for example, 1 megabyte or more of XML is sent from a server to a client, with high-CPU creation and parsing, and equally-unacceptable latency the result.

SOA really is the answer to this, as companies like Amazon.com will attest to. According to an article based on a conversation with Werner Vogels, titled Learning from the Amazon Technology Platform, Amazon's front page is made up of more than a hundred distinct service invocations which fire simultaneously and are reassembled asynchronously into the page you see. The point here is that each of those actions involved -- like checking who you are, the status of your shopping cart, what favorites to present you, and so on -- have been decomposed and abstracted into fine-grained, light-weight services.

A much misunderstood but highly-touted feature of this type of architecture is the speed of replacing any of those services with an alternative provider. Personally, I have trouble imagining it being possible to replace any sufficiently complex service simply by making a url change, but there's still a nugget of truth to this point. By keeping services extremely granular and general purpose, it's practically innate that such a service will be easier to make changes to and deploy -- a smaller code base means fewer dependencies and potential for conflict.

In this regard, SOA becomes a manifestation of the best practices in software development that have been argued for years:

-- Smaller, more manageable code bases
-- Clearer division of labor
-- Abstraction of systems logic from delivery and access
-- Discrete, unit-testable components
-- Improved performance for both server and network resources

So in the end, while SOA may have begun life as a broad marketing term, there is a concrete definition, the value of which should resonate with technical and business-minded people alike.

J9 Technologies is going to SOAworld!

2008-09-25T16:00:00.000-07:00

We are hitting up as many conferences as we can without pulling out our hair in pre- and post-conference angst, this is yet another one that we will be giving bottle openers away at! And that is not all. We will have bookmarks! (In case you still read that paper stuff – and lest you forget about the digitized ordeal when you do.) And of course, you can get a free trial for our new LoadRunner Protocol Add-ins, or a login to our cool On Demand Testing Suite.

Here is the plug from the SOAworld conference organizers. When we find out our booth number, we will be sure to post it here first.

The 14th International SOA World Conference & Expo 2008 West will take place on November 19-21, 2008 at The Fairmont Hotel in San Jose, CA.

The most important business benefit that service-oriented architecture (SOA) can provide is the ability to respond swiftly to change: changes in the market, the supply chain, strategic processes, and regulations. Make sure that you are able in 2008/9 to declare your own company's SOA journey toward such responsiveness and agility a success. Come to SOA World Conference & Expo 2008 West and learn just how much SOA can do for your business and for your developers.

There is no other event in the USA with as rich and varied opportunities for learning, networking, and tracking innovation occurring in the IT infrastructure, architecture, and standards communities. The speakers are all seasoned SOA practitioners with a passion for helping to usher in the new era of IT agility made possible by standardized best practices, pervasive communication & data protocols, and a general movement toward openness and interop in the industry.

StarWest 2008 Conference Update - We will be in Booth #9

2008-09-25T11:42:00.000-07:00

As you may already know* the folks at J9 Technologies are pleased to announce that we will be attending the StarWest 2008 Conference in Anaheim, CA next week. The conference runs for days and days, but we will be participating as an exhibitor, so as far as we are concerned the most important part of the conference goes from Sept. 30th through Oct. 2nd. We will be showing off our new Protocol Add-ins for LoadRunner, giving demos and free trial licenses for them, unveiling to our new On Demand Testing Suite, and generally schmoozing with strangers. Come by and visit us at Booth #9 and see the only one-handed bottle opener you will probably ever get for free.

And if all of that doesn’t give you reason enough, and you are still in the fence about such a proposition, here are the top ten reasons to attend this event – as compiled by the StarWest organizers themselves. Some of which may seem boring, yet informative.

1. Over 100 learning sessions: tutorials, keynotes, conference sessions, bonus sessions, and more
2. In-depth tutorials, half- and full-day options—double the number of classes from last year
3. Cutting-edge testing answers from top testing experts
4. Presentations from highly experienced testing professionals
5. Networking opportunities with your peers in the industry
6. Special events—welcome reception, bookstore, meet the speakers, and more
7. The largest testing EXPO anywhere
8. Group discounts—bring your whole team
9. The perfect balance of learning and fun in Southern California
10. All this at the happiest place on Earth—Disneyland® Hotel!

We hope to see you there!

*If you don’t already know about a certain Belated Disneyland Fetish that one marketing manager in our company has, please scroll down to the posting featuring keyword “Disneyland” to learn all about the excitement first hand.

JMS Protocol Add-in for LoadRunner is now available

2008-09-22T11:41:00.000-07:00

With the explosion of Message Oriented Middleware (MOM) applications and their SOA counterparts, QA professionals must face the challenge of creating real-world testing conditions for these complicated, multi-node, composite applications. From a testing perspective…How do you generate hundreds of unique messages every second? How do you simulate a transaction that consists of HTTP, SOAP, JMS and JDBC protocols? What is the maximum de-queue rate of the application’s Message-Driven Beans? How can you test all of the possible routes a message may take through the application without requiring the Java knowledge to hand-code an extensive test harness?

The JMS Protocol Add-in helps you tackle these problems head on. By extending the capabilities of HP LoadRunner, the JMS Protocol Add-in help stress and isolate problems in the messaging tier and ensure applications are deployed with confidence. Turn a new page on middleware performance testing In today’s multi-tier distributed applications, some of the most critical bottlenecks reside in connections to backend systems, such as a database, message queue or the mainframe. In the past, the crucial task of isolating, testing and tuning messaging operations required creation of complex scripts that rely on in-depth Java and application knowledge. This adds a
tremendous burden to the already full plates faced by many quality assurance teams.
The JMS Protocol Add-in from J9 Technologies helps you stress the messaging middleware
layer and identify bottlenecks before deploying in production. Based on years of testing and diagnostics expertise in helping some of the world’s largest enterprises,
this solution automatically captures all JMS traffic going from a Java application to the backend message queues and plays them back to directly exercise the messaging tier
for high concurrency testing.

The JMS Protocol Add-in for HP LoadRunner helps you:
• Verify messaging infrastructure
supports a high volume of messages
• Obtain an accurate picture of backend
system performance
• Isolate and eliminate performance
bottlenecks in the messaging tier

How it works
Leveraging HP LoadRunner and VuGen technology, the JMS Protocol Add-in records all JMS put and get operations and captures them in standardized scripts. These scripts can be used to simulate targeted loads in terms of concurrent users to almost any JMS provider. This easy-to-use solution helps QA teams scale up backend testing by applying very measurable and repeatable loads that expose any potential problems early in the development cycle. The JMS Protocol Add-in eliminates the dependency on the application tier for direct testing of the messaging tier. Autogenerated scripts represent real-world load, and can be easily parameterized to create accurate and realistic mix of users for stressing the messaging tier. Detailed measurements are captured and provided to the HP LoadRunner Analysis application for side-by-side metric comparisons between test runs.

Get more out of Hp LoadRunner
HP LoadRunner is the market leading performance validation solution embraced by 77% of all QA professionals. J9’s HP LoadRunner Protocol Add-ins are fully integrated into the full suite of HP LoadRunner solutions. The JMS Protocol Add-in can be used alongside all other HP LoadRunner protocols to create a more realistic and comprehensive load test. With the simplicity of the solution, any skilled HP LoadRunner user can start using JMS Protocol Add-in with no additional training, and be productive immediately. With the JMS Protocol Add-in, recording of all JMS put and get operations happen exactly as they occur in sequence, and entirely on the server side. This unique feature is very beneficial in situations where client side recording is not feasible, and allows for server-side only testing that does not require installing VuGen on the backend systems.

JDBC Protocol Add-in for LoadRunner now available from J9 Technologies

2008-07-23T17:14:00.000-07:00

Some of the most critical bottlenecks reside in connections to backend systems, such as a database or the mainframe. But testing the connections to these systems is no trivial matter. This often requires creation of complex scripts that rely on in-depth application knowledge and even Java development skills.

JDBC Protocol Add-in helps you stress the data connections and isolate any bottlenecks before deploying in production. It captures all SQL traffic going from a Java application to the backend database and plays them back to directly exercise the data tier for high concurrency testing. This solution leverages J9's expertise in testing and diagnostics of enterprise Java applications and is fully complementary to HP's industry-leading LoadRunner product. This perfect marriage helps reduce testing costs and prevent potential problems in production.

Features

Test application backend without writing Java code

Scale up and perform load tests to simulate hundreds of simultaneous users accessing the database without needing to create a front-end application. This singles out any potential bottlenecks in the critical data connection layer without putting additional burden on the QA teams.

Captures and Plays back JDBC statements

Records and replays all Java SQL statements, including all prepared statements and callable statements.

Enables Simple Parameterization

It's easy to parameterize within the generated test scripts to create a realistic mix of users and data for the high concurrency load tests.

Heterogeneous Platform Support

Works seamlessly with any database, on any OS platform.

Benefits

Shorten testing cycles

Deliver significant time savings in the testing of middleware and backend tiers.

Improve application service levels

Fully stress the critical JDBC connection to prevent potential problems after the applications are deployed.

Baseline data tier performance

With stress tests targeted exclusively at the backend, realistic SLAs can be created to measure and benchmark future performance.

Empower QA Engineers

Enable QA engineers to extensively test the JDBC connection without requiring Java knowledge and without additional training. Fully leverage existing skills in knowledge of HP LoadRunner solution.

Supported Platforms

all common Java application servers including Weblogic and Websphere.

all common Databases including Oracle, DB2 and SQL Server

Java 1.4 and up