That question was the lead in to a discussion I had with a colleague this week. He had been interviewing someone for a performance testing role and that was the key question that could make or break a candidate. The typical response goes something like "I'd start with one user, then move on to five, then ten, then 50 then 100, then... all the way up to 4000." While the most common answer, this is entirely wrong. This kind of common yet broken testing process explains why the group of us that joined the conversation could each retell case studies of customers who had spent multiple years (and millions of dollars) on failed testing efforts.
The right answer goes like this:
a) Ask the hard questions
How many of the 4000 users are concurrent users and what is their use pattern? For example, many batch billing systems do nothing for 29 days per month, but then run through a massive number of transactions on the last day. Other systems have limited daily use until 5pm when their user community arrives home from work and then signs in. Are the users spread across multiple timezones?
If the data to discern the number of real concurrent users isn't available, that actually means two things to our project:
1) A separate project is needed to put in place tools to capture user behavior. The lack of such information can cause poor decisions in the areas of testing, capacity planning, security, and product usability design and functionality.
2) If no such data exists and the 4000 number simply means we have 4000 users in our database, we can now back into a more realistic upward bound through some basic calculations.
b) Functional performance test
Start with one user as a means of functional performance test. This enables you to validate your test cases and test scripts and flush out any immediate functional problems with the application(s).
c) Longevity testing, peak testing, failover testing
There are a variety of other tests with greater pertinence and validity in understanding the application's serviceability than simply running through the same script with a randomly increasing number of virtual users.
d) Load and Performance testing
If we've determined that simply starting with one user and continuing to double isn't the right process for load testing our application, then what is the right heuristic for getting to the Nth user? The answer is that it doesn't really matter, as we've determined, in effect, all of the above through the answers to our questions about the user community. If we have 4000 users in our database but don't know how and when they use the application, a test of 200 users as a top number is just as valid as a test of 2000 users. Using these numbers though, one can arrive at some guidelines by looking at the length of a user day. For example, if our application is used by an internal business customer that only works standard business hours in the eastern time zone, then we can surmise a roughly 8 hour work day, 5 days per week. Take 4000 users, divided by 8 hours, we can take an educated guess that there are 500 users per hour. Take an 8 hour day, multiply by 60 to get 480 minutes, divide the 4000 users by 480 and we can surmise that at any one minute interval there are likely to be 8 users on the system. In the absence of further information about our user community, we now have real, actionable numbers to test against. Rather than the dozens and dozens of incremental tests we were potentially facing, we can now break our cases into one user, 10 users, 500 users, and anything above that is essentially to discover the upward bound of our capacity.
These steps are a productive tool to improve the quality of your testing, as well as a great way to gain new insight into the candidates you interview.
Thursday, July 2, 2009
Monday, June 29, 2009
OVIS to BAC migration, anyone?
J9 Technologies, Inc. Announces Migration Solution for OpenView Internet Services Users
J9 Technologies, Inc. announces a limited-time migration solution for Hewlett Packard customers currently using OpenView Internet Services (OVIS). As of December 31, 2009, current OVIS customers must migrate from OVIS to HP's BAC and SiteScope solutions in order to receive licenses, customer support, patches, and updates from HP. As a result of the Mercury acquisition, HP announced that it is dropping support for the OVIS solution in favor of the Business Availability Center (BAC) and SiteScope solutions. HP is offering, free-of-charge, a license exchange from OVIS to BAC and SiteScope.
J9 offers expert services to streamline the customer's license acquisition process, pre-migration planning, end-to-end migration from OVIS to BAC, and identifying best practices for ongoing utilization of the new BAC solution. With J9, the migration process is not just one of conversion, but evolution. J9's solution reduces a customer's time to value risk during the migration process.
J9 works with each customer individually to determine their current state, identify gaps and provide a plan for migration. Throughout the process, J9's experts partner with the customer to ensure the migration not only replaces their current state, with no gaps, but allows the customer to be positioned to expand their capabilities utilizing the enhanced BAC platform. J9's services expand beyond OVIS migration solutions to offer a complete range of basic and advanced BAC implementation services and training programs designed to enable the customer to take full advantage of the power of the BAC platform.
Business Availability Center brings a robust platform to address a broad range of application environments. BAC's rich feature-set offers customers an enhanced platform addressing their Business Service Management needs and initiatives, including application diagnostics, service level management, business transaction management, and superior dashboarding capabilities. The BAC platform is capable of supporting a wide range of application environments beyond standard web-based applications, resulting in enterprise-wide coverage for the customer's most business critical systems, such as ERP/CRM, E-Mail, SOA, Web 2.0, and client-based systems.
About J9 Technologies
J9 Technologies is a certified Hewlett Packard Gold Partner, specializing in Business Service Management and Application Performance Lifecycle management.
Offering strategic consulting, training, installation and ongoing support for HP software products such as SOA Systinet, Service Test, Business Availability Center, Diagnostics, LoadRunner / Performance Center, Real User Monitoring (RUM), TransactionVision. With a focus on application diagnostics, composite application management, and business transaction management, J9 Technologies strives to ensure that customers can quickly identify the root-cause of issues and minimize mean time to resolution.
For more information:
J9 Technologies, Inc.
24 Roy St., Box 211
Seattle, WA 98109
Tel: (866) 221-8109
Fax: (206) 374-2901
www.j9tech.com
J9 Technologies, Inc. announces a limited-time migration solution for Hewlett Packard customers currently using OpenView Internet Services (OVIS). As of December 31, 2009, current OVIS customers must migrate from OVIS to HP's BAC and SiteScope solutions in order to receive licenses, customer support, patches, and updates from HP. As a result of the Mercury acquisition, HP announced that it is dropping support for the OVIS solution in favor of the Business Availability Center (BAC) and SiteScope solutions. HP is offering, free-of-charge, a license exchange from OVIS to BAC and SiteScope.
J9 offers expert services to streamline the customer's license acquisition process, pre-migration planning, end-to-end migration from OVIS to BAC, and identifying best practices for ongoing utilization of the new BAC solution. With J9, the migration process is not just one of conversion, but evolution. J9's solution reduces a customer's time to value risk during the migration process.
J9 works with each customer individually to determine their current state, identify gaps and provide a plan for migration. Throughout the process, J9's experts partner with the customer to ensure the migration not only replaces their current state, with no gaps, but allows the customer to be positioned to expand their capabilities utilizing the enhanced BAC platform. J9's services expand beyond OVIS migration solutions to offer a complete range of basic and advanced BAC implementation services and training programs designed to enable the customer to take full advantage of the power of the BAC platform.
Business Availability Center brings a robust platform to address a broad range of application environments. BAC's rich feature-set offers customers an enhanced platform addressing their Business Service Management needs and initiatives, including application diagnostics, service level management, business transaction management, and superior dashboarding capabilities. The BAC platform is capable of supporting a wide range of application environments beyond standard web-based applications, resulting in enterprise-wide coverage for the customer's most business critical systems, such as ERP/CRM, E-Mail, SOA, Web 2.0, and client-based systems.
About J9 Technologies
J9 Technologies is a certified Hewlett Packard Gold Partner, specializing in Business Service Management and Application Performance Lifecycle management.
Offering strategic consulting, training, installation and ongoing support for HP software products such as SOA Systinet, Service Test, Business Availability Center, Diagnostics, LoadRunner / Performance Center, Real User Monitoring (RUM), TransactionVision. With a focus on application diagnostics, composite application management, and business transaction management, J9 Technologies strives to ensure that customers can quickly identify the root-cause of issues and minimize mean time to resolution.
For more information:
J9 Technologies, Inc.
24 Roy St., Box 211
Seattle, WA 98109
Tel: (866) 221-8109
Fax: (206) 374-2901
www.j9tech.com
Wednesday, June 24, 2009
J9 @ HP Software Universe 2009
We are just catching our breath after few a long and fruitful days in Las Vegas, where HP held it's annual Software Universe conference. We exhibited, we rolled out our OVIS to BAC migration offering, we chatted, we happy hour-ed, we presented, and after it all, we slept. For a long time.
And now here we are, in all of our glory, having emerged from the gambling flames of software sales and services just a little bit tougher, a little bit wiser, and a feeling lot more connected. Thanks to everyone who came by the booth (and picked up one of our lovely one-handed bottle openers pictured above) and who attended our happy hour event. It really was a blast getting to be face to face with all of these people we work with everyday. See you next year!
Monday, June 8, 2009
What are you testing?
Yet another Monday after a red-eye flight, at another new customer in some city far from home. I'm ushered into a conference rooms with a team randomly checking their watches and Blackberries, each tense, nervous, and wishing they were somewhere other than this. A project manager begins the briefing, which everyone has heard a dozen times before: the project is past its scheduled delivery, application performance is terrible, the users are angry, the sponsors are talking about pulling funding, and the order for the past month has been forced 15 hour days and weekends. This afternoon they want to begin another series of load tests and they ask me if our team of consultants will be able to help them. My only question is: What are you testing?
It's a simple question. I find myself asking it several times per day with customers and sadly the usual answer is a blank and distant stare. The reality is they don't know, but it's not their fault. To be honest, I'm not entirely clear myself on how the state of the industry reached this point. The trend though goes like this:
1) Testing happening just prior to deployment, often not even starting until the same week.
2) Tests being run by an operations team or perhaps by a lone consultant, without the involvement of a business owner who understands the functionality of the application.
3) A random mishmash of tools, some properly licensed, some not, downloaded and used without any training or guidance.
4) Test cases being made up on the fly - "Hey, what if we try this?" without any rhyme or reason as to the validity of the test.
5) Little or no written record of the test cases executed or their results, leading to duplicate efforts, errors, and a lack of objectivity.
It all really boils down to that same question -- if you can't answer, objectively, the reason for a particular test run, then your efforts are for nought. Once you are on that treadmill of test run after test run, it's impossible to get off without a hard reset; the only objective test at that point will come when the application is introduced to the real world.
Once an organization has reached this point, where production firefighting is a daily routine, the only way to bring order back is to reverse the list above. Testing must be an integrated part of any development or implementation project. Developing a written, comprehensive, repeatable test plan requires input not only from technical teams but also from end users and business analysts who understand the use cases -- both common and edge. Only then will you know what you are testing.
It's a simple question. I find myself asking it several times per day with customers and sadly the usual answer is a blank and distant stare. The reality is they don't know, but it's not their fault. To be honest, I'm not entirely clear myself on how the state of the industry reached this point. The trend though goes like this:
1) Testing happening just prior to deployment, often not even starting until the same week.
2) Tests being run by an operations team or perhaps by a lone consultant, without the involvement of a business owner who understands the functionality of the application.
3) A random mishmash of tools, some properly licensed, some not, downloaded and used without any training or guidance.
4) Test cases being made up on the fly - "Hey, what if we try this?" without any rhyme or reason as to the validity of the test.
5) Little or no written record of the test cases executed or their results, leading to duplicate efforts, errors, and a lack of objectivity.
It all really boils down to that same question -- if you can't answer, objectively, the reason for a particular test run, then your efforts are for nought. Once you are on that treadmill of test run after test run, it's impossible to get off without a hard reset; the only objective test at that point will come when the application is introduced to the real world.
Once an organization has reached this point, where production firefighting is a daily routine, the only way to bring order back is to reverse the list above. Testing must be an integrated part of any development or implementation project. Developing a written, comprehensive, repeatable test plan requires input not only from technical teams but also from end users and business analysts who understand the use cases -- both common and edge. Only then will you know what you are testing.
Thursday, June 4, 2009
This is a plea, continued...
Shortly after I wrote the last entry, about tools that get purchased and not used, I realized that I hadn't really described the reasons why this happens. Sure, there are reasons like insufficient training, inadequate tools for the problems at hand, or organizational ownership and politics. But the most common reason is far more human and harder to mitigate: apathy.
A friend of mine is a Six-Sigma and Toyota Process expert. He knows and understands quality and process control methods inside and out. I asked him about the barriers to organizational adoption of these practices. His answer was very telling. It wasn't money, time, tools, training, or anything measurable. Instead his answer was very simple -- because people don't like being told what to do.
There's a parallel here. Unless a tool, process, practice, or even idea fits within someone's world view, they simply aren't going to adopt it. It's even worse if they feel like something is being forced upon them, rather than being an engaged participant. Even if they feel invested in the tools, good old fashioned apathy can take hold.
Why? Incentives and consequences. Its just tremendously easy to become apathetic if there are no incentives or consequences to our actions. Many times after J9 consultants find a problem with a customer's application, we hear rumblings of "oh yeah, I wondered if that was a problem." Easier to turn a blind eye then to invest ones self in learning a tool, exploring a problem, and coming up with a solution. Especially since in so many organizations that live by firefighting, volunteering is akin to accepting blame for the problem.
So, the guiding points here appear to be:
1) It's imperative that all types of users are included in the selection and implementation process so that they feel invested.
2) Management has to continually communicate intentions and uphold the standard.
On this second point, here's a really example of how this can be done in a way that emphasizes the point yet stays away from simply giving orders. Say for example it's a Monday morning triage meeting and an outage that happened over the weekend is the topic. The questions isn't simply "What happened?" but also "What did the tools tell you happened?" This is a means of continuous communication about the expected tools and the expected method, plus a means of continual assessment of the value proposition -- if your tools are proving less than useful, then perhaps there is a legitimate reason why they aren't being used.
A friend of mine is a Six-Sigma and Toyota Process expert. He knows and understands quality and process control methods inside and out. I asked him about the barriers to organizational adoption of these practices. His answer was very telling. It wasn't money, time, tools, training, or anything measurable. Instead his answer was very simple -- because people don't like being told what to do.
There's a parallel here. Unless a tool, process, practice, or even idea fits within someone's world view, they simply aren't going to adopt it. It's even worse if they feel like something is being forced upon them, rather than being an engaged participant. Even if they feel invested in the tools, good old fashioned apathy can take hold.
Why? Incentives and consequences. Its just tremendously easy to become apathetic if there are no incentives or consequences to our actions. Many times after J9 consultants find a problem with a customer's application, we hear rumblings of "oh yeah, I wondered if that was a problem." Easier to turn a blind eye then to invest ones self in learning a tool, exploring a problem, and coming up with a solution. Especially since in so many organizations that live by firefighting, volunteering is akin to accepting blame for the problem.
So, the guiding points here appear to be:
1) It's imperative that all types of users are included in the selection and implementation process so that they feel invested.
2) Management has to continually communicate intentions and uphold the standard.
On this second point, here's a really example of how this can be done in a way that emphasizes the point yet stays away from simply giving orders. Say for example it's a Monday morning triage meeting and an outage that happened over the weekend is the topic. The questions isn't simply "What happened?" but also "What did the tools tell you happened?" This is a means of continuous communication about the expected tools and the expected method, plus a means of continual assessment of the value proposition -- if your tools are proving less than useful, then perhaps there is a legitimate reason why they aren't being used.
Saturday, April 25, 2009
This Is A Plea
This is a plea: please, please use the tools you've invested in. Nothing makes a J9 consultant sadder than to see customers throw their money away on tools and training. Second to that is to see customers try to build tools on their own when there are other options available.
Two cases in point here: HP Diagnostics and HP TransactionVision.
The Diagnostics product has been around for many years, coming into the HP fold in January 2007 as part of the acquisition of Mercury Interactive. Many customers are resistant to using this product, believing either that its overhead is extreme or that free/cheap tools can provide the same information. The truth is, Diagnostics adds very minimal overhead to any production system if it is configured correctly. It has been safely deployed in high-volume production settings to dozens and dozens of J9's customers. Many free or inexpensive tools offer point solutions, but none are capable of providing the complete environment monitoriing and historical data collection capabilities of Diagnostics.
Many customers have recently shared with us that they are so desperate for business transaction reporting and monitoring that they are considering building their own solutions. TransactionVision came to HP via an acquisition of Bristol Technologies, which had been a partner of Mercury Interactive. When we demonstrate it, the feedback is almost always a sense of relief "Thank goodness I don't have to build this for myself!" Sensors are deployed for Java, .Net, WebSphereMQ, CICS, and Tuxedo, with reporting and monitoring integrated into HP Business Availability Center.
So, please, don't build what you can buy, and if you are going to buy, please don't let it become shelfware.
Two cases in point here: HP Diagnostics and HP TransactionVision.
The Diagnostics product has been around for many years, coming into the HP fold in January 2007 as part of the acquisition of Mercury Interactive. Many customers are resistant to using this product, believing either that its overhead is extreme or that free/cheap tools can provide the same information. The truth is, Diagnostics adds very minimal overhead to any production system if it is configured correctly. It has been safely deployed in high-volume production settings to dozens and dozens of J9's customers. Many free or inexpensive tools offer point solutions, but none are capable of providing the complete environment monitoriing and historical data collection capabilities of Diagnostics.
Many customers have recently shared with us that they are so desperate for business transaction reporting and monitoring that they are considering building their own solutions. TransactionVision came to HP via an acquisition of Bristol Technologies, which had been a partner of Mercury Interactive. When we demonstrate it, the feedback is almost always a sense of relief "Thank goodness I don't have to build this for myself!" Sensors are deployed for Java, .Net, WebSphereMQ, CICS, and Tuxedo, with reporting and monitoring integrated into HP Business Availability Center.
So, please, don't build what you can buy, and if you are going to buy, please don't let it become shelfware.
Monday, April 20, 2009
When Java Memory Issues aren't.
What I'm going to write is likely an unpopular viewpoint, but one that needs to be expressed. One of the most frequent sentiments expressed by customers when we are doing an application analysis is that their Java application is suffering from crippling memory issues. I'm not going to go on record saying that memory issues don't happen in Java -- I'm quite certain that there are situations where legitimate memory management issues within applications and the virtual machine happen. But, a more common scenario is something like customers placing very large objects into user http sessions, then being surprised when their application server slows to a crawl and crashes out of memory. Guess what? That's not a memory issue, that's a poor architecture and design issue.
Normally about this time I hear a chorus of "but, but, but! Why isn't the VM managing my memory correctly?!?!?!" It's a virtual machine, not a miracle worker. In the case above, the http session is being abused and overused. There are several options to that particular problem, depending on the business requirements:
a) Reexamine necessity of all data being stored in the session and reduce.
b) Move much of the data to a separate cache with more granular control over cache clear/timeout strategy.
c) Externalize session data entirely to a separate user session server with purpose-built hardware and cache timeout strategy.
All of the above are overkill you say? Sure beats running out of memory it seems to me. About this time I often hear someone pipe up about "Why don't we just force garbage collection?" or some similar pearl of wisdom, such as explicitly finalizing objects. Both are flawed -- you'll recall that the vm specification states that marking an object to be finalized does not explicitly force it to be finalized on any particular schedule, and any suggestion of forced garbage collection is specifically not recommended. Generally the response I hear to this is some grumbling about a lack of control over memory management. Therein lines the rub: If you need explicit memory management due to the nature of your requirements, then Java and .Net are not for you. And, as demonstrated above many issues that at first appears to be memory related are in fact not.
Normally about this time I hear a chorus of "but, but, but! Why isn't the VM managing my memory correctly?!?!?!" It's a virtual machine, not a miracle worker. In the case above, the http session is being abused and overused. There are several options to that particular problem, depending on the business requirements:
a) Reexamine necessity of all data being stored in the session and reduce.
b) Move much of the data to a separate cache with more granular control over cache clear/timeout strategy.
c) Externalize session data entirely to a separate user session server with purpose-built hardware and cache timeout strategy.
All of the above are overkill you say? Sure beats running out of memory it seems to me. About this time I often hear someone pipe up about "Why don't we just force garbage collection?" or some similar pearl of wisdom, such as explicitly finalizing objects. Both are flawed -- you'll recall that the vm specification states that marking an object to be finalized does not explicitly force it to be finalized on any particular schedule, and any suggestion of forced garbage collection is specifically not recommended. Generally the response I hear to this is some grumbling about a lack of control over memory management. Therein lines the rub: If you need explicit memory management due to the nature of your requirements, then Java and .Net are not for you. And, as demonstrated above many issues that at first appears to be memory related are in fact not.
Subscribe to:
Posts (Atom)
