Browsermob Load Testing Service Is Fantastic

December 2, 2009 - zenoconsultingzenoconsulting


I recently built a Google Web Toolkit (GWT) application for a client. I approached testing from a number of different perspectives:

I first read about BrowserMob on Hacker News and I read founder Patrick Lightbody's interesting post on Why Load Testing Ajax Is Hard. This intrigued me, but at the time, I hadn't had a need for load testing AJAX applications…until I built one. The client I built this app for already had a contract with a different online load testing service, and as such, it made sense to see if we could use that. I tried it, and it didn't. The record and playback features of this particular service (which will remain nameless), did not work at all for the GWT application.

BrowserMob seemed easy enough to try. You sign up for an account and you get 500 free testing credits. They also allow you to import Selenium scripts created with Selenium IDE. So, the first thing I did was install Selenium IDE and record a simple login/logout script. I had it working in about 10 minutes, and I was able to record and playback the script. Then I signed up for BrowserMob, and uploaded the script. BrowserMob converts the script to an easy to use JavaScript format that is simple to understand and well documented. You can manually edit the script inline, and then validate it. It will run against your website and tell you if it fails or succeeds. If it fails, they show you a screenshot of the website at the point of failure so you can visually inspect it. This is extremely helpful in figuring out what went wrong. You can also add debug logging statements to your script which also let you know when/where something went wrong.

Once we had a staging environment deployed and the site was accessible via a public URL, I ran several small load tests to tune my scripts. Now we had a couple scripts that we wanted to exercise on a larger scale, so we purchased a chunk of credits from BrowserMob. BrowserMob has a number of tune-able parameters when you set up a load test that determine the cost of a particular test:

  • Duration of the test
  • Number of users
  • Type of user: real or virtual

Real users mean real web browsers spawned on Amazon EC2 instances in the cloud. Virtual users mean the HTTP traffic is first recorded from a real browser and then replayed. I ran into some technical difficulty getting a virtual user test to run — this was mainly a side effect of the security features of the application and the use of GWT-RPC. In a nutshell, for security purposes, a server side security token is generated after login, and all future requests require the passing of this token. This is to avoid cross-browser scripting attacks. A simple replay of the HTTP traffic will fail on the server side because the token will no longer be valid. I spoke with Patrick about this, and we both believe that this is a technical hurdle that could be overcome — but in the interest of expediency, I was going to go for real browser testing initially. The cost up front of testing with real browsers is higher, but it is really the most realistic test you can probably execute and provides you an initial baseline of what your app can handle.

I had had a few exchanges with Patrick at this point and he was always extremely prompt and informative in answering all my questions. At this point, I thought I was ready to go, so I created a test and scheduled it. You have to schedule your test at least 15 minutes out — presumably the time it takes to fire up the EC2 instances and prepare the infrastructure for testing. As I was waiting for my test to kick off, I got a call from Patrick. He had called to inform me that some of the parameters in my scheduled test may not be optimal for the test I was trying to achieve. We had a good conversation, and he clarified for me some of the test input parameters and provided some good advice on setting the test up. I was able to stop the test before it ran and chewed up my credits, and reschedule it with better parameters. This was indeed first-class service! He could have simply let the test run, let us chew up our credits and learn the hard way, but instead he went out of his way to ensure that we got the most benefit for our investment.

I ran the test, and watched it scale up in real time. BrowserMob allows you to see all the variables in a real-time graph as the test executes (e.g. # users, response time, failures, throughput). Very quickly, ~20 users I started to see failures. I stopped the test (allowing us to keep the credits we had not yet burned up), and analyzed the results. I saw the problem right away — a particular component in the server side code was not using a thread safe class. This was a quick fix, and we were back in business firing off the test again.

This time, we had much better results, although it just took longer to uncover a new problem. We had scaled up to about half the total users I was looking for, when we started getting failures. I let the test run a bit longer to see if the ratio of failures was increasing…it was. I stopped the test, and we went off to analyze the problem. In this case, we uncovered an issue in the use of a particular Spring Framework class SimpleJdbcInsert. From the Javadoc of that class:

A SimpleJdbcInsert is a multi-threaded, reusable object providing easy insert capabilities for a table. It provides meta data processing to simplify the code needed to construct a basic insert statement. All you need to provide is the name of the table and a Map containing the column names and the column values. The meta data processing is based on the DatabaseMetaData provided by the JDBC driver. As long as the JBDC driver can provide the names of the columns for a specified table than we can rely on this auto-detection feature. If that is not the case then the column names must be specified explicitly.

Indeed, this class makes doing an insert easy. All you have to provide is a map with column names and column values. The problem is that every time you execute this, the driver has to query the JDBC metadata. This turned out to be a very expensive proposition as we scaled up. The database server's CPU was pegged on this metadata query as the number of users scaled up, and we started to experience a lot of HTTP/Selenium timeouts on the client side.

Again, this was an easy fix. We replaced this with a simple INSERT statement, and removed the use of SimpleJdbcInsert.

There is real value in finding these problems out quickly, and simply. BrowserMob is the key to making this happen. Their service is simple to use, and being able to quickly discover these issues is a testament to the real value it supplies. I also can't say enough about their first class service. Specifically, Patrick is extremely helpful, and he goes out of his way to make sure the customer is getting the most out of their credits. I'll be recommending their service for all future load testing needs, and I wanted to make a point of describing my positive experience here so others may learn about their great service.


Backlinks


Add a New Comment
or Sign in as Wikidot user
(will not be published)
- +
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License