As I alluded to in my excuses for not posting in ages blog post the other day I am working on a new Magento hosting review for a company called MageMojo. In doing so I’m trying out the performance testing tool called Siege. Siege was mentioned several times at the Magento Developers Paradise, so I have been wanting to try it ever since.
This is a little article that summarizes the steps I took to do the testing, in case you want to test your own Magento stores.
Update: I have since created a tool that will allow you to run testing like this from the comfort of your own web browser. Check out the automated Magento performance testing at MageSpeedTest.com.
Get a Server
Being all the way over in New Zealand pretty much means performance testing Magento from here is out of the question. I fired up an Amazon EC2 small instance for the purposes of testing for a few hours, 20c – I’ve said it once, I’ll say it again, commodity computing is the future . I won’t bore you with a tutorial on starting an EC2 server, there are enough perfectly good guides already online.
You could optionally use a server you already have, or run this from your own computer at the office or home. It would arguably be a fairer test of latency, not running the tests in a production data centre!
Definitely do not run it from the server that is running Magento, that would be a poor test; a) because there would be no latency and b) because the server would be using resources to run the tests, and serve the webstore, that’s not very fair on it!
Install Siege
wget http://www.joedog.org/pub/siege/siege-latest.tar.gz tar xvzf siege-latest.tar.gz cd siege-2.71/ # if you're on a fresh ubuntu, get this package first sudo apt-get install build-essential ./configure make && sudo make install |
Now you can begin a siege on your store, but first, an important warning…
Too many requests on your production server could easily make it unusable for other visitors, so be careful. Start small, with low numbers of concurrent users for short durations and keep an eye on your servers memory usage. You can halt an in-progress siege with ctrl+c at any time.
Oh and I’m sure the people who read my blog don’t need to be told this but; if you use Siege maliciously to _performance test_ other people’s websites (from multiple Amazon EC2 instances) that’s not performance testing anymore – that’s a denial of service attack, I strongly advise against it!
Run a Basic Siege Test
Now you can run your first Siege test. In this test we just hit one URL with some number of concurrent users (10) for some amount of time (1 minute):
siege -c10 -t1M http://www.google.com |
Run a More Complex Test against Magento
First grab a bunch of categories and products from the sitemap.xml (you have a sitemap, right?) and put them into a urls.txt file.
#categories curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 0.5 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' > urls.txt #products curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 1.0 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' >>urls.txt |
And then run the test like this for 30 seconds with 25 concurrent users, adjust to suit:
siege -i -c25 -t30s -f urls.txt |
Note: I bet a 100 million dollars there is a more elegant way to grab the urls from a sitemap. Someone post one in the comments so I can buy an island.
Looking at Results
Siege results look something like this:
Transactions: 149 hits Availability: 100.00 % Elapsed time: 13.87 secs Data transferred: 1.26 MB Response time: 0.37 secs Transaction rate: 10.74 trans/sec Throughput: 0.09 MB/sec Concurrency: 4.02 Successful transactions: 149 Failed transactions: 0 Longest transaction: 0.42 Shortest transaction: 0.36 |
The most import numbers to look for are Transaction rate and Response time they will give you an indication of how quickly your server is responding and how many connections it can handle per second.
I hope this little demo has been helpful. I’ll be following up with the full test results for MageMojo later this week, I also compared the Nexcess results – very interesting.
I’d be very interested to hear from anyone with tips and ideas for performance testing, likewise if you try this and have any trouble feel free to ask for help.
Edit: Pefromance…doh….
Edit: Updated Siege URL and version, thanks to Eric@MageMojo for letting me know.
I just invested 99 million dollars in getting the world’s top coders to try and come up with a better way to extract the URLs from the sitemap. Your method held up as the best method. They held their heads in shame and admitted defeat. Please PayPal me the 100 million dollars asap, as the interest on the 99m is killing me already…
Haha, I think we should wait a couple of days! There might be some other contender.
Might be cheating, but what about good old php:
php -r ‘$x=new SimpleXMLElement(file_get_contents(“http://domain.com/sitemap.xml”));foreach($x->url as $n) echo $n->loc.”n”;’
Hi Ashley,
Think i have seen that method somewhere before 😉
using siege to identify caching mechanism and server environment is also possible by using the -g parameter.
siege -g http://enterprise.procontractors.eu/
@Vinai Apologies this took a while to show up – somehow got flagged as spam on the way through, I’m pleased to see such a great one liner! Thanks for stopping by.
Inspired by this article I checked out siege more thoroughly. Thats how I stumbled over sproxy by the same author. It’s a proxy that saves all requests (GET and POST) in the siege url.txt format. That way it is possible to simulate user session with siege. The Sessions-ID needs to be passed as a GET parameter, though, Cookies don’t seem to work.
hey how’s the testing result for the Magento Stores hosted on Amazon EC2? I’m using a Large instance.. 7GB of memory on a fedora box.
my AVG response time is over 1 second.. after:
1. fooman speedster
2. compliation
3. APC cache
(now fooman’s not activated on the site, as I’m changing CSS atm)
this is why I’m considering spending some money to get this:
http://www.tinybrick.com/magento-modules/sales-tools.html
you think it’s worth it? Or kristof heard you have a good speed extension coming out as well..
cheers
nathan
( 0508 NATHAN ) my free call
@Nathan,
if you can tune the PHP APC settings to apc.stat = 0 and apc.shm_size = 512 (thats Megabytes) you will have better results then with the Magento Compiler module (turn in off when using apc.stat = 0).
Whenever you change a php file you will need to restart apache or flush the apc system cache by calling apc_clear_cache() though.
hi thanks a LOT, there are some huge differences.. from 1 sec wait time to 500ms wait time..
You just saved me $500 US…haha..
Merry Christmas everyone.
cheers
nathan
@Nathan @Vinai – uncanny timing! I have been doing some testing on different EC2 setups these last couple of days. Mainly I’m trying to set up a cluster that most of the time just has 1 large server in it, but can ramp up an extra one or two when the first is under load (like when you send out a newsletter, or get a blog review/coverage).
I’m going to write up my setup instructions and publish more thorough results of the EC2 clustering side of things when I get a spare moment (mid 2012).
In terms of optimization on the actual servers I did only the very basics (APC+more mysql memory). I’ll def. try your
statsuggestion Vinai sounds like a quick win.My initial testing with 1-3 large instances behind a load balancer, with one of the 3 large instances being the ‘mothership’ mysql server:
The good thing about this setup is that if you use the EC2 auto scaling, it could add up to say 10 larges and probably scale pretty linearly. The next improvement would be to pull out the mysql server but I have read that it does not improve performance by much so it’s probably not the bottleneck.
Stay tuned for full setup and results of my experiments and please, keep the tips and results coming. It’s so hard to know if we’re on the right track in isolation.
Cheers,
Ashley
Hi Ashley,
try
cat sitemap.xml | xpath -q -e “//loc/text()” > urls.txt
for a nice way to extract urls from the sitemaps.
Bye,
Fabrizio
Hi Fabrizio, very nice I haven’t had experience with xpath on the commandline. The current Magento specific algorithm grabs only specific priority URL’s – but I am seeing a need to get all URL’s so this may just be the best way.
Thanks!
Hi Ashley,
grab urls with a specific priority with following command using xpath:
curl –silent http://…/sitemap.xml | xpath -q -e “/urlset/url[priority=0.5]/loc/text()” > urls.txt
Bye,
Fabrizio
Nice one Fabrizio!
Hi – tool looks great, but once the pages are cached, aren’t you just testing the server, rather than Magento?
Is there a way to start a session, add a product to the cart & checkout in order to test actual customer transaction time?
thx!
Currently writing an free extension that will produce profiles for siege based on visitors url profiles. It will be configurable in the admin. For instance creation of sessions like:
Sesssion Name: Standart Checkout
Url Path:
Browse Catalog – Product View – Add to Cart – Checkout – Billing – Shipping – Shipping Method – Payment – Review – Place Order – ….
@Ashley: Thanks for explaining the tool and for creating magespeedtest.com
@Ivan: That would be huge for my team. Will you comment here when it’s ready?
Hi Ashley,
Great tool at http://www.magespeedtest.com.
It says that the tests are done from 4 data centres on 3 continents – can you please tell us which continents? I looked through this post, couldn’t find the detail (I may have missed it). Like you, I’m in the southern hemisphere. So a US host can sometimes end up having slow latency from here.
Thanks for your help,
TIm
Ashley, forget my last question – I just saw the answer!
The answer for everyone else’s benefit:
North America
Europe
Asia
(and soon, another Europe and one in South America!)
Ashley,
One more question – can I use MageSpeedTest to test my site if it has a different domain to my email? In other words, can I test our Magento site while it’s still on the devserver (to see the impact of adding extensions etc) – rather than wait till the site is live?
Example scenario:
My email is email@mydomain.com
Devserver is http://www.mydevserver.com
Thanks,
Tim
Yes, you can test any, but they are rate limited and I may ask you to prove you own/run the server to keep testing it if your tests are sustained. I haven’t had to yet, mind you.
Hi Ashley,
I’m a premium member for the speed test and I keep getting several server time out errors when testing 20 users, 45 secs. Is this normal? Any ideas how this can be fixed?
Thanks for your help
AJ
Example:
** Preparing 20 concurrent users for battle.
The server is now under siege…
[[1;32malert[0m] socket: -869157120 select timed out: Connection timed out
[[1;32malert[0m] socket: -848177408 select timed out: Connection timed out
I am searching for Magento Monitoring. Can you advise something?
I want to track if magento is ready for selling LED stuff…