Comments on: MageSpeedTest.com with Siege – Performance Testing Magento Made Easy

By: Ashley

Ashley — Wed, 05 Jan 2011 09:04:14 +0000

Hi, J.T. Happy New Year to you too!

That’s a good idea having the file dictate the thresholds. It could also have:

NotifyEmail a@b.com
NotifyThreshold 10

So that it emails a@b.com if 10 tests have been run or something (logging the IP that initiated them).
I need to clone myself…

By: J.T.

J.T. — Tue, 04 Jan 2011 10:33:48 +0000

Perhaps the HTML verification file could contain the actual number of tests, or the time frame, you allow.

Or a simple XML file:

AllowFrom DateTime
AllowTo DateTime
AllowTestsPerHour 10

But that’s making it quite elaborate already. The default behaviour could simply be as you have it now with some optional advanced functionality for us nerds.

The performance versus hosting fees is a neat though potentially depressing idea. If sub one sec response and over 20 transactions is possible, I sure have my work cut out!

Happy 2011 by the way!

By: Ashley

Ashley — Mon, 20 Dec 2010 20:40:12 +0000

@J.T. Good idea to shorten up the 10 minute delay. The Google style ‘place an html file in the root of your store’ approach is probably the best.

The kicker is that once that file is there, anyone could initiate a test against you. I could make the file contents email specific – which would give a little security by obscurity. So the file contents+domain+email must all match in order to do unlimited tests.

Maybe the correctly coded html file could be what is sold in a bid to make money from this little experiment!

I’ll have a tinker with it this week, but definitely will take your feedback on board.

@Paul – Good question, it’s a little bit hard to mark a single test as good or bad – if you’re paying $20/month then the definition of good is very different to if you are paying $500/month.

Here’s the best one I’ve seen, one to strive for. I won’t say who it’s from but they shared it with me via Skype yesterday. I’m sure when they are ready, they’ll publish their results.

Response time:		        0.68 secs
Transaction rate:	       24.80 trans/sec

This is an enterprise solution though, you’ll struggle to get performance like that for $50/month!

How about when people elect to publish their results they can optionally include how much they are paying for the hosting. That way I could do some sort of graph that shows price/performance and then others can see where they sit on the graph.

I’ll add it to the feature ideas list – it’d be good to put results in context.

Good ideas guys, please keep them coming.

By: Paul

Paul — Mon, 20 Dec 2010 13:57:59 +0000

Are there any guidelines or recommendations as to what is acceptable?
Perhaps you could rate the results good bad or ugly when they are published.

By: J.T.

J.T. — Mon, 20 Dec 2010 10:47:32 +0000

Used it, like it!

Apart from the fact I got terrible results, which will ruin my Christmas holiday as a perfectionist just can’t tolerate crap server performance.

A third party testing tool that doesn’t need setting up is great as performance testing otherwise can be difficult to get going.

And when this is out of beta, you can expect reliable testing, that is the same every time.

So nice tool indeed, thanks for making it available! I’d even pay a one-off fee to lower the 10 minute delay. As I now need to test some 100 combinations of server tweaks and speed tools, I’ll end up with a lot of waiting for the 10 minutes to pass.

I understand it’s to avoid DOS attacks of sorts so perhaps you can do something like Google Webmaster Tools and have us upload a unique html file or meta tag value to proof the server is ours, and then it allows more heavy testing.

Anyway, I always want more more more, don’t mean to be ungrateful! 🙂