Any general web development topics are covered here, ranging from ecommerce web stores, search engine optimisation or store optimization to framework discussion.
I’m forever having to look up various types of obscure GQL syntax to use in the App Engine console. I thought I’d start a blog post where I note them down, for my own sake. I’ll update this as I go.
SELECT * FROM EntityName WHERE __key__ IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ') |
SELECT * FROM EntityName WHERE ancestor IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ') |
More to come, if you find one I’m missing let me know
In this post I’ll run through the quick steps required to update your AWS Auto-Scaling group to use a new EC2 AMI. This post builds on the guide covered in my earlier post on setting up an Auto scaling group for an elastic load balancer, which in turn was built on the WordPress clustering guide. You could say this is part 3 in a rather long-running series.
There’s not much too this process thankfully.
The easiest way I have found to do this is actually via the EC2 console. All I do is start a new instance on my current AMI, log into it and make any changes required (install/update packages etc) and then from the EC2 console, select the instance.
You don’t often need to log into the slave nodes, so if you need a refresher on how to do it, this command is all you should need:
ssh -i path/to/your/key.pem user@your-public-ip-of-the-node.compute-1.amazonaws.com |
Continue reading Updating your AWS Auto-Scaling AMI to a new version
As you may know, we run our OrderPipe ecommerce dashboard on Google’s App Engine – I’m a big fan of the platform, but it has some traps for new (and old) players. A recent issue required us to bulk delete entities on App Engine, and for that it seemed the best tool for the job was the ‘Mapper’ part of Map Reduce. It was a good experience learning a) how it works and b) applying it, I thought I’d document a full Map Reduce example, because the worked examples I found were all based on an older version of the library.
You might be interested to read a bit about why I had to use Map Reduce – it was my own fault. A subtle bug in a recent release caused an additional Account to be created under certain situations – it made it past our staging server, because that certain situation happened infrequently enough to not cause an issue, but once it got to production, where we have many thousands of orders arriving daily, we started seeing a lot of zombie accounts being created. My first thought was we were being attacked, but alas, it wasn’t an attack, just a bug! The nett result, we were left with 10’s of thousands of unwanted entities.
In App Engine, the datastore is highly scalable, but it’s not relational, you can’t just run a simple query to delete all rows in a table with a particular created timestamp. There are simple tools to blindly delete all entities of a type, but not entities that meet a particular condition, and more importantly, not related entities that also meet a particular condition. That’s where Map Reduce comes in, basically we shard the entities and create multiple parallel workers to break the collection into smaller parts and process through them quickly. This seems like an inefficient approach until you consider scaling beyond a single database server, or even a single database cluster.
Continue reading App Engine Java MapReduce Example: bulk deleting entities
In this post I’ll run through two ways that we improve the default Magento ImportExport module when importing products to Magento stores. The two improvements I’ll cover are: importing directly from a Google Docs spreadsheets and improving the multi-line format to make generating the import files easier.
The ImportExport module itself is such a drastic improvement on the old DataFlow module it’s almost unbelievable, so let me first start by saying it’s not half bad to start with. Andreas does a brilliant job of describing the current file format, it’s in German but Google Translate plus the excellent example tables will make it quite readable. Vinai’s presentation at Magento Imagine 2011 is also required reading to understand how the new module works, and most importantly how easy it is to extend.
The two things I didn’t really like about the current functionality were the need to get a single local file to the server for processing and the multi-line format for configurable products and multi-store/site/category/image products. Here’s the changes made to improve on these.
Continue reading Improving Magento ImportExport module with Better file format and Google Docs integration
I’ve had a fun & games sort of day working through this Magento SSL offload on Amazon’s ELB and I thought it’d be worth documenting what’s involved, in the process I’ll give some steps to get it working and explain what the new (since version 1.6.2) Offloader header config option is for and how it helps.
Firstly, a bit of background – if you just want the quick fix, skip right to it. The Internet can beam magically onto your computer screen one of two ways: http and https. http means the things you see everyone else can see too, the content is unencrypted and sent from the server to your computer. If you’re at a wifi hotspot, what you do online is being broadcast to everyone around you. Conversely, an https connection means the content is securely encrypted, and the endpoint server you are connecting to, is probably who they say they are. So we want https in public places, or when financial or personal information is involved – and if we’re paranoid, we want it all the time.
Continue reading Magento SSL Offloading with Amazon ELB