Category: Google App Engine

Posted on

ScaleConf Wellington 2017

I went to and presented at ScaleConf in Wellington last week and was really impressed with the topics and info shared. Here’s a quick round up of my highlights from ScaleConf, a one-day conference with a focus on building and running large scale web apps.

All of the presentations were excellent and I learned a great deal from all the speakers, there were four that were real highlights for me that I wan to call out specifically and make some notes from:

Sarah Dapul-Weberman, Engineer from Pinterest
Sarah described her project at Pinterest transpiling Python code to Javascript. For a site with 100’s of millions of users, this sort of migration is seriously hard and Sarah had such a cool approach to basically writing a throw-away tool to convert the code, then a second tool to test the result using shadow traffic (a technique I’d not heard of before) until the re-written code generated the same responses and actions as the original.

Kumar Srinivasamurthy, Tech Lead from Bing.
Being the second largest search engine means serious scale, Kumar shared some fascinating insights into what it’s like behind the scenes of 100,000 server data centres, I particularly liked his anecdote on how hard it is to actually turn off a datacenter, with redundancy even at the utility level.

Andrew Schofield, CTO from Timely
Andrew gave a good overview of the growth of Timely, the phases they went through and challenges. I was inspired by what Timely have created both for software and team/work environment and culture. If we can build A2X up to that level, and create the same team, I’d be very happy.

Pam Rucinque, Engineer from ThoughtWorks
Pam detailed here project for a production mobile backend using AWS Lambdas, so it was very close to my own interest in server-less architecture. Having built a few Lambda functions for internal projects, I was really interested to see the sort of challenges that go into making it a fully production ready platform. The tool support for setting up the functions has improved a lot too, I think this is going to be a very interesting area of computing over the new few years.

If you’re interested, my slides are here, I presented A2X and our architecture on Google App Engine, what I liked about it and how it helped us go from prototype to 1000’s of users.

This is me presenting an entirely made up quote:

DG174dqUwAAJfrw.jpg-large

If you’re building a fast-growing or at-scale web app, I’d recommend checking it out next year! Thanks to Christian and the team of putting it together.

Posted on

Google App Engine GQL queries in the Datastore Viewer console

I’m forever having to look up various types of obscure GQL syntax to use in the App Engine console. I thought I’d start a blog post where I note them down, for my own sake. I’ll update this as I go.

Look up entities by encoded key string

SELECT * FROM EntityName WHERE __key__ IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

Look up child entities by encoded key string

SELECT * FROM EntityName WHERE ancestor IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

More to come, if you find one I’m missing let me know

Posted on

App Engine Java MapReduce Example: bulk deleting entities

As you may know, we run our OrderPipe ecommerce dashboard on Google’s App Engine – I’m a big fan of the platform, but it has some traps for new (and old) players. A recent issue required us to bulk delete entities on App Engine, and for that it seemed the best tool for the job was the ‘Mapper’ part of Map Reduce. It was a good experience learning a) how it works and b) applying it, I thought I’d document a full Map Reduce example, because the worked examples I found were all based on an older version of the library.

Background

You might be interested to read a bit about why I had to use Map Reduce – it was my own fault. A subtle bug in a recent release caused an additional Account to be created under certain situations – it made it past our staging server, because that certain situation happened infrequently enough to not cause an issue, but once it got to production, where we have many thousands of orders arriving daily, we started seeing a lot of zombie accounts being created. My first thought was we were being attacked, but alas, it wasn’t an attack, just a bug! The nett result, we were left with 10’s of thousands of unwanted entities.

The Problem

In App Engine, the datastore is highly scalable, but it’s not relational, you can’t just run a simple query to delete all rows in a table with a particular created timestamp. There are simple tools to blindly delete all entities of a type, but not entities that meet a particular condition, and more importantly, not related entities that also meet a particular condition. That’s where Map Reduce comes in, basically we shard the entities and create multiple parallel workers to break the collection into smaller parts and process through them quickly. This seems like an inefficient approach until you consider scaling beyond a single database server, or even a single database cluster.
Continue reading App Engine Java MapReduce Example: bulk deleting entities

Posted on

MageSpeedTest.com: New locations, Magento Extension, REST API, Featured Host and Pricing

Way back in 2010 I set up a quick little wrapper around siege for testing Magento stores and for the last nearly 2 years I’ve added features such as multiple data centers, premium subscriptions, monitoring alerts, and automated host comparisons. In this blog post I’ll walk you through an update on the new features and improvements to MageSpeedTest.com over the last few months.

I’m really happy with the site and how it helps users identify and improve Magento performance. I often see test results coming through in the order of 2-3 transactions per second and >5s per transaction. Identifying these sort of slow stores early is important – customers are simply not going to wait at least 5 seconds to see a product page these days.

At the opposite end of the performance spectrum I see public tests coming through with incredible performance profiles. For these developers and server admins it’s a way to consistently test server tweaks and tuning to get the absolute best customer experience.

In this post I wanted to quickly cover my latest Magento Speed Test changes. It’s a big update because I haven’t posted anything in a while.

Here’s the cliff notes:

  • New Locations: Japan, South America and Australia
  • A Magento Extension to make testing easy (and more soon!)
  • A REST API for accessing your monitoring results
  • A pricing update, $5 up to $9 per month
  • Featured Magento hosts
  • SSL support – test https:// sites

For more information on these items, please read on.
Continue reading MageSpeedTest.com: New locations, Magento Extension, REST API, Featured Host and Pricing

Posted on

Sending Apple iOS Notifications via Urban Airship on Google App Engine

In this post I’ll show you how to send an Apple Notification from Google App Engine via Urban Airship. I was mightily impressed how easy it was to do earlier this week. Their docs are excellent, and I love when apps use curl for their API examples – it means the API is simple, and the examples are perfectly transparent.
Continue reading Sending Apple iOS Notifications via Urban Airship on Google App Engine