Posted on

App Engine Java MapReduce Example: bulk deleting entities

As you may know, we run our OrderPipe ecommerce dashboard on Google’s App Engine – I’m a big fan of the platform, but it has some traps for new (and old) players. A recent issue required us to bulk delete entities on App Engine, and for that it seemed the best tool for the job was the ‘Mapper’ part of Map Reduce. It was a good experience learning a) how it works and b) applying it, I thought I’d document a full Map Reduce example, because the worked examples I found were all based on an older version of the library.

Background

You might be interested to read a bit about why I had to use Map Reduce – it was my own fault. A subtle bug in a recent release caused an additional Account to be created under certain situations – it made it past our staging server, because that certain situation happened infrequently enough to not cause an issue, but once it got to production, where we have many thousands of orders arriving daily, we started seeing a lot of zombie accounts being created. My first thought was we were being attacked, but alas, it wasn’t an attack, just a bug! The nett result, we were left with 10’s of thousands of unwanted entities.

The Problem

In App Engine, the datastore is highly scalable, but it’s not relational, you can’t just run a simple query to delete all rows in a table with a particular created timestamp. There are simple tools to blindly delete all entities of a type, but not entities that meet a particular condition, and more importantly, not related entities that also meet a particular condition. That’s where Map Reduce comes in, basically we shard the entities and create multiple parallel workers to break the collection into smaller parts and process through them quickly. This seems like an inefficient approach until you consider scaling beyond a single database server, or even a single database cluster.
Continue reading App Engine Java MapReduce Example: bulk deleting entities

Posted on

Opening (and closing) jQuery Mobile Collapsible sections programatically

We use jQuery Mobile for OrderPipe, it’s a fantastic framework and I have a lot of love for it. One thing that has bugged me about the way our settings section works though is after making a change within a collapsible section (like inviting a user for example) the section would be closed afterwards. It was surprisingly easy to fix that and this blog post will run through a quick example of allowing URL’s to map to open sections quickly and easily. It assumes Spring MVC on the server side, but you could do it with any server side technology.

Firstly on the server side, you could do this with query parameters, or URL paths (or cookies if you really wanted I guess) – we use paths, because I think it looks cleaner.
Continue reading Opening (and closing) jQuery Mobile Collapsible sections programatically

Posted on

Java App Engine integration testing with Spring MVC and OpenPersistenceManagerInViewFilter

In this post I’ll quickly show the technique I used for integration testing of Java services where the services rely on Spring data repositories being injected. We’re using Google’s App Engine to host and run our mobile sales dashboard for Magento/Amazon/Shopify so the tests will use the integration testing environment provided in the SDK.

Starting with a number of great articles on the subject I was able to quickly get a basic integration test up and running for App Engine with Spring. Our problem was related particularly to the PersistenceManager.

In the real application each web request is assigned a ThreadLocal PeristenceManager which is kept alive for the duration of that request, and always closed at the end, thanks to the OpenPersistenceManagerInViewFilter provided by Spring. So this is great when you’re calling data access methods on repositories from services in a web container, but means that if you’re trying to call those services during an integration test, the different repository calls will trigger exceptions that the objects involved belong to different PersistenceManagers.

The solution was to emulate the behavior of the OpenPersistenceManagerInViewFilter before and after a web request, in the setUp and tearDown of the tests themselves. For that I checked out the code for the filter and simply applied the same logic in the test.

First we make sure the Test class is setup to use Springs Junit functionality like below. Also note we are specifically setting the context to use the two test xml configurations. This is important because we actually wire a LocalPersistenceManagerFactoryBean in this Test class.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations =  {"classpath:application-config-test.xml", "classpath:servlet-config-test.xml"})
public abstract class BaseSpringJunitTest
extends AbstractJUnit4SpringContextTests {
 
	@Autowired
	LocalPersistenceManagerFactoryBean pmf;
 
	// Standard App Engine integration testing helper
	protected final LocalServiceTestHelper helper =
			new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig(), new
					LocalTaskQueueTestConfig(), new LocalMemcacheServiceTestConfig());
 
	// ...
}

Most of the test xml configuration is standard, the important bit is wiring in the bean, which should look familiar from your usual container deployment:

<bean class="org.springframework.orm.jdo.LocalPersistenceManagerFactoryBean" name="persistenceManagerFactory">
	<property name="persistenceManagerFactoryName" value="transactions-optional" />
</bean>

The setUp() just creates a new PersistenceManager (from the PMF we wired in) and stores it for use during the test (the same way the filter stores it for a web request).

@Before
public void setUp() {
    helper.setUp();
    PersistenceManager pm = PersistenceManagerFactoryUtils.getPersistenceManager(pmf.getObject(), true);
    TransactionSynchronizationManager.bindResource(pmf.getObject(), new PersistenceManagerHolder(pm));
}

In the tearDown() we’ll just do the same thing the filter does after the request.

@After
public void tearDown() {
	PersistenceManagerHolder pmHolder = 			(PersistenceManagerHolder)TransactionSynchronizationManager.unbindResource(pmf.getObject());
    PersistenceManagerFactoryUtils.releasePersistenceManager(pmHolder.getPersistenceManager(), pmf.getObject());
	helper.tearDown();
}

Once you have that class setup, you can extend it for your actual testing like this:

public class TestAccountService extends BaseSpringJunitTest {
 
	@Autowired
	private AccountService accountService;
 
	//..
 
	@Test
	public void testSubscribeNull() {
		accountService.subscribe(null, null);
		// check that subscribe behaved OK for null parameters...
	}
}

That’s it, go forth and test!. App Engine sure does make local integration testing and developing a local copy of your app easy and painless – I suggest you try it.

PS: This is the first Java development related post I have written in a long time, apologies to the Magento developer readers out there, it may not be the last.