Sending Emails by Queue in Magento 1.9.1

My last blog post was over 6 months ago, so I think I’m technically way past due for a new post. In this post I wanted to take a tour through the new functionality in Magento 1.9.1 with regards to sending emails from your store.

The astute among you will have noticed there were some changes to email in Magento 1.9.1 from the Magento 1.9.1 release notes. Two things jump out as requiring further investigation:

1. “all Magento e-mails (including order confirmation and transactional) are now queued and sent according to your configured cron schedule”
2. “boasts responsive default email templates so customers can read your order confirmation emails and newsletters on any device”

The first change sounds worrying to me, after years of emails from merchants struggling to get their Magento store to send emails – anything which makes it _harder_ seems like a bad idea to me. However, we’ll look through the changes and see what’s new.

The second change sounds really good – I was tinkering on a responsive email extension myself, so it’s pleasing to see it has already become core functionality. I’ll take a look through the template changes, and how they affect your store in the second part of this blog post series.

Using cron to send Magento Emails

Magento email sending all boils down to the Template class Mage_Core_Model_Email_Template, you can see below the change for queuing has been to introduce a check for an available queue in the template class, and if available to enqueue the message with all of it’s data and return immediately.

if ($this->hasQueue() && $this->getQueue() instanceof Mage_Core_Model_Email_Queue) {
    /** @var $emailQueue Mage_Core_Model_Email_Queue */
    $emailQueue = $this->getQueue();
    $emailQueue->setMessageBody($text);
    $emailQueue->setMessageParameters(array(
            'subject'           => $subject,
            'return_path_email' => $returnPathEmail,
            'is_plain'          => $this->isPlain(),
            'from_email'        => $this->getSenderEmail(),
            'from_name'         => $this->getSenderName(),
            'reply_to'          => $this->getMail()->getReplyTo(),
            'return_to'         => $this->getMail()->getReturnPath(),
        ))
        ->addRecipients($emails, $names, Mage_Core_Model_Email_Queue::EMAIL_TYPE_TO)
        ->addRecipients($this->_bccEmails, array(), Mage_Core_Model_Email_Queue::EMAIL_TYPE_BCC);
    $emailQueue->addMessageToQueue();
 
    return true;
}

If there is no queue, the send() function proceeds as it used to, builds the mail object and sends it. Extensions like SMTP Pro and MageSend that hook in to the Template class to change the mail transport, will inject their transport at this point.

Slight Tangent: Are all emails actually being queued?
You might be thinking, as I was at this point, Under what circumstances will there be no queue? Good question. According to the release notes, “all Magento e-mails” are now queued, so I expect that the queue must be being set somewhere for all outbound emails. Let’s dig into this and find out.

Firstly, where the queue object is instantiated with a getModel() or getSingleton() call it would be referenced by it’s handle core/email_queue. So we grep for that handle, to find the places where it is instantiated. Oddly, I only find two places where the queue is created and set on the Template object. Both are in Mage_Sales_Model_Order, functions queueNewOrderEmail() and queueOrderUpdateEmail(). So my gut feeling at this stage is I have missed some obvious place where the queue is being set for *all* other emails (ignoring the obvious question, if it’s being set for all emails, why set it explicitly here?).

So, let’s try and find calls to the setQueue() method, assuming the core developers have not used a setData() – they must have called setQueue somewhere, to ensure the queue object is being used. Strangely I only find reference to Template.php and Mailer.php, both setting the queue based on what was passed to it.

OK, so I’m not sure all emails are being queued, let’s ask the community to double check me and continue with our analysis.

When Emails are queued…
If the message is queued, then the email data is saved in the Magento database where it will be later fetched and an email message will be reconstructed from the data and sent. The data is stored in two new tables, one for the email data and one for the recipients:

<email_queue>
    <table>core_email_queue</table>
</email_queue>
<email_recipients>
    <table>core_email_queue_recipients</table>
</email_recipients>

When and how is the queue cleared? For that Magento have added two new cron jobs:

<core_email_queue_send_all>
    <schedule><cron_expr>*/1 * * * *</cron_expr></schedule>
    <run><model>core/email_queue::send</model></run>
</core_email_queue_send_all>
<core_email_queue_clean_up>
    <schedule><cron_expr>0 0 * * *</cron_expr></schedule>
    <run><model>core/email_queue::cleanQueue</model></run>
</core_email_queue_clean_up>

That means the send() function on the queue will be called every minute (why */1 and not just * I wonder?) and the clean up once per day at midnight 00:00.

The send() function will grab a chunk of size MESSAGES_LIMIT_PER_CRON_RUN (default 100) emails data out of the DB table and send the emails one by one that have been queued since the last run. The send itself is handled in the same way the Template class does in the pre 1.9.1 code.

$collection = Mage::getModel('core/email_queue')->getCollection()
    ->addOnlyForSendingFilter()
    ->setPageSize(self::MESSAGES_LIMIT_PER_CRON_RUN)
    ->setCurPage(1)
    ->load();

A couple of things I wish the core team had done differently here. 1) to allow configuration of the rate of email sending from either the XML file or via the admin panel. It’s important for most 3rd party email sending services to control the rate of email sending to avoid being flagged as spam.

And 2) to re-use the email sending functionality from the Template class – instead of basically copy-pasting the Template class sending code in the Queue. This has the effect that extensions like SMTP Pro or MageSend – or any that try to modify the passage of outbound emails – will now need an extra override, one that is totally unrelated to actual sending, just to set the transport before sending.

Lastly, 3) the email queue table could also act as an email log (if all emails are sent using it) and even allow users to easily resend the emails. All of these are items I will try to address as 3rd party functionality in the next releases of SMTP Pro and MageSend.

I also think it’s important to test whether cron is running as part of the self test/setup – as it’s one of the areas of Magento that often is not handled correctly. I wonder why Magento just doesn’t do an auto-cron on front end page views like WordPress, but that can be a post for another time.

I hope this little tour through the new functionality helps you to understand the new email sending queue functionality in Magento 1.9.1. In the next post in this series I’ll take an in-depth look at the new responsive email templates shipped as part of Magento 1.9.1 – how they work, how you can customize them and things I’d like to see improved/changed (if any).