I’ve made a new Magento extension for submitting your store sitemap to Google, Yahoo, Ask and Bing. My new extension can submit automatically whenever the sitemap is regenerated, or on-demand from the admin UI. More about how I made it and how it works below, but first a little background.
How Sitemap Submit came to be
As of last week I have been taking a train to work in a far off land called Pukekohe a couple of times per week. Pukekohe itself is not traditionally known as a high-tech startup hub – but in 2010 the number of small high-tech software engineering companies has grown considerably (probably at least 200%).
What all this means is I get a couple of hours a day, two times a week where I’m literally stuck in front of my laptop as the train I’m on rocks and weaves it’s way across country to WWA HQ. This new Magento extension I have made is the first of hopefully many what I’ll now call Public-Transport-Projects. The extension will manually or automatically submit your Google sitemap file from your Magento store to several major search engines (Google, Yahoo, Bing and Ask).
How to use Sitemap Submit
To start with, you’ll need to install it – head over to the Sitemap Submit extension page on Magento connect to get started with that.
Once installed, It’s dead easy to use – basically you generate your sitemaps the usual way using Magento’s built in functionality – to do this go to the Catalog -> Google Sitemap section of the admin interface as shown in the screenshot below.
Once you have your sitemap.xml generated you can click on it and view the details. One this edit screen you will see the new Submit button, which when clicked, my extension will submit your sitemap to Google, Yahoo, Ask and Bing. You can see in the screenshot below where the button should be.
There is also a configuration option for automatic submission (In System->Config->Catalog->Google Sitemap). What this means is that whenever your sitemap is generated or regenerated (by you or by cron if you have it configured to do so) then the sitemap will also be submitted to the search engines.
The only other configuration to be aware of is the yahoo app id, to use the api you need to add an app id – read all about the Yahoo API and the app id. If you omit an ID, the extension will just skip notifying Yahoo, no big deal.
How Sitemap Submit works
I won’t bore you with too many details, basically it’s made up of an observer that watches out for the regeneration and a service object that actually does it (plus the usual various Magento extension things, like a controller and a helper.
The only tricky thing was the core Sitemap.php object didn’t fire a generating event, which if you read my explanation of Magento events and observers you’ll know is quite important. I override and slightly alter the default Magento sitemap as shown below:
public function generateAction() { parent::generateAction(); Mage::dispatchEvent('sitemap_sitemap_generate', array('sitemap'=>$this)); } |
That fires the event which is then picked up by my observer and in turn my submission service kicks off. It basically runs through each search engine and submits the sitemap using the required URL’s like so :
//ping google if($this->_pingGoogle($url)) { $msg = $msg . "Pinged Google successfully<br/>"; } else { $msg = $msg . "Failed to ping Google.<br/>"; } |
… and then I actually make the request like this:
protected function _makeRequest($ping) { $curl = new Varien_Http_Adapter_Curl(); $curl->setConfig(array( 'timeout' => 20 )); $curl->write(Zend_Http_Client::GET, $ping, '1.1'); $data = $curl->read(); if ($data === false) { return false; } if ($curl->getInfo(CURLINFO_HTTP_CODE) == 200) { // Mage::log($data); // uncomment to debug raw submission response return $data; } else { Mage::log("Submission to: " . $ping . " failed, HTTP response code was not 200"); return false; } //TODO: handle timeout? } |
And that’s about it, you can read the full story in the open-source PHP code, in fact I’d invite you to do so, and give me some feedback on it!
The Sitemap Submit Feature Roadmap
I’ll be looking to extend the extension further over the next month or so in two ways. One is to add many more search engines that have an open sitemap submit function. The second is I want to enable more SEO capability and allow store submission to the many Magento store directories and Magento showcases like Magento Parade. I’m also, as always, really keen to hear feedback on fixes and features that you would like to see in the product.
When signing up for the Yahoo API what url should be used for the application entrypoint? Just the root website or is something more specific required?
Would be nice if you could also build in sitemap splitting support for websites with more than Google’s 50K URL / 10MB limit.
Like this:
http://www.offshoresoftwaredevelopmentindia.com/blog/2009/06/13/magento-extension-google-xml-sitemap-split-module/
Not sure if these two extensions would work together or not.
Not able to get 0.2.1 to show up in admin. (?)
Can you confirm the files are installed in app/code/community/Aschroder? Also you have cleared your cache?
Got it…Cache. Thanks
How to work in multi websites ? generate it , always main site url.
Good question – thanks to Filipe my extension will submit the correct Magento store URL to Google, Yahoo, Bing etc however it looks like a display bug in the Magento Core renderer (Mage_Adminhtml_Block_Sitemap_Grid_Renderer_Link) that causes it to use this:
$this->htmlEscape(Mage::getBaseUrl('web') . $fileName);when it should probably use:
Mage::app()->getStore($sitemap->getStoreId())->getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK) . $fileName;
The
htmlescape()I’m not sure about – can any URL characters actually mess with HTML? I guess there could be spaces etc in the url path, perhaps I should escape my url too – anyone confirm this?Hi Ashley,
I’m having a minor issue with your Sitemap extension and was hoping you could help. When the sitemap is submitted to Google, Yahoo, etc. it’s being submitted as DOMAIN/index.php/sitemap.xml when the actual address is just DOMAIN/sitemap.xml – is there a simple way I can remove the “index.php” bit so that the various search engines can fine my sitemap rather than a 404 error? Thanks!!
Hi, Please check the base url in your system configuration settings – perhaps the index.php is coming from there? Let me know.
No, nothing in there. I know I’m stuck with the index.php for most of my URLs until I can get the Base Rewrite to work correctly, but that’s not the case with the sitemap. Anyway, I’ll fiddle around and figure it out eventually, just thought you might have an idea. Thanks!
oh – it might be a bug then – If url rewrites are off then I need to return the base URL without index.php, I’ll look into it when I get a moment. Thanks for the heads up.
Anytime. Keep up the great work!
Finally managed to get my rewrites working properly, so the sitemap extension is behaving just as it should for me now. Good luck w/ the potential bug fix & thanks again for a great addon!
Hi Ashley,
I have tried the plugin on Magento 1.2.x, but cant see the set-up forms etc. Does it work on this platform?
Thanks,
Gary
I should have mentioned that the specific issue is that I can enter the Yahoo API code etc under the configuration part, but that when I run the Submit option for the sitemap, I get the following error.
Fatal error: Call to undefined method Varien_Http_Adapter_Curl::getInfo() in /home1/myweb/public_html/mywebsite/app/code/community/Aschroder/SitemapSubmit/Model/Submit.php on line 126
However, manual generation does work as expected.
Gary
Hi Ashley,
I’ve just recovered from having my site hacked (grr, PHP code injection) & am concentrating on hardening security wherever I can. I noticed last night that the sitemap.xml file generated by your extension always sets itself to 666 permissions, meaning that it’s world writable. It’s easy enough for me to change the perms, but a bit of a headache to have to remember to redo it every time it gets generated. Is there something in the code I could change to make the file always generate with 644 permissions?
Thanks!!
– Ken
Sorry to hear about your hack – was it a vulnerability in Magento or some other PHP script? God suggestion to limit the sitemap permissions.The sitemap.xml is actually generated by the built in Magento code, I just took a look though and I think I could add something to my extension to tweak the permissions. I’ll add it to the list of features for the extension, and update it when I’m next on the train.
Hi Ashley,
Thanks 😉 It was a nightmare to cleanup, but I think we’re more solid than ever now, so I ‘spose it was for the best. It was actually vulnerabilities in an old WordPress installation I hadn’t updated or removed & wasn’t using – laziness on my part pure and simple.
I’ll keep my eyes here for updates & will just alter perms manually in the meantime.
Thanks again!
– Ken
i cant find the configuration option for automatic submission
It’s in System->Config->Catalog->Google Sitemap – I should make that clear in the blog post, thanks for the heads up.
Hi
I just installed this on 1.4.1.1 and it’s not showing up in the admin.
Is it not compatible with that version?
I see the Yahoo Notification API is being shut down in Dec 2010. Is there are plan to support the new Update Notification service?
http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html
Hi Ashley, I tried to install the extension using Connect but got this error..
“Invalid package identifier provided: http://connect20.magentocommerce.com/community/ASchroder_SitemapSubmit”
any suggestions..?
thanks
Um I think you’re trying to use connect20 – but that’s for Magento 1.4.2 right? I haven’t tested the extension with that version of Magento, it probably doesn’t get put on connect20 until marked as compatible (I’m guessing…)
Yahoo API Key #We’re shutting down this service in December 2010. For further details, please see the Deprecated Services blog post.#
And Now??
Thanks a lot for your the great work
Thanks for this great extension.
I note that this extension don´t generate sitemap for AW-Blog contents. I think that this is very important.
Hey Andy,
Trying to get your extension installed for toggle.co.nz.
Mage Connect keeps falling over. Can you flick me through the source to info@toggle.co.nz
Cheers bud.
Todd
Make that Ashley…
Great extension Ashley! Is there anyway to confirm that the sitemaps are submitted? Mine are being regenerated by Mage’s cron daily, but I’m not certain if they are successfully being submitted to Google etc…
Is this compatible with Magento CE 1.5?
Hi, Rose – I haven’t tested it on a 1.5 site yet, you could give it a go and let us all know?
Rose,
Did you test this on 1.5+ yet?
Does it work on CE 1.5?
This extension looks to be a great time-saver. Much appreciated. Are there any plans for a new version incorporating the new method Yahoo is using or site maps?
Hi,
I sort of got sidetracked in my attempt to upgrade to 1.5, I was setting up to upgrade and was checking all the extensions I have installed. If you check over at magento connect it seems many people are having difficulty installing this with 1.5. Maybe you can take a look and see if you can help us out and get it to work in 1.5. It is a great extension and I think we would all apreciate if we could continue to have this great functionality in 1.5.
In advance, Thanks so much for all you have done in Magento. I just tried your speedtest very nice. Thanks
My question is like “Rose”. If you can update your extension with 1.5 or even 1.6. thanks
example of a sitemap http://www.postpot.com/sitemap/sitemap.php
Unfortunely it not works on 1.6.x. I wasn’t able even to install it via connect2.0. Can you, please, provide the files? We could try to make it work on new releases.
Thanks.
Works fine on 1.6 and 1.5
becareful to use Mage_Core_Model_Store::URL_TYPE_WEB for the correct link generation
How did you install this in 1.6? There’s no extension key for 2.0 on the magentoconnect website.
My website: http://www.charngnoi.com is running 1.7… do you know if this would work on 1.7?
Thanks!
Kim
Would love for this to work on Magento CE 1.7, but I cannot get it to install on my site http://www.speelal.nl
Developer, please pretty please with sugar on top; would you consider developing this further?
Regards,
Cper
Great extension, really helpful thanks!
The link is not working (page not found)
http://www.magentocommerce.com/extension/2816