This is a blog post about how I structure my Magento store development, and how that has been made easier thanks to the efforts of Colin Mollenhour and his script modman. Hopefully after reading this, you’re all inspired to go set up a similar project structure, because it gives a repeatable, hopefully less upgrade ruined Magento experience. Certainly if you’re not inspired, I’d like to hear about how you are structuring your projects – the more talk of this, the better the overall quality of Magento stores in the wild, I hope.
History
Over the last nearly 2 years I’ve tried numerous project structures to make developing multiple Magento stores less painful. My background before getting in to ecommerce development has been in web development for large (think 100+ developer) web projects, and small, agile ones (2-10 developers). I know and appreciate how important project structure is for both teamwork, and rapid, predictable development. I thought it’d be illustrative to briefly plot my path through failed attempts at getting a workable Magento project structure. Begining with…
Not structuring your store at all
The first attempt at this was the ‘no-structure’ approach mistake – being exposed to only WordPress and Drupal PHP apps (and being of the opinion that all PHP scripts were just a little hackish), I thought that I’d be fine just installing Magento and getting started. I was for about 1 month, until I upgraded…
The reality is Magento is a lot more complicated than other popular PHP apps, and if you plan on actually engineering it, rather than just hacking it, you should structure it for that.
Sadly, it’s a very easy situation for a beginner at Magento to sort of fall in to – especially as most shared hosts will install Magento for you and basically set you up for some heartache when you go to upgrade, especially if you cut corners when customizing.
The whole store in version control structure
I’ve also tried having an entire Magento store checked-in to version control. This works OK, but doesn’t lend itself to sharing the same customizing effort across multiple stores (repeating bug fix and improvement effort), which at World Wide Access is a factor I’ve quickly realized we need to consider as we add more suppliers and product ranges.
Along side this approach I tried maintaining a ‘vendor branch’ which is described here. It’s a lot of work, and it turns out still causes problems when you try to upgrade – the best approach, I have found, is just to do all customizing as extensions.
I want to be clear here that I am talking about separate stores as in completely independent Magento installations, we create webstores for our suppliers, each is separate – with different inventory, and customers, so the Magento sub-store is not suitable for our purposes.
This is quite different to the Magento feature of websites, stores and store views. This allows you to create similar stores all contained within the same installation. If you use sub-stores in a single installation, then they all share the same code, extensions etc. On a personal note, the few times I have used this feature, I have been burned by parts of the code that incorrectly handle the store-scope, or extensions that do not properly use the store code – I’ve been guilty of it myself, actually.
Symlinking extensions into vanilla Magento installs
Fast-forward to about 6 months ago, and I had changed my development technique considerably. I started developing all my customizations exclusively as extensions, and all store specific stuff only within a package (and themes within it). These changes would be symlinked in to the Magento store on an as needed basis.
This meant the underlying Magento install was no longer part of my project, it just existed, and had to be installed, in order for the stores to function. This is very tenuously analogous to an app server and web app in the Java world I used to know and love (what have I become!). One difference between the two is that Sun doesn’t hose their API in some obscure way every point release, but Varien will get there – I’m sure.
Anyway, when set up this way each Magento store symlinks to the various components it needs from the repository. So if a store needs automatic inventory updates from Warehouse A, then I simply symlink in that extension, if some other store needs inventory from Warehouse B, then I symlink the Warehouse B extension. This can be seen in the diagram below (thanks to Google docs):
Now, if you need both warehouses, you symlink in the warehouse manager extension, and it will handle that for you. This allows me to develop each extension as a separate self-contained project, and share that effort easily between multiple stores. It drives you to create more generic solutions to customizations, so that they can be easily applied to different stores.
The diagram is slightly simplified in that I actually keep the production versions outside the code repository, on our server. In this way, when you’re ready to release a new and improved version of the Warehouse A extension, you can simply export and re-link like in this example:
svn export ../working/WarehouseA WarehouseA.`date +"%m-%d-%y"` ln -nfs WarehouseA.`date +"%m-%d-%y"` WarehouseA |
Now all stores pointing to the production symlink for Warehouse A will get the new one when they next refresh their cache – which you can script too if you want to force them to update all at once. In addition, development stores can point to a development working copy of the extension for testing too.
In the same way each store would point to it’s version of a theme and package project, which is versioned in the same way. You can release a production version of these the same was as I describe above for extensions.
This worked pretty well and up until about a month ago it was where I was up to in terms of maintaining sets of common customizations across multiple stores. In fact the detail of how it works and the benefits in terms of genericty was to form part my presentation at the Magento Developers Paradise conference in April – which is now in October, thanks to the volcano in Iceland.
One of the things that pained me a little was the constant adding of symlinks everywhere throughout the store structure (For example in app/code/community, etc/modules, skin, app/design etc ). Thankfully this has lately been made easier by a script called modman.
Using Modman
Enter Colin’s brilliant modman script. It’s like he knew exactly a tool that would make my life easier, and made it available for everyone!
Using Modman for Magento
Modman automates the creation of my symlinks from within a nested SVN working copy hidden away in the root of a Magento store. It can do a lot of other cool stuff too, mind you, but for me the primary benefit is the notion of a store specific set of symlinks – the mapping of which is itself kept in version control and maintained automatically.
I was going to put a big modman tutorial here – but this post is already very long, and Colin has documented the script well. Unless I hear otherwise I’ll assume you can all follow his instructions clearly. What I will do though, is prepare a from scratch Magento project setup guide as my next blog post – some time mid-2011 at this rate!
In any case you should download and try it yourself – you can get a free source code repository from a number of places online.
The Benefits of Structuring your Magento Development project
There are a few benefits worth noting to this approach:
- Repeatability & Team work: All changes to Magento are made on the basis that they will be operating on a vanilla Magento install. Multiple developers can set up the environment easily, and will expect the same results, every time.
- Reusing code & minimizing effort and duplication – if you fix a bug in your custom extension, you only need to fix it in one place, then the changes propagate to all stores using it.
- Genericty becomes more valuable: You strive to create custom extensions that will have use for all your stores. You get by-products of customization you can sell (or give away).
In addition, the modman script has saved me a great deal of time, and given an added element of repeatability to the development and deployment of each store. I think using it will make the whole process of setting up a structured Magento development project a lot easier. As a result I’d like to see more stores adopting a similar approach to Magento development – so hopefully an end-to-end tutorial will help with that.
Outstanding issues
One thing I’m not convinced of yet is the use of externals in SVN to allow each store to ‘point’ to it’s required extensions. What I want to achieve is for the external to point to the equivalent of my ‘production version’ of the extension. The problem is you can’t have a tag in SVN called ‘production’ and keep re-tagging your HEAD revision to production each time you’re ready to release. Instead I have to move the current production release to a .date version and copy the new one in. Mildy annoying – not a big problem though.
This is shown in a contrived little example graphically:
I’d love to hear from anyone with a better solution to this part of the structure, maybe I could check in symlinks, but I’m not sure how that would work within the context of modman projects.
Conclusion
I’ve tried to provide some background for my Magento development process, how it’s evolved and why it’s arrived to where it is today. I covered how it works now and explained the benefits I get from it, and the still unresolved issues I found with it. It’s a summary of what I’m doing for Multi-Magento store development, where all customizations are via extensions and frontend packages. I’d really love to hear from other developers who are solving these problems, particularly those doing it better ways! So please, don’t be shy.
PS: One other thing, if you are using the script on Mac OSX – be careful something in the script doesn’t seem to like the broken symlinks detection. I had to make a couple of changes, which mean that old symlinks won’t get cleaned up on Mac OSX – not a big deal.
I just changed the find -L $root -type l -delete command (in two places) to:
#The echo out this command and run it manually if you have to. echo "find -L $root -type l -delete" |
Slightly confused reading this…
When you speak of multiple stores, do you mean multiple installations of Magento? In your diagram, Store A is a full separate/vanilla code base compared to Store B, on the same server?
With one installation, we have multiple websites/stores/shops already and one custom modules is available to all of them by default.
The fact one Mage install can do multiple websites which can do multiple stores each which can do multiple store views each has always, at least semantically, been a source of confusion. I think this otherwise great blog post could do with a little clearing up in this department so we’re all on the same wavelength.
As far as my structure goes, I’m also still trying different strategies. Currently getting a VMware based routine going but will (also) try this ModMan system. Thanks for the heads-up!
Thanks for the feedback Johan, re: stores in magento vs separate store installations – I’ll do a pass over and clarify, it is confusing.
Re: Using Virtual Machines – I know Kristof (FOOMAN) uses them too, I’ll see if I can’t lure him over here to offer his thoughts on it. So you’re using a fresh VM as a place to develop/test, or you deploy to a VM, for production?
Ashley, I’m glad you find modman useful and thanks for mentioning it on your blog.
On the production issue I don’t know if I fully understand the issue, but it seems like you could use the typical trunk/tags methodology here.. E.g.:
-Production
–Paypaltweak
-Development
–Paypaltweak
-Tags
–Paypaltweak
—07-08-10
svn copy Production/Paypaltweak to Development/Paypaltweak
svn copy Production/Paypaltweak to Tags/Paypaltweak/07-08-10
hack on Development/Paypaltweak until satisfied
reintegrate/merge Development/Paypaltweak to Production/Paypaltweak
modman Paypaltweak update
FYI, I also use VMs for development. My desktop is a Windows machine but I run VirtualBox with a Linux guest which runs the database server and a web server and hosts the code, then I hack on the code via my Windows-based IDE over Samba (nfs or ssh+fuse would work on Mac). For IDEs that do code completion I keep a copy of Magento core code on the Windows filesystem and include that code in my project (rather than include it from the VM which is too slow).
As mentioned by Ashley I am also using VMs (Virtualbox) but just for development. Initially my main motivation for using them was to easily rollback upgrades (files + db) by taking snapshots of the whole VM. This works quite well if you are only hosting 1 project on each VM.
I am using VMs for hosting all my local projects, 1 vanilla Magento install with sample for each extension. Currently my main focus has shifted to automation and repeatibility. Fabric works a treat to (python-) script stuff you do over and over again via SSH: http://docs.fabfile.org/0.9.1/ (and you can ssh into your vm)
I think in the end it will come down to two different ways of structuring projects. One for developing extensions (just holding the extension code) and one for developing/maintaining complete stores.
What has been working really well for me so far for extension development has been to keep only the structure of the extension in my project folder. My IDE (Netbeans) can automatically copy the files to a second location after each save. Essentially allowing me to keep one “clean” folder to package and version control and the second one inside a Magento install for immediate testing of results.
I will put up a post over at MageBase once I have found better words to describe it.
If I understood one of your motivations correctly you want to be able to deploy to all your stores at once since they all link to the same “production ready” code. One problem I see with this is that unfortunately some extensions require different code depending on which version of Magento you are running. This in turn will then require you to keep all your stores in sync version wise and you’ll also need to update all of them at the same time. Or alternatively version the symlinks
This is an excellent post and a good introduction to the benefits of symlinking with Magento. How soon can we expect a follow up?
Soon! stay tuned, I have about 3 blog posts in my drafts, exciting times ahead.
Any of you familiar with Hudson in combination with SVN? Apparently this allows even finer configuration, monitoring and scale better too.
http://wiki.hudson-ci.org/display/HUDSON/Subversion+Plugin
Have you tried just adding all the core folders as svn:externals? This is what I do for new SilverStripe projects and I’d love to do something similarly simple with Magento.
Hi, Barry thanks for stopping by. Question: svn:external pointing to a repo you control, or to the main Magento one? If it’s to a repo you control, then maybe that could work. The number of directories that you _should_ change is quite low though, so it might be less work to bring the customizations into a standard Magento install.
To the main Magento one. So I would add separate external entries for: app, downloader, errors, etc…
Basically, I only want to check-in code for my theme and customisations to my own repo. Everything else (the core folders) I would add as extensions. As you say, I shouldn’t be editing those anyway.
correction: …Everything else I would add as *externals…
This sounds quite similar to the way I was thinking, although i was thinking of using SVN externals rather than symlinks.
You do address that and are correct in that you can’t re-tag in Subversion – but this is a problem which Git overcomes. You can have a designated stable branch (typically called ‘master’), and you can tag within a branch if you so wish (so you might have stable-1.0 and stable 2.0 for different sites). Or jsut use the HEAD revision. Any unstable developments can be done in other branches – thus not polluting the stable branch.
You would then set up a Git repo for each project and use submodules to pull in the Magento core and the various extensions you need, referencing the correct branches/tags.
@Steve Lorek – I need to look at Git don’t I? All the cool kids seem to be using it! Thanks for stopping by and the tip, I think modman has support for other SCM’s so it might be a quite easy switch.
Great article! There is a lot in here I’ll be incorporating into my development methods. I’m currently struggling with an issue related to a multi-store development/engineering process I hope you might have some advice on. We have two unique e-commerce sites and are in the process of migrating the non-Magento site to the Magento site. The final result will be two unique domains, but one admin and code base.
However, I’m not sure how to proceed given that one database supports both a live site and a development site. As we develop the new Magento site offline, the live site’s DB information is being changed. Conversely, the information added on the development platform isn’t reflected in the live site. The problem is that two databases are constantly changing, and those changes need to be merged at launch of the second site.
I’ve considered developing the new Magento site on the live server under a development domain (in order to share a DB with the live site), as well as trying to merge both versions of the database when we’re ready to go live with the second Magento domain. However, both seem like poor and/or complex solutions. Based on your experience, could you offer any recommendations for this situation?
Hi, Andrew
Thanks for the feedback. That’s a tricky situation.
Normally with new store’s I’d set up CMS blocks/pages via a module sql resource but in your case there’d be a painfully large amount of configuration required.
I don’t have a really good solution to your problem, I cheated a bit when I added a wholesale substore to an existing live retail store. I actually configured it as a substore on the live site, but developed the template and code changes (installing Vinai’s Customer activation module for example) through modman as normal.
Maybe you could do a dump from core_config for the specific store, but there’s lots of other settings to worry about too (i.e tax).
If you come up with a solution that works well please let us know.
Ashley,
Thanks for the reply!
After a few days of research, I’ve also come to the conclusion that there isn’t an easy/elegant way to do this. What I think I’ll have to end up doing is developing the custom theme / code changes off the live server to integrate when stable as there aren’t too many database changes involved and those that are can easily once the 2nd site is live.
Once I have a second, stable storefront up I can populate it with data (products, categories, etc) on the live site under a temporary test domain name, which will ensure that the data for both sites is reflected in the one live DB.
Again, not perfect, but the least complicated solution I’ve seen…..
As much for my future reference as anything else
This is a good line to delete all symlinks in a directory, which works well if you need to start from scratch on your modman setup, assuming of course that you haven’t put any other symlinks in the magento installation
find . -maxdepth 1 -type l -exec rm -f {} \;
Thanks Dane!
How did I not find this blog post until now. This is fantastic. Looks like it’s been a few years since you originally wrote this. Do you still use more or less the same approach?
I started off symlinking individual extensions (without modman), and that got a bit complicated, then moved towards a single repository for the entire project. That’s been nice, and it’s always comforting knowing you can rollback the entire project in one `git checkout` if you need to. But I’m beginning to work on an extension that I need to deploy to multiple Magento instances, so I think it’s time to go back to modman’ing it up.
Thanks!