Composer based Drupal projects can be fun

As a team that had spend time and money for years in developing best practices for staging, deployment and operations of many Drupal sites on version 5 to 7, the all new Drupal 8 came with some surprises and challenges in 2015, when first alpha and then beta releases started to see the light. Where all of Drupal core  up to Drupal 7 had been exposed in the main project folder, all the contributed code from the Drupal community and third parties was located deep down in the code tree with custom code for the specific project sometimes was hard to separate from other contributions. Consistency was just a dream, e.g. contributed installation profiles had to be in the same folder as installation profiles from core.

Drupal service providers got used to that. We even managed the odd scenario where someone had placed an add-on module in a profile's subdirectory, in core's module directory or even in a modules directory of one of the sites hosted by that Drupal installation. And of course, the aforementioned best practices were sophisticated enough to take care of all those Drupal flavored oddities.

Then this Symfony based Drupal 8 beast came around the corner and let alone the directory and file structure had completely changed - much to the better, just to make that crystal clear right upfront. Here are the highlights:

• Drupal core went into a core subdirectory and as long as you're not hunting bugs or even maintaining Drupal core, you no longer have any reason to ever again look into that folder. This is core, it is great, it does its job, it's there for you. And you can leave it as is. All your development, additions and what more goes into different places. Nothing to do with Drupal core anymore. Fantastic.
• A new directory called vendor arrived with a huge number of third party packages or libraries from the PHP community that not only serves Drupal but also all sorts of other PHP based platforms and applications. Again, this is a place where a lot of great code is living, but nothing that most Drupal agency have to worry about while building web applications based on Drupal.
• Right at the center of a Drupal project, there are 4 directories which are for us. Here is where the additions are installed and development is happening. Right in the root of the web project, not deep down in the tree:
  • /libraries
  • /modules
  • /profiles
  • /themes

Just looking at that structure at a first glance, this was so exciting to me. It's been so obvious and I couldn't agree more: this was a great step in the right direction. NB: Yes, I know that Drupal 8's re-architecture has been much more than this and I'm far from trying to down-play that achievement, it's just that this blog post focuses on the resulting structure and what that meant for our workflows.

But wait, how do I update Drupal core then?

What looked so great right at the beginning, caused major headaches a couple of weeks later, when the next alpha release was made available and suddenly highlighted the fact that the previously successful update and deployment strategies wouldn't work anymore. Calling drush pm-updatecode didn't work and threw error messages all over the place. Running Composer wasn't something I had done too often before and I didn't even know how to first update the composer.json file to get started. At least that approach didn't seem to make sense. Download the new Drupal core tar file and copying that over the existing code tree? Somehow a terrifying approach that I never felt was anywhere near to my keyboard and terminal.

Reaching out to the community at that point was a challenge. Everyone has been super busy. Digging deeper it became more and more obvious that this question had not been addressed yet: how does the life-cycle of a Drupal 8 site look like.

The best advise at the time was this: download the latest release of Drupal core, extract that in a temporary directory. Then delete the /core and /vendor directories in your web root and move those 2 directories from the just extracted archive into your project. After that run a series of different commands and with a bit of luck, you should be ready to go with an updated core. What?

Right, there is a bit of research, trial and error, discussions and some projects with real life experience required to get to an all new workflow for maintaining Drupal sites from a staging and deployment perspective. And many Drupal specialists have worked on this for the last 12+ months with the conclusion that a Composer based workflow is not just the best we can get, it really makes a lot of sense - and it's great fun!

Composer is a great tool

To my surprise there is currently a lot of negative stories out there in the wild about Composer, especially in the Drupal community. This is mainly driven by the fact that we as a Drupal community currently have to learn quite a lot of new things, all at once. We've all heard about or even got involved in this huge effort of Drupal 8 development that took several years. And some great people even left the community because they disagreed with the approach and felt that the object-oriented architecture would be too big of a hurdle for the many self-educated module developers.

From my own experience I can only tell, the adventure is amazing. You only need to let go and be prepared to learn new stuff. Once in there, developing services, plugins and controllers is so simple and you can get great support from IDEs and other tools.

In this context, all of the sudden we also get hit by Composer. It needs our attentions and steels our productivity. I believe that this is the reason why we currently read so much frustration about it in our community. This will change to the better. Quite soon, as far as I'm concerned.

Reading a lot on how others are doing things, I came across a nice project on GitHub that started a Composer template for Drupal projects. It worked well straight away and what became obvious very quickly was the fact that the directory structure of a Drupal project would get changed again - once more to the better.

Taking that template as a blue print and trying to fit that into our own workflow which is influenced by some GitLab and Ansible components, we started to tailor that towards a solution that meanwhile got into production in half a dozen projects and works like a charm. You can find the project at Drupal 8 Project Template on our GitLab repository.

Here is how such a structure looks like:

- Project Root
  - /config
  - /console
  - /drush
  - /files
  - /settings
  - /vendor
  - /web
    - /core
    - /libraries
    - /modules
      - /contrib
      - /custom
    - /profiles
      - /contrib
      - /custom
    - /sites
    - /themes
      - /contrib
      - /custom
    - index.php
  - .gitignore
  - composer.json
  - composer.lock

There are only a couple of changes from the structure of Drupal 8 described above:

• There is a project root with the Drupal root being a subdirectory of that which is named /web
• The /vendor directory is now in the project root and no longer in the Drupal root
• The 3 directories /config, /files and /settings are on the project level as well and that way clearly separated (more to that in a minute)
• The CLI tools (Drush and DrupalConsole) have their own locations too

All the contributed modules, themes, profiles and libraries got into their /contrib directories in the Drupal root and our custom code for the project also has their own subdirectories in there.

What goes into version control

From the structure above, what's specific to a customer project is pretty clear:

• composer.json and composer.lock
• config and settings
• all of the custom directories

Everything else is reproducable and in no way unique to any specific client project. For that reason, we've decided to only commit those components into the customer project repository. But we do not want to conceal that there are discussions around that topic going on in the community. Just mentioning that so that you are aware that the described approach is we've selected, it's certainly not the only possible one.

Reproducable? Yes, from just the 3 components listed above the whole Drupal site can be setup, ready to go either for production or for another developer joining the team. Almost. What's missing in the repository and still isn't reproducable is the files directory. This is where images and documents are stored and those are managed differently in our workflow. As there is a lot of binary data most of the time, and pretty big files as well, there are various options on how to handle those file. This is not within the scope of this blog post and we'll cover that elsewhere. BTW, the same applies to the database.

How to use all this: the workflow

All of what's in the repository is what you need to setup the Drupal site. Very lightweight and nice to handle. But before describing how to setup the site on a different host, let's have a look how to get started. It requires only one single command:

composer create-project lakedrops/d8-project [DIRNAME] --stability dev --no-interaction 

Just replace [DIRNAME] with the destination, where to create the new project and a few minutes later, you've got a fully functional Drupal environment and development of the customer project can begin right away. Even the git repository got created locally and you can push that to any remote repository of your choice. Source code of the project templates is Open Source and you can find it, clone it or contribute to it over at our GitLab repository.

When ever you want to setup that project on another host, either for another developer, for testing or even for production, simply checkout the project from your Git repository and call the following command:

# Production environment
composer install --no-dev --optimize-autoloader --prefer-dist

# Development environment
composer install --optimize-autoloader --prefer-dist

This is installing all components (Drupal core, modules, themes and all vendor packages) in one go and uses exactly the same version for each of those components that you've committed to in you development environment. This is possible because composer stores all details about installed components in its composer.lock file so that installing from that repository will give you exactly the same result as last time.

To update existing components or to add new ones, we edit compose.json in our development environment and then call this:

composer update 

That updates all the components and their dependencies automatically and updates the details in composer.lock again. Committing to Git and re-deploying on your other hosts like described above is all you need to do. OK, to be fair there are additional tasks like running Drupal's update routines, flushing cache and other maintenance work and we strongly advise to also automate all those tasks. But those are required in any case, not only within a Composer based workflow environment and therefore won't be covered by this blog post.

This is pretty much everything that's required. The full workflow for a Drupal agency also needs to define the flow for settings, configurations, files, databases, stages and other areas. Those are interesting topics too, but the Composer part of the workflow is based on the building blocks described above. We think, this is enjoyable, hope you have fun with it too.