Big DRD Update available: Automatic site updates and a ton of other improvements

Today's new release of Drupal Remote Dashboard (DRD) introduces a big new feature which allows anyone to manage Drupal site updates in a protected and automated framework without having to get any third party involved in the process. Just in time for Independence Day, July 4th 2017. For Drupal site owners this brings independence from external service providers without compromising quality of service or reaction time.

Over the last couple of months we've spend a huge amount of time to not only work on the advent of this significant new feature, which we had in the back of our mind from the very beginning. We've also introduced a number of other new features, improvements, cleanup and bug fixes. This blog post gives you an overview of what you're getting when you download the fresh release from drupal.org and start testing it on a simple Drupal 8 site of your choice.

Before we go into more detail, let me emphasize the fact that the dashboard is Drupal 8 only - however, the remote sites you can manage, monitor and now automatically update are supported for all 3 significant Drupal versions starting from 6 to 7 and of course 8.

What's coming with DRD 3.4

Overview

• New features
  1. Framework to update a Drupal core fitting into any existing workflow
  2. Link to start a new remote session in the browser
  3. Write a log per domain for all actions being performed
  4. Download files or database dumps from remote sites
  5. New actions to modify domain properties in the dashboard
• Improvements
  1. No core patching required anymore
  2. Projects list is much faster than before
  3. Reduce dependencies
  4. Better routing schemes
  5. Better SSH support
  6. Code cleanup
  7. Better console output for CLIs (Drush and DC)
• Bug Fixes

In detail

Update framework

Well, what's the point of monitoring your Drupal sites to find out about available updates, security relevant or not, and then not being able to easily apply those new versions of core, modules and themes to your remote sites. Starting today, DRD offers you the framework to go full-circle and close that loop. It comes with a flexible framework to run site updates with a plugin based and therefore extensible system that fits into any existing workflow and deployment method - or helps you to build one if none was in place yet.

The way it works - and yes this is far too complex to explain all details in an overview post - is that we broke it down into 6 phases which are highly configurable, customisable and extensible. The phases are:

1. Storage
   We always need to know where to find the project that should be updated. This is not about the live site, of course DRD knows exactly where that belongs to. Here we're interested in the project that exists somewhere for development purposes and if there wasn't one, a plugin to grab a copy from the production server is also provided as one of the options. Other choices are Git and a static local copy somewhere in the file system. Other plugins to support more VCSs will follow as the requirement will be demonstrated.
2. Build
   How should the updates being applied to the working copy that the previous phase helped us to make available. DRD currently supports composer, drush make or directly copying updates into the Drupal working directory.
3. Process
   What should be done when new code got applied to your Drupal code base? That depends on your Drupal sites and deployment methods. You may have to run database updates, flush cashes, export new configuration, sync your features or what ever else, you name it. As a first plugin DRD supports the LakeDrops workflow for Drupal 8 and you should consider that as a blue print which can help you to write your own process plugin.
4. Test
   You want to test the update before deploying that to your production system? What a great idea. This phase is there for that - but to be honest, we haven't completed any of the plugins yet. We do hope to get some feedback from the commmunity such that we get a better understanding on how this is best utilised for the DRD user base.
5. Deployment
   This almost final phase is about the question on how to get your updated working copy to your production server. Well, most deployment environments today are probably Git based and therefore already have deployment covered - in those circumstances DRD won't do anything and you will just continue to utilize your existing deployment environment. But DRD provides the updated and tested input such that you can safely trigger the deployment what ever tool that may ever be based upon. For those who don't have such tools in place (yet) or not for every client or site, this phase offers plugins that can do the job. For now, we've included an rsync plugin which uploads the changes to the production server. But of course we're interested to learn about additional methods that are in use and want to turn them into additional plugins as well.
6. Finishing
   The final phase is to clean up behind us such that nothing is left behind or potentially disclosed to non authorised eyes.

Everything that's happening in the above 6 phases is being logged in detail and the DRD UI provides you with a new tab to review everything that was going on. Of course, you don't want to run this new kid on your production site? We wouldn't do that either. That's why there is a dry mode which does everything temporarily in a working copy and skips the upstream storage and the deployment. As it doesn't clean up the working copy finally, it leaves you with an updated version of the Drupal site for your review. You can easily look at all the changes and how it's being done. If you like the result, just run the whole thing again, this time without the --dry-run option.

Oh, you also have those odd sites where you need that really special script that needs to run at some point in the process? And you believe that's why you can't possibly automate your update process? Well, here is another one for you: DRD's update framework allows you to write any number of additional scripts and hook them into the process documented above at any given point. Supported languages for those scripts are PHP, Bash, Python, Drush and Drupal Console.

Well, the sky is the limit. OK, we've got way to go, but the foundation is solid and dynamic at the same time. With help from the community, either in terms of ideas or contributions (code, documentation, training, etc.) we strongly believe that DRD Update Framework is going to become a powerful and popular tool in Drupalverse.

Open remote site as authenticated user

Opening the remote site with a single click from the dashboard has always been possible. But more often than not we've found ourselves in a situation where we had to authenticate ourselves remotely as well which turned out to be unnecessary, especially as we already came from a place where we are well known and authorised to manage that site with full access. It turned out, that opening the remote site and already being logged in as user 1 is not only convenient but also trustworthy at the same time. You now find a new icon next to the domain or site name in the domain list which pulls a one time login link from the remote site and redirects you to that which will go through the full authentication process on the remote site and gives you admin access.

Activity logging

A lot is going on in a busy dashboard and we wanted to make sure that we keep logs of everything that's going on. Instead of implementing our own thing we're utilising the watchdog API from Drupal core. For every action that's being executed on any of your domains, DRD is now logging everything around that activity including the response from the remote sites. If you've also enabled the dblog module on your main DRD site, you will then be able to access the activity log from the DRD UI directly when viewing any of your domain detail pages. The new tab shows you the details from the dblog system but filtered for the current domain you're looking at.

Database download from remote sites

Well, we needed that feature for the update workflow described above. But as it was there already, we've made it available as a stand alone action as well. When calling the drd-database action with one of the available CLIs, the remote site will create a dump of its own database, encrypt the file and make that available for download to the calling dashboard which will immediately pull and decrypt the databse to a local file for further usage. Not sure, what use cases you're going to come up with, but we've a couple in mind that may be handy.

Modify properties of existing domains

By design, DRD allows you to create Drupal core entities for the dashboard, but the domain entities and their details get collected from remote sites. There is no point in creating domain entities manually in the dashboard when we're interested in monitoring and managing real domains in a remote Drupal core instance. However, in practice things are sometimes changing and some of those changes may have sent us into a dead end. For example, if a Drupal core only hosts one single domain and for what ever reason, the URL of that domain changed, DRD had no means to grab those changes from the remote site. It simply wasn't accessible anymore until we've updated the URL in the dashboard. For situations like that, this new feature became relevant and we've implemented that for both of the supported CLIs.

Many improvements and bug fixes

For all the details on general improvements and bug fixes, you may want to consult the changelog available in the DRD code base. Some significant pieces are for sure the reduction of the dependencies which also allowed us to get rid of the core patches that were only required because of some of those dependencies. We've achieved that by re-writing the project list from scratch and now that is even more powerful - and much faster - than in any previous version of DRD.

Other improvements are a more logical routing system which will always provide you with a meaningful breadcrumb no matter where and how deep you navigate in DRD and its entities and logs. Speaking of logs, we've also improved the output of the activities when using DRD from your command line by either using Drush or Drupal Console.

Last but not least, we're now supporting SSH connections to remote sites even without the PHP-SSH2 module being available. This increases the feature set for everybody, even for those who are on shared hosting or other limited plans. A big round of code cleanup and bug fixing makes 3.4 the biggest update of DRD to date and we're hoping to get a lot of feedback so that we can further improve this platform and make it the Swiss-Army-Knife for everyone in Drupal.