773-525-8255 Contact Us

Always Developing on Production

Always Developing on Production

Using Chef and Vagrant, we (Promet Source) were able to standardize the platform that developers use to make Drupal sites consistent with the target production platform that the Sysadmins manage.  In this sense, *our developers are always testing on production*

There is a great amount of interest in things that make managing platforms easy and consistent.  If we want to try and practice “DevOps”, we all need to be involved in what other teams are doing.  Using Chef and Vagrant, we (Promet Source) were able to standardize the platform that developers use to make Drupal sites consistent with the target production platform that the Sysadmins manage.  In this sense, *our developers are always testing on production* by creating a local virtual machine using the same platform configuration as the end product.  


The operations part of the process relies heavily on Chef.  Chef is a configuration management tool that will control nearly every aspect of your platform configuration by code.  This is how all of our servers are created and automatically configured.  Doing so will allow you to keep all of your platforms consistent and easily manageable regardless of how many servers there are.  Any changes in configuration (e.g. increase php memory_limit) are changed via Chef and impact either one server, servers with a specific characteristic, or all servers.  Any tweak to Apache or PHP will be automatically updated across all machines to which those changes apply.  Any parameter that is changed manually will be changed back and prevent servers from continually "drifting" away from how they were initially created.  Not using configuration by code when managing multiple systems is just not tenable.


Things you've done that should be in configuration management:

  • Needing to install a package manually.
  • Manually adding the user accounts for your team on a server.
  • Changing any relatively common configuration parameter to make something work right.


The development part of the equation is accomplished in a similar fashion.  Developers use Vagrant to manage their virtual machines with Chef identically to the way the operations team uses Chef.  This implements a standard platform for your development team to use.  Much in the same way the Sysadmins have standardized the server platforms, the developers have standardized their local dev platform and process since every developer is building Drupal sites the exact same way.  As an added bonus, all of these tools are free.


The development process now:


  1. Squashes fetishizing or unique developer local platforms.  The production server is not a macbook running MAMP.
  2. Enables developers to join the project quickly.  The git repo ships with the pieces you need to run the entire LAMP stack.  Clone the repo, start vagrant.
  3. Holds the teams accountable.  If the site didn't build from the ground up correctly, the change wasn't properly implemented or the change comes with a requirement from another team.




Ops Path


The Operations team has created a server platform suitable for running a Drupal site.  In Chef, these assets are managed from the command line using the "knife" command.  Here's an example of how I create a server on Rackspace thanks to a plugin that allows me to communicate to my account using an API Key.


knife rackspace server create -r 'role[lamp]' -E dev -I f9a2ad96-355a-4696-a8f9-58d210050e59 -f 4 -N test.domain.com -S some.domain.com


What is all that?  I'm telling Chef to go to my Rackspace account and create a server with the role "lamp".  The server will be tagged as a "dev" environment within Chef and use a specific server image available from Rackspace.  A role in Chef is a profile or run list for groups of nodes within your network.  The GitHub page for the knife-rackspace plugin can provide more details -- https://github.com/opscode/knife-rackspace.  In the example, the lamp role contains:


name "lamp"

description "LAMP Stack base role for servers"









This particular role has five tasks.  It configures settings common to all of our servers (base), Apache, PHP, MySQL and the Nagios NRPE settings.  At this point, the nagios servers finds the new node and automatically starts monitoring it.  All that being said, let's focus on the "apache2" role.  Here's what's inside the apache2 role:


name "apache2"

description "Apache2 role"












 :apache => {

   :ext_status => "true",

   :listen_ports => ["80", "443"],

   :traceenable => "Off",

   :serversignature => "Off",

   :default_site_enabled => "false",

   :servertokens => "Prod",

   :timeout => 15,

   :prefork => {

     :startservers => 10,

     :minspareservers => 10,

     :maxspareservers => 24,

     :maxclients => 100,

     :maxrequestsperchild => 10000





Very simple default install.  All servers with this role will have modules installed and enabled automatically.  Common security settings will be automatically instituted.  I've configured dozens and dozens of web servers like this, yet haven't typed "apt-get install apache2" in a year.  Similar scenario with PHP and MySQL.  Each parameter in php.ini and my.cnf is going to reference an attribute the Chef server maintains for all PHP and MySQL installs.  When I change the role in the Chef server, all servers with those roles update in due time automatically.


This provides consistency between all managed servers for ourselves and our clients.  Configuration updates will automatically be processed for projects that are in somewhat of a dormant phase but remain monitored, updated with new users, changes in best practices, etc.


Dev Path


The Dev team implements the similar settings through their Vagrantfile by way of Chef-Solo.  Using Chef-solo, a localized implementation of Chef configurables will be implemented run-time when the vagrant box is started.  Let's say the developer doesn't care about the Apache settings in great detail, but they do want to manipulate PHP.  Inside their Vagrantfile they tell Chef to:


 config.vm.provision :chef_solo do |chef|

   chef.json = {

       :apache => {

         :default_site_enabled => "true"


       :php => {

         :memory_limit => "196M",

         :upload_max_filesize => "30M",

         :post_max_size => "10M",

         :max_execution_time => "300",

         :max_input_time => "300",

         :realpath_cache_size => "64k",

         :conf_dir => "/etc/php5/apache2",




These are all simple settings but are potentially necessary for standing up a Drupal site.  Plus, the Vagrantfile is in the repo, so everybody who downloads the project has the same settings!  This makes the creation of the Drupal site *and* platform repeatable for all developers.  No shared development servers need to be created to do initial development so no servers need to be managed (and paid for) in the early stages of a project.  The primary motivating factor for this came from the book The Phoenix Project where the team in the novel begins to package the environment with the product for developers. For example, does your organization primarily use Ubuntu 12 but the client wishes to eventually deploy this to a Debian 7 machine? The developers simply need to tell Vagrant to use a Debian 7 Vagrant box:


config.vm.box = "opscode_debian-7.1.0_provisionerless.box"

config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_debian-7.1.0_p..."


If you don't already have this box, the URL for where to get it is provided.  Now it doesn't matter if your developers have Macs, Windows or Ubuntu machines.  They are all developing on what the production machine will be and how it will be configured before adding one line of code to the new Drupal site.


When the Vagrant box starts, the contents of the repository and dependencies will be automatically installed on the virtual machine using a custom cookbook handled by Berkshelf (http://berkshelf.com/).  All Drupal configuration is managed through the Drupal module "features" and up and running in minutes.  This goes back to what I said about accountability.  If the site isn't configured correctly on the initial build, then the work to build the site wasn't done correctly only because the process not repeatable not because we are bad at making sites.  


Jumping In


There is much more to developing a workflow and building an application using Vagrant and a few Chef cookbooks beyond the few code snippets provided.  This isn't intended to be a copy and paste exercise, just a brief intro of a few methods used to put together the entire chain and how they have served us well.  To try the same, familiarize yourself with Vagrant (http://www.vagrantup.com/), begin reading up on Chef (http://gettingstartedwithchef.com/) and tailor your own application cookbook to work with those projects.  Here is a link to ours: https://github.com/promet/drupal_cookbook.




The final domain for your project may not point to your dev server, but it is going to be treated like production by somebody.  Your PMs and clients are going to check work completed on either some sort of dev or staging environment and they expect it to work the same when it's on production.  Establish "production" on staging and on your actual development platform to reduce the opportunities for error throughout the development cycle.  This never leaves to question aspects of the project like operating system, PHP version, Apache and MySQL parameters when developing and testing.  Those details should be established as part of the specs and reqs process before development even begins.  Additionally they should be packaged with the product, not on some wiki page somewhere.  Once they are, dropping in a few of the blocks mentioned here makes "always developing on production" possible.