We aim to encourage a culture of experimentation at MxM but also a culture of responsible decision making. Everyone needs to understand that technology decisions have the potential to impact the whole team and the ongoing budget for the support of the application in question.
With that in mind we try to optimise for tooling that is flexible, understandable, open source and/or managed as a service to smooth the process of handover between developers on a project and to clients/third parties who may take on active development in the future.
2016 was a big year for the technology team at Made by Many. We evaluated a lot of new technology and made some changes/additions to our default stack. Here’s a look back at some of the changes we’ve made in the last 12-18 months or so.
Docker has matured a huge amount in the last 12 months and so has the tooling. With the release of DLite and later Docker for Mac, plus better solutions for CI (more on this later) and AWS; using docker in development and production became much more viable.
Previously, we built debian packages as a unit of deployment and used Vagrant on our development machines. Docker provides a solution that has all the things we liked about debian packages, has declarative dependencies for development and production and allowed us to move away from our bespoke tooling built for debian packages to open source tools built for docker.
We started using Terraform in late 2015 but over the last 12-18 months it's become a part of our standard toolset. Terraform allows developers to define infrastructure as code which can then be edited, reviewed, and versioned in the same repository as the application codebase.
What we like about Terraform:
- It’s cloud agnostic. Should we decide to move from AWS to Google Cloud Platform or elsewhere in the future, we’re not bound by our toolset.
- HashiCorp Configuration Language (HCL), the syntax of Terraform configurations. It’s lightweight and rapidly grokable by humans which compares more favourably with the complex DSL’s offered by the likes of CloudFormation.
- It has distinct plan and apply stages that allow changes to be previewed before they’re executed.
- Terraform manages infrastructure state by building a dependency graph of dependencies and their relationship. This means that only the “diff” of changes is applied to your production environment.
In June last year Segment released Segment Stack, their boilerplate configuration for AWS with Terraform. It’s an excellent starting point for most infrastructure projects and is easily customisable. Stack comes with default configurations for networking, auto-scaling, ECS, CloudWatch logging/metrics and a bastion host for SSH access.
What we like about Stack is that it’s a sensible set of defaults and that it’s trivial to take the pieces you want and leave the rest.
Biscuit is a multi-region HA key-value store for AWS infrastructure and application secrets. It allows us to share project specific passwords and secrets amongst the team without having to commit unencrypted keys directly to GitHub. Unlike Vault from HashiCorp, it doesn’t require a server and works with our existing IAM profiles.
What we like about Biscuit:
- Encryption keys are stored in KMS and can be replicated across multiple regions/availability zones for high availability.
- Biscuit works with IAM and grants access to our secrets and specific KMS operations with IAM roles and users. This means we’re managing profiles and access in one place (IAM) and that we can also enforce multi factor authentication.
- Biscuit helps us to manage different keys for multiple different environments with a label to easily switch between development, staging and production
Since we made the move to Docker we took the opportunity to reevaluate our approach to CI as part of our build pipeline. In the past we’ve rolled our own Jenkins which has been knarly to say the least. First we looked at Concourse, which had first class docker support and was configurable with a simple YML file (a breath of fresh air after the manual plugin mayhem of Jenkins). But Concourse is still a solution that we would have to manage ourselves. A SaaS CI provider would be much easier to hand over to clients in the long run.
Early in 2016, Codeship launched Jet (now known as Codeship Pro) their CI platform with native docker support.
What we like about Codeship Pro:
- Fully customisable workflows configured with simple YML files that can be committed to source control (similar to Concourse).
- Jet (which is now the name of the Codeship Pro CLI tool) can be used to locally debug and test builds
- Continuous delivery integration for our docker images with EC2 Container Service
One of the biggest additions we’ve made to our default tech stack in the last 10 years is Elixir, and doing so was not a decision we took lightly. In the past we’ve relied heavily on Ruby on Rails, having been well suited to rapid web application development and versatile enough to be suitable for a wide range of different solutions.
Ruby and Elixir are both similar in syntax and are both readable and understandable at first glance. This, amongst other similarities, makes elixir equally well suited to rapid application development but additional language features make it even more versatile than Ruby.
We started experimenting with Elixir in the later half of 2015 and as of this writing, have one Elixir application in production, another in pilot and a third in development. Embracing Elixir has allowed us to think differently about how we would build applications for high availability and approach complex problems like dealing with unpredictable traffic and an unreliable network in a large distributed system.
What we like about Elixir:
- Elixir is fast and much more performant than Ruby. It has the concept of processes which are trivial to manage and provide a concurrency model without the need to rely on other technologies like Redis which would be our go-to with Ruby but which come with additional ops cost.
- The performance gains over Ruby make Elixir much easier to scale. Caching and horizontal scaling can be considered much later than with a Rails app.
- Elixir applications are designed around fault tolerance with no shared state which helps applications to remain highly available in the face of errors.
- Phoenix is an Elixir framework that would feel familiar to anyone used to working with Rails.
- The Elixir community and the resulting ecosystem of tools, techniques and libraries is expanding rapidly.
React native has been another key addition to our stack in 2016. We’ve been building native IOS and Android applications for many years and have also experimented with hybrid apps and technologies such as Xamarin.
We were attracted to react Native as it’s immediately familiar to our team with years of experience in React. Through initial experiments we found that performance was comparable with fully native code whilst allowing for much more rapid cross platform development with smaller teams - perfect for rapid prototyping and shipping to market faster.
There is a tradeoff of course, React Native is no where near as versatile as native IOS or Android when it comes to certain animations and behaviours. Our Hackaball app is a good example of where our design goals and desired interactions meant that react Native was not the best fit.
However, for many applications React Native works well and allows us to get from prototype to pilot to production much quicker and with a smaller team than we would with native code. To date, we have released two React Native applications into production, have one in pilot and another under development.
What we like about Flow, particularly in comparison with something like Typescript is that it’s a simple type checker, not a compiler and so it fits in better with the rest of our existing frontend toolset (ES6, Babel, Webpack, React). Also, it’s possible to add Flow incrementally to a codebase, rather than requiring a big refactor which was a big plus for us.
This year we’ve made a concerted effort to make our hiring process even better. From the language of our job ads to questions we ask at interview and our overarching assessment criteria. Most importantly we favour the characteristics that identify great software developers rather than specific experience in particular languages or techniques.
With this in mind we made three new hires from late 2015 to early 2017. Each of them has a diverse career history,bring a unique perspective and distinct skills to the team. Each of them was new to the technologies in our preferred stack but they have already learnt an incredible amount in the short time that they’ve been at Made by Many.
We’ve been working hard over the past year (and in fact earlier) to build up an even better culture of responsibility within our development team. That means that developers need to understand the full lifecycle of a product, be accountable for quality in their work, champion the user experience and be able to run the application in production as well as on their development machines.
For us DevOps and QA are much more than individual roles. They are facets of a culture that everyone on the development team should aspire to.
A big part of 2017 will be to further educate and foster understanding of the tools mentioned above and ensure that everyone on the team is capable of thinking about the tradeoffs involved in choosing technology and running applications in production