Skip to main content

Automation pipelines for everything

When most developers or even DevOps engineers think of automation pipelines, the first thing that comes to mind is a traditional CI/CD pipeline, one that gets triggered by a code push, which builds a piece of software, and then deploys it in a test or production environment. It's perhaps the quintessential idea of the pipeline, and as a result most DevOps tools are built around that workflow. There's nothing wrong with that, and in fact that process is likely to remain the most popular type of pipeline to be used in any organization. But pipelines, and automation in general, is much more powerful than just a way to speed up software development. Someone once said that any task which needs to be done more than twice should be automated. In the analog world, there are ways to automate tedious processes, but these ways tend to be costly. You need motors, cameras, robots and so on. Thankfully, in the digital world you can automate almost any repetitive process. In this post I'll go over a few examples that I've personally automated in my work life.

Terraform deployments

When you first start working with infrastructure as code, you quickly learn that environments should be deployed using code templates. You write up a template which describes the environment and then run a command to deploy that environment. Then when you come in and edit that document, the next time the environment is deployed, only the changes will be deployed. Whether you're using Terraform, Cloud Formation or any similar IaC tool, the workflow is the same. And as you may have noticed, the process of deploying resources this way is very similar to the CI/CD process for software development. So it makes complete sense to build deployment pipelines for your infrastructure as well. Also, pipelines don't have to only have build, test and deploy steps. In the case of infrastructure especially, there are a lot of advantages in using some of the more advanced pipeline tools like approvals to confirm that the changes have been approved by the right stakeholders, some logic to ensure that naming conventions are respected, and even things like blue/green deployments and auto scaling.

Scheduled tasks

While most pipelines get triggered by a code commit, you can trigger them in many different ways. You could run a pipeline manually, or you could schedule them. I have dozens of automated tasks that run on a schedule as part of a pipeline. Some pipelines run every day, like the one that checks whether my cloud costs are within an expected range, one that runs every week like the one that connects with my accounting software to check whether the needed payments have been processed, or once a month like the one that produces all the relevant invoices. All the DevOps tools have an option to run pipelines on a schedule, and it's worth thinking whether those reminders that you need to take care of constantly shouldn't instead be automated pipelines.

Software updates

While your CI/CD pipelines may deploy updates to software your organization produces, what about things that come from outside? I use several pipelines in relation to code that I don't personally produce. One pipeline makes sure that all of our servers have been updated correctly. Another makes sure that our SSL certificates are up to date. I also have a pipeline that validates my backups are being processed correctly. The entire infrastructure of my company, along with my own devices and online accounts, are being taken care of by some type of automation so that I only need to get involved if something goes wrong.

Automation is useful for a myriad of tasks, especially for a digital native, someone who can code and is comfortable with tech. DevOps tools like Jenkins, Azure DevOps or AWS Code Pipeline are all solutions that can be adapted to far more than just software development. Next time you find yourself repeating a task, ask yourself whether it makes sense to automate it using a pipeline.