Scaling like a cloud provider: How we reliably automate infrastructure

More than 500 VMs, multiple data centres, customer environments, mobile routers in public transport and an engineering team of around 12 people. This does not work if every configuration and every change has to be done manually.

At onway, we therefore rely heavily on Ansible. Ansible is an open-source tool that describes configurations and workflows in readable YAML files and executes them automatically on any number of systems. Not because automation is exciting in itself (it is!), but because it solves a concrete operational problem: it manages many systems reliably, traceably and consistently.

The first article in this series focused on NetBox as the source of truth. This is where we maintain VMs, IP addresses, VLANs, VRFs and other infrastructure information. NetBox describes what exists and what the environment should look like. But this does not configure anything yet. Ansible takes over this part: it reads the data from NetBox and applies the desired state to the systems.

Manually adjusting a single VM is not a problem. With several hundred VMs - distributed across two data centres, with routers, switches, firewalls and ESXi hosts - manual work quickly becomes error-prone, especially when changes also need to be documented properly.

In operations, the same questions come up again and again:

• Which systems are affected by a change?
• Do all VMs have the same baseline configuration?
• Has a security update been rolled out everywhere?
• Are there systems that deviate from the desired configuration?
• Can our OPS team safely execute recurring tasks themselves?

If these questions can only be answered manually, operations become slow. It becomes even more difficult when documentation and the effective configuration start to diverge. This is exactly why we combine NetBox and Ansible.

A new VM is first recorded in NetBox: name, IP address, role, environment and further details. Ansible does not work with a static host list, but directly against this source of truth. If the data in NetBox is correct, Ansible can work with it.

Our model is deliberately simple:

• NetBox describes the desired state.
• Ansible applies this desired state to the systems.
• Rundeck makes the automation easy to execute for Engineering and OPS.
• E-mail and Logstash ensure that changes remain traceable.

A typical use case is the provisioning of new VMs: setting the network configuration, creating users, connecting monitoring and configuring services. With Ansible, these steps are executed in a standardised way instead of being worked out manually every time.

This is especially important for new customer environments, which can quickly consist of 20 or more components, such as routers, switches, firewalls, ESXi hosts and multiple VMs. We can put such environments into operation within a few hours, with fewer manual errors and a reproducible setup.

Even after provisioning, a VM has to remain correctly configured over time. For this, we use Ansible roles. Based on the VM function maintained in NetBox, a role describes which configuration a system needs.

An excerpt from our almost 50 roles:

• `base` for general baseline settings
• `monitoring` for integration into monitoring
• `snmpd` for SNMP configuration
• `sssd_login` for central login integration
• `sshd` for SSH configuration
• `mpp` for our mpp appliance

A baseline configuration should not be implemented manually 500 times. It should be defined cleanly once and then reliably applied to all relevant systems. This makes operations more predictable and reduces differences that cost time during troubleshooting.

At onway, Ansible runs via the Rundeck web interface. Predefined jobs can be started, scheduled and logged there without having to work directly on the command line. This is important because automation must not only work for the people who built it. Our OPS team can run defined jobs via Rundeck without knowing every technical detail. This reduces dependencies on individual engineers and makes recurring tasks more accessible.

If an automation only works on one person's laptop, it is of limited use in operations.

When a critical vulnerability is published, you need to know quickly which systems are affected. Updates then have to be rolled out in a targeted way and the changes need to be traceable. With more than 500 VMs, this is not something you want to collect manually.

One example from our operations was a critical nginx vulnerability. Within an hour, we were able to identify which systems were affected and roll out the corresponding updates. Without a clean data basis and automation, this would have been significantly more time-consuming. First, you would have to check where nginx is installed, which systems are productive and which customer environments could be affected. The risk of overlooking individual systems would be high.

With NetBox and Ansible, we can work in a more structured way:

• identify affected systems
• roll out updates in a targeted way
• trace changes
• check the state of the systems

Security is therefore not only a question of the right patches. It is also about how quickly and controllably you can react when it matters.

Not every change is critical. Some are technically simple, but operationally tedious. Without automation, such tasks are easily left undone.

A good example from our own operations was our rebranding from WLAN-Partner to onway: the MOTD had to be adjusted on all VMs. Technically, this was not a big task, but doing it manually across many systems would have been disproportionately time-consuming. With Ansible, the change was rolled out in around 30 minutes and probably would not have happened manually at all.

Systems change during operation: updates, hotfixes, tests, manual interventions. Over time, VMs can start to differ from one another without this being visible at first glance. This is called config drift. It is often only noticed during the next change or while troubleshooting.

That is why we run Ansible across all VMs once a day. If changes are detected, we receive an e-mail with the affected systems; in addition, we write the changes to Logstash. Logstash is used for central log aggregation, collecting events from different sources and making them searchable. Consistency is not achieved by installing a system cleanly once, but by regularly checking the desired state and restoring it when needed.

At onway, Ansible is not an end in itself. We use it because our infrastructure is too large and too dynamic to manage manually in the long term. The initial effort - building roles, maintaining data, testing workflows - pays off in operations: less repetitive work, more time for architecture, troubleshooting and customer solutions.

For our customers, this means faster provisioning of new environments, consistently configured systems and a significantly faster response to security incidents. For us, automation is not an end in itself, but a tool for operating infrastructure efficiently, securely and reliably.

Do you operate complex IT or network infrastructure and want to automate recurring tasks more reliably?

onway supports companies in connecting source of truth, automation and operational processes in a meaningful and practical way.