DevOps Best Practices with a Focus on CI/CD

At XOOTS, we attach great importance in excelling at DevOps. To understand DevOps, we need to comprehend the value chain of: Idea, code, build, deploy, manage, and learn. DevOps is an approach that combines software development and IT operations to streamline the process of delivering high-quality software quickly and efficiently. It unites two contentious groups, software engineers who are measured on the frequency of change, and IT operations who are measured on system stability. DevOps transformation brings velocity (i.e., how quickly product/applications can move through the process), improved quality, and responsiveness to customer demands. Let’s examine application delivery as a value chain of:

1. Ideation: user stories
2. Coding: programming the idea from user stories
3. Build: packaging the code into executable so that it could be deployed in cloud/VM/Bare Metal, perform unit test cases,
4. Deploy: build images
5. Manage: production or pre-production
6. Learn: continuous improvement to become faster with higher quality

Continuous Integration (CI) and Continuous Delivery (CD) are key components of DevOps that help automate the software release process. This best practice guide provides practical, hands-on advice for implementing CI/CD best practices, drawing on resources like the Google Cloud DORA Enterprise Guidebook and Google Cloud’s “What is DevOps” research. While a DevOps transformation goes beyond tooling, the right tools remain an essential component of success. Choosing the right CI/CD tools can streamline workflows, automate repetitive tasks, and improve collaboration within teams. Effective tooling is a critical enabler for a DevOps culture that values continuous learning, experimentation, and improvement.

This best practice guide assumes the use of the Twelve-Factor methodology (12factor) as a manifesto / best practices describing the rules and guidelines to build cloud-native applications. Like any best practice guide, this guide is a bit opinionated because we are passionate about DevOps, and it is based on our experience as practitioners. 

How to excel at Continuous Integration (CI): At XOOTS, we believe there are 4 ingredients in the secret sauce of excelling at CI. CI is a mechanism to merge and test code changes on an ongoing basis, often achieved by a tool like Jenkins. 

• Ingredient 1- Version Control: Use a centralized version control system (e.g., Git, Mercurial) to store and manage code. This ensures that the entire team has access to the latest code changes and can collaborate effectively.
• Ingredient 2- Automated Build and Test: Set up an automated build process that compiles the code and runs unit tests whenever changes are pushed to the repository. This helps catch errors early and ensures that the codebase remains in a releasable state.
• Ingredient 3- Code Review: Implement a code review process to maintain code quality and share knowledge within the team. Encourage team members to provide constructive feedback and learn from one another.
• Ingredient 4- Branching Strategy: Adopt a consistent branching strategy (e.g., GitFlow, trunk-based development) that fits your team’s workflow. This simplifies the management of code changes and promotes efficient collaboration.

How to excel at Continuous Delivery (CD): At XOOTS, we believe there are 3 key ingredients in the secret sauce of excelling at CD. CD is the attempt to speed up and automate deployments, where an operator can push out multiple deployments in a week across numerous services, and know the exact condition of the applications and infrastructure in the course of the deployments. 

• Ingredient 1- Deployment Automation: Automate the deployment process to reduce human error, speed up the release cycle, and ensure consistency across environments. Use tools like Jenkins, CircleCI, or Google Cloud Build or Spinnaker to automate the deployment pipeline. Spinnaker is an open-source continuous delivery platform developed by Netflix to handle CD operations at high scale over its cloud network. It is a cloud-native pipeline management tool that supports integrations into all the major cloud providers, eg Amazon Web Services (AWS). CD takes software delivery a step further by automatically testing the software and pushing it into production using techniques such as canary testing and blue-green testing. Spinnaker works alongside the tried-and-true workhorse of Jenkins. 
• Ingredient 2- Infrastructure as Code (IaC): Manage infrastructure configuration using code (e.g., Terraform, Ansible, or Google Cloud Deployment Manager). IaC enables version control, repeatability, and consistency across environments.
• Ingredient 3- Configuration Management: Use configuration management tools (e.g., Chef, Puppet, or Google Cloud Config Connector) to manage application and system configurations. This ensures consistency and reduces the risk of configuration drift.

Monitoring and Logging: At XOOTS, we believe in importance of comprehensive monitoring and logging and we believe there are 5 key ingredients in the secret sauce of excelling at monitoring and logging: 

• Ingredient 1- instrumentation: Application code needs to be instrumented with standardized logging, standardized events, and these to be matched a standard catalogue that multiple micro services can use. It is important to standardize the log and event messaging and commodities it. 
• Ingredient 2 – Distributed Tracing: Distributed tracing is essential because we have a lot of moving parts in micro services e.g., we need to leverage Load balancing, routing. These features are commoditized with services like K-native (and ISTIO), these types of technologies are being embedded in orchestration layer and which also enable CI/CD. 
• Ingredient 3 – AIOps: AI for IT operations or AIOps is about application of AI, Machine Learning (ML) and analytics for IT operations. AI Ops provides the context and where AiOps shines, is that it gets all data and sends to SRE or Operational team via ChatOps. This means they will have the context when they receive this information, and the actions are recommended based on history of previous incidents and corresponding actions. It acts and automates by suggesting options allows SRE to resolve the issue faster.  
• Ingredient 4 – Observability: The best way to look at observability is to look at as a triangle of logging, metrics, and monitoring.  The logs could be at 3 levels: OS level, container level or application level. This is where good instrumentation of logging in your code, means more useful logs which can be aggregated and filtered to get to application-level information and of course use tools to alert and dashboard this data to send alerts to IT operators and/or security teams, using tools such as Splunk. The goal is to have the fixes automated based on actionable insight. Automation is key for any observability solution, for example, a bug is introduced, monitoring tool flags that, next thing is context of how services work with each other and finally the recommended action to fix the bug.  Observability is different to APM (Application Performance Management) or Monitoring, but they are all part of the secret sauce. 
• Ingredient 5 – comprehensive monitoring and logging: Building on ingredients 1, 2, 3 and 4, it is important to implement comprehensive monitoring and logging to track application performance, system health, and security events, using APM (Application Performance Management) tools, to collect, analyze, and visualize metrics and logs.

How to excel at release management: At XOOTS, we believe there are different ingredients in the secret sauce of excelling at release management depending on the type of release process. There are different type of deployment strategies which are explained further in the next episode of the making of XOOTS. These at high level are:

1. The Basic Deployment 
2. The Multi-Service Deployment
3. Rolling Deployment
4. Blue-Green Deployment 
5. Canary Deployment 
6. A/B Testing
7. Feature flags

Techniques such as canary releases and feature flags enables the gradual roll out of new features or changes. This allows you to test the impact on a small subset of users before deploying to the entire user base, reducing the risk of unforeseen issues.

Cultivating a DevOps Culture: Cultivating a DevOps Culture: At XOOTS, we believe in the importance of DevOps culture, and we think there are 8 key ingredients in the secret sauce of excelling at DevOps culture:

• Ingredient 1 – Collaboration and Communication: Encourage open communication and collaboration between development and operations teams. Break down silos and promote a shared understanding of goals and responsibilities.
• Ingredient 2 – the importance of KPIs: KPIs play a crucial role in measuring the success of DevOps practices. According to the DORA State of DevOps Report, organizations that ‘shift left’ on security are 1.8 times more likely to meet or exceed their business goals. To calculate the return on investment (ROI) of software delivery, DORA examines two categories: cost and value. Reducing costs involves cutting the time it takes to resolve outages and avoiding downtime. On the value side, DORA considers enhanced efficiency through the reduction of unnecessary rework, a reduction in toil, and the potential revenue gained by reinvesting the time saved in new offer capabilities. If engineers spend 60% of their time releasing new code instead of innovating, a company loses 60% of its potential value. By shifting this balance, organizations can reap significant benefits.
• Ingredient 3 – Continuous Learning and Improvement: Continuous learning and improvement is about People, process, tools across the lifecycle of change: Idea, code, build, deploy, manage and learn. As mentioned above, to enable continuous learning it is important to focus on KPIs, instrumentation to gather the data for measuring how fast, how defect free etc, and being able to trace investment to ROI. Fostering a culture of continuous learning and improvement by regularly reviewing processes, identifying areas for improvement, and implementing changes, using retrospectives and post-mortems to learn from successes and failures is an organizational level objective. The key KPIs to enable continuous learnings are:
  • Deployment frequency: understanding the number of deployments to production measured in days, weeks or months 
  • Delivery lead time: all about how long it takes to get from code to deployment – measured in days 
  • Change volumes: number of story points that are actually packed into releases 
  • Meantime to recovery: how long it takes to figure out any defects, and how long once a defect found in production, to deploy a fix – measuring deploy to deploy times
  • Learning: Once we have imperial data we can identify bottlenecks to improve, Data driven decisioning and improve on bottlenecks

• Ingredient 4– Company Culture: A strong company culture plays a critical role in successful DevOps adoption. DORA’s research indicates that high-performing organizations are more likely to have a culture that encourages employees to take calculated, moderate risks without fear of negative consequences. This culture of continuous learning and experimentation goes hand-in-hand with the ‘fail fast’ philosophy. By fostering a culture that values exploration and experimentation, teams can run daily experiments to move towards their target conditions or key results. Employees should ask themselves the following questions every day:

1. What is the target condition?
2. What is the current condition?
3. What obstacles do you think are preventing you from reaching the target condition, and which one are you addressing now?
4. What is your next step, and what outcome do you expect?
5. When can the results be evaluated to see what can be learned from taking that step?

In addition to understanding ‘why’ DevOps is essential, it’s crucial to know how to implement, improve, monitor, and avoid common pitfalls associated with DevOps capabilities. DORA’s DevOps Capabilities website provides valuable resources to help organizations up-level their capabilities and measure them effectively.

• Ingredient 5– Shared Ownership and Accountability: Promote shared ownership and accountability for the entire software delivery process. Ensure that all team members understand their roles and are empowered to make decisions.
• Ingredient 6 – Focus on Releasing Code and Not Building Code: Focus on Releasing Code and Not Building Code: Prioritize the quality and speed of delivery as the most important aspects of any application. 
• Ingredient 7 – Fund as a Product and Not a Project: To ensure that releases are given priority, funds need to flow into a product as a whole rather than being allocated to individual projects.
• Ingredient 8- Establish a Culture of Automation: Support a culture of automation by embracing new automation technologies and avoiding any infrastructure or application code that cannot be automated
• Ingredient 9- You Build it You Own it:Create ownership within the team that built the code. Every piece of code — microservice, pipeline, and infrastructure — should have a clear owner, and avoid creating separations between the teams that manage and the teams that build the code

Conclusions: By implementing the above, the organisation is able to unleash their digital capabilities by continuous delivery from ideation to continues deployment and feedback loop. It modernises the ways of working by addressing the culture aspects and bringing development and operations together. It allows organisation to optimise the legacy systems – big monolith systems deep into the organisation, which are difficult to modernise, by retrofitting DevOps. DevOps is not just about implementing tools and processes; it is also about understanding the ‘why’ behind these practices. By implementing these CI/CD best practices within a DevOps framework, your organization can improve software quality, accelerate release cycles, and enhance overall productivity. Keep in mind that these practices should be adapted to fit your team’s specific needs and workflows. As your team gains experience with DevOps and CI/CD, continually iterate and improve your processes to further optimize your software delivery pipeline.