Accelerate Innovation by Reducing Software Delivery Time
Faced with stiff competition, our customer needed to increase the speed of delivering enhancements to their end customers while simultaneously reducing production failures caused by defective updates.
The primary bottleneck they faced was a lengthy manual approval process to move updates from staging to production. However, shortening the approval process had been proven to increase problems in production.
Their IT architecture is complex, aggravating the problem. They deploy a broad range of microservices-based applications on Kubernetes, as well as a large number of monolithic applications. Their CI/CD system was built using Jenkins, plugins, and custom scripts.
Moving more quickly was a key goal. They were able to process only 50 to 100 updates per month, and their goal was many hundreds of updates per month. Of course, like all companies, they are also under pressure to reduce costs. The process of verifying updates was estimated to have annual direct costs of more than $1M.
Every significant deployment is evaluated as they move to production. The analysis requires 3 expert engineers, including at least one technical lead and one product engineer, and it takes an hour or more to decide whether to move the deployment forward.
The analysis process for every update was time-consuming because of the mountains of data generated. With hundreds of thousands of concurrent users, there is a tremendous amount of metrics and logs created. Consistently finding the “needle in the haystack” that indicates a potential problem is very difficult, even for experienced engineers. As the team’s frequency of updates increased, the severity of the problem grew until they were nearly at a breaking point.
Canary deployment is a technique to reduce the risk of introducing a software update in production by slowly rolling out the change to a small subset of users.
Canary was an essential part of British mining history: these humble birds were used to detect carbon monoxide and other toxic gases before the gases could hurt humans (the canary is more sensitive to airborne toxins than humans). The term “canary analysis” in software deployment serves a similar purpose. Just as the canary notified miners about any problems in the air they breathed, DevOps engineers use a canary deployment analysis to gauge if their new release in CI/CD process will cause any trouble to business.
You can consider the following general definition of a canary release deployment: canary deployment is a technique to reduce the risk of introducing a software update in production by slowly rolling out the change to a small subset of users before making it available to everybody.
Learn how to install OpsMx Enterprise for Spinnaker (OES) on the Azure VM cloud platform using lightweight Kubernetes K3s.
OpsMx provides enterprise Spinnaker that comes with a variety of extensions on top, enhancing easier Spinnaker adoption for enterprise customers to deliver their applications at higher velocity, confidence and trust. Here in this blog, we show you how to install OpsMx Enterprise for Spinnaker(OES) on the Azure VM platform using K3s. Let us first briefly understand what is OES, Azure VM and K3s.
In this blog, we will explore how to integrate Kayenta with Spinnaker to scrape metrics from Prometheus and perform Automated Canary Analysis in your CI/CD pipeline using the Spinnaker pipeline.
We expect the reader to have a basic understanding of how Spinnaker works as a Continuous Delivery tool and orchestration platform for deploying new software application releases, new feature updates, change requests, or bug fixes. We also expect that you know how to create CI/CD pipelines with Spinnaker.
Canary analysis is a two step process where we assess the canary based on metrics and logs to infer if we are going to promote or roll back the new version
Today, many applications run in a dynamic environment of microservices, which makes the software integration and delivery process complex. The only way to successfully deploy updates consistently is to use solid automation with tools like Spinnaker.
Canary analysis is a two step process where we will assess the canary based on a selected list of metrics and logs to infer if we are going to promote or roll back the new version. So we need to be sure that we are gathering the right information (metrics and logs) during the test process and conducting a solid analysis. (From here forward in this blog we will refer to metric and logs simply as metrics)
OES 3.5 Release features are now generally available for Kubernetes Anywhere. Learn how to improve security, include data sources for faster approvals in CI/CD.
With the pandemic year hopefully behind us, we are excited to welcome 2021 with our new release OpsMx Enterprise for Spinnaker 3.5, which extends our leadership in intelligent continuous delivery platforms. 2020 was an incredible year for OpsMx – we added a record number of customers to the OpsMx family. Our customers further validate that enterprises believe in our vision of software delivery without human intervention.
The 3.5 software release of OpsMx Enterprise for Spinnaker (OES) introduces several key enhancements that will provide tremendous benefit to any Spinnaker continuous delivery user. If you are an Open Source Spinnaker user, our OES 3.5 release will work seamlessly with your deployments.
In this blog, we will discuss what is Canary Analysis and we will also look at the different stages of Kayenta. Further, we will also explore the best practices for configuring Kayenta to perform Automated Canary Analysis on Spinnaker pipelines.
Overview of Canary Analysis
A deployment process where any change is partially rolled out and then evaluated against the current deployment i.e. baseline to ensure the latest deployment is stable is called a Canary deployment. This evaluation is done using metrics that are selected when the canary is configured.
Overview of Spinnaker microservice – Kayenta
Kayenta is the Automated Canary Analysis (ACA) platform that comes as an inbuilt feature with Spinnaker. Kayenta is responsible for assessing the risk of a Canary release and checks for significant degradation between the baseline and Canary. Canary Config is not available on the Spinnaker UI by default. Users should check the Canary checkbox in the Application Config to activate Canary Config and Canary Reports on Deck. Kayenta supports Monitoring tools like Stackdriver, Prometheus, Datadog, New Relic, e.t.c. Kayenta consists of two stages – Metric Retrieval and Judgement.
Metric Retrieval
This stage retrieves the key metrics from the baseline and canary clusters. Where the metrics are usually stored in a time-series database with a set of tags and annotations that identifies the deployment data collected from canary/baseline.
Judgment
In this phase, Spinnaker compares those metrics and renders a decision to pass or fail the canary (i.e. was there a significant degradation in the metrics?) The judgment can also be configured to continue on with a canary when the result is “marginal.”
OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business.
A series of steps that is responsible for creating automated and seamless software delivery is called a CI/CD pipeline workflow.
A series of steps that include all the stages from the outset of the CI/CD process and is responsible for creating automated and seamless software delivery is called a CI/CD pipeline workflow. With a CI/CD pipeline, a software release artifact can move and progress through the pipeline right from the code check-in stage through the test, build, deploy, and production stages. This concept is powerful because once a pipeline has been specified, parts or all of it can be automated, speeding the process and reducing errors. In other words, a CI/CD pipeline makes it easier for enterprises to deliver software multiple times a day automatically.
DevOps engineers often get confused with the CI/CD pipeline by automation of individual stages in CI/CD. Though different tools may automate each complicated stage in CI/CD, the whole software supply chain of CI/CD may still be broken because of manual intervention in between. But let us first understand various stages in a CI/CD process and why a CI/CD pipeline is essential for your organization to deliver code at speed and scale.
An enterprise must evaluate a holistic continuous delivery solution which can automate or facilitate automation of these stages, described above. If you are in the process of automating your CI/CD pipelines, OpsMx Enterprise for Spinnaker can help. If you want to know more or request a demonstration, please book a meeting with us. If you want to know more about CI/CD, subscribe to our blog. You can also download the e-book, “Continuous Delivery Planning Guide” to gain better insights and CI/CD best practices.
Cisco Systems is the worldwide leader in networking and IT technologies, dedicated to helping people connect, communicate, and collaborate. Cisco’s IT manages one of the industry’s largest and most diverse software development organizations.
Transform Continuous Delivery to Speed the Shift to the Cloud
This is a story of a large transformation – transforming the way a large organization delivers software in order to respond better to their customers. For the full story, download the document here.
In 2018, Cisco launched an ambitious initiative to transform their IT from a traditional “on-prem first” model to a modern, agile, cloud-native architecture. The goals were to improve their ability to react quickly to market demands and to further their competitive advantage.
Achieving speed in software delivery was critical for Cisco to be able to build and enhance cloud-native applications quickly and to meet their innovation goals. The previous software delivery practices were not able to keep up with the scale, speed, or flexibility requirements of the new model.
OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business.
Learn how OpsMx Autopilot AI/ML powered software can help you verify software updates, new feature release for safety & reliability across CI/CD stages.
OpsMx Autopilot is an AI/ML-powered Continuous Verification platform that verifies software updates across different deployment stages using CI/CD pipelines, ensuring their safety and reliability in live/production environment. It automates new release verification, reducing time-consuming and error-prone manual verification processes. Autopilot uses AI and ML technologies to assess the risk of a new release, find the root-cause of issues and abnormalities for instantaneous diagnosis, and provide real-time visibility and insight about the performance and quality of new deployments to avoid business disruption.
In this blog, we will discuss how Autopilot is used to analyze and improve the release and deployment of any particular product. To understand the efficiency of Autopilot, we at OpsMx, decided to use it in our own product, and we chose to use the functionalities of Autopilot on itself. This is when we came up with the name – “Autopilot on Autopilot”.
The three strategic dimensions companies leaders should consider in their CD platform selection process are Extensibility, Cost -Risk Mitigation and Culture Fit
Every company is trying to deliver software faster. The most important decision for setting up the organization for success is the CD platform. Sometimes the shiny object is not the right solution.
Responding quickly to customer needs is the new key to long term success for many companies. Since software applications are now central to so many customer interactions, speed in delivering software is a key IT competency. That is why continuous delivery is gaining such a broad following now.
Choosing a platform for continuous delivery (CD) is not simple. The number of integrations required, the process change that CD causes, and the impact of a poor choice all demand a thoughtful consideration of all the factors involved in a good decision.
Engineering and IT architects, managers and leaders have to grapple with requirements and tradeoffs as they make decisions on which Continuous Delivery Platform to choose. Rather than focus on features or technical approaches, this blog highlights 3 strategic dimensions companies leaders should consider in their CD selection process and how OpsMx Enterprise for Spinnker addresses these considerations.
The bank wanted to stay ahead of the competition by delivering services faster to customers. this was possible only by modernizing their IT infrastructure
This is a three-part series on how to speed up your CI/CD process using OpsMx Autopilot
Series 1 — “What is slowing down your CI/CD pipeline?” discusses the challenges that a typical continuous delivery (CD) system will have even after implementing a modern CD tool.
Series2 — “Use OpsMx Autopilot to Accelerate Your CI/CD Pipeline” blog, introduces OpsMx Autopilot, the intelligence layer for software delivery.
Series 3 — “How Customers Improve CI/CD Velocity Using Autopilot” outlines the experience of leading companies and provides a quick overview of the benefits that our customers have experienced.
Here are some excerpts from the blog -
Part 1: What Is Slowing Down Your CI/CD Pipeline?
Even after implementing a CI/CD process, many organizations face three significant challenges that block them from consistently, safely, and quickly delivering software.
Slow software delivery: Even with automated pipelines, changes to production services and applications still take too long to move from development to production due to the approval gates placed at stages and manual intervention thereof.
Lack of Governance and Compliance: Frequently changing external regulations, internal rules are difficult to monitor and comply with, operational and security constraints and need for best practices affects the CI/CD process.
Siloed visibility and insights: Visibility into the enterprise-wide software delivery process is lacking, and sharing best practices between teams is rare.
In today’s constant drive for faster response, a new approach is required — human decision-making simply isn’t fast enough. You can solve the issues highlighted in the blog only through the addition of a layer of intelligence to your CD system.
Part 2: How customers improve ci/cd pipeline velocity using autopilot
Autopilot is introduced as the solution to improve CI/CD deployment capabilities in this three-part series on how to speed up your CI/CD process using OpsMx Autopilot.
This is the third part of a three-part series that discusses:
Autopilot improves your CI/CD process through three major benefits:
Reduce risk and shorten approvals — shorten the data gathering drudgery of the verification process and automate the approval decision when possible.
Improve governance and compliance — ensure compliance to your corporate and regulatory policies and enable you to audit all the steps in your CI/CD process.
Generate Insights and visibility — enable the high-level management of the enterprise-wide software delivery process by sharing CI/CD metrics and DevOps best practices among teams.
Read more about the Case Studies on how OpsMx Autopilot reduces the number of risks and the “Risk of Errors” in Production that happens due to the slow manual approval process required to promote changes from dev to test and from testing to production and in another instance helped the customer reduce its Build Troubleshooting time by 75%. Autopilot is in use today at companies around the world who are improving their release cycles, reducing errors in production, and improving their governance processes.
Part -3. Use OpsMx Autopilot to Accelerate Your CI/CD Pipeline
By adding the OpsMx Autopilot intelligence layer to your software delivery process, DevOps and SRE teams managing the CI/CD pipelines can quickly make more informed decisions while analyzing the risk of releases and automatically approving them for deployment.
OpsMx Autopilot improves your CI/CD process through three major benefits:
Reduce risk and shorten approvals — shorten the data gathering drudgery of the verification process and automate the approval decision when possible.
Improve governance and compliance — ensure compliance to your corporate and regulatory policies and enable you to audit all the steps in your CI/CD process.
Generate Insights and visibility — enable the high-level management of the enterprise-wide software delivery process by sharing CI/CD metrics and DevOps best practices among teams.
The three main OpsMx Autopilot capabilities include:
ML-based automated verification — Autopilot Risk Analysis Dashboard for CI/CD pipeline enables real-time analysis, automated decision making, and approvals that reduce risk and shortens approval times.
Continuous Governance — Improve governance and compliance to your corporate and regulatory policies and enable easy audit of your CD process.
Visibility, Audit, and Insights — Audit using Autopilot to know answers to the “who did what when where and how” questions about the CI/CD pipelines, generate Insights and visibility that enables the high-level management of the enterprise-wide CD process while sharing best practices among teams.
Read more about how OpsMx Autopilot accelerates CI/CD Pipelines.
Excerpt
This a two-part blog series about Canary Deployment and Canary Analysis. In the first part, we have discussed canary as a release strategy, canary deployment stages, benefits of using canary, and constraints with implementation. In the second part, we discussed how canary analysis is performed after canary deployments are done, selection and evaluation criteria for choosing different metrics and smoke tests to perform the analysis, and canary hypothesis testing performed with incremental traffic to canary instances. Here we will discuss Canary Deployment.
Just as the canary birds notified miners about any problems in the air they breathed, DevOps engineers use a canary deployment analysis as a release management strategy in their Continuous Delivery process to gauge and control if the new software release causes any catastrophic change to the ongoing business in the live production environment.
Canary deployment stages include:
Benefits of canary Deployment Strategy include:
Constraints to Implement Canary Deployments include:
Canary deployment strategy is widely used because it lowers the risk of moving changes into production while reducing the need for additional infrastructure and organizations can test the new release in a live production environment without simultaneously exposing all users to the latest release. To know more about Canary Deployment please read our blogs — “What is Canary Deployment?” and What is Canary Analysis?
Read the success stories of Spinnaker, OES, and Autopilot used by global enterprises for their continuous delivery
Learn how to configure and connect Spinnaker with the external “Azure Cache For Redis” to store pipeline execution cache history.
Learn how a leading CRM provider achieves scale in software delivery using OpsMx Enterprise for Spinnaker
Our customer is one of the leading providers of customer service software in the world. They serve more than 150,000 customers, of all sizes, across multitudes of industries and offer a suite of CRM and support products to improve customer relationships.
Know how to update expired security certificates used for mTLS authentication between Spinnaker the different micoservices to avail uninterrupted services.
Read how OpsMx recently supported the CSR initiative for the Covid-19 response by The Akshaya Patra Foundation for social outreach and commitment towards society.
This blog will describe the power of combining Splunk and Autopilot. Autopilot is a powerful add-on to any CI/CD process running on any platform. We’ll specifically cover how OpsMx uses Splunk and Autopilot together to accelerate a CI/CD pipeline.
Most software organizations run CI/CD pipelines to move updates from dev to test to staging to prod. This CI/CD process is an intricate set of interconnected steps that deploy software to the target platforms. During the testing process, especially the performance, scalability, and resiliency tests, each application typically generates a large amount of data through log entries. On a human level, this data is complex, unstructured, and impossible to analyze. But a business can gain a tremendous advantage if this data can be properly exploited.
OpsMx Autopilot uses machine learning (ML) and natural language processing to analyze the data for you automatically so you can quickly and accurately decide whether any update should be moved forward in the pipeline. This blog will cover the ways that Autopilot can extend the value of Splunk in your continuous delivery initiatives..
Learn how to integrate Microsoft Teams with Spinnaker CI/CD pipelines for quick collaboration and feedback and faster approvals of the CI/CD pipeline stages.
Learn about the top challenges for Continuous Delivery to Kubernetes using Spinnaker CI/CD pipelines and the best practices for software delivery.
Learn how and why you should use Debian Packages for your Spinnaker CI/CD pipelines.
Learn how to configure a Spinnaker pipeline to use for deploying Windows app on Azure cloud
Learn how to create a Debian package for a spring boot application with the Netflix Nebula Gradle plugin and deploy it to GCP using Spinnaker CI/CD pipelines.
