<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Human Still by Tushar Murudkar]]></title><description><![CDATA[Field notes on AI, engineering leadership, and the decisions that keep systems reliable.]]></description><link>https://www.tusharmurudkar.com</link><image><url>https://substackcdn.com/image/fetch/$s_!0nsh!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe61462ee-7298-4049-9c3d-02fc345be067_144x144.png</url><title>Human Still by Tushar Murudkar</title><link>https://www.tusharmurudkar.com</link></image><generator>Substack</generator><lastBuildDate>Tue, 17 Nov 2026 01:52:54 GMT</lastBuildDate><atom:link href="https://www.tusharmurudkar.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Tushar Murudkar]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[tusharmurudkar@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[tusharmurudkar@substack.com]]></itunes:email><itunes:name><![CDATA[Tushar Murudkar]]></itunes:name></itunes:owner><itunes:author><![CDATA[Tushar Murudkar]]></itunes:author><googleplay:owner><![CDATA[tusharmurudkar@substack.com]]></googleplay:owner><googleplay:email><![CDATA[tusharmurudkar@substack.com]]></googleplay:email><googleplay:author><![CDATA[Tushar Murudkar]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Autonomous DevOps]]></title><description><![CDATA[From operational signals to governed action.]]></description><link>https://www.tusharmurudkar.com/p/autonomous-devops-f1d17753b569</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/autonomous-devops-f1d17753b569</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Thu, 05 Nov 2026 03:35:15 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/7fa2e94e-a6bf-46f1-8ed2-7f1cedd136d9_1024x576.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>From operational signals to governed&nbsp;action.</em></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xtvn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xtvn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!xtvn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!xtvn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb767267c-fe2c-4415-8c1c-5f041c2fb106_1024x576.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>We&#8217;ve largely automated software delivery, but operational decision-making hasn&#8217;t kept pace. CI/CD pipelines execute workflows and observability platforms detect failures. Diagnosing issues, correlating changes, assessing risk, and choosing corrective actions still consume significant engineering time.</p><p>That gap between knowing something is wrong and acting on it correctly is the next opportunity in DevOps. I got there by trying to solve the first half of the&nbsp;problem.</p><h3><strong>From Observability to Intelligence</strong></h3><p>In my past articles, <em><a href="https://medium.com/@tusharmurudkar/devops-lens-of-sdlc-c5939969743e?source=your_stories_outbox---writer_outbox_published-------------------------------------------">DevOps Lens of SDLC</a></em>, I explored software delivery as a connected engineering system. That perspective became more useful as scale increased. <a href="https://medium.com/advisor360-com/what-23-000-deployments-revealed-about-software-delivery-at-scale-9644b7056040?source=your_stories_outbox---writer_outbox_published-------------------------------------------">Analyzing 23,000 deployments across 90 days</a> revealed recurring failure patterns, environment inconsistencies, and clear opportunities to strengthen platform guardrails.</p><p>The challenge was no longer collecting data. It was turning that data into intelligence.</p><p>That led to <strong><a href="https://medium.com/advisor360-com/turning-deployment-failures-into-operational-learning-544128aa0ff6?source=your_stories_outbox---writer_outbox_published-------------------------------------------">Vizion</a></strong><a href="https://medium.com/advisor360-com/turning-deployment-failures-into-operational-learning-544128aa0ff6?source=your_stories_outbox---writer_outbox_published-------------------------------------------">, an AI-assisted RCA capability</a> that correlates logs, deployment history, code changes, and previous failures to investigate incidents and recommend corrective steps. Instead of engineers reconstructing context for every failure, the platform assembles and analyzes&nbsp;it.</p><p>Vizion answered <em>what went wrong</em>. It surfaced a harder question: once the platform understands a failure, should it act on that understanding, and how would we trust it&nbsp;to?</p><h3><strong>Closing the Decision&nbsp;Loop</strong></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yf0J!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 424w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 848w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 1272w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!yf0J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!yf0J!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 424w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 848w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 1272w, https://substackcdn.com/image/fetch/$s_!yf0J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe57339b7-4c4a-42e0-92b7-4e35a7b7cebe_1024x629.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Answering that question meant extending investigation into a complete decision&nbsp;loop.</p><p>Vizion already covered the first three stages. The rest is where trust gets built. Independent policies determine what is permitted, existing automation carries out authorized operations, and verification confirms the outcome and feeds what was learned back into future investigations.</p><p>That separation is deliberate: <em>Agents reason. Policies govern. Trusted automation executes. Independent signals&nbsp;verify.</em></p><p>Verification deserves particular emphasis. A successful pipeline run doesn&#8217;t necessarily mean the service has recovered, so verification must evaluate the actual service outcome rather than the status of the automation.</p><h3><strong>Agent Pods: Coordinating Intelligence</strong></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nlle!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 424w, https://substackcdn.com/image/fetch/$s_!nlle!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 848w, https://substackcdn.com/image/fetch/$s_!nlle!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 1272w, https://substackcdn.com/image/fetch/$s_!nlle!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!nlle!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!nlle!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 424w, https://substackcdn.com/image/fetch/$s_!nlle!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 848w, https://substackcdn.com/image/fetch/$s_!nlle!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 1272w, https://substackcdn.com/image/fetch/$s_!nlle!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F606757aa-403c-4570-8d23-0d89de6d8b1b_1024x629.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Investigation is one capability. Fixing a failure safely requires several working together: root-cause analysis, risk assessment, remediation planning and recovery verification. I organize these into Agent Pods: small, coordinated units that each own one specific operational responsibility, such as recovering failed deployments or remediating configuration drift. A pod is accountable for its responsibility from first signal to verified result. Like a cross-functional team, it is defined by what it is responsible for, not by how it is&nbsp;built.</p><p>Not every capability in a pod needs an LLM. Reasoning about an unfamiliar failure does; checking a health endpoint or applying a policy rule does not. Those belong to deterministic functions and existing platform services, which are faster, cheaper and easier to&nbsp;trust.</p><p>What makes a pod effective is how its capabilities work together, not how many agents it contains. Four design principles hold it together:</p><ul><li><p><strong>Explicit orchestration:</strong> defined handoffs, retries, timeouts and escalation paths</p></li><li><p><strong>Shared context:</strong> every capability works from the same evidence and system&nbsp;state</p></li><li><p><strong>Scoped capabilities:</strong> each one has only the tools and permissions its job&nbsp;requires</p></li><li><p><strong>Traceability:</strong> every decision, action and outcome is recorded and auditable</p></li></ul><p>Together, these make AI-driven operations composable, observable and accountable. They also make it possible to decide, action by action, how much the pod is allowed to do on its&nbsp;own.</p><h3><strong>Autonomy Is a Risk&nbsp;Decision</strong></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xiR-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 424w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 848w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 1272w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xiR-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!xiR-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 424w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 848w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 1272w, https://substackcdn.com/image/fetch/$s_!xiR-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4806e17-33bb-484d-8af8-e6bc63d8b802_1024x688.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><em>Autonomous DevOps is a maturity journey, not a&nbsp;switch</em></p><p>The right level for any action depends on evidence quality, business impact, reversibility, and available safeguards. Retrying a known transient failure may be suitable for bounded autonomy, while modifying production security policies may require explicit human authorization. More autonomy isn&#8217;t automatically better.</p><p>Moving up a level should be justified by results, not by agent activity or token consumption. The meaningful metrics are time to diagnosis, recovery time, change failure rate, remediation effectiveness and policy compliance. Autonomy has to earn its place through better outcomes without compromising control.</p><h3><strong>Where DevOps Goes&nbsp;Next</strong></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uEQq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 424w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 848w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 1272w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!uEQq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!uEQq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 424w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 848w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 1272w, https://substackcdn.com/image/fetch/$s_!uEQq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f21e9b6-0477-4683-b812-41ecde6dd64a_1024x453.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Each step builds on the last instead of replacing it, and the returns compound. Every incident a pod resolves becomes knowledge the platform keeps, so operational expertise stops living only in the heads of a few senior engineers. Recovery gets faster, investigation stops eating engineering capacity, and reliability can scale with the business instead of with headcount. Because every action runs through policy and verification, that speed comes with an audit trail rather than added&nbsp;risk.</p><p>Engineers don&#8217;t leave operations; they move up the stack. Instead of reconstructing the same context at 2 a.m., they design the guardrails, define what each pod is trusted to do, and focus on the failures no system has seen&nbsp;before.</p><p>Autonomous DevOps succeeds when a platform can determine when to act, when to escalate, and whether its intervention actually&nbsp;worked.</p>]]></content:encoded></item><item><title><![CDATA[Turning Deployment Failures into Operational Learning]]></title><description><![CDATA[How an AI bot helps a delivery platform learn from every failed deployment]]></description><link>https://www.tusharmurudkar.com/p/turning-deployment-failures-into-operational-learning-544128aa0ff6</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/turning-deployment-failures-into-operational-learning-544128aa0ff6</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Wed, 10 Jun 2026 12:05:46 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/e68c1c81-7277-4714-a557-29f88a086329_1024x576.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vklu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!vklu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!vklu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!vklu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!vklu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/69243e09-839d-4519-a489-2c801561a10d_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!vklu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!vklu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!vklu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!vklu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69243e09-839d-4519-a489-2c801561a10d_1024x576.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h3>How an AI bot helps a delivery platform learn from every failed deployment</h3><p>Every deployment leaves evidence. It shows what changed, what held, and where the system gave way. Across a large delivery platform, those signals accumulate continuously&#8202;&#8212;&#8202;in pipelines, logs, metrics, tickets, and team channels. Each failure contains a lesson, but extracting that lesson still takes time, context, and engineering judgment. Vizion was built to reduce that&nbsp;gap.</p><p>Vizion is an always-on AI Slack bot that watches CI/CD failures across QA, PreProd, and Prod. When a failure occurs, it gathers the relevant evidence, reconstructs what changed since the last successful run, drafts a root-cause hypothesis with recommended next steps, and posts the analysis back into the same channel where the failure was reported. Once an engineer approves the lesson, Vizion preserves it so future failures can be resolved with better&nbsp;context.</p><p>The goal is not to replace incident response. It is to reduce context-reconstruction time and make each failure easier for the platform to learn&nbsp;from.</p><p>What began as deployment visibility&#8202;&#8212;&#8202;understanding how deployments moved through the platform&#8202;&#8212;&#8202;is now evolving into operational learning: a system that can reason about failure patterns, retain confirmed lessons, and help teams respond with more consistency over&nbsp;time.</p><p>This is the story of how we taught the delivery platform to learn from&nbsp;failure.</p><h3>From Visibility to&nbsp;Action</h3><p>In my <a href="https://medium.com/advisor360-com/what-23-000-deployments-revealed-about-software-delivery-at-scale-9644b7056040">previous article</a>, I wrote that deployment data is not just operational reporting&#8202;&#8212;&#8202;it is architectural feedback.</p><p>Metrics alone do not improve a platform. They reveal what happened, but they do not explain why it happened or reconstruct the events that led to&nbsp;failure.</p><p>That work usually falls to an on-call engineer. Before they can act, they first have to reconstruct the operational context around the failure. That reconstruction is often where the delay&nbsp;lives.</p><p>Vizion was designed to make that context available immediately.</p><h3>One Signal, Two&nbsp;Purposes</h3><p>Every deployment produces a structured delivery signal: deployment data for reporting and an outcome signal that reflects success or&nbsp;failure.</p><p>That signal was originally built for visibility. It made deployment activity measurable across environments and gave us a more reliable view of how software moved through the platform. Vizion uses the same signal differently.</p><p>When the outcome is a failure in QA, PreProd, or Prod, Vizion treats it as the starting point for analysis. Successful runs pass through without action. Failed runs become structured opportunities to gather evidence, explain what happened, and capture what should be remembered.</p><p>The architectural point is simple: the platform does not need a separate detection mechanism to learn from failure. The same instrumentation that measures delivery can also trigger a response.</p><p>One signal serves two purposes&#8202;&#8212;&#8202;measurement and&nbsp;action.</p><h3>Vizion as an Architectural System</h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zujO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!zujO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!zujO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!zujO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!zujO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!zujO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!zujO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!zujO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!zujO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe683771c-7168-4c62-ba99-ed3c6c55e00e_1024x576.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Vizion is best understood not as a single AI feature, but as a set of deliberate architectural boundaries.</p><p>It operates as an event-driven pipeline where each layer owns a narrow responsibility:</p><ul><li><p>A <strong>listener</strong> reacts to failure events as they are emitted by the delivery platform.</p></li><li><p>An <strong>evidence layer</strong> assembles the context: logs, recent changes, the last successful run, and relevant prior analyses.</p></li><li><p>A <strong>reasoning layer</strong> drafts an explanation from that evidence and checks its output before&nbsp;release.</p></li><li><p>A <strong>delivery layer</strong> posts the analysis back into Slack, where the team is already working, and where needed, creates and assigns a&nbsp;ticket.</p></li><li><p>A <strong>memory layer</strong> preserves approved analyses as reusable priors so similar failures can retrieve the right lesson&nbsp;later.</p></li><li><p>A <strong>control layer</strong> keeps human review in place and determines what is allowed to become part of system&nbsp;memory.</p></li></ul><p>This separation is intentional. Models will change. Reasoning approaches will improve. The AI layer can be replaced over time without redesigning the whole&nbsp;system.</p><p>The durable value lives in the surrounding architecture: consistently gathering the right evidence, preserving confirmed learning, and making that learning reusable across future incidents.</p><p>The intelligence may evolve, but the architecture is what compounds.</p><h3>Context Before Reasoning</h3><p>The most important design choice in Vizion is that reasoning does not start with the model. It starts with evidence.</p><p>Before an explanation is drafted, Vizion assembles the operational context around the failure: the failed run, recent changes, logs, metrics, the last successful deployment, database migrations, and any similar prior analysis. Only then does the reasoning layer generate an explanation.</p><p>That sequence keeps the system disciplined. It reduces guesswork, makes the output easier to review, and gives the engineer a clearer starting point for&nbsp;action.</p><p>In practice, Vizion is not just asking AI to explain a failed deployment. It is building the conditions for a better explanation before the AI is involved.</p><h3>Learning Only Matters If It&nbsp;Persists</h3><p>A feedback loop has value only if the system retains what it&nbsp;learns.</p><p>Each Vizion analysis includes a confirmation step. When an engineer approves the outcome, they capture the lesson, the recommended fix, and the scope of that lesson&#8202;&#8212;&#8202;whether it applies narrowly to one repository or more broadly across the platform.</p><p>An approved lesson is the engineer-confirmed explanation and recommended fix that can be safely reused when a similar failure appears again. That approved analysis becomes reusable knowledge.</p><p>The next time a failure with the same fingerprint appears, Vizion can retrieve the prior lesson and bring it into the new analysis. A fingerprint is a stable representation of the failure pattern, not just the exact log text. That allows the system to recognize similar failures even when they do not look identical line by&nbsp;line.</p><p>One engineer&#8217;s judgment becomes context the next engineer does not have to rediscover. That is the difference between automation that repeats work and architecture that compounds learning.</p><h3>Feedback That Improves the&nbsp;Platform</h3><p>The goal is not only to explain failures. The goal is to make every interaction improve the&nbsp;system.</p><p>Engineers can refine the analysis, request more evidence, ask for a code-level fix, or approve the lesson so it becomes reusable. Each path adds signal back into the platform.</p><p>That distinction matters. The value of AI is not only in producing an answer; it is in helping the platform become more informed, more consistent, and easier to operate over&nbsp;time.</p><h3>Keeping Judgment Where It&nbsp;Belongs</h3><p>Vizion supports operational judgment. It does not replace&nbsp;it.</p><p>QA is optimized for developer feedback, while PreProd and Prod require operational ownership and ticket tracking. Vizion adapts to that difference by automating ticket creation and assignment to an on-call engineer, rather than treating every failure the same&nbsp;way.</p><p>Vizion gathers evidence, drafts reasoning, and preserves approved lessons. The decision still belongs to the engineer responsible for the&nbsp;system.</p><p>Human approval is what keeps the memory trustworthy. The platform learns only from what someone has reviewed and confirmed.</p><p>In operational architecture, that distinction matters. Assistance can scale. Accountability should remain&nbsp;clear.</p><h3>The Next Step: Reducing Failure Before It&nbsp;Happens</h3><p>Everything described so far is reactive. It reduces the cost of understanding failure after it occurs. The more interesting direction is preventative.</p><p>The next capability is a readiness assessment that evaluates a change before it reaches production, using the same signals and lessons already captured by the&nbsp;system.</p><p>Does the change include a database modification?<br>Is it backward-compatible with the version currently serving traffic?<br>How large is the change set since the last successful deployment?<br>Does the rollout approach match the risk profile of the&nbsp;change?</p><p>These are the kinds of questions a delivery platform should surface&nbsp;early.</p><p>The objective is to enhance human decision-making by presenting risk to teams while there is still time to&nbsp;respond.</p><p>That is the natural extension of operational learning: not just explaining incidents better, but helping prevent avoidable ones.</p><h3>Final Thought</h3><p>Vizion exists to amplify the engineers who own the path to production.</p><p>It gives them a system that gathers context quickly, explains failures with more consistency, and preserves confirmed lessons so the same problem does not have to be solved from scratch&nbsp;twice.</p><p>That is where delivery architecture creates lasting value: not only in moving software safely to production, but in building systems that learn from what happens along the way&#8202;&#8212;&#8202;and help teams see risk before it becomes&nbsp;failure.</p><div><hr></div><p><a href="https://medium.com/advisor360-com/turning-deployment-failures-into-operational-learning-544128aa0ff6">Turning Deployment Failures into Operational Learning</a> was originally published in <a href="https://medium.com/advisor360-com">Advisor360&#176;</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded></item><item><title><![CDATA[What 23,000 Deployments Revealed About Software Delivery at Scale]]></title><description><![CDATA[Software delivery at scale is more than running pipelines successfully.]]></description><link>https://www.tusharmurudkar.com/p/what-23-000-deployments-revealed-about-software-delivery-at-scale-9644b7056040</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/what-23-000-deployments-revealed-about-software-delivery-at-scale-9644b7056040</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Wed, 03 Jun 2026 21:04:41 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/5c314d63-fac5-4c1d-8bfd-3c4f7d31fe8d_1024x576.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2XSp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!2XSp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!2XSp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!2XSp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16c0a959-754a-4b29-9f9e-cfa6bbb0b2fe_1024x576.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>Software delivery at scale is more than running pipelines successfully. It is about understanding how change flows through the entire delivery system&#8202;&#8212;&#8202;from code to build, from build to deployment, from lower environments to production, and from failure back into improvement.</p><p>Over the last 90 days, I looked at deployment metrics across teams, applications, platforms, and environments. The goal was not to create another reporting dashboard. The goal was to understand the delivery system itself: where change moved well, where friction repeated, and where platform design could make delivery safer and more predictable.</p><p>In that period, we captured more than <strong>23,000 deployment events across 20 teams and over 200 applications</strong>. The value was not in the number alone. The value was in what the patterns revealed.</p><h3>Deployment Metrics as an Architecture Signal</h3><p>A deployment is not just a CI/CD event. It is a signal that many parts of the delivery architecture are working together: source control, workflow design, artifact management, environment configuration, approvals, infrastructure, secrets, runtime platform, observability, and release&nbsp;process.</p><p>When deployments are measured consistently, they provide a practical view into the health of that architecture. They show whether change is flowing through the intended path, where teams experience friction, and where the platform needs stronger defaults or better guardrails.</p><p>That is why deployment data should not be treated only as operational reporting. It can become architectural feedback.</p><h3>Volume Shows Movement, Not&nbsp;Maturity</h3><p>Deployment volume showed that teams were actively moving change through the platform. That is a positive signal, but volume alone does not prove delivery maturity.</p><p>From an architecture lens, the more important question is whether change is flowing consistently, reliably, and through the intended paths. Activity matters, but flow matters&nbsp;more.</p><p>A mature delivery platform should make the path to production repeatable, observable, and continuously improvable. When deployment volume is viewed alongside success rate, failure reasons, hotfixes, and environment trends, it becomes a stronger signal of delivery&nbsp;health.</p><h3>Failure Patterns Reveal Design Opportunities</h3><p>Failure patterns were especially useful because a failed deployment is often treated as an isolated event, while repeated failures usually point to a system design opportunity. The issue may be missing validation, weak pipeline guardrails, environment drift, configuration inconsistency, unclear ownership, or poor error visibility.</p><p>The answer is not simply to ask teams to be more careful. The better question is: how do we improve the delivery architecture so this type of failure becomes less&nbsp;likely?</p><p>That may mean moving validation earlier, improving deployment templates, strengthening defaults, making error messages clearer, or creating better self-service guidance for&nbsp;teams.</p><h3>Environment Flow Shows Delivery&nbsp;Health</h3><p>Development, QA, pre-production, and production environments should each add confidence before change reaches users. When deployment activity is concentrated in one part of the lifecycle, it may indicate active development, but it may also point to rework, validation gaps, promotion friction, or release pressure.</p><p>For a DevOps architect, environment flow is a key signal because environments should not just be checkpoints. They should be part of a progressive confidence model where each environment reduces uncertainty instead of simply moving change to the next&nbsp;stage.</p><h3>Hotfix Trends Expose System&nbsp;Pressure</h3><p>Hotfixes are normal in real systems and they will happen. The concern is not that hotfixes exist; the concern is when they become a&nbsp;pattern.</p><p>When hotfix activity increases, it may point to pressure in testing, planning, compatibility, rollback strategy, production readiness, or deployment controls. Tracking hotfixes helps identify whether urgency is becoming part of the normal operating model.</p><p>A healthy delivery system does not eliminate every hotfix. It understands why they are happening, how often they are happening, and whether the trend is moving in the right direction.</p><h3>Metrics Should Improve the System, Not Judge the&nbsp;Teams</h3><p>Deployment metrics need context. A team with more failures may own more complex services, deploy more frequently, carry more legacy dependencies, or operate with more constraints. Metrics without context can lead to the wrong conclusions.</p><p>Used well, metrics help identify where teams need better paved roads, stronger defaults, clearer standards, earlier guardrails, or better enablement. The purpose is not to rank teams; the purpose is to improve the delivery system around&nbsp;them.</p><h3>From Dashboard to Feedback&nbsp;Loop</h3><p>The dashboard itself is not the outcome. The outcome is the feedback loop it&nbsp;creates.</p><p>When deployment activity is visible, repeated issues become easier to prioritize, platform improvements become easier to justify, and standards become easier to measure. A mature DevOps platform should not only automate deployments. It should learn from how teams deliver and continuously reduce friction in the path to production.</p><p>That is where deployment metrics become valuable: they connect delivery activity back to platform&nbsp;design.</p><h3>Where This Goes&nbsp;Next</h3><p>Deployment metrics are one layer of the engineering delivery picture. The next step is connecting them with other signals such as change failure rate, lead time, rollback trends, incident data, security gate performance, code quality, and production reliability.</p><p>Together, these signals can provide a more complete view of software delivery health. The goal is not to measure everything. The goal is to measure the signals that help make software delivery safer, faster, and more predictable.</p><h3>Final Thought</h3><p>The biggest takeaway is this: visibility is not just an operational benefit. It is an architectural input.</p><p>It shows how change flows, where friction repeats, where standards are working, and where the platform needs to&nbsp;evolve.</p><p>That is where DevOps architecture creates real value&#8202;&#8212;&#8202;not only by building pipelines, but by designing a delivery system that continuously improves how software moves from idea to production.</p><div><hr></div><p><a href="https://medium.com/advisor360-com/what-23-000-deployments-revealed-about-software-delivery-at-scale-9644b7056040">What 23,000 Deployments Revealed About Software Delivery at Scale</a> was originally published in <a href="https://medium.com/advisor360-com">Advisor360&#176;</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded></item><item><title><![CDATA[Feature Flags for DevOps]]></title><description><![CDATA[Feature flags have emerged as a versatile tool in the arsenal of modern-day DevOps teams.]]></description><link>https://www.tusharmurudkar.com/p/feature-flags-for-devops-6a84117aa995</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/feature-flags-for-devops-6a84117aa995</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Thu, 01 Feb 2024 18:12:26 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/896c1232-9f7a-4c5b-a876-9e0baf9d2a3f_1024x530.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9MI9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 424w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 848w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 1272w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!9MI9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!9MI9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 424w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 848w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 1272w, https://substackcdn.com/image/fetch/$s_!9MI9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F121aec85-d985-4ab3-8492-c4a8d150a171_1024x530.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p><strong>Feature flags</strong> have emerged as a versatile tool in the arsenal of modern-day DevOps teams. At their core, these toggles serve as a switch that enables teams to turn specific features on or off within an application or system. This functionality makes feature flags indispensable for several reasons, including testing and experimentation, targeted rollouts, and gradual releases.</p><h3>Use cases:</h3><ol><li><p><strong>Testing and experimentation</strong>: Feature flags are an excellent tool for testing and experimentation. Teams can use them to turn specific features on or off, allowing them to test new features in a controlled environment before going public. This approach minimizes the risk of any negative impact on the overall&nbsp;system.</p></li><li><p><strong>Trunk base development</strong>: Keeping the trunk always deployable is essential to practice continuous deployment; this demands wrapping of in-progress features under a feature&nbsp;flag.</p></li><li><p><strong>Targeted rollouts</strong>: Feature flags are helpful for targeted rollouts, particularly for new features or updates. Instead of simultaneously releasing a new feature to all users, teams can gradually introduce it to a smaller subset of users. This approach allows for better control and testing of new features and minimizes the impact of any issues that may&nbsp;arise.</p></li><li><p><strong>Gradual releases</strong>: Feature flags are an excellent tool for gradual releases, which can slowly introduce new features to users over time. This approach benefits complex systems, where minor changes can significantly impact user experience and performance.</p></li><li><p><strong>A/B testing</strong>: Feature flags can be used for A/B testing, where two versions of a feature are released to different subsets of users. This approach allows teams to test which version of the feature performs better, allowing them to make decisions based on data about which version to release to the&nbsp;public.</p></li><li><p><strong>Emergency rollbacks</strong>: If critical issues arise, feature flags can quickly roll back to a previous system version. This approach helps ensure that problems can be resolved promptly, minimizing user&nbsp;impact.</p></li></ol><h3><strong>Best Practices</strong></h3><ol><li><p><strong>Naming &amp; metadata</strong>: Contextual names with traceability to who owns the flag and details on the provided feature; a good strategy is to tie the flags back to feature definition in the project management tool.</p></li><li><p><strong>Traceability to code</strong>: Feature flags should be traceable to the code they wrap; that way, we would understand the cleanup area during the removal of the&nbsp;flags.</p></li><li><p><strong>Use a centralized management system</strong>: It&#8217;s essential to have a centralized system for managing feature flags. This system should provide a single source of truth for all feature flags, allowing developers to manage them efficiently and avoid conflicts.</p></li><li><p><strong>Document feature flags</strong>: It&#8217;s crucial to document feature flags accurately, including their purpose, status, and configuration. This documentation ensures that teams understand the purpose and use of each flag and can avoid confusion or conflicts; these documents ideally should be generated during the planning&nbsp;phase.</p></li><li><p><strong>Use Automation to toggle</strong>: Toggling should be part of the automation pipelines. If the flag is a scope to a release, it should be part of automated steps.</p></li><li><p><strong>Test feature flags thoroughly</strong>: Before rolling out any feature flags, it&#8217;s essential to test them thoroughly to ensure they work as intended. The test suite should validate feature flag combinations.</p></li><li><p><strong>Monitor feature flags</strong>: Continuous monitoring of feature flags is essential to quickly identify and resolve issues that arise, minimizing the impact on&nbsp;users.</p></li><li><p><strong>Remove unused feature flags</strong>: It&#8217;s essential to remove unused feature flags to prevent confusion and avoid errors caused by unused flags; feature flags can become technical debt if not cleaned up at regular intervals.</p></li><li><p><strong>Segment Users: </strong>Group users or entities for which we would want a variation of the flags; flag can serve a different value to a segment of&nbsp;users.</p></li><li><p><strong>Security</strong>: Utilize the principle of least privileges, scope users to internal and external facing env, with the capability to view toggle and need approvals in customer-facing environments.</p></li><li><p><strong>Use feature flags judiciously</strong>: Too many can lead to confusion, clutter, and performance issues. Therefore, it&#8217;s vital to use them only when necessary and to remove them when they are no longer&nbsp;needed.</p></li></ol><p># We utilized LaunchDarkly as our feature management platform.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QaIg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 424w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 848w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 1272w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!QaIg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6788738c-d05b-4d66-947c-1d118311cf79_218x68.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!QaIg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 424w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 848w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 1272w, https://substackcdn.com/image/fetch/$s_!QaIg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6788738c-d05b-4d66-947c-1d118311cf79_218x68.png 1456w" sizes="100vw"></picture><div></div></div></div></figure></div>]]></content:encoded></item><item><title><![CDATA[Kubernetes Probes — An Essential Ingredient to Zero Downtime.]]></title><description><![CDATA[Kubernetes Probes &#8212; An Essential Ingredient to Zero Downtime.]]></description><link>https://www.tusharmurudkar.com/p/kubernetes-probes-an-essential-ingredient-to-zero-downtime-eb16cf2cf3cf</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/kubernetes-probes-an-essential-ingredient-to-zero-downtime-eb16cf2cf3cf</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Thu, 25 Jan 2024 15:07:15 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/67903d6b-c360-4d87-a86a-5025b7a07bec_1024x552.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h3>Kubernetes Probes&#8202;&#8212;&#8202;An Essential Ingredient to Zero Downtime.</h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Eiqy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 424w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 848w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 1272w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Eiqy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Eiqy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 424w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 848w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 1272w, https://substackcdn.com/image/fetch/$s_!Eiqy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F577d0138-d3ef-4558-a847-b2c0137cfb9a_1024x552.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h3><strong>Introduction</strong>:</h3><p>One of the critical requirements of Kubernetes is to keep the deployed applications highly available and free from downtime. To achieve this, Kubernetes provides various features and strategies; probes are essential for zero downtime for your application deployment. Probes are used to check the health of an application running in a Kubernetes container. They work by periodically sending requests to the container and determining whether it responds appropriately.</p><h3><strong>Probe Types:</strong></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mVJD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 424w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 848w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 1272w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!mVJD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!mVJD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 424w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 848w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 1272w, https://substackcdn.com/image/fetch/$s_!mVJD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8623d250-29e3-45dc-8bde-57c0f95d844f_1024x341.png 1456w" sizes="100vw"></picture><div></div></div></div></figure></div><ol><li><p><strong>Startup Probe</strong>: This probe is used to determine whether the application has started successfully or not. If the startup probe fails, Kubernetes will restart the container.</p></li><li><p><strong>Readiness Probe</strong>: This probe determines whether the application is ready to serve the traffic. If the readiness probe fails, Kubernetes will stop sending traffic to the container until the readiness probe passes&nbsp;again.</p></li><li><p><strong>Liveness Probe</strong>: This probe determines whether the application is running. If the liveness probe fails, Kubernetes will kill and restart the container.</p></li></ol><p><strong>Probe Attributes:</strong></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7upx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 424w, https://substackcdn.com/image/fetch/$s_!7upx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 848w, https://substackcdn.com/image/fetch/$s_!7upx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 1272w, https://substackcdn.com/image/fetch/$s_!7upx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!7upx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!7upx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 424w, https://substackcdn.com/image/fetch/$s_!7upx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 848w, https://substackcdn.com/image/fetch/$s_!7upx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 1272w, https://substackcdn.com/image/fetch/$s_!7upx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb0de9d5-77bf-4179-8ec3-b29cf74e1b0b_1024x693.png 1456w" sizes="100vw"></picture><div></div></div></div></figure></div><h3>Probe Actions:</h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!FgoS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 424w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 848w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 1272w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!FgoS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!FgoS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 424w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 848w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 1272w, https://substackcdn.com/image/fetch/$s_!FgoS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97c2d011-555d-4711-b8b9-7c21dc1b583c_1024x440.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pfbj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 424w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 848w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 1272w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pfbj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pfbj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 424w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 848w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 1272w, https://substackcdn.com/image/fetch/$s_!Pfbj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ea763a-88ec-4b69-8de3-fef3d7514f44_1024x176.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Adding a startup probe wherever you&#8217;re using liveness and readiness probes is good practice. When setting up the liveness probe delay, it's essential to consider the initial delay time. However, it can be challenging to estimate the exact amount of time needed to initialize(startup) some applications due to unpredictable response times of certain factors. Setting a short delay(in liveness) may cause incorrect container restarts, while a long delay can lead to issues without any probe to detect&nbsp;them.</p><p>Misconfigured startup probes can easily lead to restart loops. You should measure your application&#8217;s typical startup time to determine your periodSeconds, failureThreshold, and initialDelaySeconds values.</p><p>Monitor Kub events for the pod and verify if probes are working efficiently &amp; and&nbsp;tune.</p><p><strong>Role in Zero Downtime</strong>: <br>We can use these probes to ensure that only healthy containers run and serve traffic, leading toward zero downtime. Combination of these three probes and analyzing every application for its scheduling, startup, and health thresholds &amp; translating them into probes would achieve a highly available solution.</p>]]></content:encoded></item><item><title><![CDATA[DevOps Lens of SDLC]]></title><description><![CDATA[The workflow sketches a view of SDLC from a DevOps perspective.]]></description><link>https://www.tusharmurudkar.com/p/devops-lens-of-sdlc-c5939969743e</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/devops-lens-of-sdlc-c5939969743e</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Tue, 16 Jan 2024 03:20:20 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/903d2a11-4e6d-4acd-87a5-6657db37e4f2_2424x936.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Mkxz!,w_424,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 424w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_848,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 848w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_1272,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 1272w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_1456,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Mkxz!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Mkxz!,w_424,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 424w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_848,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 848w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_1272,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 1272w, https://substackcdn.com/image/fetch/$s_!Mkxz!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59b659ac-cf31-45e8-9ab2-65e86fb1d09f_2424x936.gif 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>The workflow sketches a view of SDLC from a DevOps perspective.</p><p>The DevOps lens of SDLC provides a unique end-to-end perspective on the software development life cycle. It emphasizes the importance of collaboration and communication between development and operations teams to deliver high-quality software on&nbsp;time.</p><p><strong>Requirement analysis</strong>&#8202;&#8212;&#8202;The workflow starts with requirement analysis, where the DevOps approach involves analyzing the requirements from a holistic perspective, considering the entire software stack and the infrastructure it runs on. This helps ensure the software can operate efficiently and effectively in production and provides ways to turn off or rollback features.</p><p><strong>Continuous Integration</strong>&#8202;&#8212;&#8202;Integrating code changes frequently helps to catch any issues early and allows for faster iteration. Shift-left checks like SAST allow security to play a role early in the&nbsp;cycle.</p><p><strong>Continuous delivery&#8202;</strong>&#8212;&#8202;Prescribes automating the entire software delivery process, from builds and tests to deployment and release. This helps to ensure that requirement changes are delivered quickly and reliably to end&nbsp;users.</p><p><strong>Observability&#8202;&#8212;&#8202;</strong>Involves monitoring and analyzing the performance via service level indicators of the software and infrastructure in real-time. Systematic monitoring helps to analyze issues before they escalate into problems. Observability helps in visualizing that<strong> </strong>the<strong> </strong>software is running efficiently and effectively.</p><p>The DevOps perspective of SDLC emphasizes collaboration and automation to deliver high-quality software continuously and meet end-user&nbsp;needs.</p>]]></content:encoded></item><item><title><![CDATA[Infrastructure Pipeline]]></title><description><![CDATA[The workflow sketches the end-to-end Infra build and deployment pipeline, including quality gates(detectors & risks).]]></description><link>https://www.tusharmurudkar.com/p/devops-infrastructure-pipeline-beab47e7b876</link><guid isPermaLink="false">https://www.tusharmurudkar.com/p/devops-infrastructure-pipeline-beab47e7b876</guid><dc:creator><![CDATA[Tushar Murudkar]]></dc:creator><pubDate>Thu, 11 Jan 2024 04:27:11 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/360ae550-bec3-4afb-a632-1dc0c6532f61_1024x340.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RMJ1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 424w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 848w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 1272w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!RMJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!RMJ1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 424w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 848w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 1272w, https://substackcdn.com/image/fetch/$s_!RMJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a4abede-35d2-45b4-994d-f859d4e525b4_1024x340.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>The workflow sketches the end-to-end Infra build and deployment pipeline, including quality gates(detectors &amp;&nbsp;risks).</p><p><strong>Branching:</strong> As a developer, it is recommended to have a local branch, which is essentially a copy of the mainline branch. This local branch should be used for making isolated changes and should be short-lived. While the local branch has no constraints, the developer is responsible for ensuring that the branch is always synced with the mainline to avoid merge conflicts.</p><p>The mainline branch, or the trunk, is used to assemble and build deployable artifacts. Every merge in the mainline has the potential to be released to any given environment. Therefore, it is vital to keep the mainline deployable. In other words, we must ensure that the merge has no issues. To achieve this, appropriate quality gates must be built to keep the mainline continuously deployable &#128274;. The mainline remains locked until we assess the risk of the&nbsp;change.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QWJp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 424w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 848w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 1272w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!QWJp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!QWJp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 424w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 848w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 1272w, https://substackcdn.com/image/fetch/$s_!QWJp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F442f60b2-48fc-4e1c-bdf9-67386404c462_1024x470.png 1456w" sizes="100vw"></picture><div></div></div></div></figure></div><p><strong>Modules&#8202;&#8212;&#8202;</strong>Infrastructure is coded in Terraform modules; modules are designed based on blast radius and applicability; it is essentially to understand the scope of the modules; the scope helps define policies.</p><p><strong>Infra Config Management&#8202;&#8212;&#8202;</strong>For successful zero-touch deployments, appropriate configurations must be supplied to the deployment framework; a central configuration management system can store configurations.</p><p><strong>Risk Meter&#8202;&#8212;&#8202;</strong>A risk meter would help assess the risk we carry if the specific version of the module is deployed; this would help us determine additional actions beyond certain risks. With evolving CI/CD, lean releases translate to lower risk. It determines if a standard set for quality is met; if yes, we move to the next deployment phase; if not, appropriate feedback is delivered back to the team on the expectations and gaps to be&nbsp;closed.</p><p><strong>Detectors&#8202;&#8212;&#8202;</strong>A Detector performs analysis, gathers details of changes,<strong> </strong>and maps the analysis independently for the change set. Every detector carries a&nbsp;weight.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!siCo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 424w, https://substackcdn.com/image/fetch/$s_!siCo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 848w, https://substackcdn.com/image/fetch/$s_!siCo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 1272w, https://substackcdn.com/image/fetch/$s_!siCo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!siCo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ab886d24-b892-444c-a092-a563a5cb44ed_976x616.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!siCo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 424w, https://substackcdn.com/image/fetch/$s_!siCo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 848w, https://substackcdn.com/image/fetch/$s_!siCo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 1272w, https://substackcdn.com/image/fetch/$s_!siCo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab886d24-b892-444c-a092-a563a5cb44ed_976x616.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><strong>Approve Plan&#8202;&#8212;&#8202;</strong>The objective of this gate is to generate a doc suggesting what change set is getting applied; the artifact produced by this gate needs to be <strong>carefully reviewed</strong>. This can be auto-reviewed depending on the risks meter in some environments, whereas we would need a more careful eye in&nbsp;others.</p><p><strong>Deploy</strong>&#8202;&#8212;&#8202;Deploy of infra would work against a planned artifact generated in the &#8220;plan phase&#8221;; the plan would have versions similar to&nbsp;builds.</p><p>Meanwhile, Deploy would also validate( which is <strong>intrinsic </strong>to IaC tools ) the creation of infra; for example, if an S3 bucket needs creation, Terraform would validate if an S3 bucket is&nbsp;created.</p><p>Monitoring tools can be queried to inspect the health of applications.</p><p><strong>Release Manifests&#8202;&#8212;&#8202;</strong>While each merge to mainline can be released to a different set of env, we assemble a list of modules that changed in a manifest to deployed together; this would benefit us from integration with pipeline solutions.</p><p>Early feedback &amp; velocity of reporting back failures are vital&#8202;&#8212;&#8202;risk meter returns failed assessment and logs metrics and keep mainline locked till issues are resolved.</p><p>The infrastructure pipeline is an essential aspect of any software development process. The use of quality gates, risk assessment, and detectors helps to identify issues before they become problems, while the use of monitoring tools and release manifests ensures that the infrastructure is running smoothly. Ultimately, by following these guidelines, developers can ensure that their infrastructure is robust and meets the needs of their organization.</p>]]></content:encoded></item></channel></rss>