Techsperiments

Getting Your First DevOps Job Is Harder Now — Here's What I Tell My Students

2026-09-24T10:27:15.970-07:00

I've been working in DevOps for around 14 years, and I've also been teaching DevOps and Cloud subjects at universities for the last 4–5 years.

One thing I've noticed recently: getting your first DevOps job seems much harder than it was even 2–3 years ago.

A few years back, students and juniors would send me their CVs and I'd usually have something useful to suggest. Many of them would at least get interview opportunities.

Today, the market isn't great, and on top of that, everyone has access to AI-generated CVs, cover letters, LinkedIn profiles, and so on. When hundreds of applications all look polished, it becomes harder for recruiters to figure out who actually knows their stuff.

So here are a few things I keep telling my students.

1. Don't Choose DevOps Because You Think You Can Avoid Coding

I've heard this quite a few times:

"I'm not interested in coding, so I'm thinking of DevOps."

That's the wrong reason to get into DevOps.

You'll write scripts, automation, YAML, pipelines, Infrastructure as Code, debugging tools, and sometimes proper application code. More importantly, you need to enjoy investigating systems and figuring out why something isn't working.

If you hate both coding and troubleshooting systems, DevOps probably isn't the escape route you're looking for.

2. DevOps Comes With Responsibility

Remember Spider-Man?

"With great power comes great responsibility."

That applies surprisingly well to DevOps.

The infrastructure you're managing might be running someone's entire business. Sometimes you're having dinner with your family and an alert appears on your phone. Sometimes something breaks at an odd hour. Sometimes everyone else is enjoying a holiday while you're wondering why a production node suddenly decided it no longer wants to participate in society.

Not every DevOps job has terrible on-call expectations, of course. But ownership and accountability are part of the profession. Know what you're signing up for.

3. Build a GitHub History, Not Just a GitHub Account

When I started 13–14 years ago, having an impressive GitHub profile wasn't nearly as important for many infrastructure roles.

Today, if you're a student with no professional experience, your GitHub can become evidence that you actually build things.

Don't obsess over making every project impressive. First build the habit of creating things regularly.

Write scripts
Build Docker images
Automate something with Python
Create Terraform modules
Write an Ansible role
Deploy Kubernetes applications
Build CI/CD pipelines

And commit your work.

I actually suggest turning this into a game.

When I was in school, some friends and I wanted to improve our English. We made a game where we had to speak English all the time — if someone caught you speaking another language, you lost points. We did this for years, and it genuinely helped.

Do something similar with GitHub.

Find a friend. Every day you commit something meaningful, you get a point. First person to 100 wins. Then reset and start again.

You'll miss days. You'll get bored. That's fine.

But after a year, compare your GitHub profile with the one you had before.

4. Stop Sending the Same Application to 500 Companies

AI has made mass-applying ridiculously easy — which also means everyone can do it.

I'd rather spend proper time on 10 jobs I genuinely want than blindly apply to hundreds with the exact same CV and a generic cover letter.

Read the job description. Understand what they're actually asking for. Then explain why your experience and projects match those specific requirements.

Quality over volume, every time.

5. Don't Just Ask AI to Write the Entire Cover Letter

Use AI to proofread it, improve it, or organize your thoughts if you want.

But put yourself into it.

Mention something specific from the job description. Explain a relevant project. Explain why you're interested in this particular role.

It's okay if your writing isn't perfect. I'd personally rather read something that sounds like a real person than another perfectly formatted:

"I am thrilled to apply for the exciting opportunity to leverage my passion for innovative cloud-native solutions..."

You know the type.

6. Add Something Human to Your Application

One thing worth experimenting with: a short Loom or video introduction.

Not 10 minutes. 60–120 seconds.

Something like: "I know you're probably reviewing a lot of applications, so instead of making you read another long introduction, here's a 90-second video about who I am, what I've built, and why I'm interested."

Whether recruiters watch it is another question. But at least you've given them another way to evaluate you — something that's genuinely hard to fake with AI.

7. Make It Clear You're Willing to Prove Your Skills

When appropriate, mention that you're happy to go through a technical evaluation or practical exercise if that's part of their process.

For someone without much professional experience, being able to show what you can do can be far more useful than another paragraph claiming you're "passionate about DevOps."

None of these are magic tricks.

They won't guarantee an interview, and they definitely won't guarantee a job.

But the way people apply for jobs has changed dramatically with AI. If everyone's CV looks good, simply having a good-looking CV isn't much of a differentiator anymore.

Build things. Show your work. Develop consistency. Personalize your applications. Give people evidence that there's an actual engineer behind the PDF.

Curious what others think — especially people involved in DevOps hiring.

Has the way you screen junior DevOps candidates changed over the last 2–3 years?

MeasureCamp Delhi: Where I Became a Speaker Without Meaning To

2026-09-24T10:24:55.122-07:00

Last week I went to MeasureCamp Delhi 2026. This was their 200th event and indeed a special one. Before going, I had a simple picture in my head. Rows of chairs. A stage. Some speakers talking. Us nodding along. Chai/Coffee in between. Maybe a goodie bag at the end. People looking to sell their product every now and then. Group of students clicking selfies all over. The standard conference deal we have all attended at least fifty times in our careers.

I was wrong indeed.

MeasureCamp is something called an "unconference." When I heard the word for the first time, I thought maybe they forgot to plan the conference part. It was on purpose though. There is no fixed agenda. The event starts with a short intro, sponsors get their thank you, and then they show you a big blank board with time slots and room numbers. That board is the real hero of the day. Here is a picture:

Speaker/Event Session Board

I remember pre-covid days, I did something similar. As a meetup organizer at DevOps Pune and after getting some inspiration Seattle meetup groups, I started working on Leancoffee format of meetups since we were getting lot of marketing pitches back then. Here is what Leancoffee is. Definition copies as it is from their site.

Lean Coffee is a structured, but agenda-less meeting. Participants gather, build an agenda, and begin talking. Conversations are directed and productive because the agenda for the meeting was democratically generated.

So I now assumed this Unconference was probably the same, with a new name. But again, I was wrong.

Here is what happens. If you want to speak on something, you write it on a small card, stick it on the board, and boom, you are now a speaker. No approval committee. No forms. No "we will get back to you." If people like your topic, they walk in. If they don't, you get a very empty room and a good lesson in humility.

Only one rule was strict. No pitching your company. Sponsors already pay for that privilege, so the rest of us just talk shop, not sales. Once that rule was set, every conversation in the building felt honest. Nobody was secretly selling you a subscription while pretending to chat.

Wow! It sounded good and scary at the same time. Good, because

This was different. A different learning experience is always good.
No marketing pitches. As an organizer, I know how bad this can get.
I could choose who and what I want to listen to.
So many good topics being displayed and so much to learn.

Scary because:

Embarrassment of not having any audience if you end up becoming a speaker.
Fear of missing out on good speakers if multiple good events go in parallel.

My colleague Rohan Deshpande and I went there just to attend, representing Convert.com, nothing more. No plan to speak, no slides, nothing prepared. But after the opening session, we looked at the board, looked at each other, and thought, why not? We put up a session on how AI has been sneaking into troubleshooting and internal workflows at CRO and A/B testing companies. Basically the stuff we have been quietly doing for months.

We kept it in a very informal discussion format. In fact, I sat down with the attendees just facing them. More like a group chat than a lecture. About 15 to 20 people joined. And the questions came fast. Do you actually trust AI's calls or are you just being brave? Where is this going for testing platforms? Personalization, traffic tracking, integrations with tools people already use daily. People were genuinely curious, not just polite.

Best part of the whole day, honestly. We did not plan to speak, did not prepare a single slide, and still walked away with speaker mementos. Somehow that felt more earned than if we had rehearsed it for a week.

This time the turnout was around 120 plus people. I think they expected more, but Delhi was also hosting the BRICS summit that same day, so a lot of people probably got stuck in traffic or security barricades instead of analytics discussions. Still, seven breakout rooms ran side by side the whole day, and people moved between rooms freely, following their curiosity, not any fixed schedule.

I also sat in on a few sessions myself. One on attribution problems in ads and CRO. One on moving from GA4 to Amplitude. One on data governance and AI. And one with a title that stuck with me, "How Data Lie." Every session was run by someone actually doing the work, not someone presenting a fancy case study made for likes.

But if I am honest, no single session was the best part. The people were. I met folks from Amplitude, founders running their own analytics platforms, and CRO experts who have worked with some really big consumer brands. Normally this kind of networking takes weeks of cold emails and LinkedIn requests that nobody replies to. Here it happened in five minutes near the coffee counter.

This was also the 200th MeasureCamp Delhi event, which is no small number. A big thank you to the organizers and volunteers who pulled this off. Running an unconference sounds easy on paper, "just let people talk," but making that chaos actually work smoothly takes real effort. They did a solid job. Thank you so much for all the work. Tanmay Garg Priya Dhiman Anuj B. Abhishek Deswal Sorry if I missed anyone here.

If you work anywhere near analytics, data, or experimentation, and you have never been to a MeasureCamp, go once. Go with zero expectations. That is honestly the whole point.

The CV Habit That Accidentally Saved My Job

2026-09-21T00:16:00.828-07:00

Every engineer knows this problem. You switch jobs after a year and a half, sit down to update your CV, and go completely blank. What did I even do all this time? You worked hard, you know you did, but the specifics have evaporated.

That's exactly what happened to me after my first job switch. I worked in a small company, so it wasn't like I had ten different clients with ten different stories to tell. Just one job, multiple responsibilities, and somehow none of it stuck in memory when I needed it most.

So I built a habit. Nothing fancy. Just an Excel sheet. Date, what I worked on that day, two or three lines, status: in progress or done. That's it. No fancy tool, no elaborate system. Just discipline.

It worked exactly as intended. Next time I job-hopped, I knew precisely what I'd built, what I'd learned, what to put on the CV. Simple win.

Then one day, the habit paid off in a way I never expected.

I was the only other DevOps engineer besides my boss, who lived in a different time zone. Over a phone call, he asked me to delete a list of S3 buckets, unused, just sitting there costing money. I did it. And because I'd made note-taking a daily habit by then, I logged it too: deleted these buckets, told they were unimportant.

A few weeks later, the engineering team noticed a monthly report had stopped generating. Nobody knew why. My boss dug in during his hours and found that an important S3 bucket was gone. No CloudTrail back then, no audit log, no way to trace who did what. And it wasn't just the two of us with AWS access, several people in the company had it.

When I came online, he asked me to investigate. I went straight to my Excel sheet. There it was, dated weeks earlier: deleted those buckets, was told to.

That log ended the mystery in minutes. We knew exactly what happened, when, and why. It was human error, not a hack, not malice, just a bucket that should never have been on that delete list. My boss also gave me an important lesson that day: being told to do something doesn't mean you skip verifying it yourself. As a DevOps engineer, that judgment call is on you too.

But the real hero of that day was a habit I'd started for a completely unrelated reason: keeping my CV honest.

I kept that habit for years after. What began as a fix for a bad memory turned into something closer to a safety net. You don't always know which note you write today will matter six months from now. Sometimes it's just for your CV. Sometimes it's the only thing standing between "human error" and "we have no idea what happened."

So here's my question to you: do you keep any kind of daily work log? Not for performance reviews, not because someone told you to, just for yourself. And if you don't, after reading this, would you start?

The Day I Missed My First IPL Match (And Learned the Most Important Lesson of My Career)

2026-09-20T23:40:29.090-07:00

I still remember the date almost. Not exactly, because it's been years. But I remember it was a Friday. I remember it was IPL season. And I remember I had a ticket in my pocket for my first ever live cricket match.

I was an intern back then. QA team. I'd been handed a project because I "liked Linux." That's it. That was my entire qualification. I'd played around with LAMP stacks, set up WordPress a few times, run enough commands to feel dangerous. So when they needed someone to set up a QA environment for a PHP application, my name came up.

Simple task, they said. Get the environment running. Make it accessible. Client's in Australia, they need to see it live if things go wrong.

Nothing about that sentence was simple.

The Setup From Hell

I started with VirtualBox, because that's what I knew. My boss shut that down immediately. Had to be AWS. Public URL. Client access.

New OS setup. New cloud platform. New tool called Composer that I'd never touched. I did what every self-respecting engineer with no documentation does: I Googled my way through it.

Every install hit a wall. Every wall meant a new Jira comment. Developers would reply with fixes: use this version, open this port, set the database up like this. I'd write it all down on a random notepad. Spin up a new instance. Try again. Snapshot everything because god forbid I lose progress. Scrap the instance. Repeat.

Three weeks of this. Three weeks of two steps forward, one step into a wall.

The Comment That Ruined My Friday

Mid-day, that Friday, I saw it. A comment on the ticket from the client. Something like "not happy with the delay."

As an intern, that hits different. You don't have the thick skin yet. I read it and felt genuinely sick.

I made a decision right there: I'll fix this today. Match or no match.

6 PM came and went. 6:30 too. Office was emptying out. I was still at my desk, notepad open, trying combination number forty of the same setup.

That's when my boss's boss walked past. Not my direct manager, the guy above him. Saw an intern still sitting there at that hour and got curious enough to ask what was going on.

I showed him the comment. Told him I couldn't crack the setup.

The Question That Changed Everything

He sat down next to me. We started fresh, new instance, working through it together. At some point he noticed my notepad, full of scribbled notes from old chats and Jira threads.

"What's this? Who gave you this?"

I explained. Historical Jira comments, cobbled together over three weeks.

Then he asked the question I should have asked myself on day one:

"Do you not have documentation for this setup? Surely someone who built this environment before has notes. Did you ever ask?"

I hadn't. It genuinely hadn't occurred to me that documentation was something you could ask for. I thought my job was to figure it out, silently, on my own, because asking felt like admitting I wasn't good enough.

He looked at me and said something I've never forgotten: learn to say no, or learn to ask for help, when you need it. You're an intern. You're not the highest paid person in the room. Nobody expects you to know everything. But dragging on for three weeks because you were too afraid to ask a question? That helps nobody.

The Email

We tried the setup a couple more times together. Then we did something I hadn't thought to do in three weeks: we wrote an honest email to the client. Explained the real story. No documentation had ever been shared. No process existed. That's why this dragged on, not because anyone was slacking.

We left the office around 10 PM. I walked out past people who'd already watched their match and were heading home. My ticket was still sitting unused in my bag.

I won't lie, I was gutted about the match. But I was also carrying something heavier than disappointment. I'd learned something that had nothing to do with Linux or AWS or Composer.

The Payoff

Next morning, a full documentation package landed in my inbox from the ops team on the other side. Turned out it existed all along. Nobody had thought to hand it to an intern, and I'd never thought to ask.

With that doc in hand, I set up the entire environment in one shot. Automated it with a shell script right after, just so nobody after me would have to go through what I did.

Why This Still Matters

That lesson didn't stay in that internship. Years later, at a different company, a server went down and the CEO's email landed straight in my inbox asking what DevOps was doing about it. I told him straight: DevOps isn't magic. Some things need process, not miracles. That confidence to say it plainly came from that Friday night.

Since then, I've made it a habit. If I know the answer, I say yes upfront. If I don't, I say so, without the fear of looking incapable. That fear is expensive. It cost me three weeks and a cricket match. It doesn't need to cost you anything if you learn the lesson early.

So here's the takeaway, for every intern, every junior engineer, every person new to a team: ask for the documentation. Ask for help. Say no when you need to. Nobody's going to think less of you for it. If anything, they'll respect you more for not wasting three weeks pretending you have it figured out.

I missed a cricket match to learn that. You don't have to.

Why Nagios Still Has My Heart (Even Though I Use Prometheus Now)

2026-09-17T22:32:24.500-07:00

Let me take you back to 2012. I was young, curious, and had just been handed the keys to a Nagios server. One server, a handful of clients. Nothing fancy. Just config files, cron-like discipline, and a dashboard I had to check every single morning like it was my first cup of chai.

That daily ritual taught me something no course ever could. Logging in, scanning the trends, noticing when something looked "off" before it became a full-blown crisis. Disk filling up because some user decided the server was their personal Dropbox? Caught it. CPU spiking for reasons nobody could explain? Went hunting for the process before it became a 2 AM phone call. This wasn't fancy AIOps. This was just paying attention, the old-fashioned way.

Every new server that got provisioned, I added to Nagios myself. Every new process, I wrote a check for it myself. There was no abstraction between me and the machine. If something broke, I knew exactly why, because I built the thing that was watching it.

Then I moved organizations, and Nagios grew up with me. Suddenly it wasn't one server and a few clients, it was a clustered setup monitoring 5,000 servers. A team of 10 DevOps folks, SaaS tools watching Nagios itself (yes, we had to monitor the monitor), and Icinga entering the picture as the better-looking cousin of Nagios. That's when things got serious.

This is also where I learned Nagios could do more than just "is the server alive." We started writing bash scripts and plugging them in to parse logs, actual business logs, to generate reports for sales and marketing. Number of deals closed, tracked through log monitoring. When those reports failed, guess who had to debug the script at odd hours. Character building, as they say.

I last touched Nagios seriously about two years ago, mostly for bare metal setups. Since then, I've worked with Prometheus, New Relic, Zabbix, the works. And they're good. Genuinely better dashboards, better scaling, better everything on paper.

But here's the thing nobody tells you about Nagios: it keeps you rooted to the system. You're not looking at pretty graphs from a distance. You're inside the OS, understanding processes, permissions, disk layouts, the works. When I automated Nagios setup with Ansible and later with Fabric in Python, things broke constantly. And every time they did, it was my old shell scripting instincts and my bare-metal Nagios days that bailed me out. That knowledge doesn't come from reading a Prometheus exporter's YAML file. It comes from getting your hands dirty.

I recently came across a blog from another engineer reminiscing about Nagios, and it hit me that I'm not alone. There's a whole tribe of "Nagios oldies" out there who get it. We know the newer tools are shinier. We use them daily. But there's something about Nagios that built our fundamentals in a way nothing else quite has.

Better monitoring tools may exist. But Nagios will always be the one that taught me how to actually understand a server, not just watch it.

Disney Did It With Cartoons. My AI Did It With My Kid's Bedtime Story.

2026-08-24T23:51:00.000-07:00

I saw a video a while back about old Disney animation and how they fooled viewers in the last couple of years. Same character movements, reused frame by frame, across completely different characters and different movies. Different faces, different worlds, same underlying motion. It worked for years. Nobody noticed, or nobody cared, because the backgrounds and story kept changing enough to sell it. Here is the video I am talking about. This was deliberate for many reasons as you read more about it, you will know. Some reasons, I understood were, nostalgia, saving time or maybe laziness too.

I didn't think much of it until I saw the same thing happening with AI.

I was working on generating some images recently. I kept feeding new context, new prompts, expecting newer results. The first couple of images were just fine. Then I started noticing a pattern of same poses, same layouts, same compositions repeating underneath a different backdrop. Not identical, but close enough that it felt like the AI found something that worked and decided to stop being creative.

I told myself, okay, maybe that's fine for images. Then I tested it on something that mattered more.

I use AI sometimes to teach certain concepts to my kid. I asked AI to write a short story. It was about living and nonliving things. It came out lovely. She enjoyed it, I enjoyed it. All good, right?

The next day, I asked for a story about vertebrates and invertebrates. Same structure. Same pattern with just a bit of find and replace may be 😀. Practically the same story with a few words swapped out. If I had outsourced that to a person and gotten this back, I wouldn't have accepted it. And here's the thing, my kid is five. Even she would note this eventually. Kids get bored of the same story wearing a different costume faster than we think.

Now this is not about things impacting my personal work. The part that actually worries me is documentation.

I use AI heavily for documentation and writing. And I've caught the pattern there too. Same structure repeating article after article, same phrasing patterns, same rhythm, regardless of what the topic actually needs. It's probably efficient. But it's also lazy. And I worry people will lean onto this instead of catching it, hand AI a rough context, ask for a full end to end document, and publish whatever comes back without questioning why every article somehow reads the same.

Here is what I found being the technical reason to be. This comes down to how models pick outputs. They default to the highest-probability path unless you push them elsewhere, so once a structure works well in one response, it becomes the easy option for the next, especially in the same session where earlier context keeps shaping what comes after. Lower temperature and conservative sampling, which most tools default to for reliability, make this worse on purpose. It's not laziness the way we mean it. it's optimization doing exactly its job: minimize risk, maximize consistency. Creativity was never the objective. Which is exactly why the paragraph-by-paragraph, keep-pushing-back approach works, it forces the model off the safe path it would otherwise default to.

When I see this happening, here's what works for me instead. Don't ask for the whole thing to be written or documented at once. Go paragraph by paragraph. Feed fresh context deliberately every time. Question, Argue and provide newer/better context every time. And at any time something feels repeated, say so. Tell it directly. That nudge matters more than people think, it's not just fixing one document, it's teaching the model what you don't want. Some would comment and ask to create a new session for improvement here. I agree with it, but how I and sometimes even you may work, sometimes we need the session memory to exist for various reasons, so we use the same sessions for a long time.

Disney may have eventually moved past reusing the same animation cycles. Or even if they re-used some, they had some valid reason like ‘nostalgia’. They could have even stopped doing this because audiences quietly noticed this and started expecting more, and also maybe that the reused motion stopped being good enough. I think we're also at the exact same point with AI generated content, whether that's documentation, images, or bedtime stories. It'll get away with repeating itself for a while. Until the audience notices. And audiences always notice eventually, even five year old ones will.

So if you're using AI for content, don't just accept the first draft's texture as final. Push it. Ask it to genuinely think differently, not just swap a few nouns. Creativity isn't AI's job to protect. It's ours.

#AIAgents #Documentation #ContentCreation #DevOps #CreativeWork

✏️ drafted with ai assist, because practicing what I preach.

Documentation Drifts. Here's How to Catch It.

2026-08-05T23:47:00.000-07:00

"Documenting the documentation."

Yes, I know how that sounds. Documentation about documentation about documentation. Somewhere an intern is crying. Actually that's the phrase that hit me mid-conversation yesterday.

In DevOps we talk about monitoring the monitoring system. You don't just set up alerts and walk away. You watch the watcher too, because if your monitoring breaks silently, you find out the hard way. We handle it mostly by using a 3rd party monitoring Saas tool or a completely different monitoring tool that is on a different infra to monitor our monitoring. Many a times, we also setup multiple monitoring tools to monitor each other.

Turns out documentation has the exact same blind spot, that I realized yesterday.

Here's the problem. Documentation itself isn't static, but the system around it should be. A proper guideline of how the docs should look consistent across.

Think about what happens with two or three writers on the same knowledge base.

One person is deeply technical, every article turns into a dense reference doc.
Another writes for humans first, simple language, easy to follow, but a senior engineer might find it too light.
A third writes in a fun, engaging tone that people love reading but that doesn't actually answer the question they came for.

None of these people are wrong. But put them in the same wiki with no shared rules, and the documentation stops looking like one system. It may end up looking like three different people arguing.

That's what documenting the documentation fixes. Not the content. The structure around the content. Heading formats. Whether an index is required. How many bullets before it's too many. Screenshot rules, including the boring but important one, don't reveal internal or customer details in a screenshot(Nothing says 'security incident' like a screenshot where someone forgot to close the tab with the customer PI sitting right there). Whether emojis are fine or banned. Which kinds of diagrams are expected. Small decisions, but decided once, so nobody has to reinvent them article by article. Is humor ok. How much humor is ok? Procedure to build the context, etc.

Two things break without this:

When someone leaves the team, whoever replaces them has nothing to stand on. No rulebook, no shared style, just vibes and old examples to guess from.
Even with the rules written down, you can't assume people will keep following them. Rules you hand someone on day one quietly erode by month six. Rules handed over on day one have a shelf life shorter than office milk.

So the system needs an audit layer too. Not a one-time handoff. A recurring check.

This is where it gets easier than it used to be. You write your rules once, in something like a CLAUDE.md, plain and specific. Then you let AI do the boring, repetitive part; scan every new article, and periodically scan the old ones too, and flag anything drifting from the standard. Wrong heading structure, missing index, a screenshot that shouldn't be there, tone that's drifted too casual or too dense. The AI doesn't decide what's true. It checks what's consistent. Humans still review before anything changes.

Monitoring the monitoring system caught on in DevOps because someone finally said the obvious thing out loud, watch your watchers. Documentation needs its own version of that sentence.

Document the documentation. Then audit it like you mean it.

#Documentation #DevOps #KnowledgeManagement #AIAgents #TeamProcess

✏️ drafted with ai assist, because practicing what I preach.

CRO: The Career Stream Nobody Told You About

2026-07-27T02:16:00.000-07:00

When I ask any of my student what they want to be? I mostly get one of the five answers: Developer, QA, Project Manager, Business Analyst, and now DevOps too.

Nobody says “CRO”, because nobody knows it exists. A few years ago, this was the state with DevOps too :)

What Is CRO?

Conversion Rate Optimization. The job to figure out why people visit a website and leave without buying, signing up, or clicking the button, and then fixing it. Not by guessing but by testing, perhaps even by some coding.

Here’s what you already know.

Dev builds the product.
QA makes sure it doesn’t break.
BA figures out what the business needs.
CRO figures out why real humans aren’t doing what the business hoped, and runs experiments to fix it. With lots of tools again.

Tech, psychology, and data, all in one job.

Classic one: “But I’m Not Great at Coding”

If you don’t want to code at all. Stop reading here. Some coding will always be needed in all tech. But Relax! You don’t need DSA or LeetCode for this. What actually helps here is:

Basic HTML/CSS, enough to know what “move the button above the fold” means. You learn this mostly in the initial classes, of course I am making some assumptions here.
A little JavaScript, enough to talk to developers and state some requirement. Better if you could do it yourself.
Comfort with numbers, enough to know 40% improvement means nothing if only 20 people saw it. Some exposure to good tools like Google Analytics.

That’s it. CRO rewards curiosity about human behaviour, not coding depth.

A Day in CRO

Look at heatmaps. Watch session recordings of real users struggling. Check analytics reports. Understand human behavior. Form a hypothesis. Run an A/B test. Wait patiently (no declaring winners on Day 2). Read the results honestly. Tell a non-technical team what it all means.

Partly a detective work, partly experiment design and most part is communication.

Why Should you do This

All freshers applying for a dev role are competing with a thousand others who did the same DSA course and built the same to-do app. CRO is comparatively empty. Businesses don’t jsut need hi-tech people, they also need a mix of both who get the tech and the human side of a website, and almost nobody of your age is positioning for it yet.

Bonus: skills from Dev, QA, BA, and DevOps all transfer. So, you are not just starting from zero. You are switching lanes.

CRO in India, Right Now

The most exciting part for Indian students is here. India is now the world’s second-largest home for A/B testing software companies, behind only the US, according to Tracxn data. Some of the well-known tools in this space are all built by Indian founders. This isn’t a niche American thing you’re late to. It’s actively being built here, right now, with real jobs behind it.

And it’s moving past “just buy a tool” into “build a culture around it,” with practitioner meetups and roundtables popping up in Indian cities. Translation: the ecosystem is young enough that showing up early actually means something. Some fo the recent events around it is something I keep posting about and you will hear more about it from me.

Reference: Best A/B Testing Tools in India

Start This Week

Install Microsoft Clarity (free) on any personal project and just watch how people use it
Try Convert.com’s 15-day free trial to see what an actual A/B testing tool looks like from the inside
Learn the four experiment types: A/A, A/B, split URL, multivariate. At least start with the easiest ones i.e. A/A and A/B.
Write one hypothesis: “If I change X, then Y will happen, because Z”. This is important. If you master this, you know how the product and its users/buyers interact.
That’s pretty much it. (Of course more but I don’t want to scare you :) )

I teach DevOps and cloud subjects to students, and work at Convert.com. Curious about CRO? Drop a comment.

Credential bleed: what happens when your MCP server trusts the wrong thing

2026-07-26T23:31:30.731-07:00

I set up an MCP server for MySQL last month. Took maybe twenty minutes. FastMCP, running on port 9100, exposing a handful of database tools to any agent that connected to it.

Then I asked myself a question I should've asked before writing a single line of code. What exactly is this server trusting, and who gave it permission to trust that?

Turns out, the answer was: whatever connected to it.

Here's the thing about MCP servers that nobody warns you about when you're following a quickstart guide. The server doesn't inherently know the difference between "an agent I authorized to run read queries" and "an agent that happened to find my endpoint." If your auth model is weak or missing, every tool you expose is available to every connection that shows up. Credentials don't need to be stolen. They just need to be reachable.

I built this specific setup as a demo, deliberately, to see how bad it could actually get. Not theoretical. A real MySQL MCP server, real credential handling, real tools exposed. And what I found is that credential bleed isn't some exotic attack. its the default state of a lot of MCP setups people are shipping right now, without realizing it.

Think about what an MCP server actually does. It sits between an agent and a real system, translating natural language intent into real actions. Real database queries. Real Terraform applies. Real file writes. Somewhere in that translation layer, credentials have to live. And if that layer trusts the connection instead of the identity, you don't have an access control system. You have an open door with a "please don't" sign on it.

This is the same failure mode I saw with a Terraform agent demo I built separately. An ambiguous prompt walked straight through terraform_plan into terraform_apply and created a real S3 bucket in AWS. Nobody meant for that to happen. The agent didn't misbehave. The boundary that was supposed to stop it simply wasn't there. Same root cause, different door.

The uncomfortable part is that this scales with adoption, not against it. The more useful MCP servers get, the more tools get exposed, the more valuable the thing sitting behind that unguarded connection becomes. A vulnerable server nobody uses is a curiosity. A vulnerable server your whole team routes agent traffic through is a real incident waiting on a slow Tuesday.

None of this means don't build MCP servers. It means stop treating the connection as the identity. An agent proving it can reach your server is not the same as an agent proving it should be trusted with what's behind it. Ephemeral credentials, scoped tools, intent validation before execution, these aren't nice-to-haves you add later. They're the actual product. Everything else is just a demo that works until it doesn't.

I'll probably write up the full technical breakdown of the MySQL demo separately. For now, if you're standing up an MCP server this week, ask the boring question first. Not "does it work." Trust what, exactly.

I built a pipeline to auto-post my Telegram channel everywhere. Then I stopped.

2026-07-26T01:42:21.088-07:00

Auto-posting my Telegram channel posts everywhere sounded like a dream. Then I actually built it.

The plan was simple. Write one post on my Telegram channel (@securedevops). Let a pipeline pick it up. Reformat it for each platform. Push it out to LinkedIn, Twitter, Instagram, Facebook, Dev.to. Zero manual work after hitting publish once.

I built it in n8n. A webhook catches new posts from the channel. A dedup check makes sure nothing gets posted twice. Then a handoff point where ChatGPT would rewrite the post per platform, and push it live everywhere.

Sounds clean right? It mostly worked. Until it didn't.

Here's where it broke. I was using ngrok's free tier to expose the webhook. Every time my machine restarted, the URL changed. Which meant every time I restarted, I had to manually go re-register the new URL with Telegram. A pipeline that's supposed to save me time now needed me to babysit it every morning. Not exactly "fully automated."

And the more embarrassing part. the actual multi-platform posting, the part that was supposed to save the most time, never got finished. I built the capture layer. I built the dedup layer. I stopped right before the part that mattered most.

Thinking about it now, this wasn't really an automation failure. It was a sequencing mistake. I built the fun part first. Webhooks, dedup logic, the architecture diagram in my head looking clean. The boring part, a stable public URL, I left for later. Later never came.

This keeps happening in infra work and I don't think its a coincidence. The AI reformatting step, the multi-platform posting, that was the exciting 20%. Getting a reliable endpoint that survives a restart was the unglamorous 80%. Everyone wants to build the 20%. Almost nobody wants to build the 80%. But the 80% is usually what decides if the thing actually runs unattended or not.

So the pipeline still sits there. Half built. Working, technically, if I remember to re-register the URL by hand.

Anyone else have an automation project that stalled right at the boring part? curious if this is a me problem or a universal one.

#DevOps #Automation #n8n #BuildInPublic #AIAgents

Part 3: How to Decide Which CRO Test to Run First

2026-07-24T02:12:00.000-07:00

Because the loudest idea in the room is not always the best experiment.

In the last post, I wrote about the main experiment types: A/A, A/B, Split URL, and multivariate testing. I also introduced the formula that quietly changed how I think about experimentation:

Success = Chance × Frequency

Once that clicked, I assumed the next step would be simple.

Pick an idea > Run a test > Measure it > Repeat.

Then I actually tried to do it. Because once people around you start thinking in CRO terms, ideas come from everywhere.

The designer wants to test the hero image. The founder wants to rewrite the pricing page because “it does not feel premium enough.” Someone from sales heard that a competitor changed their CTA and now wants to copy it immediately. Someone read a blog at midnight and suddenly wants to add a chatbot, a quiz, a floating banner, and possibly a dancing mascot.

Before you know it, your experiment backlog looks like a Jira board after a chaotic sprint planning session.

Lots of tickets.
Very little structure.
Everyone convinced their idea is the one that will change the company’s destiny.

This is where prioritization comes in. Not because prioritization makes you smarter, but because it protects the experimentation program from becoming purely opinion-driven. And in CRO, opinions can multiply faster than credit card debts.

The Hidden Cost of Running the Wrong Test

At first, “let’s just test this” sounds harmless. CRO is about experimentation, right? Just run everything. But every test has a cost.

Even when the testing tool makes setup easy, a test still consumes traffic, time, design effort, analysis attention, and usually a couple of weeks of your roadmap. If your site has enough traffic to run two meaningful tests this month, and your backlog has twenty ideas, running the wrong two means you delayed the better ones. That is not a small cost.

This felt immediately familiar from Devops work. You do not deploy every feature request just because someone suggested it in a meeting with confidence and a nice slide deck. You triage. You look at impact, risk, effort, dependencies, and what the system is actually telling you before you start making changes.

CRO needs the same discipline. An idea is not valuable because it sounds smart in a meeting. It becomes valuable when it is connected to evidence and scored against other ideas in the backlog.

The HiPPO Problem

Before getting into frameworks, there is a force you need to understand. It quietly kills many CRO programs. It is called the HiPPO: the Highest Paid Person’s Opinion. Every team has one. In fact there is a rude saying about opinions. “Opinions are like…” (well… Google it)

The senior leader who says, “I think the button should be green,” and suddenly the next sprint is about button colors. The founder who saw a competitor’s landing page and wants to rebuild yours to match it by Thursday. The VP who has “a feeling” about the checkout flow. The stakeholder who says, “Can we just test it?” in the same tone people use before creating six months of technical debt.

To be fair, these inputs are not useless. People close to the business often have genuine intuition worth exploring. Sales teams hear objections. Support teams hear confusion. Founders understand positioning. Product teams know what users struggle with. The problem is not the idea. The problem is when the idea skips the queue without being scored.

In DevOps, we do not prioritize incident response based on who is shouting loudest in Slack. At least, we should not. We prioritize based on system impact and what the metrics are actually saying.

CRO needs the same structure. A prioritization framework is your polite defense against the HiPPO.

You are not saying: “No, your idea is bad.”

You are saying: “Great, let’s score it the same way we score everything else. If it ranks high, it goes first.”

That small shift changes the conversation. The framework depersonalizes the argument. The number does the heavy lifting. Very convenient, because numbers do not get awkward in meetings.

Start With the Funnel, Not the Idea

Before scoring individual test ideas, figure out where to look first. Not every page deserves equal attention. A page with very little traffic will take forever to produce statistically meaningful results. A page with high traffic but no meaningful business impact may still not be worth optimizing ahead of a page that sits directly in the conversion path.

Press enter or click to view image in full size

So before asking: “What should we change?” ask, “Where are we losing people?”, and then. Map your funnel.

For an e-commerce site, it might look like this:

Landing page > Product page > Cart > Checkout > Purchase

For SaaS, it might look like this:

Homepage > Pricing page > Signup > Activation > Paid conversion

For lead generation, it might look like this:

Landing page > Form view > Form start > Form submit > Qualified lead

Then look at where the biggest drops are happening. If 70% of users are abandoning at checkout, testing a homepage headline is probably not the first move.

It might be interesting.
It might even win.
But it is probably not where the biggest leak is.

Analogy in DevOps: If the database is causing 90% of your latency, shaving 20ms off a frontend asset is not the priority. Sure, the frontend improvement may look nice in a performance report. But the database is still sitting there like an overloaded elephant in the server room.

Find the bottleneck first. Then fix it. In CRO, high-traffic pages close to revenue, signup, checkout, or lead generation are your hot paths. Start there.

Small disclaimer: Read at your own risk. This may make you hungry

The Two Frameworks: ICE and PIE

Once you know which pages to focus on, you need a way to score the ideas sitting in your backlog. Two common prioritization frameworks in CRO are ICE and PIE. Both are simple, both use overlapping dimensions and both are easier to remember if you stop thinking about frameworks and start thinking about food. Which, honestly, improves most business concepts. ;)

ICE: Ordering Food at Midnight

ICE stands for:

Impact
Confidence
Ease

Think of ICE as ordering food on delivery app at 11 pm. You are hungry. You are tired. You are already in bed. This is not the time for adventure. You are not trying to discover a hidden culinary gem. You are trying to avoid sleeping angry. You see a biryani place.

Before you tap “Order,” three thoughts happen automatically.

First:

“How good is this actually going to be?”

If it is bad biryani, why bother?

That is Impact.

Second:

“Can I trust this place right now?”

Will it arrive hot, on time, and with raita? Or will it arrive cold, late, and missing the one thing that emotionally holds the whole meal together? You check the rating, reviews, delivery time, and whether someone recently complained that they received only rice and sadness.

That is Confidence.

Third:

“How easy is this?”

Is the restaurant open? Is delivery available? Will it reach in 25 minutes, or will the app say “arriving soon” until you lose hope? Will it ask you to verify your phone number again even though it has known you for five years?

That is Ease.

ICE is practical and slightly paranoid. It wants evidence before committing.

In CRO terms:

Impact asks: If this test wins, how much could it move the business?
Confidence asks: How sure are we that this is a real problem worth testing?
Ease asks: How difficult will this be to build, QA, launch, and analyze?

Score each dimension from 1 to 5. Then average the three. The highest-scoring ideas usually go first. ICE works especially well when you already have research data: heatmaps, recordings, analytics, surveys, support tickets, or past experiment learnings. Low confidence should lower the score, no matter how exciting the idea sounds.

Because “I saw this on a competitor site” is not research. It is browsing with ambition.

PIE: Choosing What to Order When You Have Options

PIE stands for:

Potential
Importance
Ease

Now stay in the same food delivery app. You are still hungry. It is still 11 pm. You still should have eaten earlier like a responsible adult, but here we are. This time, you are not evaluating one biryani place. You are deciding what category of food deserves your attention first.

Biryani? Pizza? Rolls? South Indian? Chinese? That one “healthy bowl” option you add to cart and then remove after seeing the price? PIE helps you decide where the biggest opportunity is.

First:

“How good could this get?”

Could this be the kind of meal that fixes your mood, your evening, and possibly your belief in humanity?

That is Potential.

Second:

“How important is this meal?”

Is this a casual snack, or have you skipped dinner and now your stomach is sending production-level alerts?

That is Importance.

Third:

“How easy is it to get?”

Is it nearby and deliverable in 25 minutes, or is the restaurant far away, closing soon, and likely to cancel after making you wait?

That is Ease.

PIE is more optimistic than ICE.

ICE asks, “Can I trust this specific choice?”

PIE asks, “Where is the biggest opportunity?”

In CRO terms:

Potential asks: How much improvement could this page or funnel step realistically have?
Importance asks: How valuable is this page or funnel step to the business?
Ease asks: How simple or difficult is the test to implement?

PIE is useful when you are comparing broader areas of opportunity.

For example, if you are deciding whether to focus on the homepage, pricing page, checkout, signup form, or onboarding flow, PIE can help you decide where the biggest opportunity may be.

So, in food delivery terms:

ICE helps you decide whether to trust one restaurant.
PIE helps you decide which food category is worth exploring first.

ICE vs PIE: The One-Line Difference

Press enter or click to view image in full size

Here is the easiest way I remember them:

ICE helps you decide whether to trust one restaurant. PIE helps you decide which food category is worth exploring first.
ICE is about confidence. Can I trust this specific option to deliver? PIE is about opportunity. Which area has the biggest upside if I focus there? Same food app. Different thinking.
With ICE, you are asking: “Should I order from this biryani place?” With PIE, you are asking: “Should I even be looking at biryani first, or is pizza, rolls, South Indian, or Chinese the better opportunity tonight?”
Both end with Ease, because nobody wants unnecessary complications at 11 pm. Especially not when they are hungry and the app is already saying, “Restaurant is closing soon,” like it is adding pressure to your life.

Which Framework Should You Use?

Use ICE when your team already has research data and wants to prioritize based on evidence. ICE is useful when you are evaluating a specific hypothesis. For example:

“Should we move the CTA above the fold on the pricing page?”

You already have heatmap data. You know users are not scrolling far enough. You want to know whether this specific idea deserves to be tested next. That is ICE.

Use PIE when you are earlier in the process and still deciding where the biggest opportunity is.

For example:

“Should we focus on the homepage, pricing page, checkout flow, signup form, or onboarding?”

That is PIE.

At Convert.com, the Compass feature includes both PIE and ICE inside the hypothesis builder. When you create a new hypothesis, you can choose a framework, score each dimension on a scale of 1 to 5, and use that score to sort your backlog.

That helps because the score becomes visible to everyone. It is much harder to push your pet idea to the top when the number is sitting right there on the screen, judging you quietly. A visible score is like the delivery rating on a food app. You can still order from the suspicious restaurant. But now everyone can see that it has 2.3 stars and seven reviews mentioning “never again.”

Turn Ideas Into Hypotheses Before You Score Them

Press enter or click to view image in full size

One thing both frameworks require is that your ideas are written properly before scoring.

“Change the button color” is not a hypothesis.

“Move the CTA above the fold” is not a hypothesis.

“Make the page look more premium” is definitely not a hypothesis.

That is a mood.

A proper hypothesis looks like this:

If I change X, then Y will happen, because Z.

The “because” is the most important part. Without it, you are not testing an assumption. You are just making a change and hoping the graph goes up, which is basically the CRO version of deploying to production and whispering, “Please work.”

Or, to stay with the food delivery analogy, it is like ordering from a restaurant with no ratings because the photo looked nice. Could it be amazing? Maybe. Could it be regret in a plastic container? Also maybe.

For example:

If we move the main CTA above the fold on the pricing page, more visitors will start the demo request flow, because heatmaps show most users are not scrolling far enough to see the current CTA.

Now you have something worth scoring.

You know:

The page
The change
The expected outcome
The evidence behind it

Even if the test loses, you learn something specific. A backlog full of vague ideas becomes a dumping ground. A backlog full of hypotheses becomes a learning system.

Or in food terms:

“Order something nice” is not useful.

“Order from a highly rated nearby restaurant because delivery time is low and reviews mention fresh food” is much better.

Specificity saves you from bad decisions. And occasionally, bad paneer.

The One Thing ICE and PIE Get Wrong

ICE and PIE are useful, but they share a blind spot. They often treat every idea as if it exists in a vacuum. But it does not.

Every idea exists inside the history of your previous tests. If you have run five tests on your pricing page and four of them lost, your confidence score for the next pricing-page idea should reflect that history. Something on that page may be resisting your assumptions.

Maybe the traffic quality is different from what you expected.
Maybe the real problem is upstream in the funnel.
Maybe people already understand the pricing page, and the actual friction is in signup.
Maybe your pricing page is innocent and has been framed by bad hypotheses.

The same applies to food delivery. If you have ordered from the same restaurant four times and three times they forgot the raita, you should not treat the fifth order like a fresh mystery.

You have history. Use it. Past experiment results should feed into how you score new ideas. They should not sit inside a spreadsheet named something like: final_results_v3_really_final_updated_NEW.xlsx

This is why documentation matters as much as the framework itself.

After every experiment, write down:

The hypothesis
The result
The confidence level
The primary metric
The key learning
What it suggests about the next test

Even a losing test is useful if it tells you where not to look. Without this history, your backlog resets to zero every quarter. With it, your scoring gets more accurate over time because you are building a picture of what your specific audience actually responds to. A CRO backlog without documentation is like a food app that forgets every bad order you ever had. You keep making the same mistake. And somehow, the raita is still missing.

What to Do When Everything Scores the Same

Sometimes three ideas all score 4.2 and suddenly you are back in the same argument. This is normal. Frameworks help, but they do not eliminate judgment. A few tiebreakers can help. Pick the page with the most traffic. More visitors usually means faster results and less time waiting for statistical significance.

In food delivery terms, this is like choosing the restaurant with enough recent orders and reviews. If only two people have ordered from it since 2021, the rating may not tell you much.

Pick the test that is fastest to build. A high-scoring idea that takes three weeks of development work may be worth doing after a slightly lower-scoring idea you can ship in a day. Remember the frequency side of the success formula.

This is the difference between ordering something that arrives in 25 minutes and choosing a dish that says “preparation time: 95 minutes.”

Technically, it may be great. Emotionally, you may not survive. Pick the test that teaches you the most regardless of outcome. Learning value is underrated. A test that answers a fundamental question about your audience is worth more than a test that optimizes something already working reasonably well.

In other words, do not only ask:

“What might win?”

Also ask:

“What will we learn even if this loses?”

That question saves a lot of time. And occasionally, a lot of ego.

Define Success Before You Launch, Not After

One mistake that shows up in beginner backlogs is choosing the primary metric after the test has already started. That is dangerous. If you look at enough metrics, something will always appear to have improved.

Clicks went up.
Scroll depth changed.
Form starts increased.
Time on page looked different.
Someone in Nebraska spent 14 minutes reading the footer.

But what was the experiment actually supposed to improve? Every test needs a primary metric before launch.

Demo requests.
Purchases.
Trial signups.
Form completions.
Revenue per visitor.

Pick one. Commit to it. Let that metric make the call. This is like deciding what “good food” means before ordering. Are you optimizing for taste? Delivery speed? Price? Portion size? Not waking up with regret? If you decide after the food arrives, you can justify anything.

“The biryani was cold, but the container was sturdy.”

That is not success. That is coping. You can still watch secondary metrics for learning, but the decision should rest on the primary one. And alongside your primary metric, define your guardrail metrics. These are the things you are not directly trying to improve, but cannot afford to break. For example:

If you are optimizing demo signups, lead quality should not drop.
If you are optimizing checkout completion, average order value should not fall.
If you are optimizing form completion, spam submissions should not explode.
If you are optimizing clicks, downstream conversions should not suffer.

This is the CRO version of SLOs. You define what success looks like and what failure looks like before you are emotionally invested in the outcome. A test that increases clicks while damaging qualified leads is not a win. It is noise wearing a party hat.

Or in food delivery terms: A restaurant that delivers in 12 minutes but sends the wrong order did not “win on speed.” It failed dinner.

A Practical First Backlog

If you are starting from zero, here is the workflow I would follow.

Map your funnel and identify the biggest drop-off point.
Use heatmaps, session recordings, analytics, surveys, and any sales or support feedback you can access.
Write down every test idea without judging too early.
Convert each idea into a proper hypothesis using the “If X, then Y, because Z” format.
Score each one using ICE or PIE.
Pick one high-scoring, reasonable-effort test.
Define the primary metric and guardrail metrics before launch.
Run the test.
Document the result and the learning.
Then re-score the backlog based on what you now know.

That last step is what most teams skip. And it is what separates a backlog that compounds from one that stagnates. A CRO backlog should not be a parking lot where ideas go to quietly disappear. It should be more like a living system.

Or, to stay on brand, a well-maintained CI/CD pipeline for learning. Every test should make the next decision smarter. Just like every food order teaches you something.

Restaurant is reliable.
Delivery is slow on weekends.
The “extra spicy” option is not a personality test you need to pass.
Never trust a place where every menu item has the same stock photo.

Learning compounds when you write it down.

The Real Takeaway

Prioritization is not about finding the cleverest idea. It is not about the biggest redesign. It is not about the most senior opinion in the room. It is about running the tests that have the best combination of meaningful impact, strong evidence, reasonable effort, and clear learning potential. Done consistently, this turns CRO from a collection of random experiments into a learning system. You find the biggest leaks, collect evidence, write hypotheses, score ideas, run the most valuable tests first, document what happened and then repeat.

The DevOps parallel is hard to miss. Good infrastructure teams do not just ship more changes. They improve how they decide what to change, how they measure it, and how they learn from it.

Good CRO teams do exactly the same thing. And the food delivery parallel is also hard to miss. The goal is not to order everything on the menu. The goal is to make better choices each time, based on what you know, what matters, what is realistic, and what you learned last time.

That is prioritization.

In the next post, I want to go deeper into reading results without fooling yourself: statistical significance, confirmation bias, and why the most dangerous moment in CRO is when a test is almost winning.

I work with Convert.com and write about the intersection of DevOps and experimentation. If you are building your first CRO backlog or trying to move your team away from opinion-driven testing, drop a comment or reach out. Happy to compare notes.

From a zip file in 2020 to an AI-powered wiki in 2026. Here's what happened in between

2026-06-24T23:46:00.000-07:00

Someone on the team once asked about a feature deprecated back in 2020.

45 minutes of digging through backups, old slack threads, a google doc nobody had touched in years, and the answer eventually showed up inside a zip file. Like a digital archaeological dig. Except less cool and more frustrating.

That moment made it clear to me that something needed to change. This post is about what the team figured out together.

Part 1: An old hero called MDWiki

Before getting to the AI part, here's a tool that ended up being central to the whole thing.

Years ago, there was a situation many people would remember. I was in a pilot batch at a new office location for an MNC. Fresh start, no internal tools, no documentation platform, no proper handover process. Knowledge transfer from HQ was slow, questions were piling up, answers were scattered everywhere.

MDWiki turned out to be the answer. Its an old tool, almost ancient by tech standards, but brilliant in its simplicity. plain markdown files in a folder, and it renders a full documentation site. no database, no server setup, no hosting complexity. just files.

Hosting it on OneDrive meant the whole company could sync it locally. suddenly every team had a shared wiki that worked offline, updated automatically when anyone saved a file, and needed zero infrastructure to maintain. Lightweight, fast, and it worked. sometimes old tools solve new problems better than anything shiny.

Part 2: The scale problem every growing team eventually hits

Fast forward to now. At a certain point, documentation at our company had simply outgrown the way it was being managed. Knowledge contributed by multiple team members over years. text files, images, screenshots, video recordings, architecture diagrams, slack conversations, meeting notes, product threads. Rich, honest knowledge built up over time by people who genuinely cared.

But here is what happens at scale. When knowledge lives across too many places, it becomes hard to navigate. some topics had multiple versions, each slightly evolved from the last. naturally, over time, some overlap crept in. This is not a people problem. Its a growth problem. every team that ships consistently runs into this. knowledge compounds faster than anyone can manually organise it.

And when someone moves on, a little undocumented context moves with them. not because they were careless, but because there was no system to capture it before it walked out the door. that gap needed closing, and honestly it was on us as a team to fix it sooner.

Part 3: Andrej Karpathy walked in (virtually)

That's when we came across Andrej Karpathy's LLM-Wiki idea. if you don't know Karpathy, he was a founding member of OpenAI and Tesla's AI director. when he writes something, the internet pays attention.

His idea is elegant. Stop treating your knowledge base as a place humans write into. treat it like a git repo. let the LLM own the writing layer. humans write exactly one file by hand, a schema file that defines the rules, structure, and conventions. Everything else? the LLM reads raw sources and builds the wiki, maintaining it as new content comes in.

The wiki becomes a persistent, compounding artifact. it gets richer with every source added. cross references are already there. contradictions get flagged. the synthesis reflects everything that has been added, not just the last thing someone remembered to update. Brought it to the team. We decided it was worth trying.

Part 4: What we built together

We combined LLM-Wiki with MDWiki and added a CI/CD layer on top. Here is how the whole thing works.

Three layers.

Raw folder. all source material goes here. old docs, slack exports, meeting notes, screenshots, video transcripts, architecture diagrams. untouched, immutable. source of truth.

CLAUDE.md. the one file written by hand. it teaches Claude how MDWiki works, what the folder structure looks like, title formats, how to handle overlapping topics, what goes where, and when to flag something for human review instead of auto-placing it. the rulebook for the entire system.

The wiki itself. Claude reads whatever lands in the raw folder and writes, organises, and cross-links the markdown files in MDWiki format. every document lands in the right place, correctly formatted, connected to related pages.

And the most important part, the human review layer. Any team member can drop a document into the raw folder and raise a pull request on GitHub. a reviewer steps in before anything gets merged. they check:

Is this information still accurate or has the process changed?
Does this conflict with something already in the wiki?
Is there any sensitive information, credentials, or customer data that shouldn't be here?
Is this still relevant or does it describe something no longer in use?
Does the framing match our internal style and audience?

Once the PR is approved, Claude adds the knowledge to the wiki, formats it for MDWiki, cross-links it with related content, and updates the index. A GitHub Actions pipeline deploys the whole thing to Google Firebase. every approved merge triggers a deployment. the team sees the updated wiki within minutes, no manual publishing needed.

Part 5: The Guardrail reminder

This one bears repeating, and it came from a hard conversation within the team early on. Claude has write access to the knowledge base. and we made sure it cannot merge its own PRs.

Not because we don't trust the model, but because nobody should. a recent research paper tested 19 LLMs on long document editing workflows and found that even the best models, GPT, Claude, Gemini, silently corrupt an average of 25% of content over time. not crashing, not warning anyone. just quietly getting things wrong as the workflow gets longer.

The human review step is not a formality. its the most load-bearing part of the whole pipeline. we almost skipped it to move faster. Glad we didn't. Automation without a review layer is not a knowledge base. its a very confident source of misinformation.

If you're building something similar or thinking through your own documentation architecture, drop a comment. happy to share more on the setup.

Ref

#Documentation #DevOps #LLMWiki #KnowledgeManagement #GitHub #MDWiki #AIAgents #GitHubActions #Firebase

Note for student readers

If you're in college and hunting for internships, stay with me for one more minute. This entire system works because of one thing: markdown. MDWiki renders it. Claude writes it. GitHub versions it. Firebase serves it. the whole pipeline has markdown at the center.

Markdown is not a niche skill. its the common language of documentation across DevOps, software engineering, product teams, and now AI pipelines. When you write a README.md on GitHub you're not just leaving notes for yourself. you're showing every hiring manager who lands on your profile that you think about the person coming after you. that's rarer than it sounds.

Most students push code and skip the README. don't be most students. Simplest thing you can do this week: go to your best GitHub project and write a proper README.md. not just "this is my project." write what it does, what problem it solves, how to run it, and what you learned building it.

That one file will do more for your internship hunt than a fancy resume template. Markdown today. Opportunity tomorrow.

✏️ drafted with ai assist, because practicing what I preach.

Part 2: The Experiments That Actually Move the Number

2026-06-22T01:23:00.000-07:00

If you read the first post in this series, you now have a working definition of CRO, a sense of why conversion rate matters, and hopefully a free account on some combination of Clarity, GA4, and Convert.com(sshhh… I work for them).

Maybe you even opened a heatmap, stared at the colors for a while, and thought: “Okay. Now what?”

This post is that now what.

Once you understand the basics of CRO, the next challenge is figuring out what kind of experiment to run. A/A test? A/B test? Split URL test? Multivariate test? They all sound similar at first, but they solve different problems at different stages.

And if you are coming from a technical background, the temptation is to overthink the setup before shipping anything. I know, because that is exactly what I started doing.

So let’s break down the main experiment types, when to use each one, and how to think about them without getting stuck. I will also share a formula from a CRO course I have been working through that quietly changed how I think about the whole thing.

The Four Experiment Types, Explained Without Jargon

1. A/A Testing

Before you trust your first experiment result, run an A/A test.

In an A/A test, both groups see the exact same page. No headline change, no button change, no layout change. Nothing at all. The purpose is not to improve conversions. The purpose is to check whether your measurement setup is actually trustworthy.

If your tool is configured correctly, both groups should show roughly similar conversion rates over time. If they do not, something is off with how traffic is being split, how visitors are being tracked, or how conversions are being recorded.

The DevOps parallel is straightforward. You would not set up Prometheus alerts on a misconfigured scrape target and then make infrastructure decisions based on that data. An A/A test is the same idea: confirm the instrument is working before you start using it to make calls.

A lot of teams skip this because it feels boring. There is no variation, no redesign, no winner to announce. But if your measurement layer is broken, every A/B test after this becomes questionable. You may spend weeks debating a result that was never reliable in the first place.

On Convert.com, setting up an A/A test takes about three minutes. Do it before anything else, and leave it running for at least a week.

Press enter or click to view image in full size

2. A/B Testing

Once your A/A test passes, you are ready to actually experiment.

An A/B test compares two versions of something. Version A is the original, called the Control. Version B is the changed version, called the Variant. You split traffic between the two and measure which one performs better on your chosen metric.

The element you change could be a headline, a CTA button, a hero image, a form layout, a pricing message, anything on the user journey. The constraint is this: change one meaningful thing at a time.

If you change the headline, button color, pricing layout, and image all in the same test, and the variant wins, you have no idea what actually caused the lift. You got a result but no learning.

The DevOps analogy that clicked for me: a clean commit. If one commit touches ten unrelated files and something breaks, debugging is painful. If one commit changes one logical thing, the cause is much easier to trace.

A good A/B test starts with a hypothesis in this format: “If I change X, then Y will happen, because Z.”

For example: “If I change the CTA text from ‘Buy Now’ to ‘Get Your Copy Today,’ then more visitors will click through to checkout, because the new copy feels less transactional and creates a stronger sense of ownership.”

The “because” is the part most people skip, and it is the most important part. Without it, you are not testing a hypothesis. You are just nudging things and hoping the graph goes up.

Press enter or click to view image in full size

The confusion between A/A testing and A/B testing

I was always confused as to why A/A and A/B testing are needed. If you are also in the same state, here is a crude example that may simplify it.

A/A Testing: The Paranoid Caterer

You are catering a wedding. You make one giant pot of biryani. You serve it to Table 1 and Table 2 from the same pot, same quantity, same everything.

Table 1 finishes 95 plates. Table 2 finishes 30 plates.

You have not changed anything. Both tables got identical food. So why is Table 2 eating so little?

You investigate. Turns out the waiter assigned to Table 2 was busy on his phone and forgot to refill the serving bowls.

The biryani was not the problem. The delivery system was broken.

That is an A/A test. Before you start testing new recipes, you confirm that the serving system itself is not lying to you.

A/B Testing: The Sensible Caterer

Same wedding. You make two versions of biryani. Version A has the usual spice level. Version B has a little more pepper because your cousin who went to Bangalore said people there like it spicy now.

You serve Version A to the left side of the hall and Version B to the right side. You count which side finishes faster, asks for seconds more often, and complains less to the mother of the bride.

One change. One comparison. One winner.

You do not also change the raita, the naan, the serving bowl size, and the background music at the same time. Because if the right side loved it, you will never know if it was the pepper or the fact that the DJ on that side was finally playing something decent.

So:

A/B test: “Which version is better?”
A/A test: “Can I trust the test setup?”

3. Split URL Testing

A standard A/B test usually changes something on the same page. The URL stays the same and the testing tool modifies what users see. Split URL testing is different: users are sent to entirely different URLs.

Control: /pricing. Variant: /pricing-v2. Traffic is split between them.

Use this when the change is large enough that modifying the existing page gets messy. Major redesigns, completely different landing page structures, new checkout flows, testing a long-form page against a shorter one. If it is a minor copy tweak, A/B is faster and simpler. If you are rebuilding the page from scratch, Split URL is cleaner.

The infrastructure analogy is blue-green deployment: two versions running simultaneously, traffic routed between them at the experimentation layer instead of the load balancer. Same principle, different domain.

One thing to watch carefully: tracking consistency. Both pages need to fire the same analytics events and conversion tracking correctly. A common mistake is having GA4 or pixel tracking set up on the original page but missing or misconfigured on the variant. When that happens, the variant looks like it is losing when the actual problem is your observability. As DevOps people know well, bad telemetry makes healthy systems look broken.

Press enter or click to view image in full size

4. Multivariate Testing

A/B testing changes one element at a time. Multivariate testing (MVT) changes multiple elements and tests every combination of them simultaneously.

Say you want to test two headlines and two hero images. MVT creates four combinations: Headline 1 + Image A, Headline 1 + Image B, Headline 2 + Image A, Headline 2 + Image B. Traffic is split across all four.

The benefit is finding the best combination without running four separate sequential tests. The downside is traffic dilution: more combinations means each version gets a smaller slice of traffic, which means it takes longer to reach a reliable result.

This is why MVT is better suited for high-traffic pages. On a low-traffic site, running multivariate tests is like spinning up a four-node Kubernetes cluster to host a static HTML page. Technically possible. Practically painful.

If you are just getting started, stick to A/B tests until you have enough traffic and enough process confidence to justify MVT.

Press enter or click to view image in full size

Confused between Split URL and Multivariate?

Here’s the crude example:

Split URL Testing: The Ambitious Caterer

This caterer is not making a variation of the existing menu. He has built two entirely separate stalls on opposite ends of the lawn.

Stall A is the traditional setup: biryani, raita, gulab jamun, done.

Stall B is the “fusion experience”: biryani bowls, deconstructed raita in shot glasses, and gulab jamun on a stick with a QR code to leave a Google review.

Half the guests are directed to Stall A and half to Stall B. You measure which stall has a longer queue, higher plate counts, and fewer people quietly throwing food in the dustbin.

This is a Split URL test. You are not tweaking one element on an existing page. You have built two completely different pages and you are routing real traffic to both to see which one works.

The infrastructure cousin of this is blue-green deployment. Same idea, different catering budget.

Multivariate Testing: The Caterer Who Has Lost the Plot

This person is testing:

2 types of biryani (chicken vs. mutton)
2 types of raita (boondi vs. cucumber)
2 types of bread (naan vs. rumali roti)
2 types of dessert (gulab jamun vs. rasgulla)

That is 16 combinations. You need 16 separate tables, each getting a different combination, and enough guests at each table to statistically prove which combination is the winner.

The wedding has 200 guests. You need at minimum 1,600 guests for the data to mean anything.

You have invited 200.

You wont get that big venue on any of the wedding seasons, but lets just say… you did.

This is multivariate testing on a low-traffic website. The math does not work. The experiment runs for six months. By the time you have a result, the wedding is over, the couple has moved cities, and nobody remembers what the original research question was.

Run multivariate tests only when you have enough traffic that dividing it sixteen ways still leaves each slice statistically meaningful. Otherwise, stick to A/B.

Heatmaps Are Research, Not Experiments

This was one of my early mental shifts, and it is worth being explicit about it.

A heatmap does not tell you what will win. It tells you what might be wrong.

Click heatmaps show where users are clicking (and where they are not). Scroll heatmaps show how far down the page users travel before leaving. Session recordings are anonymized playbacks of real user journeys, where you can watch someone hesitate over your pricing table, try to click something that is not a link, or abandon a form halfway through.

None of these tools give you a winner. They give you a hypothesis.

Here is how the workflow actually connects: you open a heatmap for your pricing page and notice that most users never scroll past the first fold. Your main CTA button is near the bottom. Now you have a possible problem and a possible hypothesis: “If I move the CTA above the fold, more users will see it and click it.” You run an A/B test to confirm or deny it.

The heatmap gave you the question. The A/B test gives you the answer.

That distinction matters because it stops you from treating research tools as decision-making tools. Heatmaps, recordings, analytics, and surveys help you decide what to test. Controlled experiments help you decide what to change.

The Formula That Changed How I Think About All of This

Success = Chance × Frequency

Success is the number of winning experiments you produce in a year. Chance is your win rate, the percentage of experiments that produce a meaningful positive result. Frequency is how many experiments you run per month.

Now run the numbers with me.

Team A wants every test to be excellent. They spend weeks on research, debate every design decision, and get thorough sign-off before anything ships. Their win rate is high, say 100%. But all that effort means they can only run one test per month. Twelve winners a year.

Team B is less precious about each individual test. Their research is decent but not exhaustive. Their win rate is only 25%. But they run eight tests a month. Twenty-four winners a year.

Worse hit rate. Double the output.

This is not an argument for carelessness. Bad experiments waste time, and random button-color testing is still random button-color testing. But it is a clear argument against perfectionism as a strategy. The team that ships eight decent experiments a month will outlearn and outperform the team that ships one perfect experiment a month, almost every time.

If this sounds familiar, it should. DevOps made the same argument against waterfall delivery years ago. Smaller releases, shipped more often, create faster feedback loops. You learn sooner, recover faster, and stop treating every release like a quarterly ceremony. CRO has the same rhythm: define the hypothesis, design a clean test, ship it, measure it, document the learning, and move on. The goal is not to win every test. The goal is to learn fast enough that your wins compound.

A Note on Statistical Significance

Every testing tool gives you some version of a confidence percentage as your test runs. This tells you how likely it is that the result you are seeing is real and not just random variation. Most practitioners aim for 95% confidence before calling a winner.

But one of the most common beginner mistakes is stopping a test early because the numbers look exciting on Day 2. Early data is noisy. Weekday and weekend behavior can differ. Email campaigns, seasonality, and traffic source changes can all skew early results. A variant that looks like it is winning after 200 visitors may look very different after 4,000.

Let tests run for at least one full week, preferably two, before drawing conclusions.

And here is the question I now ask before trusting any result: not just “Did the tool say 95%?” but “Do I trust how we got to 95%?” The number matters. So does sample size, test duration, traffic quality, and whether the tracking was set up cleanly to begin with.

What Should You Run First?

If this feels like a lot, start small. Here is the beginner loop I would follow:

Run an A/A test to confirm your measurement setup is reliable.
Pick one high-traffic page with a clear conversion goal.
Use heatmaps, session recordings, or analytics to find one point of friction.
Write a hypothesis using the “If I change X, then Y, because Z” format.
Run one clean A/B test.
Let it run long enough to collect meaningful data (at least one week).
Document what happened, win or lose.
Repeat.

That loop is what CRO actually is, more than any specific tool or technique. Observe, hypothesize, test, learn, repeat. The experiment types are just the mechanics. The real skill is building a habit of running the loop consistently, and gradually running it faster.

In the next post, I will get into prioritization: how to decide which test to run when you have more ideas than time, and how to build a backlog driven by data rather than whoever spoke loudest in the last meeting.

I work with Convert.com and write about the intersection of DevOps and experimentation. If you are a technical person finding your footing in CRO, drop a comment or reach out. Happy to compare notes.

From Apache Benchmark to A/B Testing, A DevOps Guy’s CRO Journey

2026-05-11T01:19:00.000-07:00

When my manager first mentioned A/B testing, I nodded confidently. I knew exactly what it was.

Except I didn’t.

In my head, “A/B testing” meant Apache Benchmark, the ab command-line tool DevOps engineers use to hammer HTTP servers and measure throughput. Requests per second. Latency percentiles. That kind of A/B testing.

What followed was a humbling unlearning. Concepts like Conversion Rate Optimization, variations, Split URL tests, A/A tests, I was reading all of this through a server engineer’s lens and none of it made sense at first. Why run experiments? Why not just change the code, redeploy, and see what happens?

That was my first mistake. And if you’re a technical person stepping into the world of CRO, you’ll probably make it too.

So, What Actually Is CRO?

Let me skip the textbook definition and use a story instead.

Say you’re selling a bestselling book online. You’re running ads, offering discounts, the traffic numbers look decent, but conversions are low. People are landing on your page and leaving. You’ve checked the backend, the server is healthy, the page loads fast. As a DevOps engineer, you’d call this “nothing to fix.”

But something is broken, just not in the infrastructure.

Conversion Rate Optimization (CRO) is the practice of understanding why visitors don’t convert, and systematically fixing it through experimentation. You introduce a change, a different headline, a reordered checkout flow, a new CTA button color, mark it as Variation A or Variation B, measure which one converts better, and iterate.

Simple math:

Conversion Rate = (Conversions / Total Visitors) × 100

If 10 out of 100 visitors buy your book, your conversion rate is 10%. A CRO practitioner’s job is to move that number, but more importantly, to understand why it moved.

That last part matters. A random change that boosts conversions is luck. Understanding the reason behind the boost, that’s CRO.

The Hidden Cost of Ignoring CRO

Here’s a pattern that plays out more often than it should:

A business pours budget into ad campaigns. Traffic surges. Conversions don’t. They spend more on ads. Still nothing. Eventually someone realizes the problem wasn’t the ad, it was the landing page. Confusing layout, unclear value proposition, a checkout flow that made users feel unsafe.

By then, they’ve burned through their budget chasing the wrong problem.

CRO tools exist to surface these issues before they become expensive. Heatmaps show you where users click (and where they don’t). Session recordings let you literally watch a user struggle with your UI. Exit-intent surveys catch people right as they’re leaving and ask them why. These aren’t vanity metrics, they’re diagnostic tools.

As someone who spends a lot of time on observability and monitoring in DevOps, I found this framing familiar: if you can’t measure it, you can’t fix it.

Before You Start: Is CRO Even Right for You?

This is a question I wish someone had asked me earlier.

A/B testing requires traffic. Without enough visitors, you can’t achieve statistical significance, the confidence level that tells you a result is real and not just noise. Most CRO practitioners aim for 95% confidence, ideally 99%.

Here’s an analogy that clicked for me:

Flip a coin 3 times, get heads all 3 times. Is the coin rigged? Hard to say, could be luck. Flip it 1,000 times and get heads 900 times. Now you’re pretty sure something’s off.

That “pretty sure” feeling, measured as a percentage, is your confidence level.

If your site gets 200 visitors a month, running a statistically valid A/B test is going to take a very long time. Many A/B testing platforms offer sample size calculators, use them before committing to a test.

A Framework for Getting Started

When I started approaching CRO with more structure, things got clearer. Here’s the sequence that worked for me:

1. Define a Goal First

Don’t open a heatmap tool and start poking around without knowing what you’re looking for. Start with a concrete goal:

Increase revenue by X% in Y days
Grow the customer base by Z%
Improve feature adoption on a specific page

Each goal points to a different strategy:

GoalLeverIncrease revenueUpsells, pricing page clarityBigger customer baseBetter UX, reduced frictionIncrease engagementFeature discovery, onboarding flow

2. Audit Your Site

At a D2C roundtable conference I attended, participants were split into groups and given websites to audit, armed with CRO cards listing what to look for. Are visitors able to find what they need? Is the checkout frictionless? How long does the full journey take?

One presenter, Gursimran Gujral (Co-Founder at OptiPhoenix), walked through a heatmap that had driven several of their key CRO decisions. Seeing a real example in action made the whole framework concrete.

Auditing before testing gives you hypotheses. Hypotheses give your tests direction. A good hypothesis follows a simple structure: “If I change X, then Y will happen, because Z.” That “because” is what separates informed testing from random tinkering.

An example: If I change the CTA button text from ‘Buy Now’ to ‘Get Your Copy Today’, then more visitors will click through to checkout, because the new copy feels less transactional and creates a sense of ownership.

3. Pick the Right Tools

You don’t need everything at once. Start with one tool per job:

User behavior: Microsoft Clarity (free) or Hotjar, heatmaps, session recordings
A/B / Split URL / Multivariate testing: Convert.com, robust experimentation platform
Funnel analysis: GA4 + LookerStudio, track where users drop off

Analytics alone won’t get you there. You need a dedicated testing tool to run controlled experiments.

4. Run Tests with Patience

One of the most common beginner mistakes is stopping a test too early because the early numbers look good (or bad). Let tests run for at least a week to account for day-of-week behavior differences. Trust the confidence interval, not your gut.

Resources Worth Your Time

These are the ones I actually found useful, not just names I’m dropping:

Ruben de Boer’s courses: practical, tool-focused, good for getting hands-on fast
CXL’s CRO Fundamentals: more rigorous, worth it if you want depth over speed
MeasureSchool on YouTube: excellent for GA4, GTM, and LookerStudio workflows
Baymard Institute: research-backed UX analysis, great for e-commerce audits
Robert Cialdini’s Influence: The Psychology of Persuasion: understanding why humans behave on pages is half the battle

For practice: volunteer your CRO skills to NGOs or small businesses. Real traffic on real sites teaches you things no course can.

The Real Takeaway

CRO isn’t a bag of tricks. It’s a discipline built on measurement, hypotheses, and patience, things that, honestly, aren’t that different from good DevOps practice.

You instrument. You observe. You form a hypothesis. You test. You learn.

The difference is that in DevOps, you’re optimizing for uptime and throughput. In CRO, you’re optimizing for human behavior. And humans, unlike servers, don’t always behave the way the logs suggest they should.

That’s what makes it interesting, and why I’m still learning.

If you’re a technical person who’s crossed over into CRO or is curious about it, I’d love to hear your experience. Drop a comment or reach out.

I said AI will mess up your docs. Scientists said hold my beer.

2026-05-10T23:44:00.000-07:00

Last week I wrote about why you shouldn't trust AI blindly with your documentation.

I thought I was only being careful. Turns out, researchers just proved it, with actual numbers.

A paper dropped on arxiv last month(got posted in Hackernews yesterday). three researchers tested 19 AI models, gave them real document editing tasks across 52 different domains, coding, music, science, legal, you name it. And they just watched what happened over long workflows.

The results are not great.

Even the best models out there, GPT, Claude, Gemini, the ones we all use and trust, corrupted an average of 25% of document content by the end of long workflows. Is that 1 in 4 documents going silently wrong???

And here is the part that really got me. It didn't crash. it didn't say "hey I think I made a mistake."(It never does :) ) It just kept going. confidently. while quietly getting things wrong in the background. And the longer you let it run? the worse it gets.

Now here's the twist that I didn't expect.

Giving the AI more tools(which some debated earlier), making it a full agent with access to search, file systems, everything, didn't help. performance didn't improve. the corruption still happened.

Also, weaker models tend to just delete content. which is bad, but at least you can see it.

Stronger, smarter models? they don't delete. they corrupt. they change the meaning, subtly, in ways that look fine on the surface. that's actually scarier.

So last week when I said, build the guardrails before you turn on the automation, I wasn't just being cautious. now I have a research paper basically saying the same thing now.

Think about what this means for documentation specifically.

If your company is planning to let AI write or update your internal knowledge base, and you're thinking "its fine, we're using a good model", this paper is saying 25% of that content could be silently wrong before anyone even notices.

Not broken. not obviously wrong. just, quietly corrupted.

So who's building the review layer?

Because based on this research, the AI definitely isn't going to tell you it made a mistake.

You can read the paper here : https://arxiv.org/abs/2604.15597

#AIAgents #Documentation #LLMs #KnowledgeManagement #AgentGovernance #DevOps

✏️ Drafted with AI assist, because practicing what I preach.

Docs as code. AI as author. Humans as reviewers. What could go wrong?

2026-05-08T23:40:00.000-07:00

Last week we audited our company docs.

502 articles. 77 duplicates. Nobody knew which one was correct.

And this wasn't because people were lazy. Everyone was busy. Docs just... don't update themselves.

So we started asking, what if an AI did this?

What if instead of waiting for someone to update a document after a product change, an AI just does it automatically. Reads the release notes, finds the affected articles, rewrites them, sends a PR for human review.

Sounds great right?

It actually is. I genuinely think this is where documentation is going. Andrej Karpathy (the AI guy from Tesla and OpenAI) wrote about this — he called it the LLM-owned wiki. Humans write the rules. AI does the writing. Humans approve before it goes live.

Clean. Simple. Scalable.

But here's the thing nobody talks about.

In our own test, the AI gave us wrong pricing information. confidently. Like it had no doubt. it cited some random blog as a source and we almost missed it.

If we had set it to auto-approve, our customers would have read wrong info.

So before you get excited and just "let AI handle the docs", ask yourself a few things:

who is reviewing what the AI writes? (if its one person, you just created more work, not less)
what rules has the AI been given? does it know what it can and cannot touch?
Is someone watching it over time, or only at the start?

This is not about being anti-AI. I love where this is going.

But we don't deploy code to production without monitoring, access controls and alerts. why would we treat our company's knowledge base any different?

Automation without oversight is not progress. its just faster mistakes.

The teams that get this right will build the guardrails first. then turn on the automation.

What's your take — are you using AI for internal docs? how are you handling the review part?

✏️ Drafted with AI assist, because practicing what I preach.

SkenAI's role in DevSecOps CI/CD pipeline

2021-01-22T08:35:00.003-08:00

The term DevOps has pretty much revolutionized the entire IT industry. We have heard of different versions/definitions and nomenclature for different methods of adoption of the culture by different organizations. Some well known are NoOps, GitOps, ChatOps, TechOps, SecOps and there are surely many others out there. However, the introduction of all these “Ops” summarize a common pattern and understanding of speeding up the traditional Software Development Life Cycle using automation via tools, technologies and processes that would allow developers to focus only on code. As per organizational need whichever part of their process seems to be operating slow and introducing unnecessary delay, they would want to “Opsify” it.

Over the last few years we have seen a lot of new tools being introduced in the DevOps world intended to bring in more efficiency and speed in the Software Development process. We have seen equal movement in the risk, threat and security field as well. New tools are subject to new languages, versions, APIs and SDKs. With every such new innovation, we also invite a possibility of new threats and vulnerabilities. So we typically hear about new vulnerabilities in different libraries of different source code. We then get into the phase of version upgrade, patching, etc to make sure that our environment is clean and safe. This process however traditionally has been extremely slow. Many times security threats and vulnerabilities are identified in production and take a long time to get fixed. This is when you need to introduce DevSecOps in the flow.

Eliminating the possibility of having vulnerabilities introduced at code level itself can help in the long run and save a lot of man hours. This is where DevSecOps ideally must get introduced. And, a typical DevSecOps pipeline would look like this:

Identifying of vulnerable libraries being used.
Secrets stored
Static code analysis
SAST
DAST
SCA
Image verification
OS vulnerabilities
System software packages

Most of this is curbed prettty much in the pipeline with some well known tools in the market that parse your source code and identify different threats.

Based on the language you need to do some research and get the best code scanners available to scan your code and resolve the issues identified. There are many opensource scanners available out there that help you do all the scanning and threat/vulnerability identification. Once you are sure which tool you need to use, you simply integrate it with you CI system by installing it there and setting up appropriate binary paths to be used while executing the build. This effort saves considerable time in the long run and helps you focus more on product and less in security.

However, setting up the security scanners could be time consuming or may need additional knowledge of that language. Also, since new vulnerabilities keep coming in, these scanners also keep getting upgraded all the time. It is important to stay updated with them and also update your CI system components periodically. This is where Sken.ai solution comes into picture. Sken offloads the responsibility of setting up a DevSecOps pipeline to a large extent.

You don't need to be remembering different ways to setup code scanners
You also need not worry to update them periodically.
Moreover, you do not even need to know what programming language is being used in the source code. Sken is intelligent enough to parse the code and automatically select the best scanner based on the languages found in the source code.
Docker is the only requirement.
Once you insert Sken's credentials, it will automatically pull the best scanners most updated docker image and scan your code completely, provide result quick and you are good to go.
The scanners will automatically look for SAST, DAST, SCA, secret leaks etc and provide you all of the data in the results to be fixed.
You don’t have to be a security expert, its Saas. It does it all for you.
Easy integration with most of the CI tools; Jenkins, Bamboo, TravisCI, Github, CircleCI
Free 300 test per month on private projects
Opensource scanners integrated, no costing or licensing issues of proprietary application scanners
Bonus: AI driven, scan results in prioritized scan types, less false positives and reduced noise.
Consolidated visuals for all apps and scans, you can prioritize vulnerabilities based on risk rating.

Here is a GIF that can help you have an overview of how to quickly integrate and get started.

Sken.ai DevSecOps walkthrough

2021-01-22T08:28:00.002-08:00

Sken.ai Introduction:-

Sken.ai is a Continuous Application Security Testing and vulnerability scanning tool. It helps in static code analysis, Source code analysis, code review, code complexity, and penetration testing. It is a new and automated DevSecOps tool that provides multi-language scanning features in one place. It provides comprehensive Saas based continuous application testing software developers and DevOps, without the need for any security expertise. It helps to Find and fix all types of application security issues within your DevOps CI/CD cycle.

It provides a SAAS orchestration layer that integrates continuous application security testing into your DevOps CI/CD workflow automating the use of any open source security scanners across all scan types; SAST, DAST, SCA, MAST, container scanning, license compliance, and secrets detection.

What will you learn from this article/blog:-

How to provide security in your CICD pipeline?
What is SAST/DAST/SCA/MAST and how it works?
How sken.ai works? Which features and scanners sken.ai provides?
What vulnerability scanning and penetration testing is?
List of Top IDE’s used by developers provided by DevOps?
List of version control tools provided by DevOps?
List of Top CI/CD pipeline tools provided by DevOps?
What web application security scanner is?

Who will need this tool:-

There are lots of open-source application security testing tools available in the market. So it's very hard and confusing to select what exactly we need. Here automation comes further there are many open-source plugins and security tools are available to provide and scan our application for security tests before the development state goes to production.

Sken.ai provides automated scanning features and Generally, it is used to find out the bugs at early stages of the development of software (coding stage). With this code review, the quality of the software gets improved and the bugs/errors in the program code decrease. The Code Review Tools automates the review process which in turn minimizes the reviewing task of the code. It provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.

What does Sken.ai contain:- Here are some detailed points that I have discussed about the Sken.ai. It helps to scan every application in any language from any CI/CD Platform (such as Jenkins, Bamboo, etc). It uses all scan types SAST, DAST, SCA, Secrets, and more to scan at the coding time or the runtime. As I said before it enables security by simply adding a 2-line code snippet into CICD to enable the complete range of tests. It provides open-source scanners packaged and managed, There's No need to set up, install or update scanners. It selects & auto-discover scanners as per scanning context.

Sken.ai performs the most comprehensive security scanning and testing of applications during the CI/CD process. This includes not only static source code analysis (SAST) and software composition analysis (SCA), but also dynamic security testing (DAST) to uncover run-time security holes as well. This ensures that all the common avenues hackers use to execute security breaches are discovered before going to production. By providing the relevant scan type at the right stage in the DevOps and CI/CD stage, Sken.ai covers more vulnerability and compliance holes than anyone's capability alone. It uses all major development languages & frameworks including Java, PHP, Python, Microsoft .NET, NodeJS, Ruby, GO, and more.

Sken.ai applies machine learning and data science to normalize and prioritize all the application security issues found across testing categories. It provides AI-Based Analysis, and it reduces noise up to 97%. It Aggregated scan findings View and filter findings across multiple scans. It tracks bugs through code changes and provides standard risk ratings across all scan types. DevOps and AppSec teams can manage their entire bug fix and mitigation processes with Sken.ai, including prioritization and risk scoring, bug verification, task assignment through resolution. It is Intelligent & prioritized to fix critical findings first. It helps users to Manage findings from the Sken.ai portal.

Here are some Open Source Scanners included in Sken.ai internally:

SCA: OWASP Dependency Leaks.
SAST: Find-Sec-Bugs, NodeJScan, Brakeman, Bandit, Gosec, PHPCodeSniffer, ES Lint, TSLint.
DAST: OWASP ZAP
Secrets: Gitleaks, Trufflehog.

Static application security testing (SAST), or static analysis, is a testing methodology that analyzes source code to find security vulnerabilities that make your organization’s applications susceptible to attack. SAST scans an application before the code is compiled. It’s also known as white box testing. SAST takes place very early in the software development life cycle (SDLC) as it does not require a working application and can take place without code being executed. It helps developers identify vulnerabilities in the initial stages of development and quickly resolve issues without breaking builds or passing on vulnerabilities to the final release of the application.

A dynamic analysis security testing tool, or a DAST test, is an application security solution that can help to find certain vulnerabilities in web applications while they are running in production. A DAST test is also known as black-box testing, it is performed from the outside.

Traditional security scanners are proprietary, expensive, and difficult to use. They also need a team of app sec specialists to operate them, thus adding to the total cost. Sken.ai uses an innovative approach that uses a combination of popular, effective, and widely used open source security scanners to dramatically reduce the cost. Sken.ai provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring.

Vulnerability scanning is a continuous process that fits throughout an app’s software development life cycle (SDLC). Frequent scanning, especially in the early stages of development before upstream components are layered onto the source code, makes it much easier to fix issues before they become costly problems.

Sken.ai also provides IAST(Interactive Application Security Testing) Scanners are DAST scanners with some SAST capabilities. They test whether known code vulnerabilities can be exploited in the running app. Developers often layer API scanners over other tools, such as SAST and DAST, to ensure their APIs stay secure, even after a code or interface change. API scanning focuses on uncovering existing and potential security vulnerabilities like SQL Injection, Content-Type Missing, Misused Exception Handling, and Parameter Tampering.

For development, DevOps helps users to automate source code by providing IDE's like Microsoft Visual Studio, NetBeans, IntelliJ Idea, Eclipse, PyCharm, etc. For version control management Git, GitHub, GitLab, CVS, SVN, and for CI/CD pipeline DevOps provides Jenkins, CircleCI, Bamboo, Travis CI, etc. It enables security with a simple, 2-line code snippet to enable the complete range of tests. It provides Comprehensive testing across all scan types SAST, DAST, SCA, Secrets, and more. Perform security testing for all major development languages/frameworks. Sken.ai application developers are notified of security vulnerabilities as they commit and build their code. Also, Sken.ai helps app sec and DevOps respond to the vulnerability alerts, prioritize the issues found, and fix them during the code and build process.

A CI/CD pipeline is a series of steps that must be performed to deliver a new version of the software. Continuous integration/continuous delivery (CI/CD) pipelines are a practice focused on improving software delivery using either a DevOps or site reliability engineering (SRE) approach.

A CI/CD pipeline introduces monitoring and automation to improve the process of application development, particularly at the integration and testing phases, as well as during delivery and deployment. Although it is possible to manually execute each of the steps of a CI/CD pipeline, the true value of CI/CD pipelines is realized through automation. A comprehensive set of application security testing (AST) tools help you test for and remediate security vulnerabilities in your CI/CD pipeline.

SecDevOps (also known as DevSecOps and DevOpsSec) is the process of integrating secure development best practices and methodologies into development and deployment processes that DevOps makes possible. SecDevOps consists of two distinct parts:
1. Security as Code (SaC).
2. Infrastructure as Code (IaC).

Cybercriminals take no prisoners. According to cybersecurity experts, SecDevOps, or the process of integrating secure development best practices and methodologies into the development and deployment processes, is the best way to do just that.

Here automation comes further there are many open-source plugins and security tools are available to provide and scan our application for security tests before the development state goes to production.

A website vulnerability is a weakness or misconfiguration in a website or web application code that allows an attacker to gain some level of control of the site, and possibly the hosting server.

A web application security scanner is a software program that performs automatic black-box testing on a web application and identifies security vulnerabilities. Scanners do not access the source code; they only perform functional testing and try to find security vulnerabilities. Various paid and free web application vulnerability scanners are available. Security Testing is a type of Software Testing that uncovers vulnerabilities of the system and determines that the data and resources of the system are protected from possible intruders. It ensures that the software system and application are free from any threats or risks that can cause a loss.

Ruby on Rails is the most popular open-source web application framework. It’s built with the Ruby programming language. Rails help you build websites.

Ruby on Rails is an open-source web development framework, which provides Ruby developers a time-saving alternative to develop code. It is a collection of code libraries, which offer a ready-made solution for repetitive tasks like developing tables, forms, or menus on the website. Ruby on Rails also comes with a unit testing setup called RSpec, which is very easy to learn. It uses some other scanners to scan application source code. Static analysis security vulnerability scanners are used in Ruby on Rails for scanning. So here, we can use any static analysis security tools to scan ruby on rails.

Sken.ai aligns with the security framework OWASP ZAP. It is an open-source tool which is offered by OWASP (Open Web Application Security Project), for penetration testing of your website/web application. It helps you find the security vulnerabilities in your application. Here are the OWASP top 10 security threats that your website/application might face: SQL injection, Broken authentication and session management, Cross-site scripting (XSS), Broken access control, Security misconfiguration, Sensitive data exposure, Insufficient attack protection, Cross-site request forgery (CSRF), Using components with known vulnerabilities, and Underprotected APIs. ZAP provides the following features: active scans, alerts, API authentication, and verification, etc.

Zap is used as a proxy server, it allows the user to manipulate all of the traffic that passes through it, including traffic using HTTPS. It helps you find the security vulnerabilities in your application. Here is a list of the main features: Intercepting Proxy, Automated Scanner, Passive Scanner, Brute Force Scanner, Fuzzer, Port Scanner, Spider, Web Sockets, REST API.

ZAP is an easy-to-use tool. Following are some more reasons for using ZAP:

1. Ideal for both beginners and professionals

2. Cross-platform - works across all OS (Linux, Mac, Windows)

3. Reusable

4. Can generate reports of the results, etc.

A penetration test, also known as a pen test, is a simulated cyberattack against your computer system to check for exploitable vulnerabilities. In the context of web application security, penetration testing is commonly used to augment a web application firewall (WAF). Penetration testing in simple terms is a simulation of a process a hacker would use to launch an attack on a business network, attached devices, network applications, or a business website. The purpose of the simulation is to identify security issues before hackers can locate them and perform an exploit.

Mobile application security focuses on the software security posture of mobile apps on various platforms like Android, iOS, and Windows Phone. Mobile application security testing involves testing a mobile app in ways that a malicious user would try to attack it. Effective security testing begins with an understanding of the application’s business purpose and the types of data it handles. From there, a combination of static analysis, dynamic analysis, and penetration testing results in an efficient holistic assessment to find vulnerabilities that would be missed if the techniques were not used together effectively.

List of tools available in the market for all types of scanning & the sken.ai is provide same scanning features in one place:

DAST Scanning Tools:-

SAST Scanning Tools:-

Combine Scanning (SAST/DAST/SCA) tools and alternative to sken.ai tools:-

Cloud Security Scanning Tools:-

Ruby on Rails application scaling tools:-

SCA Scanning Tools:-

OWASP Dependency Leaks.

How is it different than other tools in market:-

I have used the sken.ai scanning tool, I have mentioned some features which I liked,

1. There is "No App Security Expertise" required to use this scanner.

2. "Commercial Scanners are expensive" so mid-markets can decide to use open-source scanners. However, Open-source scanners are extremely unmanageable. There are too many of them, so it is confusing to select what you'd exactly need.

3. This includes code level scanners like SAST, SCA, Secrets, License, Test/Deploy level scanners like DAST, Containers, API, Serverless, etc. It is extremely difficult for mid-markets to operationalize scanning with these multiple types of scanners.

The mid-markets need a product that:

Provides app security with no required app-security expertise.
Is affordable yet manageable.
Supports comprehensive scanning across the application, yet simple to understand and use.

Sken tackles each one of the requirements of the mid-markets. There are three pillars in Sken, and they correspond and tackle these three core requirements of the mid-markets.

Sken provides a product that can be used by DevOps with no app security experience.
Sken packages open-source scanners in a SaaS orchestration layer and automates them in CI/CD. This SaaS orchestration does a bunch of the heavy lifting, but it makes it so simple for the end-user to operate.
Sken does all scan types. There includes

Code-level scanners like

SAST (Static Application Security Testing) for scanning source code across multiple languages.
SCA (Software Composition Analysis) for scanning open source libraries that are included in your application.
Secrets for scanning open passwords
License for scanning license files

Test level scanners like

DAST (Dynamic Application Security Testing)

Deployment level scanners

Containers
API, etc.

It provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.

Why should you use Sken.ai:- Because of the number of different features sken.ai provides, It helps in static code analysis, Source code analysis, code review, code complexity, and penetration testing free of cost. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.

What are the integrations possible in sken.ai:- Sken.ai integrates with the CICD pipeline with 3 simple steps. 1. Add Sken.ai in CICD. 2. Write sken.yml as you need. 3. Review issues in the Sken.ai portal. It uses automated open-source scanners providing tools like SCA, SAST, DAST, Secrets management, and many more. Those are included:-

SCA: OWASP Dependency Leaks.
SAST: Find-Sec-Bugs, NodeJScan, Brakeman, Bandit, Gosec, PHPCodeSniffer, ES Lint, TSLint.
DAST: OWASP ZAP
Secrets: Gitleaks, Trufflehog.

Here is the demo of sken integration with jenkins and travis-ci:-

Jenkins integrate with sken:- https://drive.google.com/file/d/1OxpziWQZ6lEZ4Q6c7WYWAw3DTNf9qLd6/view?usp=sharing
Travis CI integrate with sken:-

https://drive.google.com/file/d/1-zEbJ0rwzkR_vpvDkBvTXY4MwRXZcOFX/view?usp=sharing

DevSecOps Pune Meetup 4

2019-03-09T11:19:00.001-08:00

9th March 2019

This is our 4th DevSecOps meetup. More and More and More swag sponsors added to the list. New sponsors: Elastic, Sonatype, Synk along with previous sponsors i.e. Polyverse and Cloudneeti already into the list. The meetup head count was usual and exact. We have been hitting the correct and intended audience.

Qualys Pune was the venue for the meetup. This is the 2nd time we organized the meetup at Qualys. We started at 10.30 am. We had a huge list of topics this time and I was pretty sure not all of them are going to get discussed since the most voted topics were very interesting. The following topics got discussed:

Machine Learning for Security: This topic got discussed for a good 40 minutes although none of us was an ML engineer and only understood theoretical ML concepts. The topic was chosen more for a discussion as the person suggester of the topic just had the curiosity to see if this can be done. The topic unfolded as people discussed and understood how ML works and how is it being used in the industry. Different tools like TensorFlow, Pandas, etc got discussed. Once we had a clear picture of ML, we moved to understand what security breaches we experienced in our Ops lives in industry. Based on our experience, we framed logic as to how ML algorithm could be written by Data Science engineers by studying security auth logs and application logs. We also discussed how ELK stack could be used to prove a security attack on system and further. We agreed to the part that as an Ops person we can only best provide inputs and prove security issues in systems. An ML engineer should be the best person to provide inputs on what ML algorithms could be used to mitigate security issues.

Ansible for DevSecOps CI/CD pipeline: Ansible was suggested multiple times in the past meetups as well but always went not noticed and never got discussed. In this session, we started with Ansible and went on an on for a good 1 hour. We understood how Ansible is best used in different organizations. We also discussed how a bad code in Ansible can mess up and not make any difference between olden days shell script. We discussed Idempotency feature of Ansible. We also discussed as to how Ansible is being used by some firms for Provisioning, Configuration Management and Deployment altogether. We agreed how many used Puppet for multiple jobs and later moving to a different tool was difficult for them. Hence using different tools for specific different jobs could be useful. We discussed a typical CI/CD workflow with Terraform for provisioning, Ansible for CM and Fabric/Capistrano or different language specific deployment tools for easy deployments and rollbacks. We also did a small white board presentation on how CI/CD can be used for all 3 purposes. We also discussed DR strategies, Cloud managed services like auto-scaling services etc.

Git for securing code : The suggester was a Developer and knew exactly how was not being used in the best manner to ensure security and highlighted some best ways like code review practices, git hooks to be used in Jenkins, linting analysis using pre-commit hooks, static code analysis with SonarQube before merge. We discussed this in details already in CI/CD pipeline above as well.

Burpsuite : Burpsuite was a tool that many knew but never got a chance to use as such. The suggester of the topic had extensive experience using it and spoke about multiple possibilities and usecases that could be achieved using Burpsuite. He also highlighted on how precautions need to be taken before using this tool for testing giving examples of one of his projects. Burpsuite indeed was very interesting for many members and we decided to have a demo session on this.

Takeaways from this session for speakers to prepare were:

Session required on ML in security
Demo on CI/CD in DevSecOps
Demo on Burpsuite

Many topics that did not get discussed were:

Nexus scan in Jenkins pipeline
SonarQube for secure code analysis
Pentesting with Python
IOT Security
Metasploit for pentesting
Cloud security
Prowler
Automobile software release cycle and missing lacking devops chain
Maven
Regulations

In the end Shirish wanted to highlight the recent All Intel Chips Open To New 'Spoiler' Non-Spectre Attack

Some clicks :)

Source build Envoy proxy on Ubuntu 18.04

2019-02-20T13:36:00.002-08:00

sudo apt-get update
sudo apt-get install openjdk-8-jdk build-essential autoconf libtool cmake ninja-build
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install bazel
wget https://dl.google.com/go/go1.11.5.linux-amd64.tar.gz
tar -xvf go1.11.5.linux-amd64.tar.gz
sudo chown -R root:root ./go
sudo mv go /usr/local
echo "export GOPATH=$HOME/go" >> ~/.profile
echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin" >> ~/.profile
git clone https://github.com/envoyproxy/envoy.git
cd envoy/
bazel build --package_path %workspace%:/home/<user>/envoy/ //source/exe:envoy-static

RELAX, ITS GOING TO TAKE LONG TIME

To generate the example configurations run the following from the root of the repo:

mkdir -p generated/configs
bazel build //configs:example_configs

RELAX, ITS GOING TO TAKE LONG TIME

I wrote an Ansible playbook too for this. I will publish it later on Github

Easily SCP/Rsync through bastion host or SCP/Rsync through multiple hops

2019-02-20T13:18:00.000-08:00

Often we work in environment where we need to copy files or directories from a local system to another server that can be accessed only through a Bastion host. In such cases, typically we transfer from local machine to Bastion and from Bastion to the intended server. This is time consuming, repetitive and unreliable too. There are many ways you can make this automated. I found a way to get this done through SSH tunneling. Here's how it works:

There are 3 machines involved here:

localhost
Bastion host
Intended server

1. Create an SSH tunnel from localhost to the intended host through bastion. The tunnel will be created from port 1234 at localhost. You may choose any other port.
ssh -L 1234:<intended_server>:22 <user>@<bastion-host> cat -
2. In a new tab initiate the file/directory transfer using the tunnel port
scp -P 1234 <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

As I did this, I realized SCP is very slow in getting the transfer done due to its linear and sequential file transfer behavior. Hence, I used Rsync which made it pretty fast due to its delta based transfer algorithm

rsync -avz -e "ssh -p 1234" <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

DevSecOps Pune Meetup 3

2019-02-09T22:37:00.001-08:00

9th Feb 2019

This is our 3rd DevSecOps meetup. I am glad to see the head count getting even better. Another Goodies sponsor added to the list. Cloudneeti Software sponsored T-shirt and Mugs to best presenter.

For this meetup, we got an exact count again with 2-3 last minute exits. Cloudneeti Software was the venue for this meetup. We started sharp at 10.30. Surprisingly the topics discussed this time were pretty advanced and much looked out for. The following topics got discussed:

Kali Linux for security : The topic was super hit and got discussed for pretty long hours. We spoke for a good 20 minutes on Kali Linux and only stopped because the next topics were also equally interesting. We spoke about the old distro Backtrack and how Kali got introduced. Others added their inputs as to how Kali is used by different SecOps people in industry. Overall we all agreed that Kali was a really vast topic to be just discussed and this rather needed a presentation + demo with so many tools within.
OWASP Top 10 web app security risks: As we discussed Kali, we also spoke about OWASP top 10 in the same discussion and this got prolong for another 20 mins. Not all of us were aware of all OWASP top 10 attacks hence we Googled it just for making notes, however with time limitations we could only discuss the most famous SQL Injection and a little about XSS. We also concluded that this needed a bigger session.
CI/CD in DevOps Pipeline: DevSecOps CI/CD pipeline was discussed shortly in 1st meetup but we did not deep dive in it then. In this meetup we actually went to the depth of Jenkins, Git, SonarQube, Static code analysis, Container Image security, SecOps role to play, Vulnerable libraries being used by devs and how to resolve this. How Github has inbuilt vulnerability analysis done now and many more details. We also drew the following CI/CD architecture diagram for DevSecOps pipeline.

Ansible secure key rotation: This was more of a question to the forum as to how this could be done since Chef and Puppet use clients and use their own keys for security, while Ansible uses SSH. We agreed to the fact that the keys must get rotated and many companies do follow this. Ansible's authorized_keys module helps you to rotate keys was the perfect answer suggested.
Security Patching at scale in cloud: This was again a question and the answer was many companies did this using cloud native tools to create a golden images and other tools to get into the image to verify if the images were CIS compliant this is not just for images but also for Cloud environment and resources in general. Once again Prowler was discussed to evaluate the cloud environment.

Takeaways from this session for speakers to prepare were:

Kali Linux hands on demo
OWASP Top 10 risks
CI/CD in DevSecOps

A few topics that did not get discussed were:

Metasploit for Pentesting
GDPR automation
Deploying Software securely
Cloud security trends 2018-19

An important Topic that Budhram did put forth as to what minimum qualifications and expertise do companies look for in a fresher candidate to call him an eligible candidate for DevOps Engineering further. This was a good debate cum discussion that we all spoke about in the end.

This turned out to be a long event in spite of small number of attendees. Ashish, Budhram, Shrikant and Dhiru got goodies to keep discussion more happening and actively participating in all topics as well. Some others got stickers.

Some clicks :)

Copying ssh keys easily

2019-02-09T19:55:00.000-08:00

I use VMs/Vagrant a lot in my day work for all sysadmin/devops automation. One of the problems that I always face with the systems is to authorize my server for the 1st time with my master host. If I am using 10 VMs I need to authorize them 10 times? I wrote a small script to automate this process:

Create a "list" file and add all IPs and hostnames for the VMs in it.
Create a "password" file to write your SSH password in it, you may choose to write the password in the bash, however I feel this gives me the flexibility to add the bash to my source code if need by putting password file in a.gitignore
Next create a shell script that will read the IP addresses and the hostnames from the "list" file and password/s from the password file(I generally keep the same password for all VMs for simplicity)
Remember 2 commands are useful here ssh-copy-ip and ssh-keyscan. Here's how you use them:

    ssh-keyscan -H <IP> >> ~/.ssh/known_hosts                                                                                                                                
    sshpass -f <password> ssh-copy-id -i ~/.ssh/id_rsa.pub <USER>@<IP>

The ssh-keyscan command command is for gathering the public ssh host key of a VM host specified. After collecting the publich ssh-key it adds it to your localhost. You can verify this by checking the contents of "~/.ssh/known_hosts"

The ssh-copy-id command copies the public key of your default identity (otherwise use -i identity_file for other identities) to the remote host. You can verify this by checking the content on ~/.ssh/authorized_keys in the VM host.

The final script looks like this with a loop:

#!/bin/bash                                                                                                                                                                 
user="vagrant"                                                                                                                                                              
for ip in `cat ./list`; do                                                                                                                                                  
    ssh-keyscan -H $ip >> ~/.ssh/known_hosts                                                                                                                                
    sshpass -f password.txt ssh-copy-id -i ~/.ssh/id_rsa.pub $user@$ip                                                                                                      
done

That's how my "list" file looks like:

consul-server1                                                                                                                                                              
consul-server2                                                                                                                                                              
bootstrap-server1                                                                                                                                                           
client1                                                                                                                                                                     
client2                                                                                                                                                                     
client3                                                                                                                                                                     
client4                                                                                                                                                                     
client5                                                                                                                                                                     
192.168.3.111                                                                                                                                                               
192.168.3.112                                                                                                                                                               
192.168.3.121                                                                                                                                                               
192.168.3.151                                                                                                                                                               
192.168.3.152                                                                                                                                                               
192.168.3.153                                                                                                                                                               
192.168.3.154                                                                                                                                                               
192.168.3.155

I purposely add IP as well as hostname as I keep using them interchangeably. I also came to know about ansible authorized_keys module that does the ssh-copy-id task:

- name: Set authorized key for user ubuntu copying it from current user
authorized_key:
    user: ubuntu
    state: present
    key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_rsa.pub') }}"

However, you will still need the the ssh-keyscan here. This script goes handy for ops who keep destroying their local environment and use a new one.This is available on Github: https://github.com/iamrawtion/ansible-autossh

AWS Tagger

2019-01-20T05:49:00.001-08:00

Image credits : jdhancock

Tagging in AWS is often not considered useful by many users. Tagging of resources in cloud and DC not only helps us identify resources but it can also do multiple other wonders that one might have never thought about. We don't tag resources in cloud for many reasons, laziness being the topmost reason.

Lets see why tagging is important:

Identification and Isolation: Tagging allows identification of resources as to what purpose a specific resource may have been created for. It also allows you to separate resources from each other. e.g. separating different environments.
Automation: When you tag resources with certain values you can ensure that your automation scripts only addresses certain intended resources and not all. e.g execute security patches on certain systems that need to be compliant.
Costing: You can identify based on tags as to which resource is costly and also make business decisions based on the results received.
Define ownership: You can also understand based on proper tags as to who are the stakeholders for a certain resource or group of resources.
Versioning: Sometimes when you need certain resources to be preserved based on its state, you may also versionize them based on tagging. Although AWS provides versioning mechanism for a few services, it may not be applicable to all of them.

In many organizations although the importance of tagging is understood a lot later. Until then its too late to start tagging and it becomes almost always a manual process to tag all the resources. Or you may need to write complex programs to identify systems and tag them as per your requirement. Thankfully, AWS Tagger comes to rescue if you have a requirement to tag your AWS resources. You may also bulk tag them to avoid a lot of manual work. So how do we do this.

Its a 3 step process to Bulk tag resources:

Collection : This is a simple process. Here all you need to do is, collect all the resources in a file. Hereafter you may process this data. AWS Tagger heavily depends of resource ID's of all the resources you create. Resource ID's are further used to implement all the tags. To get the resource ID's for all the resources, simply login to your AWS account and navigate to https://resources.console.aws.amazon.com/r/tags . On this page, you are given a field to enter the region for the resource you want to choose and all choose the types of resource. Choose "All resource types" here and click on "Find Resources" button. Click on the "Download" button to download the CSV data generated.
Identification and filtering: I recommend this step particularly to filter the data so that AWS Tagger can act on individual resources. Here you may use your excel skills to separate data based on resource types.
Tagging: Once the resources are separated, you may start executing AWS Tagger scripts as per the documentation provided on their Github page.

DevSecOps Pune Meetup 2

2019-01-18T19:27:00.001-08:00

For the 2nd DevSecOps meetup, we already had our first swag sponsor. For DevOps Pune, I received swags from Docker and Ansible. Hashicorp was also planning to send a few.

With DevSecOps our 1st swag sponsor was Polyverse :) I couldn't resist posting these.

For this meetup, I changed the RSVP format to get an exact count. I was expecting to start soon at a bigger location. We couldn't risk wasting of resources. Lean coffee needs logistics to take care of and hence needs to be addressed really well. Everyone who RSVP' at the meetup page was informed to Call/SMS/WhatsApp the organizers to book a slot. So the Meetup page RSVP meant nothing.

Qualys Pune, was our venue, logistics and refreshment sponsor this time. When I 1st told them about the meetup, this is how the they arranged the seating :) :

I told them about the format and later we changed the seating to best suit the format.

We had 12 RSVP and 10 attendees this time with just 1 last minute informed cancellation. That was a perfect number . We also made sure that the refreshment we take was a packed and long lasting one so we don't waste food.

With 10 attendees we stared pretty much on time and this time we had a huge list to discuss and the participants were from mixed domains unlike last meetup. This time we had QAs, Support Engineers, DevOps, Consultants and Developers altogether.

The following topics got discussed:

Understanding PKI - Public Key Infra (How SSL Works?)
CIS Benchmarking
SOAR (Security, Orchestration, Automation and Response)
AWS Compliance
Securing serverless in Azure (Function as a Service)
Debian Linux and Contributing to it
Microservices with an example

SSL and PKI got discussed a lot since Muneeb Shaikh really explained concepts that we were unaware of that goes behind PKI formation and how public and private keys work. A 5 minutes discussion prolonged to a good long 30 minutes with inputs from everyone.

SOAR was a new process and concept that Rahul spoke about and worth reading for everyone. CIS compliance was a hit one this time too. We understood that it was topic worth presentation hence added to the DevOps Pune meetups Agenda. Later we spoke about AWS compliance and how Prowler could be used to ensure compliance in AWS. We also discussed importance of good naming conventions and Tagging in AWS. There was a chaos when we spoke about both serverless and microservices resulting to a debate topic that we discussed at the end of the meetup. The final closure was on Debian systems and how we would contribute to it.

Takeaways from this session for speakers to prepare were:

PKI and SSL encryption
Microservices

A few topics that did not get discussed were:

Types of Security and importance of each
GRC - Governance, Risk and Compliance
Security Testing with Selenium
Achieving CI/CD with Ansible

This turned out to be a long event in spite of small number of attendees. Muneeb got Polyverse T-Shirt to keep PKI discussion happening and also actively participating in all other topics as well. Some others got stickers.

Some clicks :)