Winterdom

Kondex: Teaching Claude to use it

Wed, 16 Sep 2026 00:00:00 +0000

In my previous two posts, I covered how kondex builds its index and how you can query it. There’s one part I haven’t talked about yet, though: an index is not very useful if the LLM never thinks of using it. In this post, I’d like to go over how I got Claude Code to actually reach for kondex, which turned out to be surprisingly hard, and what I learned along the way.

If you look back at the architecture diagram in the first post, you’ll notice a box labeled “kondex plugin”, which contains a SessionStart hook and a skill. Let’s see why both of them are there.

Starting with a skill

Claude Code supports skills, which are just a folder with a SKILL.md file describing what the skill is for and how to use it. Claude sees a list of the available skills (only their names and a short description), and loads the full contents of one when it decides it is relevant to the task at hand.

Creating a skill was the obvious thing to try, so the first thing I did was write a skill, originally just called kondex, with a full reference on how to use the tool: the commands, the ref syntax, the flags, and so on. That didn’t work, however. Claude kept doing what it always does, grepping through the repository and reading whole files.

This led me to wonder if the skill name was just too abstract. Yes, the skill description is obviously important, but one would guess a name like kondex does not say anything about what the tool does. So I renamed the skill to find-declarations, and rewrote the description so that it started with the questions it could answer. Unfortunately, that did not seem to make much difference either.

Measuring it

At this point, I decided to stop guessing and measure what was going on. The setup was fairly simple:

A fixed question with a known answer, run against our monorepo. I picked one that kondex is very good at answering: which columns of one of our connectors can be filtered server-side?
A headless claude -p run for each configuration I wanted to test. Each configuration differed from the others by exactly one command line flag (--plugin-dir, --append-system-prompt, or an environment variable).
Counting the kondex calls, greps and total tool calls in each session transcript.
Three runs for each configuration, since the model is not deterministic and a single run doesn’t prove much.

Here are the results, averaged over the three runs:

What was installed	Runs that used kondex	kondex calls	Greps	Tool calls
Nothing	0/3	0.0	8.3	12.3
The skill	0/3	0.0	7.7	9.7
Skill + usage notes in the system prompt	3/3	7.7	0.0	8.0
Skill + `PreToolUse` hook (advisory)	3/3	5.0	3.7	11.0
Skill + `PreToolUse` hook (blocking)	3/3	7.3	3.0	14.3
Skill + usage notes from a `SessionStart` hook	3/3	5.0	0.7	6.0

The first two rows are the interesting ones. A session with the skill installed used kondex exactly as often as a session with nothing installed at all, which was never. And the question was not a tricky one either: it was taken almost word for word from the skill’s own description.

Note: Every configuration got the right answer, including the one with nothing installed. Claude can find the answer by grepping, it just takes longer and uses more context doing it. So this experiment was strictly about whether Claude used the tool or not.

Why didn’t the skill work?

The best explanation I have is that it’s a matter of timing. Claude chooses skills from that list of names and descriptions, but it is not looking at the list when it decides to grep for something. That usually happens many tool calls into a task, in the middle of tracking something down, and not at the start, when it is deciding how to approach the problem. So no amount of rewording the description was going to help.

On top of this, I would presume that built-in tools probably just take precedence over whatever skills describe.

What does work is having the guidance in context before the first search. You can see this in the third row of the table: putting the same usage notes directly in the system prompt got kondex used in every single run, and the greps went away entirely.

The two attempts using PreToolUse are also interesting. A PreToolUse hook runs before each tool call, so I wrote one that caught greps that looked like a search for a declaration, and suggested (or required) using kondex instead. This does get kondex used, but the hook only fires after Claude has already decided to grep, so you pay for the abandoned attempt: 11 to 14 tool calls, which is worse than having nothing installed at all. A good attempt, but ended up having to scrap it.

Why a plugin?

So the guidance needs to be in context right from the start of the session. There are a few ways to do that, but each of them has a problem:

--append-system-prompt: This works, but nobody is going to type that every time they start Claude Code.
A CLAUDE.md file: Adding the instructions to the repository would also work, but that means committing kondex-specific instructions into every repository that uses it. For our monorepo, that means a pull request against a code base shared by the whole team, for a tool that not everyone has installed.
A SessionStart hook: This runs when a session starts, and can add text to Claude’s context, which is exactly what we need. Hooks can also be distributed as part of a Claude Code plugin.

That’s the last row of the table, and it had the best results of all: kondex got used in every run, there were almost no greps, and Claude needed half the tool calls it used with nothing installed, because it skipped the exploratory grepping altogether.

What’s in the plugin

The plugin lives in the kondex repository itself, and it’s pretty small:

.claude-plugin/
  marketplace.json          makes the repository installable as a marketplace
claude/
  .claude-plugin/
    plugin.json             the plugin manifest
  hooks/
    hooks.json              registers the SessionStart hook
    session_context.py      the hook itself
  fragment.md               the usage notes the hook puts in context
  skills/
    find-declarations/
      SKILL.md              the full reference

Installing it only takes a couple of commands inside Claude Code:

/plugin marketplace add <path-to-kondex>
/plugin install kondex

Or, if you just want to try it for a single session:

> claude --plugin-dir <path-to-kondex>\claude

The hook is registered with a few lines of JSON:

json

{
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python \"${CLAUDE_PLUGIN_ROOT}/hooks/session_context.py\"",
            "timeout": 20
          }
        ]
      }
    ]
  }
}

The session_context.py script doesn’t do much more than that:

It runs kondex status to check whether the session is starting inside a git repository that kondex can work with. If kondex is not installed, or this isn’t a git repository, it returns nothing, so a session somewhere else doesn’t pay for anything. In my case, the tool will always be present, but it didn’t hurt to add just in case.
Otherwise, it reads fragment.md and returns it to Claude Code as additionalContext.

Using kondex status for the check was on purpose: it is the only command that reports on a repository without indexing it, so it’s safe to run at the start of every session.

Note: Make sure you use the additionalContext field. A hook can also return a systemMessage, but that is only shown to the user in the console, and never reaches the model.

If you want to turn the hook off without uninstalling the plugin, you can set the KONDEX_SESSION_CONTEXT environment variable to off.

The downside:: kondex supports only a specific subset of languages. Having the plugin enabled when working on a repo that uses an unsupported language would probably not… be a great user experience. Something I might look into improving later on.

The fragment

The fragment.md file contains the text Claude actually sees at the start of every session. It’s about 150 lines long, and it starts like this:

## Finding declarations

This repository is indexed by `kondex`. **Use it before grepping for a class
name or opening a source file to look up a declaration.** A query costs ~0.5s
and a few hundred tokens; reading a 4,000-line file to find one signature costs
~50,000.

After that comes a short cheat sheet with every command and its most useful flags, followed by a few paragraphs covering the things Claude kept getting wrong.

Note: This isn’t free. The fragment adds roughly 2,300 tokens to the context of every session in a repository. Considering that a single read of a large source file can cost 20 times that, I think it’s a good trade, but it’s something to be aware of.

The skill is still part of the plugin, but it’s now more of a fallback. In the configurations that used the fragment, the skill was only loaded zero or one times out of three, because the fragment already had what Claude needed. A skill costs nothing until it’s used, so it is still a good place for the full reference that doesn’t fit in the fragment.

Something I really wished Claude Code had better support for was reporting metrics for plugin usage. Unfortunately, the OpenTelemetry-based instrumentation in Claude Code is useless unless your plugin comes from a “known marketplace”. Sigh.

Trying it on real work

The results in the table come from a single question, on a single repository, with three runs for each configuration. That’s a good start, but I wanted to see what would happen on real work. So I took an actual ticket from our backlog and had Claude Code investigate and fix it twice: once with only the skill installed, and once with the plugin.

The good news is things worked as expected: The fragment injected by the hook did what it was supposed to do, and Claude went from never using kondex to calling it 10 times, the first one before it did any grepping.

Unfortunately (and it was surprising), it kept grepping just as much as before. Both sessions ran 32 searches over the source code, and used about the same number of tool calls and output tokens. Even the time was almost the same (49 minutes against 48). So the “half the tool calls” result from the table did not carry over to a real investigate-and-fix task at all.

Claude Code is fantastic at researching its own session transcripts, so of course I had it go through the transcripts for these attempts. I found three reasons for this:

About half of the remaining greps were looking for references rather than declarations: who calls this method, or where is this helper used. kondex only indexes declarations, so no prompt was going to fix that. (This is also what eventually led to the experimental callers command I skipped in the last post.)
The fragment was missing two flags: --sig and --limit. Twice, Claude was using kondex correctly, then hit the default limit of 30 results, or couldn’t see the declaration it needed, and went back to grep. Both flags were documented in the skill, but since the skill was never loaded, Claude never saw them.
kondex indexes the content staged in git. While Claude is editing files during the fix, those files are stale in the index by design, so grep is actually the right tool for them.

After adding the missing flags to the fragment, I tried again with a different ticket. This time Claude ran 13 greps instead of 32, and the whole session took 23 minutes instead of 49. It’s a single run on a different task, so I wouldn’t read too much into the exact numbers, but it was a nice step in the right direction.

Designing the output

Once Claude was using the tool regularly, it also became clear that the output of kondex is part of the prompt too (or rather part of the model input tokens), and it needs to be designed with that in mind. Here are a couple of examples from those same sessions:

Counting results: In one session, Claude asked for every implementation of a very common method. kondex correctly returned 180 results, but when a tool result is that large, Claude Code saves it to a file and only keeps a preview of the beginning in context. The total count was at the end of the output, so it got cut off, and Claude’s final report said there were around 30 implementations, when the real number was 145. This is why long listings now show the total count at the top as well, and why the --count flag exists.
Using line ranges: Every result from kondex includes a line range, and the fragment told Claude to use it to read only the lines it needed. Claude did use the start line, but across 19 reads, it used the end line only once. In one case, it read the same file six times in overlapping chunks, 28.7 KB in total, to get to a method that was only 944 bytes long. The problem is that the Read tool takes a starting line and a number of lines rather than a start and an end, and that bit of arithmetic was enough to lose it. Rewording the instructions wouldn’t have helped much here, but adding show --body did have a significant impact.

Conclusion

The surprising conclusion from this was that the skill on its own turned out to not not be sufficient to get Claude to use kondex, since Claude doesn’t look at the skill when it decides to grep for something. Adding the usage notes at the start of every session through a plugin hook is what finally made the difference, and it took lots of testing and careful measuring of each attempt to discover this.

Kondex: Searching the index

Tue, 15 Sep 2026 00:00:00 +0000

In my last post, I introduced kondex, a declaration indexing tool I wrote for our team. Today, I’d like to spend some time going over how to use the index to navigate the code base.

Note: kondex will automatically detect a stale the index on any command and do an incremental scan if necessary.

Search

The first entry point to the index is the search command, which uses a case-insensitive substring match on the declaration name and finds matching declarations:

> kondex search OnViewportWidthChanged
method  Winterdom.Viasfora.Rainbow.RainbowLines#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Rainbow/RainbowLines.cs:154-157
method  Winterdom.Viasfora.Text.CurrentLineAdornment#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Core/Text/CurrentLineAdornment.cs:83-85
method  Winterdom.Viasfora.Text.PresentationMode#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Core/Text/PresentationMode.cs:28-31
method  Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Rainbow/Util/ToolTipWindow.cs:76-81

4 results

Note: The result of a search in kondex lists what we call a ref, which will usually take the form of Type#member(parameter types), with constructors using the <init> name.

Members declared outside of a type will be owned by the module/package they are declared in:

Go functions will be owned by their package (e.g. scan#Run())

Python / TypeScript functions at module level will have an empty owner (e.g. #UseThis())

When searching, exact names will rank first, then prefixes, then any other matches. A term containing ., # or ( is matched against the full ref rather than the name, so search 'Rainbow#On' works.

There are a couple of interesting additional filters that can be used for search. First, you can narrow down results to only declarations of a specific kind:

> kondex search --kind class Rainbow
30 results for "Rainbow" (kind class); more matched

class  Winterdom.Viasfora.Rainbow.Rainbows                          src/Viasfora.Rainbow/Rainbows.cs:4-19
class  Winterdom.Viasfora.Tags.RainbowTag                           src/Viasfora.Core/Tags/RainbowTag.cs:6-11
class  Winterdom.Viasfora.Rainbow.RainbowLines                      src/Viasfora.Rainbow/RainbowLines.cs:52-359
...
class  Winterdom.Viasfora.Rainbow.RainbowKeyProcessorProvider       src/Viasfora.Rainbow/RainbowKeyProcessor.cs:11-24

30 results shown; more matched — raise --limit, or narrow with --kind/--path

Note that in the previous example, kondex found over 30 matches, so the result is partial. You can do a search --count to get just the total number of results without any extra details.

You can also limit results by only searching within a specified path:

> kondex search Rainbow --path .\src\Viasfora.Core\
class        Winterdom.Viasfora.Tags.RainbowTag                              src/Viasfora.Core/Tags/RainbowTag.cs:6-11
constructor  Winterdom.Viasfora.Tags.RainbowTag#<init>(IClassificationType)  src/Viasfora.Core/Tags/RainbowTag.cs:8-10
field        Winterdom.Viasfora.Guids#RainbowOptions                         src/Viasfora.Core/Guids.cs:9
field        Winterdom.Viasfora.PkgCmdIdList#cmdidRainbowNext                src/Viasfora.Core/PkgCmdIdList.cs:15
field        Winterdom.Viasfora.PkgCmdIdList#cmdidRainbowPrevious            src/Viasfora.Core/PkgCmdIdList.cs:14

5 results

Outline

Another extremely useful feature of kondex is being able to return an outline for a file (meaning, a simplified view of the declarations in the file). Again, this saves LLM tools from having to read the file and extract them directly:

> kondex outline .\src\Viasfora.Languages\Sql.cs
src/Viasfora.Languages/Sql.cs  (11 declarations)

  class        Sql                             8-25
  field          #knownContentTypes            10-11
  property       #SupportedContentTypes        12
  property       #Settings                     13
  constructor    #<init>(ITypedSettingsStore)  15-21
  method         #NewBraceScanner()            23-24
  class        SqlSettings                     27-43
  property       #ControlFlowDefaults          28-32
  property       #LinqDefaults                 33-35
  property       #VisibilityDefaults           36-38
  constructor    #<init>(ITypedSettingsStore)  40-42

The outline command also accepts a folder path as an input, in which case it will produce an outline of all files in the directory (recursively). Since this could be a very large result, it will only report a number of findings (200 declarations by default). You can use --limit and --offset to narrow it down.

Members

One common question an LLM might want to answer is: What members does this type declare? That’s what the members command is for:

> kondex members Winterdom.Viasfora.Rainbow.CharPos
struct Winterdom.Viasfora.Rainbow.CharPos  src/Viasfora.Languages/CharPos.cs
  public struct CharPos

  field        CharPos#ch                    5
               private readonly char ch
  field        CharPos#state                 6
               private readonly int state
  field        CharPos#position              7
               private readonly int position
  field        CharPos#Empty                 8
               public static CharPos Empty
  property     CharPos#Char                  10
               public char Char
  property     CharPos#State                 11
               public int State
  property     CharPos#Position              12
               public int Position
  constructor  CharPos#<init>(char,int)      14-15
               public CharPos(char ch, int pos) : this(ch, pos, 0)
  constructor  CharPos#<init>(char,int,int)  17-21
               public CharPos(char ch, int pos, int state)
  method       CharPos#ToString()            27-29
               public override string ToString()

10 members

There are two options that can be very useful to members:

The --public flag will ask kondex to only list public members of the specific class. The rules for each language will determine what is considered public:
- Java / C# includes only strict public members (not protected, internal, or package)
- Go will define it based on capitalization
- TypeScript will define it based on export
The --inherited flag will ask kondex to not only list members directly defined in the specified type, but also in the superclasses (or interfaces) in the source (it will obviously not return anything that comes from external, unindexed libraries). Since kondex only does a basic lexical scan, the matching is done by name, which means this is supported strictly as a best-effort.

Show

Once you’ve found the declaration you are looking for, you can show it:

> kondex show "Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)"
method Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)
  file         src/Viasfora.Rainbow/Util/ToolTipWindow.cs:76-81
  declaration  private void OnViewportWidthChanged(object sender, EventArgs e)
  declared in  Winterdom.Viasfora.Util.ToolTipWindow

Note how this gives you the complete method declaration, along with a location (file + line range). This is the ideal case when all you care is about what the declaration looks like (for example, you just want the method signature)

One interesting feature of show is that kondex is aware of language-specific syntaxes for documenting declarations, such as javadoc comments or C#’s /// comments, and will return those as part of the show command where available:

> kondex show "Winterdom.Viasfora.Options.MainOptionsControl#Dispose"
method Winterdom.Viasfora.Options.MainOptionsControl#Dispose(bool)
  file         src/Viasfora/Options/MainOptionsControl.Designer.cs:12-17
  declaration  protected override void Dispose(bool disposing)
  declared in  Winterdom.Viasfora.Options.MainOptionsControl

  Clean up any resources being used.

  @param disposing true if managed resources should be disposed; otherwise, false.

Note: The above example is interesting in that it searches the declaration based on a partial prefix: The ref used is missing the parameter list!

There are cases when an LLM might want more than a signature. For example, it might need to read what the method implementation looks like. In that case, kondex show also has the --body argument, which gives you its source or body:

> kondex show "Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)" --body
method Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)
  file         src/Viasfora.Rainbow/Util/ToolTipWindow.cs:76-81
  declaration  private void OnViewportWidthChanged(object sender, EventArgs e)
  declared in  Winterdom.Viasfora.Util.ToolTipWindow

  76      private void OnViewportWidthChanged(object sender, EventArgs e) {
  77        this.tipView.ViewportWidthChanged -= this.OnViewportWidthChanged;
  78        if ( this.tipView.ViewportRight > this.tipView.ViewportLeft ) {
  79          this.ScrollIntoView(this.pointToDisplay);
  80        }
  81      }

This is very useful to LLM tools as it saves them the need to open the file directly and scan the line range.

Tip: In some cases, you might have multiple declarations with the same name. In that case, you will want to either specify --path or --kind to ensure you get the right one. Note that if the input to show is ambiguous, it will provide a list to disambiguate:

> kondex show OnViewportWidthChanged
"OnViewportWidthChanged" matches 4 declarations:
  method  Winterdom.Viasfora.Rainbow.RainbowLines#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Rainbow/RainbowLines.cs:154-157
  method  Winterdom.Viasfora.Text.CurrentLineAdornment#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Core/Text/CurrentLineAdornment.cs:83-85
  method  Winterdom.Viasfora.Text.PresentationMode#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Core/Text/PresentationMode.cs:28-31
  method  Winterdom.Viasfora.Util.ToolTipWindow#OnViewportWidthChanged(object,EventArgs)  src/Viasfora.Rainbow/Util/ToolTipWindow.cs:76-81
retry with one of the refs above, or narrow with --kind/--path

Terraform

I thought including a short example of using kondex on code bases including terraform scripts would be interesting, as the syntax used can be somewhat different from other languages. For example, searching:

> kondex search dns
30 results for "dns"; more matched

data_source  data.azurerm_resource_group.dns-rg                  terraform/dev/dns.tf:29-32
data_source  data.azurerm_resource_group.dns-rg                  terraform/modules/cdatakube_assets/external-dns.tf:5-8
data_source  data.azurerm_resource_group.dns-rg                  terraform/prod/security.tf:65-67
...

30 results shown; more matched — raise --limit, or narrow with --kind/--path

You can see Terraform introduces its own set of declaration kinds (data_source, variable, resource, module, output, local, provider). Showing a declaration still works the same way:

> kondex show azuread_application.azuredns-sp
resource azuread_application.azuredns-sp
  file         terraform/prod/security.tf:28-31
  declaration  resource "azuread_application" "azuredns-sp"
  members      none

  Create Azure AD App for azure dns

An interesting aspect for Terraform is that refs use the exact same syntax the configuration language uses to refer to things.

The workflow

By now, you can probably guess there’s an implicit workflow to using kondex:

search to find the name
outline or members to see the shape
show when you want the complete declaration, show --body when you need the implementation

In other words, the whole purpose is to provide a tool that helps an LLM narrow down as much as possible the possibilities before it reads anything.

One thing worth being honest about: The examples above are almost all from the Viasfora code base, which, let’s face it, is very small. The numbers don’t reflect much complexity. However, the tool can have substantial impact when operating on a large code base, particularly one that has lots of larger source files. This is where kondex really shines, as it lets the LLM avoid having to read larger files directly, or spend multiple cycles trying to figure out the exact boundaries of a declaration for a narrow read.

In the next post, I will cover something that was hinted at in the original article, but not explicitly mentioned: Why kondex comes with a Claude Code plugin!

Kondex: A declaration indexing tool

Mon, 14 Sep 2026 00:00:00 +0000

It’s been a very long time since I’ve written on this blog, but wanted to pick this up again, and thought tools would be an excellent topic.

Like most people in the Software Industry, I’ve been spending increasing amounts of time getting comfortable with LLM-based tools. This includes not only using tools such as GitHub Copilot and Claude Code, but also figuring out better ways to make our engineering team used to them and able to use them effectively.

In the next few days, I’d like to cover some of the internal tools I’ve built to help support this journey.

Note: Right now, these tools are purely internal and not published externally. However, if someone finds them useful, let me know and I’ll see what I can do.

First, I’d like to cover a tool called kondex.

What is Kondex?

Kondex is a utility tool that aims to simplify how LLM tools navigate a code base. There’s plenty of tools out there that claim to do this by indexing the code itself and then providing search operations on top of the index.

Kondex aims to solve the same issue, but from a slightly different perspective. Kondex does not apply any kind of RAG or vector-driven search. Instead, it works by building an index of all declarations (like classes, methods and other members), and then providing primitives to search through these declarations.

Why build it?

This is an excellent question. Like I said, there’s a number of similar tools out there (many open source), but after testing several of them, I couldn’t find one that would do exactly what I wanted and worked well with our specific requirements.

The biggest issues I ran into with other tools were:

Size of the code base: Our connector team works off a fairly big Git monorepo from which we build and deliver 1000s of artifacts, and contains millions of lines of code. Many of the tools I tried would need well over 10 minutes to even do the initial indexing, and even incremental indexing wasn’t very performant.
Support for Git worktrees: Our team relies heavily on using Git Worktrees. Most of the tools I tried are completely unaware of worktrees, and treat each separate branch on disk as a completely different repo, requiring generating a full index every time you created a new worktree. This was unacceptable for our use case.
Ignoring files: Our team makes heavy use of code generation. Our build process not only generates a lot of code files, but also copies stuff around while doing builds, all of which goes into a ./Release folder that’s excluded by .gitignore. The problem is most tools don’t know anything about this, and will happily index all source files on disk, polluting the index with either duplicate files or just generated code that’s irrelevant.

Solving these issues was the primary driver behind developing kondex.

Architecture

Kondex is implemented using Go. I wanted to use a language that would produce native tools for best performance. I am not an expert at Go, but that’s where Claude Code becomes very useful. It helped me build the initial working prototype fast and evolve it over time to fix issues and support new features. The following diagram shows the basic architecture:

Parsing source files is achieved through the use of Tree Sitter, which does an excellent job of abstracting the complexity of building a syntax tree; kondex uses this as the base for its declaration extraction engine. Currently, kondex supports parsing declarations from the following languages:

Java
Kotlin
C#
Go
Python
TypeScript
Terraform

Because of Tree Sitter, adding support for other languages is substantially easier now. Note: kondex also supports parsing our own metadata definition files, which is done through a custom linear scanner instead of tree-sitter.

The actual index is stored in SQLite.

Building the Index

I mentioned before one of the key problems I wanted to solve with kondex was the cost of building a full index of a repository. Solving this required doing a few tricks that ended up being a very interesting part of the project. The following figure illustrates the process of doing an initial scan for a code base:

The most obvious initial solution for speed was parallelizing the work: kondex will indeed scan input files in parallel through the use of multiple threads. However, it has a single thread writing to the SQLite database and it can be the bottlleneck. This is the reason we chose to rebuild indexes at the end of the scan, instead of ensuring they are kept up-to-date as the data is written.

Supporting Git Worktrees required getting a bit more creative. Instead of deciding the list of files to scan using the file system, kondex takes a bit of a different approach:

git ls-files -s gives us the list of all tracked files in the current worktree alongside its blob SHA.
Indexed declarations are stored per blob SHA, and shared by every worktree or branch.
Each worktree has it’s own snapshot, which tracks the path -> blob list

This means that when switching branches or worktrees, kondex is able to:

Keep a single index across branches or worktrees
Quickly figure out the files that have been changed, removed, or added from the current index and do an incremental scan, which can be done by comparing the new blob SHAs against the list already stored in the index.

Let’s illustrate this with an example. First, let’s build a full index of our current major branch:

> kondex scan
repo   C:/dev/src/<path>  (worktree: C:/dev/src/v26)
index  <path>\index.db
enumerated 145139 tracked files, 55834 indexable, 33173 under excluded directories, 46664 unique blobs (465ms)
cache: 0 hits, 46664 to read (116ms)
suspended query indexes and checkpoints for the load; both are done once at the end
  6162/46664 blobs, 77.0 MB, 129607 declarations
  12533/46664 blobs, 163.3 MB, 265517 declarations
  19370/46664 blobs, 260.2 MB, 410976 declarations
  26076/46664 blobs, 347.3 MB, 546584 declarations
  29517/46664 blobs, 399.7 MB, 615042 declarations
  37074/46664 blobs, 490.4 MB, 768798 declarations
  44792/46664 blobs, 590.0 MB, 923175 declarations
rebuilt query indexes (1.294s)

scanned v26 at c0049c880
  tracked    145,139 files (55,834 indexed)
  blobs      46,664 unique; 0 already cached, 46,597 read
  cache hit  0.0%
  read       618.1 MB
  extracted  963,379 declarations from 46,597 files
  syntax     326 files had error regions (0.70%); entities may be incomplete
  skipped    67 over size limit
  excluded   33,173 files under infoscripts/
  elapsed    18.9s  (enumerate 465ms, diff 116ms, read 15.29s, index 1.29s, write 1.18s, checkpoint 559ms)
  writer     busy 11.85s of the 15.29s read (78%), 2.58s of it committing

You can see doing a full index of over 45K files was done in less than 20 seconds. This is good enough that we can use kondex even without forcing the developer to manually trigger a full scan the first time the tool is used!

Now let’s switch to a worktree tied to a different major branch, and do an incremental scan:

> kondex scan
repo   C:/dev/src/<path>  (worktree: C:/dev/src/v25)
index  <path>\index.db
enumerated 147991 tracked files, 54909 indexable, 35470 under excluded directories, 41853 unique blobs (497ms)
cache: 34322 hits, 7531 to read (91ms)
suspended query indexes and checkpoints for the load; both are done once at the end
  2886/7531 blobs, 77.5 MB, 70259 declarations
  5915/7531 blobs, 177.1 MB, 150643 declarations
rebuilt query indexes (1.71s)

scanned v25 at 521d30840
  tracked    147,991 files (54,909 indexed)
  blobs      41,853 unique; 34,322 already cached, 7,493 read
  cache hit  82.0%
  read       225.5 MB
  extracted  192,292 declarations from 7,493 files
  syntax     55 files had error regions (0.73%); entities may be incomplete
  skipped    38 over size limit
  excluded   35,470 files under infoscripts/
  elapsed    9.85s  (enumerate 497ms, diff 91ms, read 5.63s, index 1.71s, write 1.12s, checkpoint 810ms)
  writer     busy 4.14s of the 5.63s read (73%), 714ms of it committing

Now we see the incremental scan for a different (older) branch being done in less than 10 seconds. This is a fairly high value due to the number of modified files, but branches with fewer changes will see very fast scans.

What’s next?

I will continue going over kondex functionality in a later post, by showing what you can do once an index has been built.

Azure File Storage and Billing Transparency

Wed, 11 Dec 2019 00:00:00 +0000

Azure Files is an excellent service, providing file-system (SMB) based storage on the cloud. It has been very useful for us together with Azure Kubernetes Service.

Functionality-wise, I have no complaints. While there are a few things that could be better, the service overall provides good value, and the addition of Premium File Storage is definitely a very interesting and exciting option.

I have been, however, dissatisfied with the Billing experience.

Cloud Billing Transparency

I’m a strong believer that Cloud-based offerings need to have a clear set of billing rules that customers can clearly understand and use to budget their cloud costs. There are 4 key elements that I believe contribute to having clear and transparent cloud billing for usage-based services:

The publicly available pricing information should clearly document what billing meters are used and how these relate to the prices listed.
For services that charge for communication, any related billing meters should have a clear relation to the underlying protocol operations and semantics.
The service should provide metrics that allow customers to understand their usage.
Any metrics should clearly relate to the corresponding billing meters, so that the customer can audit their bills for accuracy.

A customer that understands how your service is billed, budget properly for it, and also audit the usage is a happy customer.

The Azure Files Billing Conundrum

Why am I generally unhappy about how Azure Files are billed? It’s not so much that I object to the actual cost of the service (which is generally reasonable for most scenarios), but that I believe that Azure Files breaks several of the rules above.

Let me explain why.

Pricing Information vs. Billing Meters

The first issue I have with Azure Files is how pricing information is presented. There are two separate components: Storage and Operations.

Storage pricing is very clearly presented, but Operation pricing is very deceptive.

Billing meters are only partially related to how pricing is presented. While pricing talks about “Put, Create Container Operations” or “List Operations”, billing meters include things such as “protocol operations”, and “lrs write operations”. You can guess about how they are mapped in some cases, but not all of them.

I would much rather the pricing information was presented by outlining the meters used and what each one measures exactly.

Billing and Protocols

Azure Files operation pricing is stated in terms of the underlying operations on the REST API. This is fine if you’re accessing the service over the HTTP interface, but that’s not really the common usage for Azure Files.

Instead, you’ll mostly be accessing the service over SMB from Windows or Linux machines, and so there’s no clear relationship of how a File-System-based operation (such as Open File) maps to an operation on the REST API. This leads to significant confusion and makes it easy to budget incorrectly for the service, understimating the actual costs.

For example, I ran into a case where a service would often open a file, read from it, and then close it. I was pretty surprised that my Azure Files bill was primarily composed of “Write Operations”. Well, surprise-surprise, turns out that the file-system open a file operation is considered a write operation for billing purposes. There was absolutely no way for me to figure that out before I wasted a lot of time and a few support cases (which went nowhere) on it.

Meters and Metrics

Azure Files supports some basic metrics, which get stored in Azure Storage Tables on the same storage account. While these are useful in general, they are not very useful to audit your usage costs.

This is because the metrics, again, bear no direct relationship to the billing meters. For example, you will see things like this in your storage metrics:

user;Create
user;SessionSetup
user;Write
user;GetFile
user;QueryInfo
… and more

These bear a little bit more relationship to the file-system semantics of Azure Files access over SMB (but still only partially). The big problem is that there’s no obvious way to relate the metric values to the corresponding Billing Meters being used. We can guess about what some of it means, but since it’s essentially undocumented, it does not provide a base that’s stable enough to actually audit your bill.

It also clearly doesn’t relate very clearly to the public pricing information, either.

But beyond the obvious disconnect with pricing/meters, there’s an even more annoying issue with Azure Files metrics: They don’t provide enough information to actually troubleshoot billing issues, because there are no corresponding access logs for Azure Files.

So if my Azure Files bill grows 10x in write operations, there doesn’t appear to be a way to figure out what shares, folders, or files are actually getting accessed and thus triggering the usage increase. Even Microsoft Support appears unable to obtain this information, which means that if you run into such a case, you’re stuck trying to figure things out by trial and error.

Conclusion

While Azure Files provides an excellent platform for cloud services, make sure you understand exactly what your usage patterns are and keep in mind that the file-system nature of the service seems to be very much at odds with how the pricing information is presented. Expecting some surprises in your bill while you understand your usage patterns better might be a good idea, too.

Importing Certificates into Azure Key Vault using the API

Thu, 31 Oct 2019 00:00:00 +0000

I spent some time the last two days figuring out how to correctly import X.509 certificates into an Azure Key Vault instance using the API (through the Microsoft.Azure.KeyVault NuGet package), and ran into a few issues.

Unfortunately, neither the library, nor the underlying REST API are very well documented, so wanted to write this post to document the gotchas I ran into, in case it’s useful to anyone.

PEM Certificate Format

The Import Certificate API (and corresponding KeyVault.ImportCertificate() method) are documented as accepting the certificate to import in both PFX and PEM formats. It took me a while to figure out the right incantation to use for a PEM file, however.

There are a few separate issues here you need to be very careful about:

Private Key Format: A certificate with a private key in a PEM file could have the key stored in various format. One very common one is PKCS#1, which is identified by the key being wrapped in -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----.

If you try to import such a certificate to Key Vault, however, you will get an error. This is because Key Vault will only accept a key in PKCS#8 format, which you will recognize because it’s wrapped in -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY-----.

You can easily use the openssl pkcs8 -topk8 command to convert the private key once you know to do this.

This aspect is not documented on the API, but it is mentioned in passing in the (Key Vault documentation)[https://docs.microsoft.com/en-us/azure/key-vault/certificate-scenarios#formats-of-import-we-support].

Content Type: If you’re going to import a PEM certificate, you also need to set the policy.secret_props.contentType property to the right type:

var policy = new CertificatePolicy {
  SecretProperties = new SecretProperties {
    ContentType = "application/x-pem-file"
  }
};

Value Format: The second confusing part I ran into here is how to actually send the PEM data to KeyVault. The API documentation states that the value parameter is Base64 encoded representation of the certificate object to import. This is only partially correct.

If you’re importing a certificate in PFX format, this is correct. PFX is a binary format, so you need to base64-encode the value before providing it to Key Vault. PEM, however, is already a text format, so base64-encoding it will make no sense.

The result is that the value parameter will just be the raw text contents of your PEM file. With one major, undocumented gotcha: The content must use UNIX-style line separators (\n). Attempting to send the content using Windows-style line endings (\r\n) will just result in a confusing error such as The specified PEM X.509 certificate content is in an unexpected format. Please check if certificate is in valid PEM format.

The only way I was able to figure out this little detail was by reading the Azure CLI source code.

Updating Certificates

Another interesting scenario I ran into was attempting to import a new certificate version on top of an existing certificate as a new version. The documentation for the Import Certificate API doesn’t actually say if this is possible, but since the Azure Portal actually allows you to do it, I figured it would be a safe bet that it would work.

It is, indeed, possible to do this. However, there was one surprising error I ran into while testing this out.

Purely by coincidence, I tried importing a certificate that had a 4096-bit key as a new version of a previous certificate that had a 2048-bit key. Trying this out failed with a CONFLICT error. Digging deeper, I found out that the actual error message was Expected KeySize is 4096 but was 2048.

This made no sense at first, until I realized that Key Vault was apparently attempting to somehow reuse the key of the current certificate version rather than the one included in the Import Certificate call.

The way to avoid this error appears to be to explicitly set the policy.key_props.key_size parameter to the right value when attempting to import the new certificate version on top of the old one:

var policy = new CertificatePolicy {
  Attributes = new CertificateAttributes {
    Enabled = true
  },
  SecretProperties = new SecretProperties {
    ContentType = "application/x-pem-file"
  },
  KeyProperties = new KeyProperties {
    Exportable = true,
    ReuseKey = false,
    KeySize = theCertificate.PublicKey.Key.KeySize
  }
};

Hope you find this useful!

Defining RBAC Role Assignments in ARM Templates

Thu, 02 Aug 2018 00:00:00 +0000

It’s no secret I’m a big fan of Azure Resource Manager (ARM) templates. Getting started with ARM templates is hard, but well worth the effort, and make it significantly easier to have reproduceable, consistent deployments of your Azure resources.

One thing that I had been feeling left out, however, was being able to assign permissions to Azure resources during creation. Azure’s Role-based Access Control (RBAC) mechanism is a powerful way to control who can manage and access your resources, and having to do this through scripting was possible, but cumbersome at times.

A few days ago, I realized that you can actually create RBAC role assignments through ARM templates just like any other resource. This capability is not new by any means, I just had missed it before!

Creating an assignment

To create an assignment, you need the following information:

The ID of the role you want to assign. This is a long string that contains the subscription id and the role identifier (both GUIDs).
The object ID of the user/group/service principal you want to grant access to.
The scope at which you want to assign the role, which is going to be either a subscription, resource group, or resource id.

Here’s an example of creating such an assignment:

json

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": { 
    "monitoringUsersGroup": {
      "type": "string",
      "metadata": {
        "description": "Object ID for the monitoring users group in AAD"
      }
    }
  },
  "variables": {
    "monitoringContributorRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/749f88d5-cbae-40b8-bcfc-e573ddc772fa')]"
  },
  "resources": [
    {
      "type": "Microsoft.Authorization/roleAssignments",
      "name": "ace492e6-ebf6-48fa-8dd4-ea762489cbf3",
      "apiVersion": "2017-10-01-preview",
      "properties": {
        "roleDefinitionId": "[variables('monitoringContributorRole')]",
        "principalId": "[parameters('monitoringUsersGroup')]",
        "scope": "[resourceGroup().Id]"
      }
    }
  ],
  "outputs": {}
}

Here we grant the members of an Azure Active Directory group the Monitoring Contributor built-in role to the resource group the template is deployed to.

Also interesting here is that you don’t need to specify a location property in the resource.

Some gotchas

There are a couple of things to watch out for when doing this.

The first one is that to assign a role, you need the objectId of the AAD user/group/principal, rather than the name. This is cumbersome because there’s no way to resolve these within the ARM template itself, so you’ll always need to pass these as input parameters.

A more significant issue, however, is the name of the roleAssignment resource, which needs to be a unique GUID.

This is a problem if, for example, you’re assigning role permissions at the resource group or individual resource level, rather than globally at the subscription.

For example, in my case I was creating a template that would be used to deploy multiple copies of the same resources into different resource groups within the same subscription.

If the GUID that defines the role assignment name is hardcoded in the template, then each time I ran the template, the scope of the role assignment would get overwritten with the id of the last resource group it was deployed to. Clearly, this is undesirable.

What we need then, is a way to ensure that each deployment to a different resource group uses a different GUID for the role assignment, but at the same time, ensure that the same one is used when deploying to the same resource group.

Clearly, providing the assignment GUID as a parameter is an easy workaround, but very cumbersome.

A better workaround comes from the guid function! It takes one or more strings that are used to calculate a hash, very much like the uniquestring function; only this one generates a string in GUID format instead.

By using the guid function with the resource group id and some other consistent stuff as input, we can solve our problem in an elegant way:

json

    {
      "type": "Microsoft.Authorization/roleAssignments",
      "name": "[guid(resourceGroup().id, 'monitoringUsers')]"
    }

AzureFile Persistent Volumes Retain Issue

Thu, 26 Jul 2018 00:00:00 +0000

A bit ago, I posted about some issues around permissions when using static provisioning of Azure File volumes in Azure Kubernetes Service (AKS). In there, I mentioned that the workaround was to use explicit Persistent Volumes so that the right mount options could be created.

Since then, I’ve run into a separate, but related, issue. I started noticing that sometimes, my Persistent Volume Claims would get stuck on the PENDING state:

The notes for the PVCs would not contain any useful information. But looking at the Persistent Volumes, I noticed they would get stuck in the RELEASED state:

I could not figured out exactly what was happening until I noticed the pattern that led to it:

I’d create the Persistent Volumes
Then I’d deploy an application on the cluster that would create Persistent Volume Claims on the volumes
I’d update the application. This worked just fine and stuff continued to run.
I’d remove the application (thus deleting the Persistent Volume Claims)
I’d deploy the application again, and now the PVCs would get stuck in Pending state.

The issue of course, happened in step (4) above. Once the Persistent Volume Claim was deleted, the Persistent Volume would transitioned to the Released status. These volumes all had a default reclaim policy of Retain.

Based on my understanding, I would have expected that taking a new claim on the volumes would work. But unfortunately, this does not appear to be the case (either I’m misunderstanding the Kubernetes documentation, or there’s an issue here in the volume provider).

I also thought that maybe making this work would involve setting the retain policy to Recycle, but this appears to be unimplemented as of yet for the azureFile provider.

In any case, once this happens, the only way to get this working again is to manually delete the Persistent Volumes and recreate them again from scratch.

This is unexpected (not explicitly documented), not obvious, and certainly annoying.:w

Issues when deleting azureFile dynamic volumes in Kubernetes

Thu, 26 Jul 2018 00:00:00 +0000

I’ve been doing a lot of work lately with Kubernetes and Azure Kubernetes Service in particular. For this, I’m using the azureFile storage provider to support providing storage folders for my pods.

For one specific case, I’m using dynamic provisioning of persistent volumes, as described in the documentation. This has been working great. Almost.

Today I noticed something odd. Some of the Persistent Volumes created were stuck on a failed state during deletion:

The error message was this:

text

Failed to create deleter for volume "pvc-85ba62d5-902e-11e8-9771-2a03d0b14ac3": Couldn't get secret 6d07576ce8d89bd641b5/azure-storage-account-fd9063993902011e888c32e-secret

This actually made sense. The way dynamic persistent volumes work is this:

When the Persistent Volume Claim is created, Kubernetes creates a new Secret object containing the credentials for the Azure Storage Account. This secret is named azure-storage-account-<namespace>-secret.
When the Persistent Volume Claim is deleted, the deleter for the volume will use these credentials to delete the Azure File Share that was dynamically created, and then delete the corresponding secret.

The second part is what was biting me.

I currently deploy the applications using Helm. I deploy each one into a separate Kubernetes Namespace. This is fully supported by Helm, but has one issue: When you delete the application using helm delete, the original namespace created remains behind. I wanted to clean these up, and so was aggressively deleting them using kubectl delete namespace.

The problem here is that the secret that gets automatically created by the azureFiles provider when the Persistent Volume Claim is created, goes into the same namespace as the PVC. That meant that when I was aggressively removing the namespace, I was also deleting the secret before the volume deleter could get to it.

This isn’t very clearly documented (or at least I found no mention of this), so it might be something to watch out for.

AKS Service Principal Credentials

Tue, 24 Jul 2018 00:00:00 +0000

When creating a new Azure Kubernetes Service (AKS) cluster, you must define a Service Principal in your Azure Active Directory Tenant that will be used by the cluster to do operations on the Azure infrastructure later on.

The documentation states:

On the master and node VMs in the Kubernetes cluster, the service principal credentials are stored in the file /etc/kubernetes/azure.json

I was curious how this worked, so I set out to take a look at the file. An easy way to do this was to spawn a new busybox pod on the cluster with the azure.json file mounted as a volume into the container.

I used the following YAML template to create the pod using kubectl apply:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: test
    image: busybox
    command:
      - sleep
      - "3600"
    volumeMounts:
      - mountPath: /etc/kubernetes/azure.json
        name: azure-secret
  volumes:
    - name: azure-secret
      hostPath:
        path: /etc/kubernetes/azure.json
        type: File

Once the pod is up and running, I went spelunking into it by opening a new interactive session using kubectl exec -it test -- sh.

First, let’s look at the contents of the file through cat /etc/kubernetes/azure.json:

json

{
    "cloud":"AzurePublicCloud",
    "tenantId": "*********",
    "subscriptionId": "*********",
    "aadClientId": "**********",
    "aadClientSecret": "*********",
    "resourceGroup": "MC_dev-k-eastus-2-rg_dev-k-eastus-2_eastus",
    "location": "eastus",
    "vmType": "standard",
    "subnetName": "aks-subnet",
    "securityGroupName": "aks-agentpool-22088002-nsg",
    "vnetName": "aks-vnet-22088002",
    "vnetResourceGroup": "",
    "routeTableName": "aks-agentpool-22088002-routetable",
    "primaryAvailabilitySetName": "agentpool-availabilitySet-22088002",
    "primaryScaleSetName": "",
    "cloudProviderBackoff": false,
    "cloudProviderBackoffRetries": 0,
    "cloudProviderBackoffExponent": 0,
    "cloudProviderBackoffDuration": 0,
    "cloudProviderBackoffJitter": 0,
    "cloudProviderRatelimit": false,
    "cloudProviderRateLimitQPS": 0,
    "cloudProviderRateLimitBucket": 0,
    "useManagedIdentityExtension": false,
    "useInstanceMetadata": true,
    "providerVaultName": "",
    "providerKeyName": "k8s",
    "providerKeyVersion": ""
}

The relevant fields containing the credentials are aadClientId and aadClientSecret.

I was curious, however, as to the permissions on the file:

text

/ # ls -l /etc/kubernetes
total 4
-rw-------    1 root     root          1173 Jul 24 19:13 azure.json

As suspected, the file is only readable by containers running as root.

AKS and Azure Files Permissions

Mon, 23 Jul 2018 00:00:00 +0000

Saving this here for my own recollection later on. Warning, a bit of ranting ahead.

Recently, I’ve been running a lot of trials on top of the Azure managed Kubernetes Service (AKS). One key feature that I needed was the ability to provide services deployed to an AKS cluster with external storage. Since I needed the option to mount the storage shared between multiple pod instances, Azure Files was the way to go.

My first attempt to do this leveraged using automated static volume provisioning, as defined in the documentation. Static just means that you first create the Azure File Share on your own, and then provide the pod with the necessary information to reach it, such as the account name and key, and the file share name.

This worked, sort of. There’s a big gotcha that’s not included in the documentation, and that takes a bit of search-fu to find it.

The problem is file permissions. When using static provisioning, the Azure Files plugin in Kubernetes will default the share permissions to 0750 or 0700 depending on the version of Kubernetes in use.

What this means is that static provisioning, as described in the AKS documentation, is completely useless if the following conditions are met:

Your container is configured to run as a non-root user (which is generally the recommended approach for security), and
You need to write to the mounted file share.

If you just need to read data from the share, then this is not a problem. If you need to write to it, however, this really throws a wrench into your plans, and the documentation doesn’t tell you how to work around it.

The workaround

The workaround is described in more detail on this sample by Andy Zhang.

Create a new Kubernetes Persistent Volume for your Azure Files Share, and override the default file and directory permissions for the file share.

yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-azurefile
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  azureFile:
    secretName: azure-secret
    shareName: k8stest
    readOnly: false
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=10000
    - gid=10000
    - mfsymlinks
    - nobrl

Create a new Persistent Volume Claim for your service to request access to the Persistent Volume created in the previous step

yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azurefile
spec:
  accessModes:
    - ReadWriteMany
  mountOptions:
  resources:
    requests:
      storage: 5Gi
  storageClassName: ""
  volumeName: pv-azurefile

Create a volume for your pod out of the PVC.

yaml

volumes:
  - name: azurefile01
    persistentVolumeClaim:
      claimName: pvc-azurefile

Notice that the storage capacity is required on both the Persistent Volume and the Claim, and should be equal or smaller to your Azure File Share size.

The disadvantages

The documented workaround works just fine. However, the whole issue is frustrating on many levels.

First of all, making it hard to follow the recommended practice of running containers as non-privileged users makes no sense to me. This should be easy, and work right away without having to result to these complicated steps.

Second, the fact that the documentation doesn’t warn you about the issue, and doesn’t provide a link to the solution, is very frustrating, and causes you to lose a few hours for no good reason.

While I can sort of understand why restricting the default Unix permissions on the file share, it seems to me this is just a very leaky abstraction, than on a platform like AKS doesn’t make a lot of sense.

The most frustrating aspect for me, however, is the fact that using Persistent Volumes forces you to have to create those manually before you can deploy applications. If you just need to reference a single Azure File Share, then it’s sort of OK, but if you need to reference several, it quickly adds to a lot of work. You need to manually create at least 2 Kubernetes objects (a Secret, and the Persistent Volume) for each share.

I’m using Helm to deploy my application, and now I can’t simply deploy the helm chart. I need to manually create stuff, ensure it’s correct, and only then can I deploy it. This is just cumbersome.

I wish you could just adjust default permissions directly when creating the volume, but alas, apparently this is not possible with the Kubernetes model (or simply unsupported by the Azure Files plugin).

Update: Posted some additional findings here

Getting fixed!

Update 2018-11-15: Looks like this issue is finally fixed in Kubernetes v1.11.4 with the default permissions for Azure Files reverting to 0777. Great news.