Its massive genome isn’t the product of design or advantage, but of incidental complexity, a slow accretion of viral DNA and chromosome duplications. Inhabiting a stable, low-competition ecological niche, this fern faced little pressure to streamline its genetics, so it remains as-is despite the challenges posed by such size: slower reproduction and higher energy requirements.

Reading about this fern with a ridiculously large genome reminded me of the idea of managing software complexity discussed in Rich Hickey’s presentation, “Simple Made Easy” (I am sure you all had the same thought). Hickey distinguishes between simplicity (the absence of unnecessary interleaving or complexity) and ease (solutions that are immediately accessible or require minimal mental effort). Quick solutions seem ideal in the short term but overreliance on taking the path of least resistance can lead to accumulation of incidental complexity. Gradually, systems become less maintainable and harder to reason about, resulting in slower project velocity.

In nature, this fern exemplifies what can happen when complexity accrues unchecked. It survives due to its isolation, carrying the burden of an extraordinarily complex genome; a tolerance that most living organisms, and most software systems, cannot afford.

Incidental vs Intrinsic Complexity

In software, intrinsic complexity is the inherent difficulty of the domain: business rules, user requirements, or performance constraints. These challenges require thoughtful and novel solutions from engineers but even with strong engineering, they demand time and focus to learn and understand. Even the most well engineered software systems can seem daunting and confusing due to this.

On the other hand, incidental complexity arises from the optional choices made during implementation that make the codebase harder to work with, like layered abstractions, nests of conditionals, and excessive caches. Such complexity is often the byproduct of opting for the easy and more immediate solution.

Finding truly simple solutions is hard. They take time and may require revisiting core assumptions, data models and flows in the application. These solutions often lead to short-term velocity decreases in order to unlock longer term understandability and productivity.

Selective Pressure in Software

Just as selective pressure discourages excessive complexity among organisms through competition, software products compete against each other too. Products with limited competition or robust brand loyalty may withstand slow iteration or internal inefficiencies. However, most live in more competitive environments; customers will switch to a competitor’s product unless the software continually advances to offer new functionality at a high quality. Rival products that innovate and adapt quickly can erode your customer base. This is why it is important to stay focused on investing in the simple solutions, so your applications don’t get bogged down under their own weight.

However, the realities of software products don’t always offer the privilege of taking the time to find the truly simple solutions right away. Given deadlines and immediate customer needs, engineers (who are often aware of what the simple solution is or that it can be found) must decide to make reasonable trade-offs to balance other priorities.

For example, adopting the easy fix may be warranted to address urgent needs, customer issues or key deadlines. However, everyone must be aware of the cost paid in doing this. Increasing the system’s overall complexity, adding cognitive load and increasing the number of edge cases. Each easy fix adds incremental complexity and technical debt. This is akin to numerous examples in evolutionary biology where an organism accrues sets of mutations that enable short-term benefit but lead to disadvantage down the line like Darwin’s finches, for instance, whose beak sizes oscillate with the shifting pressures of drought and rain.

A decision not to pursue the simple solution requires the team to acknowledge the impact (which I discussed in a previous post) and decide if the team (and the product) can live with this lack of simplicity forever or must hard commit to a concrete follow up soon. Discerning which complexities are tolerable versus which require immediate work towards the simple solution is a crucial skill for sustaining development velocity and system health. The team must periodically reflect over how often they have opted for the easy path and if this is leading to systemic problems in maintainability and velocity. Without deliberate reflection regarding whether deeper, structurally simple solutions must be pursued, organizations risk developing a genome like our favorite fern’s: functional, yet increasingly burdensome.

Conclusion

Incidental complexity is inevitable in both natural organisms and engineered systems. Unlike our tiny fern, whose survival hinges on environmental isolation, software projects seldom enjoy the luxury of operating without external pressures. Vigilantly distinguishing between easy fixes and truly simple solutions is vital to ensure continued innovation, responsiveness, and longevity in a competitive landscape.  Nature has billions years to figure out this balance, your software project doesn’t.

The post Software Engineering Lessons from a Fern appeared first on Matthew Manela.

Building software is complex. Engineers juggle performance, experience, security, maintainability, testability and observability (that is a lot of balls in the air). With so many competing concerns, it’s easy to lose the forest for the trees and forget the most important reason we build software: to make an impact.

That impact might be helping a human directly, powering another service, or enabling another team. Whatever the case, the result should be an improved experience for someone.

Why are we building this?

Too often, engineers get so focused on the what that they lose sight of the why. On my teams, I put deliberate emphasis on ensuring every engineer understands the purpose of the work—and feels empowered to question it.

Asking “How will this create the intended impact?” keeps the focus where it belongs and helps the team search for the elusive pit of success.

Before starting a sprint or building a feature, every engineer must be able to explain why the change matters and what impact it will have. They should be able to paint a narrative for how the user’s experience will change and why that matters. And ideally tie this back to concrete metrics we expect to improve.

If the why doesn’t resonate with the team, it won’t resonate with customers either. That’s a signal to rethink or reframe the work. If I can’t articulate the why, I must question if I am propsing the right direction for the team.

Once the purpose is clear, engineers can (and should) continually ask during development: “Will this still achieve our goal?” Scope evolves and requirements change while building, and if halfway through someone realizes the feature won’t deliver the intended impact, that’s the perfect time to pause and reassess. It’s far cheaper to adapt or cut work in progress than to ship something that misses the mark.

In search of the pit of success

Clarity around the intended impact is the foundation for a culture that actively searches for the pit of success: a state where the user naturally falls into the impactful behavior you want to drive. That sounds obvious, but it’s hard in practice because of a core tension in software: features vs. simplicity.

Many products try to deliver value by adding features, flexibility, and configuration. While that’s useful, it often prevents users from stumbling into success. If there’s no current pulling toward a meaningful outcome, users will walk away, even if your product is technically superior.

But when engineers, product managers, and designers are aligned on the why, they can work together to deliberately design for the pit of success. This isn’t just about slapping a CTA button on a page. It’s about simplifying the surface area (UI or API) so users aren’t distracted, confused, or pulled off track.

Closing

Building software isn’t about features or technical cleverness—it’s about creating impact. When teams keep the why at the center, they’re more likely to build experiences where users naturally succeed (and choose not to build experiences where they won’t). That’s the real goal: designing so the pit of success is the only place to land.

The post Finding the pit of success appeared first on Matthew Manela.

It was hard to see my desk under the clutter of sticky notes. For years I worked in team rooms at Microsoft and relied on the ample supply of sticky notes in the supply room to organize my daily to-dos, in-progress work, weekly plans and future ideas. This was not a great organizational system, but the sticky notes were free and unlimited so why not? Over time, I adopted better practices to lessen my need for the plethora of paper but one use remained – self-reminders to combat anxiety.

I have fought with anxiety my whole life. When I began working professionally, I frequently left conversations or meetings feeling dismayed that my anxiety had taken control of my behavior. Eventually, I started using sticky notes to leave reminders on my monitor (often covering a small portion of the screen) to help interrupt the cycle of anxiety.

The messages were simple. Stay Calm. Take a breath. Slow Down. These didn’t magically solve how I felt, but they provided a moment of reflection to let my thinking brain back into the room. I did this for years, but I felt embarrassed about it since I wasn’t comfortable talking about anxiety. I hid the messages by abbreviating them so others wouldn’t know what I was doing when they saw letters S.C. stuck to my screen.

While I have a better hold of my anxiety than I used to, it remains a daily presence and I still use visual reminders (although not sticky notes) to shake me out of a spiral. Above my desk at home, I have a canvas print depicting Sisyphus pushing a boulder up a hill as a reminder that letting anxiety take control is a self-defeating process.

Let’s build something

It has been a long time since I even though about my sticky note days. But after a recent meeting where I felt frustrated over acting too much out of anxiety, the idea came back to me. Without the limitless supply of sticky notes, I decided to build an app (seemed like a good enough reason to learn the basics of Swift and Mac application development) to revisit the idea.

The concept was simple, a Mac OS app that allows pinning a customizable message to the menu bar that is always visible. I used this as an opportunity to experiment writing software using Chat Oriented Programming (CHOP) with Sourcegraph’s Cody AI agent (disclaimer: I work at Sourcegraph).

Impressively, by describing my requirements and iterating on the resulting components, I created a working app quickly with 80% of the fuctionality. However, as it often is, that last 20% took the majority of the time. A couple behaviors and features required more research and reading to get working smoothly.

Handling Menu Bar Occlusions

My first pass used Swift UI to add the text to the Mac menu bar. Swift UI makes building UX components a breeze and I also used it for the settings dialog. But using it for the menu bar limited the ability to listen to occlusion status events. The Mac menu bar is a fickle friend that, depending on the screen size and other menu bar items, can evict your menu bar item at any moment. Reacting to eviction requires listening to events and adjusting the size. This meant leaving the comfort of Swift UI behind for rendering the menu bar and implementing those parts of the application as an application delegate. In the delegate, I attempted to render the full message and if it didn’t fit, render just an emoji. Making this work smoothly took a lot of trial and error.

Persistent Popover

To account for the full message not always being visible in the menu bar, I added a pinnable message popover to view the full message when you hover over the menu bar item. I first used the built-in popover component which is very convenient, but inflexible and dissapears when a user changes workspaces (virtual desktops). To make sure the pinned message always shows, I built custom popover component using a window component and view controller.

No more sticky notes

The end result is … Quick Motivation, a simple Mac app that adds a menu bar item with customizable messages.

Menu bar message

Quickly change between custom messages

Configure new messages

Pin a message anywhere on the screen

You can install the app from the Mac App Store and view the source code on GitHub. While this is not a large or complex application, I learned a ton creating it and I hope others will find it useful.

The post Quick Motivation – A Mac App appeared first on Matthew Manela.

In this wave of excitement over Large Language Models (LLMs), one of the most interesting technologies that emerges from them is the idea of embeddings. Most people interact with LLMs (like ChatGPT or Bing Chat) by providing an input and watching it build astonishing output:

Write me a 2-line poem about a carrot whose dream of becoming a banker has gone unfulfilled.

He wanted to be a banker, but he was just a carrot.
He tried to crunch the numbers, but he could not bear it.

https://bing.com/chat

Under the covers, LLMs take the prompted text and turn it into an embedding which is a vector of numbers representing the meaning and context of the tokens processed by the model. In essence (in my non-ML expert view), the embedding represents the model’s conceptual understanding of the input. It then uses this to generate its output.

[-0.003150753309585745, -0.012415633924448732, 0.0095910591273366, -0.0013784545438399322, -0.025483625805167297, …

The first few numbers of an example embedding.

But skipping to the generated output misses the fascinating possibilities embeddings represent. Quantifying a concept numerically opens many opportunities for manipulating semantic understanding. This numerical representation enables scenarios like semantic document search, classification, and translation. Over the last year, many companies have sprung up to take advantage of this potential.

To learn more, I built a program to explore the power of embedding, understand how they work, and learn how to manipulate them. This program takes a document (PDF) as an input and uses embeddings to semantically group and name its key concepts.

For example, one of the outputs the program can generate is a 3D scatter plot where each dot represents the embedding of an individual section of the document. The color represents a grouping of all similar passages, and the legend shows the words that summarize that grouping.

This post will cover the steps followed to build this program:

Each step will share the key pieces of code needed and show examples for executing that functionality in the program so you can explore and iterate as well. The goal of this post is to demonstrate the power of embeddings and help you get started quickly, leveraging it for your own applications.

If you speak code and not text, jump right to the GitHub repo.

Technology decisions

The program is built in Python which has a rich ecosystem for exploring LLMs and machine learning concepts. It leverages the awesome LangChain library which provides an abstraction for all things language models to enable experimentation.

It also uses OpenAI‘s text-embedding-ada-002 model as the backing LLM. OpenAI API integration is not free, so I purchased $5 worth of tokens. In all my experimentation and testing, I have only used up 10 cents of that amount since the datasets are small. If any cost is prohibitive, LangChain supports different LLMs (both online and local).

Document Splitting and Chunking

The first step is taking an input PDF document and splitting it into smaller chunks. This is required since most LLMs have a character limit. Smaller blocks of text also aid in semantic analysis since larger blocks may contain many concepts and clustering will be less accurate.

LangChain supports many options and configurations for splitting a document. I experimented with three different splitters and with different chunk sizes.

1. RecursiveCharacterTextSplitter – Attempts to chunk a document by using a set of separators.
2. SpacyTextSplitter – Splits text using spaCy’s understanding of English language via its built-in tokenizers.
3. NLTKTextSplitter – Splits text using NLTK’s understanding of English language via its built-in tokenizers.

I split documents many times using each of these text splitters with chunk sizes of 100, 250, 500 and 1000 characters. The chunk size is a suggestion, and it will break the text as close to that as it can. Through this iteration, NLTKTextSplitter showed the best results, but all were reasonable.

This function takes a PDF and chunks it and returns the results:

from langchain.text_splitter import NLTKTextSplitter
from langchain.document_loaders import PyPDFLoader
import os

def parse_pdf(filename):
    path = os.path.join(FILES_FOLDER, filename)
    if not os.path.exists(path):
        raise FileNotFoundError(f'File {path} not found')
 
    chunk_size = 250
    chunk_overlap = 0
    loader = PyPDFLoader(path) 
    nltk_splitter = NLTKTextSplitter(
        chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    texts = loader.load_and_split(text_splitter=nltk_splitter)
    print(f'Loaded {len(texts)} texts')
    return texts

When using NLTK there is a onetime step to download its sentence tokenizing module by running the follow code:

import nltk
nltk.download('punkt')

This functionality is exposed for experimentation in the program through the extract command (referencing the sample pdf included). It prints out the first 100 characters of the first 10 chunks it creates.

./run.sh -m extract -f pair_programming.pdf

Creating and Querying Embeddings

Generating Embeddings

With bite-sized chunks of text produced, the next task is turning each of these chunks into an embedding. An embedding is a numerical representation of the LLM’s conceptual understanding of the chunk of text. I leveraged LangChain’s OpenAI’s embedding model (OpenAIEmbeddings) and its embeddings caching layer (CacheBackedEmbeddings).

The caching layer minimizes how much it costs (which in hindsight is very small) and improves performance in subsequent runs. It hashes each chunk of text and stores it in a file (or Redis or in memory) per hash with its embeddings.

Using these two classes, this is how to generate embeddings for an array of strings.

from langchain.embeddings import CacheBackedEmbeddings, OpenAIEmbeddings
from langchain.storage import LocalFileStore

# Create a file store to store the cached embeddings
embedding_cache_store = LocalFileStore(EMBEDDING_CACHE_PATH)

# Create an open ai embedding service
underlying_embeddings = OpenAIEmbeddings(
    openai_api_key=os.environ.get('OPENAI_API_KEY'))
    
# Wrap the embedding service in a cache
cached_embedder = CacheBackedEmbeddings.from_bytes_store(
    underlying_embeddings, embedding_cache_store, namespace=underlying_embeddings.model)

# Generate embeddings for the array of texts returned from parse_pdf
doc_embeddings = cached_embedder.embed_documents(texts)

Running it generates the cached embeddings:

With embeddings created and cached, the next step is to load and query them. This enables two key scenarios:

1. Querying embeddings by input text to find closest matches
2. Clustering a list of embeddings

Using a Vector Database

A vector database is the key technology involved in scenario #1. This is a special database that is optimized for storing and querying against vectors of values (like embeddings!). It is built to run in memory or on disk and can scale to very large datasets. LangChain makes it simple to experiment with several vector DBs and I decided to use FAISS. Creating and persisting a FAISS index to disk for the text and embeddings above is just two lines of code:

faiss_index = FAISS.from_texts(texts, cached_embedder)
faiss_index.save_local(VECTOR_STORE_PATH, fileName)

Technically, you don’t need to save the index to disk, since it can be re-built from the cached embeddings, but for large datasets it can save the re-indexing time.

With the index built there is a simple method to provide it a text query and find out which embeddings match most closely:

# Load the index
faiss_index = FAISS.load_local(
    folder_path=VECTOR_STORE_PATH, index_name=fileName, embeddings=cached_embedder)

# Query 3 closest matching chunks from the index
docs_and_scores = faiss_index.similarity_search_with_score(query, 3)
snippet_and_score = [(x[0].page_content[0:100], x[1])
                        for x in docs_and_scores if x[0].page_content]

Storing Embeddings with their chunks

In addition to the vector DB, it is useful to have a persisted list of source text, metadata and embedding. This makes it easy to do further analysis and comparison. There is not a simple way (in my experimenting) to re-construct this from the FAISS index. Instead, store a file to disk that contains all text chunks and their embeddings. To do this, I created a small object called `EmbeddedReference` that contains the embedding, its source text and the page number in the source doc the text came from. I then serialize this to disk.

embedding_objects: list[EmbeddedReference] = []
for i, embedding in enumerate(doc_embeddings):
    embedding_objects.append(EmbeddedReference(
        embedding=embedding, metadata=metadata[i] if metadata is not None else None, content=texts[i]))
with open(paths.get('embeddingStorePath'), 'wb') as f: 
    pickle.dump(embedding_objects, f, pickle.HIGHEST_PROTOCOL)

Then to read this back out and generate a list of the embeddings:

embedding_objects: list[EmbeddedReference] = None
with open(mapPath, 'rb') as f:
    embedding_objects = pickle.load(f)

embeddingsArr = np.array([x.embedding for x in embedding_objects])

Exploring the embedding model

The create command demonstrates taking an input PDF, chunking it, generating embeddings, and storing it to disk as cached embeddings, a vector store and the custom serialized model.

./run.sh -m create -f pair_programming.pdf

After the create command is run once, the query command lets you run queries against the built and persisted model. It takes an input string and returns (by leveraging the vector DB) the first 100 characters of three chunks of text in the document that are semantically closest to the input query.

./run.sh -m query -q "code clarity is a key indicator of success"  -f pair_programming.pdf

# Outputs - first 100 characters of three closest matches
# [
#('[11].”In keeping with the known characteristics of code reviews, we find practitioners citing:• Mi', 0.34571558), 
# ('Coding standards are followed more accurately with the peer pressure to do so• Team members lea', 0.3501709), 
# ('The staff agreed with my points that pair programming: - Should significantly reduce the risk of su', 0.3555439)
#]

Clustering and visualizing embeddings

Generating clusters from embeddings

With the ability to generate, store and query embeddings solved, the next challenge is to demonstrate how the embeddings represent concepts. The key idea is to cluster the embeddings and see what groupings emerged and what they represent. Python has a very rich ecosystem for number analysis and since embeddings are just vectors of numbers all the popular Python libraries like Numpy, Scikit-learn and Pandas can manipulate them. The program leverages the scikit-learn suite for its pre-built clustering algorithms. Clustering is a deep area and many of the modern algorithms can be very complex. Among the different algorithms I experimented with (HDBScan, MiniBatchKMeans, Optics), MiniBatchKMeans showed the best results.

The following code shows how to cluster embeddings and references the embeddingsArr defined in the previous example.

# Runs the clustering algorithm over an array of embeddings (which themselves are long arrays)
clusters = MiniBatchKMeans().fit(embeddingsArr) 

# Get a set of all labels (e.x. 1,2,3,4,5)
labelSet = set([x for x in clusters.labels_ if x != -1])

# Get total number of labels (e.x. 5)
labelCount = len(labelSet)

The output of the clustering algorithm is a set of labels (clusters.labels_) with a label per input embedding. This tells you which group the clustering algorithm assigned to that embedding.

But how do we know if those groupings make sense? The labels are just a number with no innate meaning. This led to the next challenge of visualizing the results of the clustering to understand what (if anything) the grouping of chunks labeled 0, 1 or 2 have in common.

Visualizing clusters

The application generates three visualizations for the clustered embeddings:

1. A histogram of the frequency of the labels
2. A scatter plot of the embeddings in 3-dimensional space with labels mapped to colors
3. A data table for inspecting the chunks and their labels

Histogram

The histogram of label and chunk counts is created using the Matplotlib library in combination with Numpy’s histogram method.

color_palette = sns.color_palette('Paired', labelCount + 1)

# Render histogram of the clusters
labels = np.array(clusters.labels_)
hist, bin_edges = np.histogram(
    labels, bins=range(-1, labelCount + 1))
fig1 = plt.figure()
f1 = fig1.add_subplot()
f1.bar(bin_edges[:-1], hist, width=1, ec="black", color=[
    (0.5, 0.5, 0.5), *color_palette])
plt.xlim(min(bin_edges), max(bin_edges))
f1.set_xlabel('Cluster')
f1.set_ylabel('Count')
f1.grid(True)

3D Scatter Plot

Generating the scatter plot is more complicated. The problem is that embeddings are 1500 numbers long and to plot in 3D we need to reduce the dimensionality. There are many techniques to do this but the t-SNE (T-distributed Stochastic Neighbor Embedding) library in Scikit-Learn worked well for this case.

t-SNE [1] is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e., with different initializations we can get different results.

t-SNE takes the 1500-dimension embedding vectors and projects them to 3-dimensionql vectors minimizing the probabilistic difference between the full and reduced data set. With the vector space in 3D, Matplotlib’s scatter chart can plot it. Each clustered label is mapped to a different color. The interesting result is that there are two representations of “similarity”. One is from the clustering algorithm represented by points in the graph with the same color. The other is by points in the graph visually close to each other due to the projection by t-SNE. Based on the analysis, the clustering algorithm’s labelling is more accurate, likely because it has more information at its disposal (all 1500 dimensions).

# Project the embeddings to 3 dimensions
projection = TSNE(n_components=3).fit_transform(embeddingsArr)

 
# Scatter plot by first projecting the n-dimensional embeddings into 3D
# then by using the cluster from HBSCAN to color the points
color_palette = sns.color_palette('Paired', labelCount + 1)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusters.labels_] 
                  
fig2 = plt.figure(figsize=(10, 10))
ax = fig2.add_subplot(projection='3d')
ax.scatter(*projection.T, linewidth=0,
           c=cluster_colors, alpha=1)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
 
plt.show()

Data Table

The last visualization is a data table to show the results in a readable manner. The Pandas library’s `DataFrame` is used to generate a table with the cluster’s label and first 100 characters of the embeddings source text.

df = pd.DataFrame({
    "cluster":  clusters.labels_,  
    "page_num": [embedding_object.metadata['page'] for embedding_object in embedding_objects],
    "snippet": [embedding_object.content[0:100] for embedding_object in embedding_objects], 
})

df = df.sort_values(by=['cluster'])
print(df.to_string())

This prints a table which I spent time analyzing and trying to figure out which concepts it was grouping by.

Semantically Labelling Each Cluster

Grouping the chunks of text semantically together and manually looking for conceptual similarities is interesting, but we can further leverage embeddings to name the concepts the clusters represent automatically. The approach I took is to generate embeddings for each word in the dictionary. Then by leveraging the vector DB we can find the words closest to each cluster. To represent the cluster as a single point in embedding space we find its centroid (the geometric average of all the embeddings). This builds on the idea that the position in the 1500-dimensional embedding space has semantic meaning, and the average of the points represent a common meaning of them all. This is an intriguing thought and worthy of further exploration to understand change of conceptual meaning by traversing the embedding space.

Generating word embeddings

I use the SCOWL database of English words (which I learned about on a previous project) as the dictionary. It provides word lists categorized by commonality (list 10 being the most common words through list 90 being the least). Combining word lists 10 through 70 yielded the best results.

This list when concatenated all together is 111,814 words long! After analyzing it, I noticed there is a very large amount of the same word with different endings (like biologist and biologists). To minimize these, I used the lemmatization processing class in NLTK called WordNetLemmatizer to post-process the word list. This results in a new word list of length 88,314. This process does not yield perfect results but helps lessen the potential dupe word meanings.

The following code shows how to lemmatize a list of words.

import os 
from nltk.stem import WordNetLemmatizer

with open(WORDS_FILE, 'r') as file:
    words = file.readlines()

wnl = WordNetLemmatizer()

stemmed_words = set([wnl.lemmatize(word.strip())
                    for word in words if len(word) > 0])
sorted_words = sorted(stemmed_words)

With the list of words saved, we can iterate over that list and generate embeddings for each one and persist them to disk in the same way as the chunks of the document.

word_embeddings = cached_embedder.embed_documents(sorted_words)

Exploring word embeddings

In the application the command create_dict generates the embeddings for all 80,000 words. This takes a long time to run so be patient with it.

./run.sh -m create_dict

Once that has finished successfully, the query_dict command takes an input string and finds top 3 closest words (and includes their Euclidean distance from the query text) in the embedding space.

./run.sh -m query_dict -q "If every instinct you have is wrong, then the opposite would have to be right.

# Outputs
# [('contrapositive', 0.3559821), 
#  ('wrongness', 0.35792178), 
#  ('instinct', 0.35800213)]

Labeling the clusters

With the word embeddings built, we generate the describing words for each cluster by calculating the centroid (the average of all the embeddings in the cluster) and finding the words closest to the centroid.

By converting the embeddings into a Numpy array we can use its mean function to calculate the array of averages. For each cluster calculate this average and pass this vector into the vector DB similarity_search_with_score_by_vector method to find the words that best describe that cluster.

# Enumerate each cluster label
for cluster in labelSet:
  print(f'Finding centroid for Cluster {cluster}')
  
  # Calculate the centroid for the cluster
  cluster_embeddings = np.array([x for i, x in enumerate(
     embeddingsArr) if clusters.labels_[i] == cluster])
  mean = cluster_embeddings.mean(axis=0)
  
  print(f'Centroid for cluster {cluster} is {mean}')
  
  # Use the centroid to find words to summarize the cluster
  word_matches = words_index.similarity_search_with_score_by_vector(
      mean, 3)
  word_and_scores = [(x[0].page_content[0:SNIPPET_LENGTH], x[1])
                     for x in word_matches if x[0].page_content]
  print(f'Words for cluster {cluster} are {word_and_scores}')
  
  # Use tuple of top two matching words to categorize the groups
  clusterCategory[cluster] = f'{word_and_scores[0]},{word_and_scores[1]}'

Initially, I labelled each cluster with the topmost closest word match, but this led to several distinct clusters labeled with the same word: programming. If the article is primarily about programming, it makes sense that core topic in each chunk of text will match closely with that word. Including the second closest word adds much more clarity on the intent of that passage. For example, in one test two chunks of text that were labeled with “programming” changed to “programming, teamwork” and “programming, testing” respectively. Both clusters of passages are still about programming, but one set focuses more on teamwork while the other on testing.

With the word labels created, we can update the graphs from earlier to add a legend naming each cluster. For example, the following shows adding the legend to the histogram:

color_palette = sns.color_palette('Paired', labelCount + 1)

# Render histogram of the clusters
labels = np.array(clusters.labels_)
hist, bin_edges = np.histogram(
    labels, bins=range(-1, labelCount + 1))
fig1 = plt.figure()
f1 = fig1.add_subplot()
f1.bar(bin_edges[:-1], hist, width=1, ec="black", color=[
    (0.5, 0.5, 0.5), *color_palette])
plt.xlim(min(bin_edges), max(bin_edges))
f1.set_xlabel('Cluster')
f1.set_ylabel('Count')
f1.grid(True)

# Addition to add a legend
handles = []
for cluster in set(labels):
    if cluster >= 0:
        handles.append(
            mpatches.Patch(color=color_palette[cluster], label=clusterCategory[cluster]))
    else:
        handles.append(
            mpatches.Patch(color=(0.5, 0.5, 0.5), label='None'))
f1.legend(handles=handles)

To generate the final charts showing the 3D scatter plot and the histogram with the categorized cluster labels, run the analyze command.

./run.sh -m analyze -f pair_programming.pdf 

Which outputs:

Conclusion

It is deeply intriguing thinking about ideas existing in a N-dimensional space and using distance calculations to find interrelations between concepts. This program only scratches the surface on the possibilities in this space. The tools and libraries that have grown around LLM’s (like LangChain) make exploration in this space easy and understandable. I am eager to see and read more and hopefully experiment further on the power of embeddings.

The finished application is available on GitHub, check it out and let me know what you think.

The post Exploring the power of LLM Embeddings appeared first on Matthew Manela.

Throughout my career as a software engineering manager, I have built a metaphorical toolbox containing methods and techniques for building strong, compassionate, and productive teams. One of the key tools is assigning engineers joint accountability to projects. For significant features or changes, I ensure that two people are accountable for delivery, quality, communication, and success of the work. This has numerous benefits and has become a critical tool in managing remote teams.

Assigning joint accountability with proper guidance and training leads to higher quality code, greater team cohesiveness, more engaged engineers, better interpersonal awareness (EQ), and higher productivity.

I landed on this concept as an avid pair programmer and built on it while running teams for years. I experimented with many techniques and learned where pair programming worked well and where it didn’t. I learned about different personality types and how individuals react based on the type of work and deadlines. My experiences led me to shift from an ideological position of “pair programming for all” to my implementation of joint accountability. Joint accountability is a way to get the benefits of pair programming but in a more flexible and general way.

Starting off with pair programming

Shortly after I joined Microsoft out of college, I landed on a team that practiced pair programming. The concept petrified me. Pair Programming is the development technique where two engineers work together at one workstation (physical or virtual). One engineer is typing while the other one observes, discusses, comments, and directs. As a young engineer with a ton of imposter syndrome, I had many thoughts when I first pair programmed:

• You mean someone is going to watch me …. type?
• What if they realize I am a fraud?
• What if it turns out everyone is just way better than me?

I was paired up with one of the most senior and confident engineers on the team. This developer was diving into an area of the code and typing while I was asking questions. I remember clearly pointing out some issue with what he just wrote. He paused typing, thought for a minute, and said, “great catch”. He fixed the issue and we moved on.

The thing that surprised me with the pairing experience was that it wasn’t just about “training the new guy”. It was about higher code quality and ensuring the team had a common baseline understanding of the whole codebase. Even as the brand-new team member, I was expected to contribute and learn while pairing. I soon understood that everyone I paired with expected me to pull my weight regardless of my experience. There was a deep trust between pair partners, and you wanted to live up to the trust given.

I worked on teams practicing pairing for 5 years and transitioned from the most junior to the most senior engineer on the team. All the while I was honing this craft and learning how it helped build a culture of quality and unity in the team.

When I eventually became an engineering manager years later, I knew I wanted to build a team that benefited from the performance and trust I experienced.

From ideology…

In researching development methodologies, you will quickly find lots of competing views and strong opinions; pair programming is no exception to this. I have talked to people who vehemently believe it is a waste of time, counter-productive, or just not the way they want to work. Similarly, I have spoken with people who argue you should always pair, and no task is not a good task to pair program on. In general, extreme stances that fail to consider nuance and trade-offs are most likely a gross oversimplification.

When I started as an engineering manager, I worked on a team and a group that did not pair program. I didn’t want to rock the boat and followed the processes the team already had for several months. However, we were struggling with knowledge silos leading to poor code reviews, work stoppages during individual engineer time-off due to lack of context, and a lack of team cohesion. When I asked my boss to let me try out pair programming (along with some other structural changes) with the team, she supported me (although with healthy skepticism). I proposed a 6-week pair programming experiment. After this period the team would decide if we wanted to continue. The team accepted this proposal (with reservations) but initially it was turbulent. The engineers weren’t used to working together and some didn’t know how to communicate 1×1 while coding. I actively coached and found a few engineers who immediately showed great skill at the practice. I leveraged them as a vanguard to pair with others and help train and teach how to pair effectively.

By the end of the 6 weeks the culture and environment had evolved, and team chemistry was growing. At our retrospective the team agreed to keep working this way. We were more aware of the work the entire team was doing, enjoying the work more and delivering higher quality code.

However, it was not all sunshine and rainbows. Even while the team worked better, there were some engineers who clearly preferred not to pair. Sometimes engineers wanted to drive items individually to test themselves, or just wanted days of focused headphone music coding time. I listened to the reservations and found ways to balance and flex the model to work through it with them. But this left me questioning what is the key to the growth and benefits we saw? Is it truly two programmers one keyboard? Or is there another way to get these benefits in a more general and systematic way where?

…To pragmatism

After years of running that team successfully with pair programming as a core tenet I decided to change teams and move to a group in Azure. During this transition my key concern was how well my existing toolbox would work in a new team, environment, and problem space. After joining, I noticed many of the same problems: Silos, lack of group culture (was more individualistic), interpersonal conflicts and lower quality deliverables than they were capable of (the team was very talented). At this point I thought this was perfect, I already had a tool to solve this and proposed pair programming as part of the team methodology but got significantly more pushback than before.

This forced me to think hard about what I was trying to accomplish. Was my goal that everyone pair program or was it actually to inculcate a culture of trust and quality? This led towards refining pair programming into a new model for this team. I proposed to the team an experiment called active/passive pair programming. The idea was that we prefer to have pairs own deliverables, but the pair decides at the start of the work which parts need “active” pairing (two engineers working directly together at one keyboard) and which parts will use “passive” pairing (they work on different independent parts but check in multiple times a day).

My fear in this tweak was that it would devolve into fully independent work streams, and we would return to silos and lack of unity across the team. To prevent this, I highly encouraged pairs to try “active” pairing at least once a day and to routinely check-in with each other multiple times a day while “passive” pairing. Even while passive pairing, I made clear that both individuals must have full context on the work.

Some engineers actively paired, but many of them did passive pairing. I saw similar results as those in my previous experiment. The team grew closer together, psychological safety increased, and we were able to deliver more with higher quality. Even while passive pairing, the pairs communicated more, got to know each other and learned to rely on each other for questions and feedback.

The Power of Joint Accountability

When I left Microsoft to join Palmetto Solar, I transitioned into my first remote only role as a manager. I started this job with a renewed concern of how to build the culture I have seen work multiple times in an environment when we are seldom in the same physical place. Building trust, cohesiveness, minimizing silos, and creating a psychological safe environment are the keys but would the same techniques work remotely?

Reflecting on the success of the passive/active pairing experiment, I realized the reason it worked was trust. I extended to the pair the trust of accountability. They were accountable for how they worked, and for the success of the feature. By placing the accountability on them as a pair, it encouraged the behavior I was looking for. That is how I landed on the latest iteration of this technique which is to assign joint accountability on significant deliverables.

When planning new work streams, I ask the team who is interested in the work, and then we decide on a pair of engineers to take on joint accountability for the feature’s success. This is not just getting the feature out the door, but the entire lifecycle: design, coding, review, telemetry, and deployment. I don’t require that they “actively” pair at all (although many do). I make clear that I expect both people to have full context on the work. If one is out on vacation, the other can support and continue the other person’s part. If there is a question from a stakeholder, product manager, or other engineer, each person in the pair must be able to handle it.

Even in a remote only work environment, the pairs build close relationships driving features. Over time they learn each other’s communication styles, support each other, and even when not pairing, they end up with many calls to break down design issues and to act as rubber ducks for each other.

As new work comes, we rotate pair members to ensure different groupings work together. Over time this increases the cumulative level of trust, interpersonal awareness, and technical breadth within the team.

Critiques and Questions About Joint Accountability

Over the years of deploying versions of joint accountability I have heard feedback from peers and engineers. One concern I hear is around rewards, if a pair delivers a feature who gets credit? As long as both people have been driving impact to the best of their ability, they both deserve acknowledgement for the success of the feature. If the pair was not functioning well and one member feels they did most the work, it leads to the other concern I hear often, handling a non-functioning pairing.

There is a common concern that the stronger or more experienced engineer will carry the pair, and the other engineer just watches and does not contribute. In my experience though, this rarely happens, in most cases both people, even with a technical or experience gap, are eager to deliver value and contribute. However, I have witnessed a few times when one of the engineers clearly was not doing their work or seemed checked out.

In fact, this is actually a huge benefit of joint accountability – problems of this nature arise and become evident very quickly. When a pair is accountable for a feature, I periodically meet with each person individually. The focus is to provide coaching to make sure they are listening, questioning, and supporting each other well. In these meetings, I hear of any conflicts or roadblocks (which are natural when people are first learning someone’s communication style). Usually, if one person feels the other is not doing their fair share, I am able to provide coaching or more pointed, redirecting feedback if needed.

Keys to implementing Joint Accountability Successfully

If you are struggling with some of the same issues I faced, assigning joint accountability may help. To start, I recommend proposing it to the team as an experiment. All process changes work better when you get buy-in from the team and make clear their input is an integral part of the change. Set an expectation for the duration of the experiment and when and how often you will iterate and tweak. I use the team’s sprint retrospective for this purpose.

For joint accountability to work you must make crystal clear three key expectations:

1. The pair is accountable for the entire lifecycle of the change/feature. From initial design and implementation to deployment, monitoring, documentation, and communication.
2. Each member of the pair must have full context on all that is being built. If one member of the pair is out sick, the expectation is the other must be able to pick up their work if needed.
3. The pair chooses how they work (pair programming, divide and conquer, etc.). Trust them to make this decision rather than dictating how they work. This trust is critical.

Once the team agrees to try this experiment and all understand the expectations, you are ready to see if it starts the journey to a stronger more cohesive team. As work progresses, hold 1x1s with each person and ask how things are going. Checking in is not solely about the status of the work, but the status of their communication and how well they are listening and relying on each other. Look for issues, communication gaps, or dissonance as opportunities to coach and guide.

Closing Thoughts

Joint accountability is a flexible and powerful way to build a team that has more trust, autonomy, and accountability. In my experience it has been a critical tool in constructing strong teams. But like any process, it may not work in all situations or for all people. The key is to listen, take feedback and make sure you are focused on the end results and not married to any specific process or methodology.

The post Joint Accountability appeared first on Matthew Manela.