The good news? Like any skill, deliberate practice builds competence[1]. The more targeted experience you gain, the more you accomplish. This article shares what works based on practical experience.

The Contractor in Your IDE

Think of AI as a type of contractor: brilliant, fast, and tireless—but with no memory, no wisdom, and no stake in your project’s success. This mental model changes how you interact with AI tools. Future articles will explore other mental models in detail.

Just as with human contractors, you must clearly communicate your needs and expectations to AI. Great results require clear context, concrete constraints, and validation checkpoints.

The harsh truth: AI excels at certain tasks, delivering results faster and cheaper than any human. Many developers initially reject this reality.

Have you thought or said something like this?

I have tried AI. It’s nothing more than a fancy auto-complete.

Or

I tried it. I can produce better code faster without its so-called help.

I thought exactly these things before I realized the problem wasn’t AI, it was my approach. More specifically, my lack of skill in working with it. If you cannot write better software faster than you ever imagined using AI, then the issue is technique, and that’s fixable.

AI as a Force Multiplier

AI isn’t a 10x developer, it’s a force multiplier. A developer understands how the project fits with the business, larger mission, and other knowledge not available to the AI. This means you must carefully focus and direct the AI to useful activities. When done well, you benefit from useful results quickly.

Here’s what that means in practice.

AI Contractor Superpowers

• Execute tasks at superhuman speed
• Recognize patterns across vast knowledge domains
• Generate creative solutions you might not consider
• Work tirelessly without reduction in performance

How AI Contractors Differ from Humans

• No memory between interactions unless you explicitly provide it
• Zero wisdom about your codebase, architecture, or constraints
• Will confidently hallucinate when uncertain, responding with convincing (and sometimes amusing) but incorrect information[2]
• Cannot make strategic decisions
• Can enter repetitive loops when stuck on a problem

While these appear to be limitations, understanding them transforms them into manageable constraints. Sometimes these characteristics can even be useful.

Your Responsibilities

• Provide robust, efficient context
• Validate every output
• Make architectural decisions with clear implementation expectations
• Iterate and refine specifications and improve problem definitions
• Redirect to ensure effective progress
• Use the best tool for each job
• Protect your private data

You must define what you want done, how to do it, and establish validations.

Data Privacy and Security

Like human contractors, AI tools require careful data management. Never assume data you share is private. Encrypt or obscure sensitive information before sharing with AI.

Tools like Age provide practical encryption without complex infrastructure. Remember that AI assistants may not follow instructions as literally as traditional software. For example, if you specify a working directory but later reference an external file, the AI might access that file despite your initial constraints.

Choosing the Right Tool and Mission

AI focus has two aspects: knowing where to apply AI and maintaining proper aim. AI is not the best tool for every job, so focus on the things it does best. I am reminded of Star Trek where Dr. McCoy, when asked to do something outside his area of expertise, often reminds the captain that he is a doctor and not trained as a bricklayer or whatever.

Just as you expect better results for a plumbing job when hiring a plumbing contractor than a roofing contractor, you need to ensure you use the best models and tools for the job at hand. Future articles will explore this in detail.

A force multiplier is useful only when it brings you closer to your goal. Think of golf. The goal is to move the ball into the hole with the fewest strokes. Beginner golfers often focus on hitting the ball as far as they can. Golfers quickly learn that hitting the ball twice as far is good only when this results in moving the ball closer to the hole. Aim is as important as distance.

Understanding AI Behavior

What we observe as hallucinations aren’t defects, they’re features of probabilistic design[2], [3]. They enable creativity but require validation loops.

Test-Driven Development still holds regardless of how the code is written[4]. When interacting with AI, we also need, Validation-Driven Prompting (VDP).

Test‑Driven Development ensures code aligns with expectations defined in tests; similarly, Validation-Driven Prompting ensures AI responses or agent actions align with expectations defined in prompts.

I recommend instructing AI agents to develop software using TDD. AI agents benefit from TDD the same way human developers do. Defining tests first helps communicate more precisely what we need.

It provides better context.

VDP helps the AI know when and how a response meets our needs. Here is a quick example.

While working on a markdown table, I asked AI to insert two new columns. Multiple prompts failed until I instructed the AI to validate each field as a data matrix. After adding that validation instruction, the AI updated the table correctly. This is just one example of many I have experienced personally.

For this series, when I use the term “AI,” I generally mean Large Language Models (LLMs) — systems trained to predict probable text outputs based on input patterns[5]. Understanding they’re probabilistic, not deterministic, explains both their power and pitfalls.

Steve Jobs famously described a computer as a bicycle for the mind. AI is becoming a mind for that bicycle. The difference between beautiful, production-ready code and total garbage comes down to how you interact with that mind. While this is true when working with any contractor, your AI contractor lacks the agency afforded a human, so you must adjust your approach accordingly.

Building Skill on a Moving Frontier

AI skill development requires deliberate practice with feedback loops[1]. But here’s the twist: the frontier moves while you’re learning it.

AI changes so quickly that some things true today are outdated tomorrow. Another challenge is grasping non-linear improvement. AI capabilities don’t grow steadily, they move, stall, and make dramatic jumps.

This Means Two Things

1. Focus on transferable skills: Context engineering, validation patterns, and mental models (like the contractor framework) remain valuable even as specific tools change.

2. Build your practice loop: Nobody knows what will work best tomorrow because those techniques don’t exist yet. You need a system for continuous experimentation and improvement.

The Context Engineering Evolution

Prompt engineering exemplifies how the field evolves. Initially popular, it was declared unnecessary as models gained automated reasoning capabilities. Then practitioners discovered it wasn’t dead—

Response quality still depends on context quality.

The “death” of prompt engineering gave birth to context engineering[6]. The net here is that no matter what you do with AI, context matters. Mastering context is a transferable skill.

What This Series Covers

This series targets developers with basic AI experience who want to move from “fancy autocomplete” to genuine force multiplication[7].

Examples will be based on my macOS setup, but translate easily to other platforms.

Next Steps

AI isn’t magic. It’s a tool that rewards skill. Like learning any technology, you need deliberate practice, feedback loops, and experimentation. The difference? The frontier keeps moving, so your learning system matters more than memorizing specific techniques.

The most important transferable skill right now? Context engineering.

The next articles will focus on context engineering, including some things rarely discussed elsewhere.

Try this now: Take your last frustrating AI interaction. How many prompts did it take until you were satisfied with the response or gave up? If you were finally satisfied, what did your last prompt provide that was missing from the previous prompts? If you were never satisfied with the response, why not?

Questions? Experiments to share? Reach out. We accomplish more through collaboration.

References

What

Alacritty does all the things I need and may even be faster than iTerm2. It is also a kick-butt Rust based project. What more could you ask for? If you say tabs, well then no, Alacritty does not support tabs. I use Tmux in place of tabs, so for me this just means that Alacritty is a better targeted solution.

I need a terminal emulator to be fast, support color, ligatures, Tmux, and VIM simultaneously and seamlessly. Alacritty does all this very well.

If I notice terminal introduced latency then it is too slow, if I do not, then it is fast enough. I use color to provide semantic clues. I expect at least 256 colors. Tmux and VIM should work together largely the way they work separately. Ligatures require more explanation.

A ligature is a single character representation of a symbolic concept resulting from the joining of two or more characters¹. It is useful in some programming languages to simplify the display of compound ASCII symbols.

Here are two ways to represent not equal.

Not Equal - ASCII

Not Equal - Ligature

I use ligatures as a way to display symbols, comprised of multiple ASCII characters, as a simpler rendering, without altering the actual source characters. Here is an example from Fira Code that shows and example of the ASCII saved in a file, on the right, and the ligature representation displayed, on the left.

This enables me to view some characters as ligatures, but not to enforce my preference on other users of a source file. I like having the control of when to obscure the actual ASCII with ligatures. When I am editing a line, I like to see the original ASCII on that line so I can see what I am editing. The iTerm2 update forced the display of ligatures all the time. The iTerm2 configuration for this feature was disabled.

In order for me to control the expression of ligatures, I need the terminal emulator to display the actual character. I use tools like enomsg/vim-haskellConcealPlus to obscure the ASCII with ligatures when that is my preference.

Notice how some of the characters are displayed as ligatures even on the cursor line. The cursor is narrow in the in the screen-shot but look at the character just left of liftIO.

How

Here is how I got Alacritty working on my Mac (Sierra 10.12.x). If you do not use Mac then your setup will be different.

Fonts

I am using two fonts, Fira Code and Menlo. I beleive that Menlo ships with Mac, but Fira Code does not. Open Font Book to ensure both are installed. To install Fira Code, simply follow the instructions here. I just downloaded the Fira Code fonts and installed them with Font Book, rather than using homebrew. Once installed you should see the following in Font Book.

Fira Code

Menlo

Tmux

Make sure Tmux is installed and reasonably updated. I found that the version from homebrew did not work fully, so I uninstalled it and installed the following from macports².

• tmux @2.5_0
• tmux-pasteboard @2.6_0

I updated my ~/.tmux.conf to change the default terminal identifier.

set -g default-terminal "tmux-256color"

Alacritty

Manually install Alacritty as shown here, i.e. do not use homebrew.

Edit the .config/alacritty/alacritty.yml configuration file.

Update the fonts

I am using Fira Code for regular and bold fonts. I am using Menlo for Italic.

# Font configuration (changes require restart)
font:
  # The normal (roman) font face to use.
  normal:
    family: Fira Code
    # Style can be specified to pick a specific face.
    style: Regular

  # The bold font face
  bold:
    family: Fira Code
    # Style can be specified to pick a specific face.
    style: Bold

  # The italic font face
  italic:
    family: Menlo
    # Style can be specified to pick a specific face.
    style: Italic

Misc

The terminal setting should look like.

  TERM: xterm-256color

Look over the rest of the config and ensure it matches your environment. I also changed the bash used to the version installed in /opt/local/bin.

shell:
   program: /opt/local/bin/bash
   args:
      - --login

The colors can be manually modified by providing the hex value.

  # Normal colors
  normal:
    black:   '0x000000'
    red:     '0xd54e53'
    green:   '0xb9ca4a'
    yellow:  '0xe6c547'
    blue:    '0x7aa6da'
    magenta: '0xc397d8'
    cyan:    '0x70c0ba'
    white:   '0xffffff'

If you are moving from another terminal emulator, like iTerm2 you can view the defined colors, for example from iTerm2ColorSchemes.com, and grab the hex value with Digital Color Meter.

Open Digital Color Meter and then menu, view, display values, and select as hexadecimal. Then fly over the color and then hit Shift - Command - c, to copy the hex value to the clipboard. Then from the clipboard, you can paste them to the .config/alacritty/alacritty.yml configuration file for the given color.

Also if you like the color schemes at iTerm2ColorSchemes.com you can find the scheme you like and read the termite config in github.

If for example you like the SoftServer scheme, you can view the hex values in github.com/mbadolato/iTerm2-Color-Schemes/blob/master/termite/SoftServer. The termite config file looks like this.

[colors]
background = #242626
cursor = #d2e0de
foreground = #99a3a2
color0 = #000000
color1 = #a2686a
color2 = #9aa56a
color3 = #a3906a
color4 = #6b8fa3
color5 = #6a71a3
color6 = #6ba58f
color7 = #99a3a2
color8 = #666c6c
color9 = #dd5c60
color10 = #bfdf55
color11 = #deb360
color12 = #62b1df
color13 = #606edf
color14 = #64e39c
color15 = #d2e0de
colorBD = #d2e0de
colorIT =
colorUL =

If manually editing is not your thing, there might be a color scheme that you can use. Nord Alacritty looks nice.

Man

I went ahead and added some color for Man pages also.

• Update ~.bashrc

# color for less and man 
export MANPAGER='less -s -M +Gg'
export LESS="--RAW-CONTROL-CHARS"
lesscolors=$$HOME/bin/.LESS_TERMCAP
[[ -f $$lesscolors ]] && . $$lesscolors

• Create or edit ~/bin/.LESS_TERMCAP

export LESS_TERMCAP_mb=$$(tput bold; tput setaf 2) # green
export LESS_TERMCAP_md=$$(tput bold; tput setaf 6) # cyan
export LESS_TERMCAP_me=$$(tput sgr0)
export LESS_TERMCAP_so=$$(tput bold; tput setaf 3; tput setab 4) # yellow on blue
export LESS_TERMCAP_se=$$(tput rmso; tput sgr0)
export LESS_TERMCAP_us=$$(tput smul; tput bold; tput setaf 7) # white
export LESS_TERMCAP_ue=$$(tput rmul; tput sgr0)
export LESS_TERMCAP_mr=$$(tput rev)
export LESS_TERMCAP_mh=$$(tput dim)
export LESS_TERMCAP_ZN=$$(tput ssubm)
export LESS_TERMCAP_ZV=$$(tput rsubm)
export LESS_TERMCAP_ZO=$$(tput ssupm)
export LESS_TERMCAP_ZW=$$(tput rsupm)
export GROFF_NO_SGR=1         

Enjoy

Here is are a few examples of the results.

This shows the way I use ligatures. Also note the italic comments.

Here is a Man page using color.

I recently purchased my first non-Model-M mechanical keyboard, the KBParadise V60 polestar.

Night

Day

I like the 60% size, the feel of the cherry MX clear key switches, and the customization potential from its ATMEGA32u4 microcontroller. The original layout of the keyboard did not match the way I prefer for use with a Mac or Linux system, but this can be easily changed. I am using the U.S. layout, other layouts likely work similarly. The following describes how to customize the keyboard from a Mac, for comfortable Mac use.

This is the layout defined by default for the keyboard. This layout shows two layers with the first layer printed on the tops of the keys, the top legend, and the second layer on the front.

The real difference is with the arrangement of the modifier keys. The important modifiers being at the bottom left. Here is the default Mac layout from an actual mac keyboard.

Mac order

Ctrl, Option/Alt, Command/Win

Windows order

Ctrl, Command/Win, Option/Alt

Tools

Updating the layout from my Mac was simple using a few freely available tools.

tkg-toolkit

The TKG toolkit is a selection of tools supporting the TMK Keymap Generator. It is used to update the keyboard’s EEPROM and firmware. It should be possible to program this keyboard using Quantum Mechanical Keyboard Firmware, but at the time of writing this, the keyboard is not fully supported by qmk.

keyboard-layout-editor.com

The Keyboard Layout Editor makes it simple to define a layout by mapping key locations to names.

TMK Keymap Generator

The TMK Keymap Generator converts a keymap string, such as produced by the keyboard-layout-editor.com, and generates a file that can be used to update the keyboard’s EEPROM. This is then provided to tkg-toolkit to flash the new layout.

Mac Keyboard Viewer

Mac OS X has a keyboard viewer that can be accessed by mouse or other pointing device. This is helpful when keyboard input is required but the physical keyboard is busy being updated.

Howto

tkg-toolkit setup

Clone the tkg-toolkit git repository. The current version of the {{page.tkg-toolkit-link}} does not contain the firmware for this keyboard, fantastic60.hex, as described in this pull request. I added it to my local repository from here.

The firmware hex file, fantastic60.hex, does not appear to be required if all you wish to do is update the EEPROM with a new layout. I feel is it wise to keep a copy of the current firmware.

Use git to clone the tkg toolkit repo

Execute the following from the command line.

clone git@github.com:kairyu/tkg-toolkit.git

Ensure that libusb-compat is installed.

brew install libusb-compat

Prepare to reflash.

From the tk-toolkit repo, do the following.

cd mac
./setup.sh
PATH=$$PATH:$$(pwd)/bin

During setup you will be prompted to select your keyboard. Select the option for Fantastic60. You will then be prompted to select your bootloader. Select the option for atmel_dfu. The setup.sh script does two things. It creates a conf directory then a configuration file named default.ini.

The conf/default.ini created by setup.sh should look like this.

Name="Fantastic60"
MCU=atmega32u4
Firmware=fantastic60.hex
Bootloader=atmel_dfu

Setting the path with PATH=$$PATH:$$(pwd)/bin is temporary, so you would either want to update your path differently or just execute this each time before you re-flash.

Setup Alternate keyboard

Ensure that you can access the Mac Keyboard Viewer. On my system, the Keyboard Viewer looks like this.

It should be possible to activate the keyboard viewer from one of these paths:

• Apple menu > System Preferences, click Keyboard, click Keyboard, then select “Show keyboard and emoji viewers in menu bar.”
• System Preferences > Language & Text > Input Sources > Check the box “Keyboard & Character Viewer.”

Define layouts

This keyboard can have multiple layered layouts. The way this works is explained in the TMK Keyboard Firmware Collection here. The keyboard shipped with two layers, but as shown in MechMerlin’s video here¹, I too add a third layer.

Define a layout for each layer using keyboard-layout-editor.com. Save the layouts by saving the raw data. This can be done using the github functionality provided in the keyboard-layout-editor.com site, or manually from the raw data tab.

The user manual describes the original layout. My customization includes three layers, numbered zero through two, to match the numbering used by [tkm-keymapgenerator-link][]. Higher numbered layers apply changes for mapped keys and are transparent for un-mapped keys. By transparent I mean that if a higher numbered layer does not define a mapping for a given key, then that key will function as defined in the next lower numbered layer, with a mapping.

I created a public gist for the raw data and a link to load each layout into the keyboard layout editor.

Each of the layers are defined against a Default 60% keyboard.

Layer 0

The first layer is named layer zero. Here I have altered the order of the meta keys on lower left and remapped the caps-lock key to the esc key. The new meta key order matches the default for a mac keyboard. I also altered the original esc key to produce the ` and ~ characters.

Make sure you save your customizations to your gist.

Layer 1

The second layer, layer one, is modified to change the space-bar to execute firmware function Fn11. The Fn11 key is not the same as the F11 key. The F11 key is a normal keyboard key. F11 activation is passed on to the operating system for interpretation, and in this layer, is still mapped to the key that is - on layer zero. I added a few more normal function keys to this layout in case they are actually used by the operating system.

Notice that in this layer the space-bar key has been re-mapped to Fn11.

Layer 2

Layer two will map only four keys.

Transparent layout

{:.no_toc}

When there are few keys to map, it is easier to start from a fully transparent layout. I created such a layout. Its gist is here and can be loaded from here.

Arrow keys defined for Layer 2

{:.no_toc}

Here all the keys are transparent except the arrow keys defined in the lower right corner.

Make a keymap with TMK Keymap Generator

• Go to TMK Keymap Generator.

• Select the Fantastic60 keyboard.

• Select layer mode Normal.

• Select 3 layers.

• Paste each of the three layers we created above into their respective text area.

You can paste the raw data or you can paste the keyboard-layout-editor.com links with the saved content. I have links for my layers in the table here. Your cursor must leave the text area in order for the URL to be fetched and the data parsed. You might need to click on the page outside a text area to trigger the action. You will be presented with errors and warnings if applicable. I have found that errors typically mean that I have pasted data incorrectly.

• Define the firmware functions.

The mapping for the Fn keys should largely be the same as the default, but with the addition of a toggle setting for Fn11. Note the action for Fn0 and Fn11. In both cases they are a toggle to a new layer as described here. This means that with each press and release of the key the active layer changes. Fn0 is being used to activate and deactivate layer one (the second layer). Fn11 is being used to likewise activate and deactivate layer two, the third and final layer.

• Click the Download .eep file button to download the .eep file.

Flashing the keymap to EEPROM

• Go back to the command line. You should be in your tkg-toolkit git repo in the mac directory. You should be there from where you executed the setup and updated the path above.

• reflash the EEPROM and provide the path to your newly downloaded .eep file. Where ./keymap.eep is replaced with the actual path and file name for the newly created .eep file.

./reflash.sh ./keymap.eep

You will be presented with a prompt asking if you want to continue.

• Open the keyboard viewer.

• Turn over the keyboard you are flashing, and push the reset button, located on the bottom of the keyboard.

• Using the keyboard viewer, Type Y to answer the continue prompt. Prevent the keyboard from being interrupted during the flashing process.

• When the flashing process is complete, you will be prompted to press any key to continue. Use the keyboard viewer to press any key.

• Disconnect the keyboard from the USB cable, and then re-connect it. Now test out each layer and ensure it is working the way you want.

• You are done. Enjoy!

Useful links

• dfu programmer
• qmk
• QMK Firmware docs
• KBParadise
• ATMEGA32U4
• KBParadise V60 polestar
• tkg-toolkit
• keyboard-layout-editor.com
• TMK Keymap Generator
• MechMerlin’s video for programming this keyboard from Windows
• MechMerlin’s video reviewing this keyboard
• Programming Manual from KBParadise for the V60 Type R Keyboard
• Homebrew package manager

What!?

Numbers are used everywhere in code, and often the type used is something like Int (Scala) or Integer/int (Java). These simple types are familiar to those of us who used languages like C but should not be used without considering their inherent limitations. Let’s imagine an example web application that during its normal operation records interesting events…

Imagine a field used to represent the key of a database table named INTERESTING_EVENT, specifically an automatically incrementing surrogate key, as is typical. (This is not the best example, in part because one would rarely need perform arithmetic on a key value, but it is simple to explain.) Let’s say that on average, only twenty INTERESTING_EVENT records are populated per second.

Is an Int or Integer safe? Let’s get some perspective…

365 days per year

8760 hours per year

525,600 minutes per year

31,536,000 seconds per year

Given a liner expansion of the INTERESTING_EVENT sequence, by twenty each second, the value would grow by 630,720,000 each year.

630,720,000 events per year

In four years that would grow to 2,524,608,000.

2,522,880,000 events in four years

The MAX_VALUE for java.lang.Integer is 2,147,483,647.

The example is contrived and unlikely but also recall that automatically incremented keys do not always grow linearly, and twenty events per second is conservative for a web server… (Insert your own web scale humor here.) But even with those constraints the key grows beyond the capacity of java.lang.Integer in less that four years. Four years are enough time to forget about the capacity constraint built into the example application.

You might forget how these simple number types work and think:

No problem, statistics or other processes are running, constantly computing metrics against Events and some computations use the key value… Surely an exception will be thrown as soon as the Integer overflows.

This is where you would be wrong.

Division will throw an ArithmeticException on division by zero, but a sum, for example, will quietly continue on an overflow. Let me repeat that… The sum will return an incorrect value on overflow, but it will not throw a runtime exception! This is serious problem. The application will quietly continue to run, but the mathematical operations on the value will be incorrect. This should be a real “What!?” moment.

Two’s complement

The JVM stores an int as a two’s complement binary (the core of java.lang.Integer and scala.Int), and thus it is often referred to as a two’s complement type.

Specifically, the JVM uses a 32-bit signed two’s complement integer for int. The most significant bit is used to designate the sign of the number. The max value for int is 2,147,483,647, and would thus be represented¹ like 0111 1111 1111 1111 1111 1111 1111 1111.

Simple sum

First lets simply add 1 to 2,147,483,646.

  0111 1111 1111 1111 1111 1111 1111 1110
+ 0000 0000 0000 0000 0000 0000 0000 0001
-------------------------------------------
  0111 1111 1111 1111 1111 1111 1111 1111

The result is what we expect, 0111 1111 1111 1111 1111 1111 1111 1111, or 2,147,483,647 in decimal.

Negative values

The JVM int is signed, that is to say both positive and negative values can be stored as an int.

Using the MSB (most significant bit) to designate sign makes it possible to store negative values using the same placeholders and arithmetic. A negative is represented with the MSB set to 1. To convert -2,147,483,647 to a two’s complement binary for int, we must find the complement of each bit then add one to the result.

1. Find the complement of each bit of 0111 1111 1111 1111 1111 1111 1111 1111.
   1000 0000 0000 0000 0000 0000 0000 0000

2. Add one

  1000 0000 0000 0000 0000 0000 0000 0000
+ 0000 0000 0000 0000 0000 0000 0000 0001
-------------------------------------------
  1000 0000 0000 0000 0000 0000 0000 0001 

The minimum value would be the value with the greatest magnitude and the opposite sign. The 32 bit, negative, two’s complement number with the greatest magnitude would be where the sign bit is set to 1 and all the other bits are set to 0, i.e.. 1000 0000 0000 0000 0000 0000 0000 0000. This means the min value² is -2,147,483,648 in decimal.

Bad arithmetic without error

Let’s see what happens when we overflow by adding one to 2,147,483,647.

                                       1                                 
  0111 1111 1111 1111 1111 1111 1111 1111
+ 0000 0000 0000 0000 0000 0000 0000 0001
-------------------------------------------
                                        0

  1  
  0111 1111 1111 1111 1111 1111 1111 1111
+ 0000 0000 0000 0000 0000 0000 0000 0001
-------------------------------------------
  1000 0000 0000 0000 0000 0000 0000 0000

Wait! We have seen this one before, it is -2,147,483,648! So in this case the result is mathematically incorrect, but a valid int!

You can try out Scala code snips in the online REPL here, and Java in the online REPL here. The following return -2147483648.

Scala

Int.MaxValue + 1

Java

Integer.MAX_VALUE + 1

BigInteger (java.math.BigInteger) to the rescue?

Clearly when arithmetic is needed, or when it is possible to overflow a field, one should use something safe like BigInteger over its primitive counterpart. Since one can never know for certain how a field will be used, selecting a safe type should be the default.

If one can be sure that arithmetic will never be performed on a field, and benchmarking shows that using a primitive is needed, then maybe it is worth the risk.

Primitive Type

Pros

Simple quick arithmetic.

Fixed memory allocation size.

Symbol operators. ( + - * /)

Cons

Arithmetic can fail without error!

BigInteger (java.math.BigInteger)

Pros

Correct arithmetic

Cons

Can not use symbol operators; must use add, subtract, multiply, and divide methods.

Arithmetic could be slower.

Higher memory allocation requirements because more information is stored and capacity is dynamic.

BigInt (scala.math.BigInt)

If you live in the Scala world you have better options, such as scala.math.BigInt. It can be thought of a Scala’s BigInteger. In Scala, characters such as + - * / can be used as method names, so this makes using scala.math.BigInt a pleasant experience.

Spire

Finally, when talking about handling numbers in Scala, we need to talk about spire. Spire makes working with numbers a beautiful experience.

//spire example
object Inspired extends App {
  import spire.syntax.literals.si._
  val y = big"2 147 483 647"
  println(s"y, using spire string interpolation = $${y + 1}")
}

Look at how clear and simple it is to create a BigInt using spire string interpolation. Using spaces makes it easier to identify the place values for each digit and results in source that is simpler to read. Spire supports additional number types, type classes and more. Let’s take SafeLong for example. It works as a nice replacement for BigInt. While it provides accurate arithmetic, independant of value, it performs better than BigInt for values inside the java.lang.Long range.³

Conclusion

A two’s complement type like int should be considered an optimization choice for Scala and Java software development, and not a default. Its performance comes at a cost and that cost is silent failure. The default choice for a whole number field should be something like BigInt or SafeLong because they ensure correct arithmetic without regard to the size of the operands. This is made far more pleasant in Scala, especially when using a library like spire.

Getting Started with Scala

When you get started with a language you need a “Hello, World”. (This example works if you install Scala, rather than using it from SBT.) So here is one:

 $$ vi HelloWorld.scala

object HelloWorld extends App {
  println("Hello, world!")
}

 $$ scalac HelloWorld.scala
 $$ scala HelloWorld
  Hello, world!

What is Scala?

The official answer is here, but to me it is simply an object-functional programming language on the JVM. The more interesting part is the functional part, but because it also supports object oriented design on the JVM, it makes for a simpler transition from Java than say moving directly to Haskell. If you are a Java programmer maybe you should avoid scala? Don’t say I didn’t warn you. ;-)

Functional

Here are some functional programming links (mostly videos). Some relate to scala more than others.

• Great link for many other links

Videos

• Erik Meijer Functional Programming Fundamentals Thirteen lectures.

• Erik Meijer Many useful videos.

• Brian Beckman Many useful videos.

• UCB SICP lectures playlist

• Bodil Stokke: What Every Hipster Should Know About Functional Programming A fun look at some functional programming terms.

• Functional Thinking with Neal Ford A good introduction to the kind of thinking used in Functional Programming. Includes examples in various programming languages (not just Scala). Gets especially pragmatic at about 31:00.

Books

• Functional Programming in Scala by Paul Chiusano and Rúnar Bjarnason This is an outstanding book. It can be treated as a course in Scala by going through the exercises in each chapter.

• SICP: Structure and Interpretation of Computer Programs, by Abelson, Sussman, and Sussman Read or re-read this one. No, seriously, do it right now. You will be glad you did.

General Scala

Links

• A Tour of Scala A nice tutorial.

• Scala School! From ∅ to Distributed Service Twitter’s excellent tutorial.

• A Glossary of Scala and Functional Programming terms taken from the Programming in Scala book

• Cheat Sheet for Scala Syntax or Cheatsheet at scala-lang.org The version at scala-lang.org is a copy of Brendan O’Connor’s Cheat Sheet for Scala Syntax.

• progfun wiki cheatsheet This originated from the Functional Programming Principles in Scala class. I can’t recommend the class enough.

• Reactive cheetsheet This originated from the Principles of Reactive Programming class. Really great class.

• Scala information from stackoverflow.com Good stuff here. Some overlap but that is fine.

• Scala style guide Be aware that some of these style decisions will save you headaches. One example is the curly brace alignment. It has been difficult to change my brace style to fit the recommendation but it is better than dealing with parsing problems.

• scala user google group

Books

• Scala by Example by Martin Odersky Useful book. This link to a PDF version.

• Programming in Scala, First Edition by Martin Odersky, Lex Spoon, and Bill Venners The link is to the first edition which is made freely available online.

• Scala for the Impatient by Cay Horstmann The link is to a freely available subset of the book in PDF format.

• The Scala Language Specification Scala Reference I consider this the Scala Reference rather than a true specification regardless of the title.

If you have some favorite links for getting started with Scala, then please share them in a comment.

A while ago I was chatting about how I found some aspects of Haskell and Go really interesting, when BeepDog mentioned Scala. Well, at the time I felt I had no need of Scala because

If I wanted to use the JVM I would use Java, and if I did not want to use the JVM I could use Go, Haskell, or whatever other language best fit the needs of the project.

But then I did not know Scala. An EPFL Coursera class, Functional Programming Principles in Scala opened and being out of excuses, I took the class. I was so impressed that I took the next one, Principles of Reactive Programming, too. There is no fitting exclamation, or emoticon to describe the experience.

I can’t list all the advantages of using Scala on the JVM, because I keep finding new ones. Here are a few just to get things started.¹

• An improved, static type system

• Native support for Functional Programming

• Native support for Object Oriented Programming

• Clever concurrency solutions

• A more concise and expressive syntax

• Runs on the JVM fully interoperable with Java.

Scala classes can call Java methods, create Java objects, inherit from Java classes and implement Java interfaces.
Scala Language Specification, version 2.8

The above advantages and more makes Scala the pragmatic choice for productive, maintainable, DRY, and performant code on the JVM platform.

So with all those advantages, why avoid it? Because something happens when you learn Scala…

I continue to experience situations where I find some aspect of the Scala syntax or some framing of a problem awkward only to have an epiphany. The result of the epiphany is an astonishing change in my way of thinking. Avoid Scala because once you learn it you will be forever changed and will never again approach problems the same way. Scala is improving, and promising languages appear all the time. Who knows where the journey will end.

If you are content with using the JVM just as you do now then please stick with Java, but stay away from Scala. ;-)

This is the second of a two article series about using machine virtualization as a development tool. It focuses on the question of how. The first article is found here and focuses on the question of what.

Using VMs

It is time now to get to the specifics of using a VM solution, in this case VirtualBox.

Guest Additions

Guest Additions are vendor specific add-ons that permit some useful features of VirtualBox. The Guest Additions should be installed into the guest OS when creating a base VM. See the VirtualBox documentation for details.

Shared Folders

One of the important features that Guest Additions permits is a shared folder. A shared folder permits the sharing of a host directory to the guest VM. This is a nice way to deploy code or otherwise share content with the VM. If content in a shared folder is stateful, such as code, then it should be managed by a version control system, such as git.

Snapshots

A snapshot saves the state of a VM such that when the snapshot is restored the VM continues exactly where it left off. When a VM is started back at a point in time of a snapshot, remounted network drives and shared folders may have different contents than when the snapshot was made because the content is sourced externally to the VM. This is in contrast to virtual drives which by default will be restored bit for bit.

Snapshots are useful for a developer to test a new release or to rollback a site to an older state to work on a bug-fix. For example, one might be working on a new release of an application, only to have an emergency bug-fix take precedence over work on the new release. It is then a simple task to snapshot the VM to save the state for the new release, then restore a clone of the VM as it existed for the code experiencing the bug, right along with checking out the matching code from one’s SCM, like git. Then when the bug is fixed a similar process is performed to resume work on the new release. This ensures that not only does the code match production, but the other aspects of the host match as well such as the database¹, web server settings, etc.

Taking a snapshot is not a big deal so when in doubt, take one. Always take a snapshot to capture current state before restoring a snapshot. Beyond that, it is a good idea to take a snapshot before making a time consuming change to the VM, and with each code promotion.

It is wise to shut-down the VM prior to taking a snapshot because when one takes a snapshot of a running VM the memory state of the machine is also stored in the snapshot. This extra data is only needed if it is important to restore the snapshot to the running state, and it can occupy as much disk space as the memory defined for the VM.

When making a snapshot, VirtualBox provides a field that can be used to label the snapshot. A description field is also provided. Be sure to use the description field to include any information that is needed to fully restore the state of the snapshot. For example, include the git tag² needed to revert contents of a shared folder.

In the following example, the snapshot is labeled to show that it was taken prior to doing some database maintenance. The description field contains the git data needed to restore a shared folder to the correct condition of the snapshot. The command git log --decorate --oneline -1 is a handy way to get the latest commit information, but ideally one should be creating an annotated tag to go with the snapshot. An annotated tag could be labeled the same way as the snapshot, thus clarifying the snapshot to tag relationship from examination of both the code and the list of snapshots.

The power of a snapshot is the ability to capture a point in time. However, to be able to make full use of the point in time, we may use snapshots with clones.

Clones

As one would expect, cloning a VM is making a copy of a VM. The source for a clone can be a VM’s current state or a particular snapshot.

Let’s consider the following where one is working on a web development project where git is being used to manage the stateful content of a shared folder:

A critical bug is found in production. We must roll-back to a snapshot to work a bug then later resume the current state…

There are two ways to handle this:

Use snapshots

1. Shutdown the VM so we do not preserve the VM run-time data.

2. Stash content of shared folder.

    $$ git stash save "Working on customer lookup when ticket ID 113245 was received"

3. Capture the full stash information for use in the snapshot description.

    $$ git stash list
    stash@{0}: On dev: Working on customer lookup when ticket ID 113245 was received

4. Take a new snapshot of the current state so we have a place to return to when the bug is fixed. Include the stash information in the snapshot description.

5. Restore the VM to the snapshot required to fix the bug. (In this case the snapshot created for the v1.0.1 code promotion.)

6. Restore the shared folder to the correct place³ for fixing the bug.

    $$ git checkout -b b1.0.1-113245 tags/v1.0.1 

7. Fix the bug…

8. Restore the VM to the snapshot made with our stash comment.

9. Restore the shared folder to the state that corresponds to the snapshot we just restored.

    $$ git checkout dev
    $$ git stash list
    stash@{0}: On dev: renaming serialized files
    stash@{1}: On dev: Working on customer lookup when ticket ID 113245 was received
    $$ git stash pop stash@{1}

Use snapshots with a clone

What if a co-worker can take on the bug fix while you continue to work on your current sprint? If that is the case, then we can simplify the process down to creating a clone VM for the co-worker.

1. Select the correct snapshot from the snapshots list. (In this case the snapshot created for the v1.0.1 code promotion.)

2. Right-click the snapshot and select clone, or click clone button (pictured as a sheep).

3. Name the clone including the source VM name, snapshot name and the ticket ID 113245.

4. If your co-worker uses the same network as you, then click the Reinitialize the MAC address of all network cards. Note that you may need to boot into the VM and update network settings to reflect the new MAC addresses.

5. Select a full clone rather than a linked clone, so that the clone can be operated independently.

6. When prompted for the what parts of the snapshot tree to clone, select Current Machine State. It is confusing but Current Machine State is the current state of the VM at the point in time of the snapshot you selected. This can result in a smaller VM because no other states, and their requisite differencing disk images, need to be incorporated into the clone.

Headless

Once a VM is fully setup with SSH and the other tools I need, I find that the GUI simply gets in the way. There are two easy ways to start a VM without loading its console or its Graphic User Interface (GUI).

1. Hold down the shift key when starting a VM from the GUI VirtualBox Manager.
2. Start a VM from the command line with VBoxHeadless --startvm <uuid|name>.

Maximize return on virtualization investment

When using VM software, as with any tool, to be useful it must save more time and headaches than it causes. There are a few things one can do to maximize return on one’s time investment.

Separate source code management from VM management and identify intersections

Keep source code and source code management on the host machine. Share code or deployments with the VM guests by way of shared folders, do not store such things on a VM’s virtual disk. This eliminates unnecessary dependencies on guest VMs by confining code management to the host machine. This does not preclude the use of remote git repositories. It simply ensures that the host is handling those dependencies.

Identify intersections of state between guest VMs and managed code. This means identifying important milestones in code state with SCM tags, such as a git annotated tag. Also identify important milestones in VM state with snapshots. Be clear that each time a tag is needed for the source code, a snapshot should be made in the VM, and vice versa. Keep snapshot descriptions updated with tags that correspond to the intersections of state between the VM and the code. See Snapshots above.

Use pre-built base VMs.

It is my experience that the most time consuming aspect of using VM software is creating the base VM. That said, creating a base VM from scratch is no more complicated than installing an OS on a real machine, and once you have done it once, I find it simpler than dealing with a real machine. A nice short-cut is to use a pre-built base VM, then simply customize it if needed. The ideal place to get a VM would be from whomever created the VM for your production environment, if a VM is used in your production environment. Pre-built VM images are sometimes called virtual appliances. Even though there are many VM software technologies, a format exists called OVF (Open Virtualization Format) that can be used to make a VM image saved in this format more compatible across different VM software providers. This is supposed to make it possible, for example, to use a VM created for VMWare on VirtualBox. VirtualBox has an import mechanism to let one make use of these OVF VMs.

The following are two places with useful VMs.

• Oracle: developer-vm
• VMWare: virtual-appliances

The place I find the most useful is vagrantbox.es, but to properly use those images we need another tool called Vagrant.

Use Vagrant

Use Vagrant to simplify and accelerate creating your own base VMs. Once setup, it is possible to create a new VM, with a single vagrant up command. The Vagrant build process is automated and controlled by a written configuration, thus making Vagrant VMs reliable, predictable and consistent.

I use Vagrant with Puppet to simplify and unify the provisioning tasks. The following example shows how to use Puppet to ensure that git is installed in the guest VM. Notice that the command does not identify a particular OS or distribution specific package manger, like rpm or yum. Part of the power of Puppet is that it can keep these specifics out of the configuration and out of the way.

package { 'git':
    ensure => installed,
    }

This lets one focus more on addressing the needs for a VM, and less on how those needs must be achieved for a particular OS/distribution.

I created a Habari development VM for GrowingLiberty.com using Vagrant. This implementation can be found in github at the following URL.

github.com/mmynsted/vagrant-centos-php

The details are documented in the Readme.md. This implementation uses a base Vagrant box from vagrantbox.es, showing that one can further simplify provisioning a base VM by using a pre-built Vagrant base box.

My preference is to use Vagrant to create a base VM, then clone it to a new VM that is no longer dependent on Vagrant. This way I am free to continue to improve my Vagrant implementation without adversely affecting actively used VMs. The example above shows how one could use Puppet as part of the original provisioning and then use traditional shell commands from the new clone. Puppet is still installed so one can interact with the VM in the way that seems most natural.

The Vagrant and Puppet combination enables one to quickly translate a VM build investment to new and changing needs.

Conclusion

A virtual machine is a software based abstraction of a physical machine. This abstraction permits reduced maintenance time, improved testing, and making better use of both remote and local machine resources. Using a local VM can become a seamless part of one’s development process, can help one be better organized, and more productive.

The use of Virtual Machines for web development is becoming ubiquitous, and with good reason.

This is the first of a two article series about using machine virtualization as a development tool. It focuses on the question of what. The second article is found here and focuses on the question of how.

What are Virtual Machines for a software developer?

A Virtual Machine (VM) is a software based abstraction of a physical machine. This abstraction makes it possible to manage and control access to the actual hardware. Hardware, that does not physically exist, can be emulated.

Here is how I recall the explosion in use of Virtual Machines…

When I first became a professional software developer I worked for IBM. The first VM I used for development was VM/CMS. It was a remote, centrally managed VM, and a sensible abstraction away from big iron mainframes with big consequences if something went wrong.

It was not too long until the PC revolution changed the way applications were used, written and delivered. The 3270 terminal (or terminal emulator) was largely replaced with the web browser and Client-Server applications. It was around that time that I displayed a picture on my office wall of a proud man standing in a giant warehouse stacked with PC servers. His arms were spread so one could get a feel of scale; a tiny man standing in such a giant warehouse with an incalculable number of servers. The picture was from a book where the man described his successful journey to replace a mainframe with PC servers. This was IBM, so I am sure the message I should have gained from the book was something like “Wow look at all the hardware that people would buy, and all the services they would need, to support their desire to join this PC revolution!”

There is something to be said for the decentralization of application development accelerating innovation, but what I saw in the picture was “Maintenance nightmare!”

If I recall correctly, the move from centrally managed mainframes to cheap distributed PC infrastructure, this PC revolution, and its requisite services, were quite profitable for IBM. There must be some irony there.

It was a while later before PC architecture servers, with enough surplus power to support virtualization, became cost effective. Until that time the big boys (think IBM, HP, Sun, and many others) continued to offer expensive, but powerful machines with hardware based virtualization (type 1 hypervisor), while the PC guys began to learn the true costs of maintaining so many physical machines.

What do you do if you want the benefits of virtualization but your hardware (architecture) does not meet the Popek and Goldberg virtualization requirements? Well, you write a software hypervisor (type 2 hypervisor) like VMWare, and change the world. This kind of VM software made it possible to emulate a number of PC servers on a larger host, with room to scale up as needs grew. Surplus power could then be traded for better maintainability, reliability, predictability and separation.

All that brings us to today where VMs (Virtual Machines) are used throughout the lifecycle of application development and deployment, especially for web applications where PC hardware is still largely ubiquitous.

There are a variety of good VM solutions, but for the sake of simplicity I will focus the rest of this article on one, VirtualBox. VirtualBox is currently free and can be found here.

What can it do for me?

OK, that is all well and good for system administrators and infrastructure planners, but what can it do for me as a developer?

Reduce maintenance time

Changing a network card or adding RAM to a virtual machine is as simple as changing a setting. This is true whether the host is a remotely maintained server, or your laptop. Virtual Machines are portable, so if the physical hardware is acting up, the VM can be moved to another host until the original is repaired or replaced. One could even continue to work on a clone while software is being upgraded on an original VM. Need a development server and all existing servers are at capacity? No problem, simply fire up a new VM.

It is quicker and easier to update a VM in concert with a closely matching production machine, than it is to update a number of physical test machines, developer laptops, workstations, etc. This is not simply a matter of scale, updating a single image rather than a number of physical boxes, but also a matter of compatibility. The compatibility becomes important if it permits one to apply the same changes to production and test VMs with tools such as Puppet, with predictable results.

Reduced maintenance and provisioning time result in increases of developer productivity, even if the developer never deals with the maintenance or provisioning him or her self.

Improve testing

There are characteristics of VMs that are useful for testing. VM images are software, so they only need to consume resources when they are needed, unlike their physical counterparts. This makes it cheap to keep many test images. Because VMs are portable and can be copied and cloned, it is simple to make them available to whomever needs them, even to many people simultaneously.

Undo changes with roll-back

VMs make it quick and safe to test dramatic environmental changes. This is because changes can be completely undone with a simple roll-back.

I recall a third party software upgrade that resulted in the upgraded server becoming completely unusable. Performing an un-install would have still left the server damaged. A roll-back restored the server to the state exactly as it was prior to the upgrade. I can not imagine the nightmare had this server not been a VM.

Better match

Testing is more efficient when all factors, other than those being tested, are consistent and thus eliminated from consideration. This way when differences are observed they can be expected to be the result of the factors being tested. Nobody wants to experience a problem in production that could not be found in test, but when these environments are different, and the differences are not the factors being tested, such problems can happen.

It is possible for one to create a VM that more closely matches production (also likely a VM) than it would be to configure one’s laptop, workstation, or even a physical test machine to do the same. This is in part because a laptop or workstation is designed to meet different needs than for example, a dedicated web server.

An example might be that one’s development workstation runs archlinux and one’s hosting provider, or a company production environment, uses Centos 6. To address the discrepancy, one could create a Centos 6 base VM that closely matches production. In some cases such a VM can be derived from the actual VM used in production.

At this point I should mention that many developers use an IDE (Integrated Development Environment) both for writing code and for local testing. I avoid IDEs. To me, they divert one’s attention from learning the language and development tools, to learning the IDE. I find that I get better results by writing a script, an editor macro or by re-examining what I am doing, than I do from using an IDE shortcut.

I worked with a developer who experienced a really frustrating and intractable code problem. We found that the problem was not with his code but was simply a flaw in the way his IDE executed it. The problem did not, and could not, exist anywhere else.

I bring this up because I find that running code, especially tests, in an IDE exacerbates the problems of run-time differences between a local development and production environment. If you are hell-bent on using an IDE, I suspect you could configure it to use a local VM for testing, etc. After all, development tools should conform to the needs of the developer, not the other way around.

Make better use of your local machine

Many developers are familiar using VMs as part of a remote, company development server pool, but few use VMs locally. VM solutions work well on local machines too. Independent contractors have been taking advantage of virtual machines for years. Visit a new client and create a fresh VM. That VM can then be customized to the client and saved for the next time the client calls. Meanwhile, the host machine is kept safe from constant alteration. This also works well when one encounters a client running a different OS. Have a Mac or Linux laptop and the customer runs Windows or FreeBSD? No problem, just use a VM of whatever is needed.

The second article in this series will focus on how to use machine virtualization as a development tool.