Will's Blog

Setting up a VLAN

noreply@blogger.com (Will Winder) — Wed, 22 Jul 2026 14:33:13 +0000

What follows is documentation for my home network VLAN configuration. An LLM was used to help with terminology. My focus is generally focused on the intent and sometimes that is misaligned with the common technical terms used to communicate some particular implementation. The machine is great at making that type of translation.

Why do you want a VLAN?

Unless you're well versed with layer 2 networking, setting up a VLAN network is quite a tedious, so there had better be a good reason to set this thing up. I have a few reasons:

Security: I'm interested in setting up various cheap IOT devices for home automation, but I don't trust them. Putting them on a separate VLAN minimizes how much damage can be done by a rogue device.

Priority: I also have IP Cameras generating a lot of traffic. I'd like to see how much traffic and have a way to lower its priority.

More security: Any employer-managed device goes on a completely isolated network.

Fun: What can I say, I've always wanted to set this up. I worked at a networking lab for years and know how this stuff works in theory and am long overdue to see how it works in practice.

Internet

Beginning from the Internet Service Provider (ISP), my public internet is delivered as light through a fiber optic cable. In order to avoid any potential issues with my ISPs troubleshooting flowchart, this entry point is completely standard and uses the provided hardware with minimal configuration: the provided Optical Network Terminal (ONT) to convert light over fiber signals to electricity over copper Ethernet wire, and the provided router with minimal configuration.

ISP Router

This is not a sophisticated router. It provides a Wi-Fi Access Point (AP) and a firewall with some basic security features. I simply disabled Wi-Fi and enabled the firewall to block incoming connections. This simplicity is very intentional. When my ISP goes down I simply patch my network out of the equation and let them fix the problem on their end.

OPNsense Firewall

Some background before jumping into this router's configuration. Since 2006 I've been running some variant of DD-WRT. In 2006 it was probably the original DD-WRT, later on I used Tomato, and most recently I was running Merlin on an Asus RT-AC68U. Originally I wanted to use these for a bridged routing feature that was not available on consumer routers at the time, later on I appreciated extra monitoring and plugin features. Custom firmware on these devices is not for the faint of heart: there is no support and every update has significant risk of bricking your router.

In early 2025, mainly for performance reasons, I switched to a mini PC running OPNsense. Finally, during my winter vacation in 2025-2026, I configured VLANs.

Configuring the VLAN

Finally, the good stuff! Setting up a VLAN is a careful balance of designing the logical topology, mapping that design to the physical hardware and cabling, and configuring all of the components correctly.

Logical Topology

Like many VLANs, my goal is to partition traffic into broad categories:

USER (192.168.10.x) - trusted devices.
INFRA (192.168.20.x) - restricted access for network devices.
IOT (192.168.30.x) - locked down for untrusted IoT devices.
CAMERA (192.168.40.x) - locked down for untrusted IP Cameras and QoS.
WORK (192.168.50.x) - locked down for company managed devices.

Physical Constraints

My network has networking "closets" connected in series. The main rack is located in the basement, everything that can go here is here. A single Ethernet cable runs to a central rack on the second floor connected to a managed switch. Finally, this managed switch connects to another manage switch in the annex rack.

The central and annex racks are configured with multiple VLANs, which means I need to use a trunking protocol. It also means I need to carefully configure each port to manage Tagged and Untagged traffic.

Hardware Availability

The last piece of the equation before getting into the physical layout and configuration is the available hardware. Each rack has a Managed Switch in order to configure traffic running across the trunk cables.

Notes on configuration order

One extremely tedious part of this project is doing things in the right order. To configure managed switches, you need to connect to them, but once they're configured you need to use a specific port and/or a specific VLAN. This makes it incredibly easy to accidentally lose access to the configuration page. My suggestion: get familiar with the factory reset feature on your switch.

This goes for OPNsense as well, but because it's also a mini-PC you always have the monitor+keyboard fallback for settings recovery.

Hardware Configuration Overview

In a VLAN network, every packet needs to be tagged. This can be done in many ways with varying levels of user convenience. The best way is to assign tags with a managed switch. At a high level, the switch will accept all incoming traffic on that port and insert a VLAN ID for all untagged traffic. Any traffic going in the other direction uses IP routing rules as usual (citation needed).

With that in mind, here is my network tree and how I've configured each port on each device. Each bullet is an Ethernet cable connected between two devices. Inside the square brackets is how the port is configured on the switch side:

main rack:

ONT/ISP router

OPNsense firewall

[USER] server
managed trunk switch

[INFRA] AP (Access VLAN: 10)
[WORK] keystone
[USER] dumb switch

keystone
TV

Central trunk

central rack:

Central Trunk

Managed PoE switch

[IOT] Ziggbee Radio
[CAMERA] Dumb PoE switch

Camera 1
Camera 2

[INFRA] AP (Access VLAN: 10)
[INFRA] Annex Trunk

annex rack (not fully configured):

[INFRA] Annex Trunk

AP (Access VLAN: 10)

Software Configuration

This is the part I should have written down in more detail 6 months ago! All of these configurations are made inside OPNsense. To the best of my ability I'll describe the end-state configurations.

Interfaces > Assignments

This is where you configure the physical interface. For my configuration, I created entries for WAN, TRUNK and Legacy. All VLAN traffic interfaces will be configured to use TRUNK.

Interfaces > Devices > VLAN

Create an entry for each of your VLANs. My configuration includes TRUNK_USER, TRUNK_INFRA, TRUNK_IOT, etc.

Services > ISC DHCPv4 [legacy]

I don't recall seeing the legacy tag before, looks like I need to update my configuration to Kea DHCP. My original configuration probably used this in early 2025.

For either approach: enable a DHCP server for all VLANs. OPNsense will route traffic according to the VLAN tags and provide addresses.

I took this one step further and configured static addresses by MAC address in the DHCP configurations. This was very convenient for the managed switches after factory resets.

Firewall

Last but not least, you can finally configure the firewall! At this stage, you may treat each VLAN interface as if it were any other interface. OPNsense gives you many advanced features to work with, which ones you use depends entirely on your goals, so I recommend reading the docs.

Misc Details & Optimizations

Cascading Firewalls: the ISP + OPNsense firewall configuration means that if you do want to allow traffic on some port, you need to configure it twice. Once in each router.

NAT Reflection: You might have some services running with a reverse proxy inside your network. This means your local service hosted at https://myservice.myproxy.com will leave your network before coming right back. Search for "nat reflection" to learn more, I opted for the settings at Services > Unbound DNS > Overrides.

Broadcast Discovery: there are an endless array of consumer devices (smart speakers, home assistants, tv streaming protocols) that use broadcast protocols for device discovery. This is a big problem if you put all of your devices on a separate VLAN from your user devices because those packets will never reach their intended recipient. There are ways to get them working (look up mDNS), but it's an endless game of cat and mouse. Take a long look in the mirror and decide if your principles are worth the hassle.

Custom cables: You need a lot of short cables to avoid creating a big mess in your networking closet. If you aren't already making your own cables, I'd suggest buying a few hundred feet of cat6, a bag of RJ45s and a crimping tool.

Next Steps

For my network, the next steps are mostly cosmetic. I'd like to get my switches and servers securely mounted to something. Probably a simple DIN rail with some 3D printed specialty mounts.

Conclusion

This was all setup about 6 months ago, since then I haven't done anything other than updating OPNsense. Overall I couldn't be happier. Everything works as it should, my network is safe, and it's still convenient enough to use that you wouldn't know it's there.

A Cautionary Tale of AI and VLANs

noreply@blogger.com (Will Winder) — Thu, 08 Jan 2026 15:44:00 +0000

This turned out to be a cautionary tale. At the beginning I was hoping to find an AI success story that demonstrated the power of how it lets you wade into the deep end of a domain and find success. It turned into a grueling week of troubleshooting rabbit holes that the AI insisted I get to the bottom of. It's a powerful thing, but it takes a mindset that I'm still trying to figure out.

I have what you might call a home lab and was thinking about adding some smart home devices. After a bit of yak shaving it became clear that I needed to setup a VLAN in order to partition the traffic. While I was at it I'd setup networks for IP cameras and my work computer. Last year I switched over to a proper firewall running OPNsense, so I figured all this would be a nice exercise to get practical experience with things I was generally already familiar with.

Shortly after firing up ChatGPT I had a high level plan and a better understanding of how to design a network. With a couple managed switches ordered, I got right to work. The AI advice sounded correct and was matched general understanding of Networking and how VLANs should be configured. You probably know this already, but the AI is very reassuring. Every step of the way it let me know how clever I was for asking the right questions. I especially appreciated the way it insisted on designing the architecture ahead of time. It gave me a lot of confidence that I was going to have a solid foundation to my network that would evolve elegantly over time. Unfortunately, the hallucinations started almost immediately.

The first problems were fairly innocuous. The advice regularly referred to OPNsense settings that don't exist. These were easy to ignore and I explained them away by saying the information could based on a different version. My human intelligence along with the actual documentation easily filled in the gaps and before I knew it, DHCP was configured and seemed to work. I plugged in a device and it got the correct IP!

Things went downhill from here as the first major issue presented itself. For some reason my device couldn't ping the firewall and the firewall couldn't ping my device. This was a bit puzzling because the link was up and the IP address was assigned in the correct range. Clearly they could communicate at some level. The AI was happy to guide me through the issue and was quite confident that this was definitely a Layer 2 problem in the managed switch. After running through a conveniently generated troubleshooting checklist, the results remained conclusive: what I was looking at was clearly a Layer 3 issue in my firewall.

At this point I was starting to realize that the usefulness of the AI was beginning to approach zero. The prescribed root cause continued to waffle between the Network and Data Link layers. Ultimately, I went off script. The AI had previously recommended I put my switch into "Basic VLAN" mode. With nothing to lose I ventured into "Advanced VLAN" mode. It presented me with a new setting called the PVID table. With some trial and error, I was able to get ICMP and DHCP to work. The AI had previously told me that "Advanced VLAN" mode was notoriously buggy... when pressed on this issue it wasn't able to back up the claim. Go figure.

With that I had my first "access" port configured, which is just the beginning of the story. I had multiple trunk ports to configure, access points and IP cameras to migrate, and a house full of people asking if I was doing something to the internet. To make a long story short the process continued for several days. By the end of it I had a pretty good idea about what I was doing and how things needed to work.

Overall, the experience left me extremely nervous about how easy it is for the AI to write code for us. None of the issues I hit were particularly sophisticated, but networks are fairly binary things. They work or they don't, and that property makes small bugs pretty catastrophic and easy to spot.

So what are the takeaways, I certainly learned a few things:

AI probably allowed me to configure this network faster than I could have done it on my own. The high level advice was sound and it helped me wire my brain to think about the problem in the right way.
The hallucinations were present at every step of the way, regardless of whether or not I was able to identify them.
AI is a powerful and dangerous tool that should be used cautiously.
You can't trust anything the AI tells you.
OPNsense is a nice piece of software.

Terminal UI the Easy Way: BubbleTEA

noreply@blogger.com (Will Winder) — Wed, 17 Jan 2024 22:11:00 +0000

For the past 18 months I've been tinkering with a terminal application that serves as an Algorand node frontend. I had a number of requirements in mind when choosing the technology: a full screen terminal application, works over ssh, extensible to allow it to evolve with different utilities and views of the system over time, be programmed with the Go programming language.

In the past I've created these sorts of programs with bash, but it's not fun to manage terminal control characters and deal with re-drawing with bash. Using bash has always led my applications to be simplistic, typically never getting beyond simple spinners and progress bars. I knew about ncurses, but didn't want to lose the portability of bash -- some of these scripts were even POSIX compliant, and honestly I just didn't want to deal with low level interfaces and C bindings.

Enter bubbletea. The team at Charm has built the tools I didn't even know I needed. There is an impressive array of documentation, example programs, and supporting tools. The "getting started" story is quite nice, there are even YouTube videos if that's your thing. Overall, my experience with these tools has been transformative and I'm enabled to build advanced terminal applications. The sort that I knew were possible, but were previously too daunting to consider.

One of my favorite things about bubbletea is that it's only one part of a larger ecosystem of libraries and tools. You can use wish to serve the UI over ssh, or glamour to render markdown files in your app, and lipgloss is great for formatting text. Then there are other tools that are generally useful on the terminal, like vhs to record demos and wishlist to manage ssh targets. I have yet to get into charm, which seems to be their flagship project.

Gotchas

This isn't to say everything is perfect. These tools can be used to quickly build a simple program, but there are no training wheels. Once you stray off the established path things quickly get complicated. For example:

managing multiple views, and their sizes.
themes across multiple components.
customizing an existing component.
input wizards.
dialogs.

This can all be done, but the framework isn't designed to handle them. At the core of bubbletea is an event handler and not much more. The examples may lull you into a false sense of security here, because it kind of looks like an application framework. Don't be fooled.

Some examples.

If you have two views that should be displayed side by side and a third that should span the entire width of the terminal, you need to explicitly set all of their sizes. While this sounds pretty obvious, the event system really isn't conducive for it. Each view is self managed, so there's no easy way to clamp them to 50% of the width, that's something you need to build yourself. Not rocket science, but not provided by the event system.

Themes are entirely managed by the View functions. How they are passed throughout the system is up to you. This becomes especially complicated when using off-the-shelf components, where you need to merge your themes with (if you're lucky) the theming capabilities of the different libraries. In some examples a global "style" object is passed around. But what to put in that global object... and how to use it. That is for you to decide. If you don't make a decision, you're gonna have a bad time (or a funky looking app).

Until recently, a table wasn't available out of the box, which was a surprising omission. Thankfully there are several 3rd party alternatives. But every bubble library has a slightly different take on how it should be used. Sometimes this leads to some quirky issues. The list bubble is designed to be used as a single page application, and does things like handle the escape key to quit the program. In one of my programs I had a list as part of a wizard and wanted escape to go back. Fortunately there is a "DisableQuitKeybindings" option, but figuring that out involved a frustrating debugging session.

Takeaway

These are not problems, they're a misunderstanding. I thought I was using a framework but that's not the case. Bubbletea is an event system with some very convenient terminal events built in. There are a number of thoughtfully designed supporting libraries that turn it into a TUI toolkit. It's a smart decision that fits well with the go ethos.
The light touch used throughout also leaves the door open for some excellent community extensions like bubblezone and many others. You should use these libraries, and your own, when building applications of any size.
The developer is responsible for knowing the exact size of each view. Take a look at how vertical and horizontal margins are passed in library code and figure out how you want to something similar in your program. My most complex program ended up having global variables that keep track of footer and header sizes to deal with the fact that I didn't have margins plumbed into every view. It's a glaring design flaw that leads to weird display issues when I modify the program. Margins should be carefully handed down to better fit with the Update function. There's room in the ecosystem for a layout-aware WindowSize event library.

# What's Next

I build that layout-aware WindowSize library and will talk about that in a future blog post.

For now take a look at the README and enjoy building terminal applications!

Decommissioning the cloud: A self-hosted future

noreply@blogger.com (Will Winder) — Sat, 11 Jun 2022 17:13:00 +0000

Cloud technology has slowly crept its way into nearly every aspect of our digital lives. And why not? It's convenient and easy to use, nothing to install or configure, and it's often free. Email clients are a great example of this. People used to live out of Microsoft Outlook or Lotus Notes, today all you need is a web browser.

This starts becoming a problem when you consider your privacy. By using these services, especially the ones that don't cost money, the price is your private information. Consider your family photos and facial recognition, are you comfortable with Google knowing exactly who you associate with? What about your private documents? Even passwords are moving to the cloud with services like LastPass and 1Password!

Another consideration is whether the service will exist in the future. If you keep notes on the cloud, how upset would you be if that service was shutdown? If you don't run it yourself you're beholden to the company providing the service.

Whether or not you're happy with all of this is a personal decision. What we're talking about is how to balance convenience against privacy and ownership. For my preferences, the pendulum has swung a bit farther than I want it to towards the cloud.

The best of both worlds.

How do you get the convenience of cloud-like services and the privacy and ownership of local storage? This, of course, is the self-hosted future. The cost of that self-hosted future is time and money. Time to install and configure different pieces of software, and money for the hardware that runs it.

The hardware side is easier today than it has ever been. Off the shelf hardware products like Synology, QNAP and Asustor are plug and play. Commodity hardware can also be configured with commercial software (Unraid) or open source software (TrueNAS, Rockstor). Any of these can be setup and configured with limited technical knowledge. This is not an exhaustive list, just an example to show that there are more options today than there have ever been in the past.

I'll give a few examples of cloud services which can be self-hosted.

Data management

After setting up a NAS, this is generally provided automatically. Enable one of the many filesharing options like FTP, Samba, WebDAV, or NFS. For the less techie "cloud" experience, you can use products like NextCloud, ownCloud or Seafile. Any one of these many options let you own your data.

Password management

This is perhaps the best example of when you might want to own your data. Every cloud based password manager is going to go into a lot of detail explaining their security model, and for obvious reasons. But do you trust them? Even if the answer is yes, do you want to place your trust a business?

One company leads the pack for self-hosted password management: BitWarden. A user of this self hosted software would feel right at home coming from any of the other password managers. It has mobile integration and browser integrations that match the competition.

Personal notes

This can be as easy or advanced as you want them. Simple files on your disk can get you pretty far, or even simple programs like Tomboy can manage your work notes. But that really doesn't compare to a cross-device cloud native note program like Evernote, Google Keep, or my favorite of the lot Apple Notes.

Today there are multiple self-hosted alternatives to these which give you cross-device access to your notes. One of the most popular open source options which provides mobile and desktop applications is Joplin. For wiki style notes, there are many other options available as well.

Other stuff

The list can go on and on as you start to decommission the cloud. Services like Spotify and Netflix can be replaced with Plex to serve your DVD and MP3 collections. Social media platforms can be replaced Mastadon. And much much more.

The penalty

It is costly to self host for many reasons. Not only do you have to buy the hardware, but you may end up investing many hours into configuring and deploying your services. Once deployed you need to consider how to backup the data.

The most important thought I would leave you with is this: you have the option of owning your data and privacy.

Where to get more information

/r/homelab is a reddit community with people building home servers.

/r/selfhosted is a reddit community all about software to run on your server.

awesome-selfhosted is a long list of software that you can host yourself.

Using a JFileChooser to browse AWS S3

noreply@blogger.com (Will Winder) — Fri, 03 Jan 2020 15:29:00 +0000

After a bit of searching I was unable to find an off the shelf solution to let me browse S3 with a JFileChooser. The closest I found was an S3 FileSystem implementation, but that doesn't seem to be used by JFileChooser. Instead you need a custom FileSystemView, which I implemented using minio as a simple client library.

The code is not incredibly complicated so I wont explain it in detail. You can see that the custom S3FileSystemView is simply passed to JFileChooser to insert the S3 logic. The most interesting method is getFiles, that is where I decided to insert the buckets at the top level, so there is one switch which decides whether to call listBuckets or listObjects. From that point my custom VirtualFile objects are carefully constructed so that the File class plays nicely with JFileChooser.

The result from selecting a file is something like s3:/bucket/path/to/file.txt which you will need to download yourself. VirtualFile could probably be extended to work like a normal file, but that was beyond my needs.

Novice to Coin Part 2 - Transactions

noreply@blogger.com (Will Winder) — Tue, 24 Jul 2018 01:08:00 +0000

We left off with an immutable data structure storing arbitrary data, you can read about that in part one, or you can view the introduction to this series here. As always, the companion code for this post can be found on GitHub.

In this post, we'll introduce transactions which are the primary feature needed to convert that data into a currency. Although a transaction simply needs to subtract coins from one account and add them to another, the way this is implemented in a blockchain is a little unusual. We'll cover all these details and more during this article before implementing them in the LynxCoin project.

This is what we'll be working towards today.

The first thing you need to understand about currencies stored on a blockchain is that there is no concept of a balance. Instead, you hold rights to transactions. Specifically, if you received a transaction for 50 LynxCoins, you now have the right to spend them.

Our updated block looks like this.

A transaction may have one or more inputs and one or more outputs. During the transaction all coins from each of the inputs will be spent, they are divided up amongst the outputs. Each output can be given to a different individual, who may then use it as an input in one of their future transactions. Let's take a look at a few examples.

Example Transactions

Suppose that Alice was given an output in Transaction-A and received 50 LynxCoins. If she wanted to give those coins to Bob, she would use her output in Transaction-A as an input while creating a new transaction. Bob would then be given an output to receive coins from that transaction.

As long as you want to give away the entire output this works fine, but what if Alice only wanted to give 10 of those coins to Bob? There is actually no way to give away part of a transaction, all of the coins from Transaction-A Output 1 will be spent following the alternative Transaction-B regardless of whether she gives Bob 10 coins or 50. The way we deal with this is the idea of "change". Since Bob should only receive 10 coins, Alice will name herself as a recipient for the remaining 40 coins. At this point, Alice's coins in Transaction-A are considered spent. If Alice needed to give Bob another 15 coins, she could now reference her output from Transaction-B while creating Transaction-C

In one final example, let's suppose Bob wanted to send 20 coins to Carol. He has 10 coins from the Transaction-B and 15 coins in Transaction-C, so he needs to use both of those in order to give Carol all 20 coins. So when making Transaction-D, Bob specifies Transaction-B and Transaction-C as the inputs, for a total of 25. He then names Carol as a recipient for 20 of those coins and himself as a recipient of the change, which is 5 coins.

Where Coins Come From

The previous examples assume that the coins were already somehow in the system but we haven't talked about where they come from originally. With the basic knowledge of how a transaction works, we can talk about a special transaction referred to as a Coinbase which is used to add coins to the system.

As we saw in the first diagram, a new block can contain more than one transaction. For most blocks, there will indeed be multiple transactions and the first of these is special. It has no inputs, instead, there is a Reward built into the system which acts as a default input. Therefore only outputs should be defined to indicate where that reward should go. In this example, we'll give it to Alice. The Coinbase transaction provides an incentive for people to create new blocks on the blockchain. It also prevents giving the power to "print" money to a central authority, in contrast, it actually acts as a way to distributes new coins across many different users which further reduces the influence that individual users have over the system. We'll talk later about why an incentive is needed for creating new blocks.

These properties are very important to a category of research called Cryptoeconomics, which studies the economics of a distributed currency like LynxCoin, Bitcoin, and Ethereum. There are many feedback mechanisms like the Coinbase reward, and they are designed to encourage people for doing the right thing and punish them for doing the wrong thing. Details of Cryptoeconomics are beyond the scope of this series (For now).

Signing Transactions

So we have a way to transfer coins between parties, the transactions are immutable thanks to the blockchain, and there is an audit trail that anyone with the blockchain can validate. These are all critical properties of our currency, but how do we ensure that the person named in an Output is the only one who can spend it? It is finally time to introduce some cryptography.

We need a way to verify that data came from Alice, Bob, or Carol in a way that nobody else could spoof. Fortunately, we can leverage Public-Key Cryptography to do just this. Like Cryptoeconomics, many details about the inner workings of Public-Key Cryptography will be left outside the scope of this series (For now). At a high level, algorithms like RSA or ECDSA work by generating Public and Private keys. Like their names suggest, the Public key is used to unlock data generated using the Private key. The locking mechanism works in two ways.

Encryption and Decryption

Suppose that Alice would like to send a message which can only Bob can read. By using Bob's Public key, she can encrypt the data. That encrypted data can then be sent over an unsecured network, now even though Eve can see the message only Bob has the Private key needed to decrypt the data:

Signing and Verifying

Encryption isn't useful for our Transactions, because rather than preventing others from seeing our message we need a way for everyone to verify that we created the message. To accomplish this we'll use a Sign and Verify routine with our Public and Private keys. The steps are similar, but by Signing our message with the Private key, all users of the system can read those messages using the Public key. Alice's Public key is the only way to unlock messages signed by her Private key, and because only Alice has access to her Private we can verify that Alice sent the message:

Signing Transactions

Using this tool, let's examine how we can secure the input to one of Alice's transactions. We now require that Alice has a Public key, available to everyone who uses the blockchain, and a secret Private key used to Sign.

To start, rather than assign an output to Alice, we'll assign it to her Public key. In fact, this key is used in the blockchain as the destination when Alice receives a transaction. This can be considered Alice's Wallet. We'll use her Private key to sign inputs, this signature is added as a new property to the input. While we're at it, let's also introduce a unique-id to identify input/output pairs. With this input/output ID we no longer need to store the value in both places, so we will also remove the value from the input. A simple transaction now looks like this:

Let's take a closer look at that signature. As we showed earlier we can sign a message, in this diagram we call that operation sig and indicate an output to be signed. The message is only half of the equation though, we also need the Private key. Looking at the input-output pair with id=2 we see that it is assigned to Alice's Public key, so the signature must be made with her Private key. Creating an input which uses output-2 now requires the following steps:

Lookup output 2.
Serialize output 2 into a message.
Sign the message using Alice's Private key.

Any user of our currency can now verify the input:

Lookup the input and output with id=2.
Serialize output 2 into a message.
Use Alice's Public key from the input to read the signature.
Verify that the serialized message from step 2 equals the signature from step 3.

Anonymity

You may have noticed that the final form of our transactions doesn't have any personally identifying information. The only thing linking a given transaction to an individual is the Public key. As it turns out, it is trivial to make new keys, so many Cryptocurrency users opt to generate a new pair of keys for every transaction. This way if someone were able to link some Public key to the user, they would only be able to see one of their transactions. It is a very curious property that all transactions are public, yet we still only have part of the story regarding who is making these transactions.

The concept of Zero Knowledge Proofs may change this one day, but I'm not currently planning to implementing that in LynxCoin.

Transaction rules

In addition to everything already talked about, there are a number of more obvious rules that our implementation must take into consideration:

Double Spending: An output may only be spent once.
Insufficient Inputs: The sum of inputs must be greater than or equal to the sum of outputs.
Unique Transaction IDs: There must be no duplicate Transaction IDs.

LynxCoin Implementation

Continuing with our implementation of LynxCoin, there are four important pieces to add: Transaction objects, Encryption utilities, Transaction creation, and Block validation. Unlike our blockchain implementation which was only a small handful of code which could be quickly verified by reading and running the code, there are many new edge cases so testing is no longer optional. So we'll introduce unit testing.

Transaction Object

Kotlin makes this easy, we'll define a few data classes. Note that the full implementation includes serialize methods, but they are omitted here for clarity.

data class Input(
    val id: Long,
    val signature: String)
data class Output(
    val id: Long,
    val value: Double,
    val destination: PublicKey)
data class Transaction(
    val inputs: List<Input>,
    val outputs: List<Output>
)

Integrating these into LynxBlock is a simple matter:

 data class LynxBlock(
     val lastHash: String,
-    val data: Any
+    val data: List<Transaction>
 )

Encryption utilities

Java includes robust implementations for encrypt/decrypt and sign/verify routines. The details are a bit cryptic, so we make some simple helper methods to perform the operations according to our needs:

fun sign(text: String, privateKey: PrivateKey): String {
  val privateSignature = Signature.getInstance("SHA256withRSA")
  privateSignature.initSign(privateKey)
  privateSignature.update(text.toByteArray(UTF_8))

  val signature = privateSignature.sign()

  return Base64.getEncoder().encodeToString(signature)
}

fun verify(text: String, signature: String, publicKey: PublicKey): Boolean {
  val publicSignature = Signature.getInstance("SHA256withRSA")
  publicSignature.initVerify(publicKey)
  publicSignature.update(text.toByteArray(UTF_8))

  val signatureBytes = Base64.getDecoder().decode(signature)

  return publicSignature.verify(signatureBytes)
}

If you look closely you'll notice that the implementation includes a SHA256 hash. It is a common convention to first take a hash of the message and compare hashes rather than fully encrypt/decrypt the message. Mainly because it is much faster to do this than to sign and verify a large message.

Transaction Creation

Before we were putting arbitrary data into the blocks, things are a bit more complicated now to deal with our transactions. Here is what the genesis block looks like, it includes a specification for the new Coinbase transaction:

-  val chain = LynxChain(LynxBlock("0", "Genesis block"))
+  val key = generateKeyPair()
+  val genesisBlock = LynxBlock("0",
+      listOf(
+          // Coinbase transaction, no inputs.
+          Transaction(
+              listOf(),
+              listOf(Output(0, REWARD, key.public)))))
+  val chain = LynxChain(genesisBlock)

Here is a larger demonstration of multiple transactions, they now include a signature:

  // Create some keys for the demo.
  val key1 = generateKeyPair()
  val key2 = generateKeyPair()
  val key3 = generateKeyPair()

  // Genesis block with coinbase reward for key1.
  val genesisBlock = LynxBlock("0",
      listOf(
          // Coinbase transaction
          Transaction(
              // No inputs.
              listOf(),
              // One output.
              listOf(Output(0, REWARD, key1.public)))))
  
  // Create chain with genesis block.
  val chain = LynxChain(genesisBlock)

  val transaction1 = LynxBlock(chain.last.getHash(),
      listOf(
          // Coinbase transaction.
          Transaction(listOf(), listOf(Output(1, REWARD, key2.public))),
          // Send the initial reward to the owner of key2 and key3.
          Transaction(
              // Sign output using key1's private key.
              listOf(Input(0, chain.getSignature(0, key1.private))),
              // Create outputs for key2 and key3.
              listOf(
                  Output(chain.lastId + 2, 50.0, key2.public),
                  Output(chain.lastId + 3, 50.0, key3.public)
              ))))
  chain.add(transaction1)

  val transaction2 = LynxBlock(chain.last.getHash(),
      listOf(
          // Coinbase transaction.
          Transaction(listOf(), listOf(Output(chain.lastId + 1, REWARD, key2.public))),
          // Multiple outputs
          Transaction(
              // Use coinbase reward from transaction 1.
              listOf(Input(1, chain.getSignature(1, key2.private))),
              // Split up output into multiple pieces to show that we can.
              listOf(
                  Output(chain.lastId + 2, 20.0, key3.public),
                  Output(chain.lastId + 3, 20.0, key3.public),
                  Output(chain.lastId + 4, 20.0, key3.public),
                  Output(chain.lastId + 5, 20.0, key3.public),
                  Output(chain.lastId + 6, 20.0, key3.public)
              ))))
  chain.add(transaction2)

  chain.print()

After reading through the example you'll notice a couple helper methods: getSignature and lastId. These help pull metadata out of the blockchain. lastId takes the last block and iterates over the outputs, then returns the largest. If there is no last block it simply returns -1. getSignature uses our encryption utilities to lookup a specific output, and sign it with the provided private key. Note that although the private key is being passed into a helper method, we only store the signature in the transaction.

Here is the code listing for these methods, it uses the Kotlin stream interface to work with data, see inline comments for operation details:

  /**
   * Return the last ID used for an output, or -1 if there are none.
   */
  val lastId get() = chain.lastOrNull()
      ?.data               // get the transaction list
      ?.map { it.outputs } // get the output list
      ?.flatten()          // flatten to output stream
      ?.map { it.id }      // get the IDs
      ?.max() ?: -1L       // return the largest ID, or -1.

  /**
   * Create a signature of the Output.
   */
  fun getSignature(id: Long, key: PrivateKey) : String {
    // Find the given output and serialize it to a string.
    return sign(find(id).serialize(), key)
  }

  /**
   * Lookup a specific transaction, throw an exception if missing.
   */
  fun find(id: Long) : Output {
    return unspentOutputCache[id] // Cache to speed up most lookups.
        ?: chain.asSequence()     // Stop processing when a match is found.
            .map { it.data }      // Get transaction list.
            .flatten()            // Get transactions.
            .map { it.outputs }   // Get output list.
            .flatten()            // Get outputs.
            .find { it.id == id } // Look for output with given ID.
        ?: throw NoSuchElementException("Not found: $id")
  }

The result from our transactions looks like the following when we print the chain object:

Size: 3

Block 0
hash: 0cf19c40407e59beb4132284e776f4f8759237ca41de14c8dd1150f89fd129e0
last hash: 0
transactions:
  Transaction: 0 (coinbase)
Disconnected from the target VM, address: '127.0.0.1:54115', transport: 'socket'
  | inputs: 0
  | outputs: 1
  |   tx: 0, value: 100.0, key: 06b80bee761...

Block 1
hash: bfdb9b282322ff4fc630f2c53ae531ead0a4f3353aac2f4f4b751266482591f9
last hash: 0cf19c40407e59beb4132284e776f4f8759237ca41de14c8dd1150f89fd129e0
transactions:
  Transaction: 0 (coinbase)
  | inputs: 0
  | outputs: 1
  |   tx: 1, value: 100.0, key: aa453d1992d...

  Transaction: 1
  | inputs: 1
  |   tx: 0, sig: NQ7IAZhyq9c...
  | outputs: 2
  |   tx: 2, value: 50.0, key: aa453d1992d...
  |   tx: 3, value: 50.0, key: caa0ff040e4...

Block 2
hash: 426ca1b9bf96171dec56d0d0f3c435c810c1145835f96ec1797fd16d1c6ecf30
last hash: bfdb9b282322ff4fc630f2c53ae531ead0a4f3353aac2f4f4b751266482591f9
transactions:
  Transaction: 0 (coinbase)
  | inputs: 0
  | outputs: 1
  |   tx: 4, value: 100.0, key: aa453d1992d...

  Transaction: 1
  | inputs: 1
  |   tx: 1, sig: eVOcNO4ozmV...
  | outputs: 5
  |   tx: 5, value: 20.0, key: caa0ff040e4...
  |   tx: 6, value: 20.0, key: caa0ff040e4...
  |   tx: 7, value: 20.0, key: caa0ff040e4...
  |   tx: 8, value: 20.0, key: caa0ff040e4...
  |   tx: 9, value: 20.0, key: caa0ff040e4...

Block Validation

Although validation is significantly more complex than before, it can be encapsulated as a single method which gets called right before adding a new block. One detail to notice is that the unspentOutputCache which was used by our find method is updated each time a block is added.

  /**
   * Add and validate a new block.
   */
  fun add(block: LynxBlock) {
    val state = validate(block)
    when (state) {
      is Valid -> chain.add(block)
      else      -> throw state
    }

    this.unspentOutputCache = unspentOutputs()
  }

To help organize the code errors are coded into a sealed class, the return type for validate is ValidationState which may either be Valid or one of the error states. This isn't strictly necessary but may come in handy later on as we build out our coin.

// Validation types
sealed class ValidationState: RuntimeException()
object Valid: ValidationState()
data class InvalidLastHash(private val expected: String, private val actual: String) : ValidationState()
data class InsufficientInputValue(val transactions: List<Transaction>) : ValidationState()
data class TransactionDoubleSpend(val transactionIds: List<Long>) : ValidationState()
data class PreviouslySpentTransactions(val transactionIds: List<Long>) : ValidationState()
data class CoinbaseRewardTooLarge(val rewardSize: Double) : ValidationState()
data class InvalidTransactionIds(val transactionIds: List<Long>) : ValidationState()
data class BadSignatures(val inputs: List<Input>) : ValidationState()

It may be interesting to read the following code, but more important is that you understand the different cases being validated. All of the cases have been described above. The code is organized with private methods performing each of the validation algorithms at the top, and a simple check which leverages these methods to generate an error when necessary.

  /**
   * Logic for validating the content of new block.
   */
  fun validate(block: LynxBlock) : ValidationState {
    /**
     * Last hash must equal previous block's hash.
     */
    fun badLastHash() = chain.isNotEmpty() && (chain.last().getHash() != block.lastHash)

    /**
     * Find inputs referencing spent transaction-ids.
     */
    fun previouslySpentTransactions() : List<Input> {
      return block.data
          // Get all the input transactions.
          .map {it.inputs }.flatten()
          // Get inputs which are NOT identified in the unspent cache.
          .filter { !unspentOutputCache.containsKey(it.id) }
    }

    /**
     * Look for duplicate transactionIds in the current transaction list.
     */
    fun doubleSpendsInCurrentTransaction() : Map<Long,Int> {
      return block.data
          // Get all the input transactions.
          .map { it.inputs }.flatten()
          // Group them by transaction ID, count occurrences.
          .groupingBy{ it.id }.eachCount()
          // Filter out the valid cases (spent 0 or 1 times).
          .filter { it.value > 1 }
    }

    /**
     * Make sure the sum of Inputs is >= the sum of Outputs
     */
    fun inputsGreaterThanOutputs() : List<Transaction> {
      // Lookup inputs and tally their values.
      fun getInputsValue(transaction: Transaction) : Double =
          transaction.inputs
              .map { this.unspentOutputCache[it.id]?.value ?: 0.0 }
              .sum()

      return block.data
          // skip coinbase transaction
          .drop(1)
          .filter { getInputsValue(it) < it.value() }
    }

    /**
     * Compute the coinbase transaction size.
     */
    fun coinbaseTransactionSize() : Double {
      return block.data[0].outputs.map { it.value }.sum()
    }

    /**
     * Transaction IDs should all be greater than the largest ID in the last block,
     * and unique in the current transaction list.
     */
    fun invalidTransactionIds() : Map<Long, Int> {
      return block.data
          // Get all the input transactions.
          .map { it.outputs }.flatten()
          // Group them by transaction ID, count occurrences.
          .groupingBy{ it.id }.eachCount()
          // Look for duplicate transaction IDs.
          .filter { it.value > 1 || it.key <= lastId }
    }

    /**
     * Verify that the signatures match.
     */
    fun badSignatures() : List<Input> {
      return block.data
          .map { it.inputs }
          .flatten()
          .filter { !verifySignature(it) }
    }

    //
    //
    // Do the checks.
    //
    //

    if (badLastHash()) {
      return InvalidLastHash(chain.last().getHash(), block.lastHash)
    }

    // Inputs must not be spent.
    previouslySpentTransactions().let {
      if (it.isNotEmpty()) {
        return PreviouslySpentTransactions(it.map { it.id }.toList())
      }
    }

    // A transaction must not be used in multiple Inputs.
    doubleSpendsInCurrentTransaction().let {
      if (it.isNotEmpty()) {
        return TransactionDoubleSpend(it.map { it.key }.toList() )
      }
    }

    // Transaction inputs > outputs.
    inputsGreaterThanOutputs().let {
      if (it.isNotEmpty()) {
        return InsufficientInputValue(it)
      }
    }

    // Coinbase reward transaction is not too large.
    coinbaseTransactionSize().let {
      if (it > REWARD) {
        return CoinbaseRewardTooLarge(it)
      }
    }

    // Validate transaction IDs are unique and increase in size.
    invalidTransactionIds().let {
      if (it.isNotEmpty()) {
        return InvalidTransactionIds(it.map { it.key }.toList())
      }
    }

    badSignatures().let {
      if (it.isNotEmpty()) {
        return BadSignatures(it)
      }
    }

    // Verify transaction signatures.
    return Valid
  }

Tests

I won't be covering the tests. They can be explored in the full source listing on GitHub. I happen to be using JUnit5, which uses the @Test annotation preceding each test method. One of the tests also uses @ParameterizedTest which is a new feature for JUnit5 allowing dynamic test inputs.

Next time

Things are really ramping up now. The blockchain concept was simple, and even though the idea of Transactions is pretty simple as well, as we found, there are a lot of small details that need to be just right. We'll continue adding on details next time as we start to look into what it takes to make LynxCoin decentralized. We'll discuss Mining, the Byzantine Generals' Problem and some of the solutions. We'll implement Proof of Work and enable some simple endpoints to allow LynxCoin to go distributed!

Novice to Coin Part 1 - Blockchain Data Structure

noreply@blogger.com (Will Winder) — Wed, 11 Jul 2018 20:34:00 +0000

This is the first post in a series building a cryptocurrency from scratch, the series introduction can be found here.

Every day new developers are learning just how simple blockchain technologies are. One of the first things you may notice is that the collection of algorithms are profoundly clever, but the mechanics of creating a verifiably immutable chain of data is borderline trivial. If you need proof, take a look at how many introductory blog posts are building a toy blockchain. These all include working implementations of a blockchain. One good reason to implement your own is to prove to yourself just how trivial the data structure is. Certainly, my first reaction to reading up on bitcoins block header was: is this all?

Of course, the answer is no. Creating a blockchain is table-stakes, and from there a series of clever algorithms allow transactions to occur without a central authority. But this structure is the heart of all that follows, so building a blockchain is a great place to start.

Like you may have guessed, a blockchain contains a series of blocks. The smallest useful block contains an arbitrary amount of data and a special hash used to verify the block that comes before it. In Diagram 1 you can see that our data structure resembles a linked list, except that the blocks do not point to each other. Rather, they verify each other. In addition, you'll notice that in order to create a new block, the existing blocks do not need to change.

The hash function used here has several important properties. There are two critical properties of a good hash function. First, for any given input to the hash function, the result must always be the same. Meaning if you and I were to hash the string "hello world", we would both get the same hash. The second critical property is that it is extremely difficult to find two inputs to the hashing function which produce the same output. Bitcoin and many other cryptocurrencies have elected to use the SHA256 hashing function which has these properties as well as several other's which we will not cover.

Therefore it would be extremely difficult to modify Block-0 in Diagram-1 in such a way that Hash(Block-0) in Block-1 would stay the same. In addition, if someone wanted to remove Block-1, the chain would no longer be valid because Block-2 contains Hash(Block-1). Because of this, like links in an actual chain, you cannot remove or change one of the blocks. Block-0 is often referred to as the Genesis Block.

Introducing: LynxCoin

It's a chain of stars, too much?

Our toy implementation will be called LynxCoin, named after the Lynx constellation. Let's begin with the block object. As we learned it should contain a previous hash and some data. In Kotlin, this can be achieved with a data class containing two variables and a hash function. Notice that the hash joins all of the data together before passing it into the hashing function, this ensures that all data is taken into account before computing the hash.

private val blockChain = mutableListOf<ToyBlock>()

/**
 * Add and validate a new block.
 */
fun add(block: ToyBlock) {
  if (blockChain.isNotEmpty()) {
    if (blockChain.last().getHash() != block.lastHash) {
      val message = "Invalid block: wrong lastHash."
      throw IllegalArgumentException(message)
    }
  }

  blockChain.add(block)
}

/**
 * Create and add a new block containing some data.
 */
fun add(data: Any) {
  add(ToyBlock(blockChain.last().getHash(), data))
}

Now tie it all together with our main function, and a helper to view the chain:

fun main(args: Array<String>) {
  add(ToyBlock("0", "Genesis block"))
  add("More data")
  add(LocalDateTime.now())
  printChain()
}

/**
 * Print the blocks.
 */
fun printChain() {
  print("Size: ${blockChain.size}")

  var i = 0
  blockChain.forEach {
    println("\n")
    println("Block ${i++}")
    println("hash: ${it.getHash()}")
    println("last hash: ${it.lastHash}")
    println("data: ${it.data}")
  }
}

Running our program produces the following output:

Size: 3

Block 0
hash: FE81582DB2E60D3933F31981CB920EA74189E9574695C0915401E6B7332FB758
last hash: 0
data: Genesis block

Block 1
hash: 7012C5E83A583F02ADCADAF2A7319CAFE45FBFA8649FE53FAFE9AD444D57AA28
last hash: FE81582DB2E60D3933F31981CB920EA74189E9574695C0915401E6B7332FB758
data: More data

Block 2
hash: BDAE7AA36CCFB6431C8788D0F354935A6E5D77987243FB075AF60EC837B14FE1
last hash: 7012C5E83A583F02ADCADAF2A7319CAFE45FBFA8649FE53FAFE9AD444D57AA28
data: 2018-07-08T15:12:13.268

Notice how the hash of each block matches the last hash in the following block, the hash for Block-0 starts with FE815, and Block-1 has the same value for last hash. As we continue we'll add additional data and constraints to these hashes which give us some very interesting results.

The code for this project can be found on GitHub.

Next time...

This concludes our introduction to the blockchain data structure. Next time we'll dive deeper and introduce the concepts of currency and transactions. Once transactions are part of the chain we can also investigate transaction validation. With this, we begin to see the underpinnings that will later allow our currency to be decentralized.

When you're ready, the next article is here.

Novice to Coin Introduction

noreply@blogger.com (Will Winder) — Wed, 11 Jul 2018 20:31:00 +0000

In this series of blog posts, the concepts needed to understand and implement the fundamentals of blockchain and cryptocurrency technologies will be introduced. During each post a new cryptocurrency concept will be introduced, then the technical details will be implemented as a practical demonstration of how they work. The currency will be implemented with Kotlin, which I've found to be expressive while maintaining readability. Future iterations of this project may include alternate implementations in other languages.

This series will be broken into several parts:

Data Structure
Currency and Transactions
Decentralization and Mining
Optimization: Merkle Trees

Data Structure

An introduction to the mechanisms that allow us to verify that data in the chain has not been tampered with. We'll begin with a high-level explanation of how it works and move onto a succinct implementation which will serve as a stepping stone for the following posts.

Currency and Transactions

Once familiar with the basic construction of a blockchain, we'll dive deeper and introduce the concepts of currency and transactions. Once transactions are part of the chain we can also investigate transaction validation. With this, we begin to see the underpinnings that will later allow our currency to be decentralized.

Decentralization and Mining

We'll discuss some of the concepts that allow us to create a trustless peer to peer network by introducing the Byzantine Generals Problem and how it is solved by Proof of Work and Proof of Stake. Our toy implementation goes distributed as we create a small web service to submit blocks.

Optimization: Merkle Trees

Now that we have a decentralized cryptocurrency we can begin looking into various optimizations to make them more usable. One of the most important ideas involves the usage of Merkle Tree's, their use allows us to validate transactions in a block without downloading the entire blockchain.

Building a REST Client with Kotlin and Retrofit2

noreply@blogger.com (Will Winder) — Wed, 18 Apr 2018 02:50:00 +0000

If you regularly work with REST endpoints you're probably used to a lot of tedium. Between carefully constructing the parameters, encoding data in just the right format, efficiently sending the requests, parsing the results and handling errors at every step along the way; there is a lot of room to make mistakes.

For simple APIs, it often makes sense to manually format the request and send it out. Checking a status code, and maybe mapping results to an object aren't that big a deal for a simple API. But what if there are 20 different endpoints? Or you need to manage authorization headers? Eventually, it makes sense to start looking for a wrapper library, or maybe making your own wrapper layer.

That's what this post is about.

For JVM developers, pairing Kotlin, Retrofit2 and Gson make it easy to bang out a type-safe client wrapper.

Let us jump right into the code:

// Create data classes for serializing inputs and outputs.
data class Results<out T>(
    val next: String?,
    val previous: String?,
    val results: List<T>
)

data class CryptoQuote (
    val ask_price: Double,
    val bid_price: Double,
    val mark_price: Double,
    val high_price: Double,
    val low_price: Double,
    val open_price: Double,
    val symbol: String,
    val id: String,
    val volume: Double
)

// Bounds, Intervals, and Spans are enum types.
data class Historical (
    val data_points: List<DataPoint>,
    val bounds: Bounds,
    val interval: Intervals,
    val span: Spans,
    val symbol: String,
    val id: String,
    val open_price: Double?,
    val open_time: Date?,
    val previous_close_price: Double?,
    val previous_close_time: Date?
) {
  // Nested data object used by Historical
  data class DataPoint(
      val begins_at: Date,
      val open_price: Double,
      val close_price: Double,
      val high_price: Double,
      val low_price: Double,
      val volume: Double,
      val session: String,
      val interpolated: Boolean
  )
}

// Define interface using Retrofit annotations and data objects.
interface RobinhoodCryptoQuotesApi {
  /**
   * @param ids comma separated list of ids
   */
  @GET(“marketdata/forex/quotes/”)
  fun getQuoteIds(
    @Query(“ids”) ids: String
  ): Call<Results<CryptoQuote>>

  /**
   * @param symbols comma separated list of symbols
   */
  @GET(“marketdata/forex/quotes/”)
  fun getQuoteSymbols(
    @Query(“symbols”) symbols: String
  ): Call<Results<CryptoQuote>>

  /**
   * @param symbol id or symbol for a pairing.
   */
  @GET(“marketdata/forex/quotes/{symbol}/”)
  fun getQuote(
    @Path(“symbol”) symbol: String
  ): Call<CryptoQuote>

  
  @GET("marketdata/forex/historicals/{id-pair}/")
  fun getHistoricals(
      @Path("id-pair") idPair: String,
      @Query("bounds") bounds: Bounds,
      @Query("spans") spans: Spans,
      @Query("interval") interval: Intervals
  ): Call<Historical>
}

This is a complete listing of the Robinhood market data endpoints. If you aren't familiar with Kotlin, this code is essentially declaring an interface and some POJOs. By sprinkling in some special annotations used by Retrofit2, it can be converted into a concrete object for accessing the endpoints. Errors are even handled by the Call objects which wrap your results.

To go from endpoint definition to concrete object we need to invoke the Retrofit2 builder. While the HttpClient doesn't really support anything besides OkHttp, there are hooks to adapt the serialization and call adapter layers. Here you can see I configure OkHttp to insert a token header, and add Gson and Enum adapters for serialization.

  // Inject the authentication header.
  private class TokenInterceptor(private val token: String) : Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
      val newRequest = chain.request().newBuilder()
          .addHeader("Authorization", "Token $token")
          .build()
      return chain.proceed(newRequest)
    }
  }

  // Build a token aware HttpClient
  val tokenHttpClient = OkHttpClient.Builder()
      .addInterceptor(TokenInterceptor("my-token"))
      .build()

  // Build the api
  val api = Retrofit.Builder()
      .addConverterFactory(GsonConverterFactory.create())
      .addConverterFactory(EnumRetrofitConverterFactory())
      .baseUrl("https://api.robinhood.com/")
      .client(tokenHttpClient)
      .build()
      .create(RobinhoodCryptoQuotesApi::class.java)

  // Make a request 
  val response = api.getHistoricals(
      idPair = "3d961844-d360-45fc-989b-f6fca761d511",
      spans = Spans.DAY,
      bounds = Bounds.TWENTY_FOUR_SEVEN,
      interval = Intervals.DAY
  ).execute()

  // Print out some data  
  for(data in response!!.body()!!.data_points) {
    println(data.high_price)
  }

And that's all! In the final code, there was some additional work to make an OAuthTokenInterceptor, but all of that complexity is constrained to the constructor. Defining the endpoints is a matter of typing out the data classes for serialization, naming your methods, and applying the Retrofit2 annotations.

All in all, Retrofit is a very concise way to define a client library in a type-safe way. The code described in this article can be found on GitHub.

Netbeans Platform: Mac OSX App Bundle and Embedded JRE with Maven

noreply@blogger.com (Will Winder) — Fri, 01 Sep 2017 21:02:00 +0000

One problem every Java Desktop developer has to deal with is that Java apps just don't act like native applications. Whether it's the JRE dependency, the swing look and feel, or just bad memories about slow performance in old versions of Java. On Mac OSX there is an additional hurdle where users expect their applications to be a .app bundle inside a DMG image with a shortcut to /Application allowing them to drag/drop the installation. The built in Netbeans Platform installer is nice, but it's a far cry from the native behavior.

With that in mind, I'm going to describe how you can really step up your Netbeans Platform Maven build to include an OSX deployment with a bundled JRE all packaged up as an APP inside a DMG image. What a mouth full.

All of the source code for my implementation in Universal Gcode Sender can be found on GitHub in this commit. The techniques used in this post were heavily inspired (and in many cases copy/pasted from) the Gephi project, which has a very nice packaging and deployment strategy.

Overview

1. Define properties - we'll expose some custom properties in our maven file in order to make it easier to understand and maintain.
2. Create and modify resource files.
3. Resource Filtering - some extra steps need to be taken to apply our properties to configuration files, so we'll have a filtering step utilizing the maven-resources-plugin.
4. Build the artifact - using the maven-antrun-plugin we'll build the Mac OSX bundle.
5. How to use it - What commands need to be run and from where.
6. Next steps - integration with CI, app signing, etc.

Define Properties

We start things off simple by defining a few properties. Here I've prefixed my app's properties with "ugs." to avoid any possible naming collisions and put them in my root POM.xml:

<ugs.app.title>Universal Gcode Platform ${project.version}</ugs.app.title>
  
<!--==== Mac OS X bundle settings ====-->
<ugs.appbundle.name>Universal Gcode Platform</ugs.appbundle.name>
<ugs.bundle.jre.url>http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163</ugs.bundle.jre.url>
<ugs.bundle.jre.version>jre-8u131-macosx-x64</ugs.bundle.jre.version>

<!-- Mac OS X signing identity - must match with a verified Apple developer certificate in the keychain -->
<ugs.codesign.identity>Developer ID Application</ugs.codesign.identity>

<!-- You probably already have this property -->

<brandingToken>ugsplatform</brandingToken>

Create and Modify Resource Files

OSX applications use an information property list file named "Info.plist" to configure the application, we will need to create such a file using the above properties. There are a number of additional configurations which can be made in this file like associating file extensions with your app, but they won't be covered here. You can read more about that in the documentation.

Create this file in your application module at applicaiton/src/main/resource/Info.plist:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
    <dict>

        <key>CFBundleName</key>
        <string>${ugs.appbundle.name}</string>

    
        <key>CFBundleVersion</key>
        <string>${project.version}</string>
    
        <key>CFBundleExecutable</key>
        <string>${brandingToken}</string>
    
        <key>CFBundlePackageType</key>
        <string>APPL</string>
    
        <key>CFBundleShortVersionString</key>
        <string>${project.version}</string>
    
        <key>CFBundleSignature</key>
        <string>????</string>
    
        <key>CFBundleInfoDictionaryVersion</key>
        <string>6.0</string>
    
        <key>CFBundleIdentifier</key>
        <string>${project.groupId}</string>
    
        <key>CFBundleIconFile</key>
        <string>${brandingToken}.icns</string>
    
        <key>NSHighResolutionCapable</key> 
        <true/>
    </dict>
</plist>

For convenience, you should also rename your launcher.conf file to your brandingToken and put it in the same directory as the Info.plist, for me this file is named ugsplatform.conf. The important part in this file is that it contains the commented out jdkhome property, it will be used later for bundling the JRE. Additional options can be configured with the "default_options" parameter:

# \${HOME} will be replaced by user home directory according to platform
default_userdir="\${HOME}/.\${APPNAME}/${project.version}/dev"
default_mac_userdir="\${HOME}/Library/Application Support/\${APPNAME}/${project.version}/dev"

# options used by the launcher by default, can be overridden by explicit
# command line switches
default_options="--branding ${brandingToken} -J-Xms64m -J-Xverify:none -J-Dsun.java2d.noddraw=true -J-Dsun.awt.noerasebackground=true -J-Dnetbeans.indexing.noFileRefresh=true"
# for development purposes you may wish to append: -J-Dnetbeans.logger.console=true -J-ea

# default location of JDK/JRE, can be overridden by using --jdkhome <dir> switch
#jdkhome="/path/to/jdk"

# clusters' paths separated by path.separator (semicolon on Windows, colon on Unices)
#extra_clusters=

Finally, we also need to create a .icns icon file, it should have the same name as your brandingToken and be located in the application module at application/src/main/app-resources/

Resource Filtering

Now that we have our resource files, we need to substitute the properties. This is done using the maven-resources-plugin. There is nothing especially fancy here, we are setting filtering to true and copying the filtered resources into the target directory:

<!-- Copy ressources ${brandingToken}.conf and Info.plist with filtering --><plugin>
    <artifactId>maven-resources-plugin</artifactId>
    <executions>
        <execution>
            <id>generate-app-conf-file</id>
            <phase>generate-resources</phase>
            <goals>
                <goal>copy-resources</goal>
            </goals>
            <configuration>
                <outputDirectory>${basedir}/target/</outputDirectory>
                <resources>
                    <resource>
                        <directory>src/main/resources</directory>
                        <includes>
                            <include>${brandingToken}.conf</include>
                            <include>Info.plist</include>
                        </includes>
                        <filtering>true</filtering>
                    </resource>
                </resources>
                <escapeString>\</escapeString>
            </configuration>
        </execution>
    </executions>
</plugin>

Build the Artifact

This is the big one, we download the JRE, create the package structure, sign the application and package it all up into a DMG image. By utilizing the maven-antrun-plugin we can do this in a procedural way.

Because we don't want to create the DMG image for every build, this is created as a profile which can be enabled independently of other features. I'll briefly describe the sections commented below to explain what is happening, but for further detail you'll need to carefully read the code.

Clean

In case the build fails, we need to delete any files which may have been left behind from a previous build. This will prevent accidentally releasing stale files.

Create folders

A special file structure needs to be created in order for the .app to be used, we start setting up that file structure by creating the folders.

Copy application

One important piece of the .app file structure is that the executable is put in a specific location. First we're copying the executable into the build directory so that we can do further modifications to it in a moment.

Copy logo and configuration files

Two more important files, copy the ${brandingToken}.icns and Info.plist files to their specified location.

Modify application script and copy to final location

We now move the executable into its final location in the .app file, which happens to be Contents/MacOS/. For OSX, the netbeans platform uses a shell script. We need to make a slight modification to this script because the relative location of the resources is different with this new file structure. To that end the replace command to update the working directory back to the bin directory so that everything works as it ought to.

Bundling the JRE

To bundle the JRE we first download it and untar the jre directory into .app/Contents/PlugIns/, some additional manipulation is needed to make sure everything works smoothly. Like resetting a quarantine bit and updating libjli.dylib

We also need to set the jdkhome variable in our .conf file, this is done with another substitution looking for #jdkhome="/path/to/jdk" and replacing it with the relative path to our bundled JRE.

Codesign JRE and App

This piece is something I haven't actually gotten to work yet. Presumably if you have your codesign identity configured properly on the build machine it should work as expected.

Create the DMG

Finally we need our DMG file complete with a symling to the /Application directory. You can take a look at the hdiutil and genisoimage commands which build the image on a OSX and Linux machines respectively.

<profile>
  <id>create-dmg</id>
  <properties>
      <skipCreateDmg>false</skipCreateDmg>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <executions>
          <!-- Create the Mac OS X application bundle and dmg file -->
          <execution>
              <id>create-app-bundle</id>
              <phase>package</phase>
              <goals>
                  <goal>run</goal>
              </goals>
              <configuration>
                  <skip>${skipCreateDmg}</skip>
                  <target>
                      <!-- Clean -->
                      <delete includeEmptyDirs="true" failonerror="false" removeNotFollowedSymlinks="true">
                          <fileset dir="${project.build.directory}/${ugs.appbundle.name}.app" followsymlinks="false"/>
                          <fileset dir="${project.build.directory}/${ugs.app.title}" followsymlinks="false"/>
                          <fileset dir="${project.build.directory}/${project.artifactId}-${project.version}.dmg" followsymlinks="false"/>
                      </delete>
                      <delete file="${project.build.directory}/${project.artifactId}-${project.version}.dmg" failonerror="false"/>

                      <!-- Create folders -->
                      <mkdir dir="${project.build.directory}/${ugs.appbundle.name}.app"/>
                      <mkdir dir="${project.build.directory}/${ugs.appbundle.name}.app/Contents/MacOS"/>

                      <!-- Copy application -->
                      <copy todir="${project.build.directory}/${ugs.appbundle.name}.app/Contents/Resources/${brandingToken}">
                          <fileset dir="${project.build.directory}/${brandingToken}"/>
                      </copy>

                      <!-- Copy logo and configuration files -->
                      <copy tofile="${project.build.directory}/${ugs.appbundle.name}.app/Contents/Resources/${brandingToken}.icns" file="src/main/app-resources/${brandingToken}.ic
ns" />
                      <copy tofile="${project.build.directory}/${ugs.appbundle.name}.app/Contents/Info.plist" file="${project.build.directory}/Info.plist"/>

                      <!-- Modify application script and copy to final location -->
                      <move file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/Resources/${brandingToken}/bin/${brandingToken}" todir="${project.build.directory}/
${ugs.appbundle.name}.app/Contents/MacOS"/>
                      <replace file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/MacOS/${brandingToken}" token="`dirname &quot;$PRG&quot;`" value="`dirname &quot
;$PRG&quot;`&quot;/../Resources/${brandingToken}/bin&quot;"/>
                      <chmod file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/MacOS/${brandingToken}" perm="ugo+rx"/>

                      <!-- Download and untar JRE -->
                      <exec dir="${project.build.directory}" executable="curl">
                          <arg line="-L"/>
                          <arg line="-C"/>
                          <arg line="-"/>
                          <arg line="-b"/>
                          <arg line="&quot;oraclelicense=accept-securebackup-cookie&quot;"/>
                          <arg line="-O"/>
                          <arg line="${ugs.bundle.jre.url}/${ugs.bundle.jre.version}.tar.gz"/>
                      </exec>

                      <!-- Unzip archive -->
                      <mkdir dir="${project.build.directory}/${ugs.appbundle.name}.app/Contents/PlugIns"/>
                      <exec dir="${project.build.directory}" executable="tar">
                          <arg line="-zxf"/>
                          <arg line="${ugs.bundle.jre.version}.tar.gz"/>
                          <arg line="-C &quot;${ugs.appbundle.name}.app/Contents/PlugIns&quot;"/>
                      </exec>

                      <!-- Remove quarantine bit set recursively on JRE -->
                      <exec dir="${project.build.directory}" os="Mac OS X" executable="xattr">
                          <arg line="-rd"/>
                          <arg line="com.apple.quarantine"/>
                          <arg line="&quot;${ugs.appbundle.name}.app/Contents/PlugIns&quot;"/>
                      </exec>

                      <!-- Get the JRE folder name -->
                      <path id="jre_name">
                          <dirset dir="${project.build.directory}/${ugs.appbundle.name}.app/Contents/PlugIns" includes="jre*" />
                      </path>
                      <property name="bundle.jre.path" refid="jre_name" />
                      <basename property="bundle.jre.name" file="${bundle.jre.path}"/>

                      <!-- Configure relative JRE path into ugsplatform.conf -->
                      <replace file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/Resources/${brandingToken}/etc/${brandingToken}.conf" token="#jdkhome=&quot;/pat
h/to/jdk&quot;" value="jdkhome=&quot;../../PlugIns/${bundle.jre.name}/Contents/Home&quot;"/>
                      <chmod file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/PlugIns/**" perm="+x" type="both"/>

                      <!-- Fix JRE by replacing libjli.dylib symlink with real file -->
                      <delete file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/PlugIns/${bundle.jre.name}/Contents/MacOS/libjli.dylib"/>
                      <copy file="${project.build.directory}/${ugs.appbundle.name}.app/Contents/PlugIns/${bundle.jre.name}/Contents/Home/lib/jli/libjli.dylib" todir="${project.bui
ld.directory}/${ugs.appbundle.name}.app/Contents/PlugIns/${bundle.jre.name}/Contents/MacOS"/>

                      <!-- Codesign JRE -->
                      <exec dir="${project.build.directory}" os="Mac OS X" executable="codesign">
                          <arg value="-fs"/>
                          <arg value="${ugs.codesign.identity}"/>
                          <arg value="-v"/>
                          <arg value="${ugs.appbundle.name}.app/Contents/PlugIns/${bundle.jre.name}/"/>
                      </exec>

                      <!-- Codesign app -->
                      <exec dir="${project.build.directory}" os="Mac OS X" executable="codesign">
                          <arg value="-fs"/>
                          <arg value="${ugs.codesign.identity}"/>
                          <arg value="-v"/>
                          <arg value="${ugs.appbundle.name}.app"/>
                      </exec>

                      <!-- Create application folder and add Applications dynamic link -->
                      <mkdir dir="${project.build.directory}/${ugs.app.title}"/>
                      <move file="${project.build.directory}/${ugs.appbundle.name}.app" todir="${project.build.directory}/${ugs.app.title}/" />
                      <symlink link="${project.build.directory}/${ugs.app.title}/Applications" resource="/Applications" failonerror="false" />

                      <!-- Create DMG (Mac OS X) -->
                      <exec dir="${project.build.directory}" os="Mac OS X" executable="hdiutil">
                          <arg value="create"/>
                          <arg value="-noanyowners"/>
                          <arg value="-imagekey"/>
                          <arg value="zlib-level=9"/>
                          <arg value="-srcfolder"/>
                          <arg value="${project.build.directory}/${ugs.app.title}"/>
                          <arg value="${project.artifactId}-${project.version}.dmg"/>
                      </exec>

                      <!-- Create DMG (Linux), only for testing -->
                      <exec dir="${project.build.directory}" os="Linux" executable="genisoimage">
                          <arg value="-V"/>
                          <arg value="${ugs.appbundle.name}"/>
                          <arg value="-U"/>
                          <arg value="-D"/>
                          <arg value="-l"/>
                          <arg value="-allow-multidot"/>
                          <arg value="-max-iso9660-filenames"/>
                          <arg value="-relaxed-filenames"/>
                          <arg value="-no-iso-translate"/>
                          <arg value="-r"/>
                          <arg value="-o"/>
                          <arg value="${project.artifactId}-${project.version}.dmg"/>
                          <arg value="-root"/>
                          <arg value="${project.build.directory}/${ugs.app.title}"/>
                          <arg value="${project.build.directory}/${ugs.app.title}"/>
                      </exec>
                  </target>
              </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

How to use it

After all that, using it is a simple matter of running "mvn package -P create-dmg". The resulting file will look something like ugs-platform-app-2.0-SNAPSHOT.dmg and be located in your application/target directory.

Next Steps

The next steps for me are to create a windows installer with a bundled JRE, and setup TravisCI (which has OSX machines available) to build my artifacts.

Netbeans Platform: Good, Bad and Ugly

noreply@blogger.com (Will Winder) — Thu, 10 Aug 2017 14:42:00 +0000

The Good

The Netbeans Platform is incredibly simple to get started with. There are extensive tutorial based guides to get you started, like this tutorial about creating a CRUD application with zero code, or making a paint app, and videos too, not to mention Geertjan's Blog which goes into many advanced examples. If you're using the Netbeans IDE you also get a huge assortment of wizards to help you along with various annotation-based integrations. The wizards have excellent ANT support and pretty good Maven support as well. There are many advanced features in the box from window management, plugins, configurable key-bindings and an auto-update framework.

The Bad

When you want to do something advanced which isn't covered by a tutorial, you quickly need to roll up your sleeves and dig into the source code. For example, if you choose to migrate from the default ANT build to Maven, this guide is lacking if you aren't very familiar with Maven. Or even if you used the built in Maven wizard, there is no documentation that you need to run with the deployment profile to build the installers (if you're stuck on this one, it's "mvn package -P deployment"). Another big pain point for me was trying to create actions at runtime with localization which also showed up in menu's, this took lots of research before eventually creating a custom service, and they still didn't work for all cases. Even integrating my build with Jenkins was a headache.

The Ugly

When you're finished with your application, you still have a Java application. No matter how convenient and productive the language is there is still a strong bias against Java on the Desktop, and that bias doesn't seem to be going away (maybe JavaFX with native bundling will help sway people). Until it's easier to bundle your Netbeans Platform application with a native look and feel that bias isn't going anywhere.

UGS 2.0 to Netbeans Platform first impressions

noreply@blogger.com (Will Winder) — Wed, 11 Mar 2015 13:15:00 +0000

For the past year Universal G-code Sender has suffered pretty severe feature creep. Lots of people had lots of good ideas, but it made the front end code confusing and hard to extend. In large part this is just because it's hard to compartmentalize a complex GUI. GRBL has had a lot of interesting features added in the past year which are begging to be leveraged, but UGS wasn't in a position to use them. With all this in mind I started trying to come up with a solution.

The first step was to decouple the "model" object from the GUI. Inspiration for this started nearly a year ago when Bob Jones implemented a web interface for UGS. Bob made a thin API layer which exposed a few simple functions and allowed him to build a completely separate web-based user interface. This pattern is known as Model-View-Presenter, and allows for multiple front ends which use the same back end code. It enhances test-ability of the business logic and helps facilitate a flexible front end. It didn't take more than a few hours to flesh out the rest of the functions this API needed so that the thin API exposed all of the back end features needed for the classic UGS GUI. After that it was a matter of re-implementing the Swing GUI to utilize the new API.

With the foundation laid it was time to consider what to build on top of it. Rugbymauri on github clued me into an idea that more experienced GUI developers have probably known for a long time: plugin based applications. Then he took it a step further and told me about the Eclipse Rich Client Platform (Eclipse RCP) and the Netbeans Platform. After some initial investigation I decided to pursue Netbeans Platform.

It took about a week for me to completely settle on the idea of a plugin based architecture. In the end, it was a perfect fit for UGS. The back end application already had features for sending status updates to any number of front end listeners. This has been used by the Visualizer from day one, it is a completely separate GUI component and it listens to the same back end events as the main GUI. Something like the Netbeans Platform would take this a step further and facilitate the creation of these modules - whereas in the classic UGS interface all these windows are managed in a monolithic "MainWindow" class.

Moving forward the concept of modules allows me to think about other features in a way that wont cause the code to feel bloated. For instance - how would a Gcode editor that can highlight some number of lines and have it displayed in the 3D visualizer? In the classic interface this would be a huge undertaking that would massively complicate the "MainWindow". On the other hand with a modular design there just needs to be a new "Editor" module, the Netbeans Platform handles sharing of the gcode file and provides APIs for binding selection events between the Editor and Visualizer. What would have been a week of effort could now be done in an afternoon or two. By the way, the Netbeans Platform also comes with all the modules created for the Netbeans IDE, so it already has GUI components for a code editor with breakpoints (Breakpoints? Maybe another afternoon).

There are a few areas that I'm not sold on yet, for instance UGS already has several observer API's and I'm not sure if I'll convert them to use corresponding Netbeans Platform features. Also the library wrapper modules are a little kludgy, especially if your project is built using a separate library jar like UGS is. Every time I rebuild my library jar I need to update the library wrapper module. The workaround I settled on was to modify my library's build script to dump the jar inside the library wrapper module, a side effect to which is that if I do a clean on that project it breaks the library wrapper.

The UGS Platform, UGS's 2.0 release, is currently in a proof of concept state. All the major integration has been done, now its a matter of creating all the different modules that exist in the Classic GUI. After that other features features of the Netbeans Platform can be leveraged for things like integrated Preferences and Configuration menu's, and default Window layout schemes.

Automated builds with Netbeans Platform 8.0.2

noreply@blogger.com (Will Winder) — Tue, 10 Mar 2015 23:00:00 +0000

I've been converting UGS to run on the Netbeans Platform over the past week, and in general its been great. The features and documentation I used while migrating UGS away from Swing were very well done. So aside from a couple hiccups, the Netbeans Platform has been great. Until a few days ago when I decided it was time to integrate my new changes with my Jenkins CI server.

It turns out that you can't build a Netbeans Platform application without connecting to the Netbeans IDE first. This has to do with the extra modules, platform components and build harness. Searching the subject online provides a lot of documentation, everything I've found is either out dated, contradictory, or I could only get it working with a new project and not my existing one.

The final solution was so simple, and minimally invasive that I wanted to create this blog post.

There are 4 interesting files for a Netbeans Platform build:

build.xml - Some documentation suggests adding a download-netbeans-rcp target here. That isn't needed. If you have a custom target you can put it here, otherwise there is no need to modify this file.
nbproject/build-impl.xml - This is loaded by build.xml, don't modify this file.
nbproject/project.properties - This file seems to identify your modules and the applications name, no need to modify this file either.
nbproject/platform.properties - Modify this file!

The Netbeans Platform has facilities built into its build script to download all required files. platform.properties is where you need to specify what it should download.

# These are used later in the build script
nbplatform.active.dir=${suite.dir}/latestStableBuild
suite.dir=${basedir}
harness.dir=${nbplatform.active.dir}/harness
# Provide some URLs
bootstrap.url=http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/netbeans/harness/tasks.jar
autoupdate.catalog.url=http://dlc-cdn.sun.com/netbeans/updates/8.0.2/uc/final/distribution/catalog.xml.gz

Thats it! It probably took me 5+ hours to find that. Now I can run "ant build-zip" on my Jenkins server and get a zip archive in return.

Easy and accurate SVG to DXF conversion using Inkscape and pstoedit

noreply@blogger.com (Will Winder) — Sat, 01 Feb 2014 19:55:00 +0000

If you've ever worked with a vector tool to design something to be carved with a CNC routing machine, you may have needed to convert the resulting SVG file into a DXF. In my workflow I often use Inkscape to generate my SVG graphic, but my CAD software is CamBam and it requires a DXF file.

Many people have tried to make the perfect Inkscape plugin for exporting DXF, there is Better DXF Export, Better Better DXF Export and Big Blue Saw's DXF Export. I've had the most luck with the last one, but it was still finicky and often broke when updating Inkscape or OSX. Not only that, but at the end of the Big Blue Saw page, Simon mentions that he doesn't even use the plugin most of the time, instead relying on a command line tool called pstoedit.

With that in mind I made a small shell script which will convert an SVG file to DXF, you can save the script and run it directly or add the functions to your startup script.

The script is available on github under the project svgToDxf. If you choose to add the functions directly to your startup script, as I have, simply copy the functions here and paste them in your .bashrc file. Now all you need to do to convert an SVG file to DXF is use the command svgToDxf MyCoolDesign.svg and the output will be MyCoolDesign.dxf

OpenSCAD linear extrude that interpolates between two shapes

noreply@blogger.com (Will Winder) — Sun, 22 Dec 2013 05:13:00 +0000

Full source code is available for this thing on thingiverse.

OpenSCAD has proven to be an easy to use and flexible tool, but at times I'm left scratching my head to figure out ways around its limitations. In this case I had two squares with rounded corners, their sizes and the corner radius for both squares are configurable. Now I wanted a square tube to join these two pieces together.

My hope was that I could get it close enough with some fancy parameters piped into the OpenSCAD linear_extrude function. It seemed possible since there are some pretty fancy parameters regarding rotations and twists. After checking out the documentation it didn't look like there was anything that would be close.

For this project I found a module by Felipe C. da S. Sanches to create squares with rounded corners. I'm sure there are many like it, but this was the first one I found and it worked as advertised.

So made a little module using this which wraps around a rectangle for one side, and another which fits into a rectangle for the other side:

This simply creates a rectangle with rounded corners and subtracts a slightly smaller rectangle from the middle, then extrudes the whole assembly some height. This piece goes around a rectangle with square corners, so the inside has square corners. Then I created a second tube designed to fit into a rectangle with rounded corners, so the outside of this tube must have rounded corners. After spacing them out I get something that looks like this:

This is where I realized I need some fancy function to extrude the bottom shape into something which meets up with the upper shape. The solution I ended up with works the same way as the 3D printer I'll print this object with: slice it up into multiple extrusions. For each extrusion I interpolate the inner radius, outer radius, width and length depending on how far into the overall extrusion we are.

With this I can now create a shape which connects my two adapters. The final product printed out nicely:

Finished product!

Fits perfectly.

A week of 3D printing with the OrdBot Hadron

noreply@blogger.com (Will Winder) — Tue, 30 Apr 2013 10:32:00 +0000

It has been about a week since my OrdBot's first successful calibration print and it seems like a good time to write down some of my initial problems, experiences and solutions. However before I get into that a review of my 3D printer stack.

Hardware
OrdBot Hadron
3/8"-12 TPI ACME Z-Axis screws
1.8 degree Nema17 X/Y motors
1.8 degree Nema23 Z motors
Qu-bd MBE Extruder (with some modifications)

Electronics
AzteegX1 v1.0 (644P processor)
Lava heatbed, I have only used PLA at this point so this is still disconnected.

Software
Mac OSX
Marlin firmware
Slic3r v0.9.9
Pronterface (March 2012 release)
Repetier-host Mac (0.56, lower versionthan linux/windows releases for some reason)

Now for the problems encountered during the past week, approximately in the order which I ran into them.

Pololu A4988 - I have used these before on my ShapeOko CNC machine. They have been so reliable that I completely forgot how to set them up, as such I blew up 3 drivers while setting up the electronics. So be careful, check the polarity and for the love of everything don't measure voltage when your multimeter is in continuity mode!

Leveling the bed - This was tedious but not particularly difficult. The LAVA heatbed has 8 M3 screws along the perimeter so calibration was just a matter of carefully tightening them until the height is just right. To measure the height I lowered the printer nozzle until it had just enough space for a piece of paper to slide underneath with little resistance, at this point I manually jogged the nozzle along the perimeter adjusting the M3 screws as necessary until the bed was level.

Setting the Z-Axis endstop - This step is ongoing and was very tedious without a way to precisely raise or lower the endstop actuator (a screw in sliding t-slot nut). I have it set fairly close, but still find myself hitting the "stop motor" button and manually turning the Z-Axis screws while the first layer is printing. Eventually I will print out something like this and dial in the Z endstop a more reliably.

Configuring steps per mm - This was the longest part of the process, and I'm still not sure why. Fortunately it wasn't difficult, just tedious. I know the steps per revolution on my motors (200), the microstep resolution for the steppers (16x), the pitch of MXL belts (2.032mm) and the number of teeth on the pulley (16). Using the equation on the buildlog.net configuration page gave me a number which ended up being way too small. My calibration object of choice was the nickel test thing on thingiverse, I just kept printing them out, measuring the insides and adding a ratio multiplier until satisfied that things were the correct size.

Pololu A4988 overheating - Once I had my machine calibrated I started trying to print larger objects that took more time. Without fail after 5-10 minutes after starting the machine would start losing steps. Eventually I noticed how hot the stepper drivers were and mounted a fan pointing at them which fixed the problem.

The notorious QU-BD MBE. This thing gave me a ton of trouble, so much so that I ordered a J-Head after the first day of trying to get it up and running. That said I've managed to get it to print very reliably and haven't yet felt the need to install the J-Head.

QU-BD pre-upgrades - Before I even had my printer assembled I performed some upgrades on the QU-BD. First is the QU-BD Modification Kit from the MakerSlide store. Secondly I hobbed the stock raptor gear with a small tap in my electric hand drill.

QU-BD overheating jam - There is a fan and heatsink on the extruder. I wired it (incorrectly) to the Azteeg X1 fan input, making it configurable in gcode (M106 and M107). My slic3r configuration toggled the fan on and off, so occasionally the cold-end of the extruder was heating up too much causing a jam. My current solution is disabling the cooling option in slic3r, but eventually I will wire the fan directly to the 12v input.

QU-BD underheating jam - My printer is in the kitchen and there is a nice crossdraft going through the room now that it is nice outside. The crossdraft was causing the nozzle to drop below 165 degrees, when this happens things usually jam. I need to wrap the hot end in some sort of insulation which will hopefully fix this problem.

QU-BD slicer jam - While printing the nautilus gear thing I was noticing frequent jams during the infill stage. The thing about this object was that there was very little amount of infill needed so the filament moved very slowly. My best guess is that the filament was sitting in one place so long that it heated up too high in the extruder which caused the jam. After reconfiguring slic3r to leave the object hollow there were no more jams on this print.

Pronterface vs. Repetier-host - This could probably be a blog post on its own. They both work but suffice to say that repetier-host is leaps and bounds ahead of pronterface in terms of usability. The one problem I had was that repetier-host allows you to configure the controller buffer size, the default is 63 which causes stuttering when printing complex surfaces with short line segments. I upped this value to 127 when I was 75% finished with the owl print (below) and you can tell just from looking at the surface of the model where I made that change because the surface quality improved dramatically. I just wish I had found that setting before printing all those feathers!

This owl is my longest print to date at 3 1/2 hours. This print was running while a draft was running past the printer, I probably fixed 5 or 6 jams while it printed which caused a several noticeable defects (most of them on the owls back).

After many attempts I finally got the Nautilus Gear to print. Now that everything is dialed in I could print one out in about an hour (~12 minutes for the two clips and ~18 minutes per gear).

Building an OrdBot 3D Printer

noreply@blogger.com (Will Winder) — Sat, 27 Apr 2013 23:01:00 +0000

I've been a fan of MakerSlide ever since building my ShapeOko CNC Mill, and have been interested in 3D printing since first hearing about the RepRap project in 2007. So when I first saw the OrdBot, I knew that it would be the printer I build. It didn't hurt that I had 10 feet of extra MakerSlide and a whole bunch of the special bearings and eccentric spacers left over from my ShapeOko build.

One of my goals was to build all the custom parts myself, the blue and black pieces in the photo above. Cutting aluminum on my new CNC machine pushed it to the limit, but worked out in the end. One of the larger OrdBot pieces is the handle, here is a shot of the ShapeOko making short work of it:

There are a couple parts that I modified or upgraded during the build. Most notably the Z axis. After reading about bent Z-rods I decided to get some ACME rods, and I wanted to use some spare Nema-23 motors that I had on hand.

The NEMA-23 motors for the Z axis were easy, I slightly modified the stock motor part to fit the new motor:

Attaching the ACME rod was a little trickier, but managed to hand fabricate a bracket with some 1/8" thick angle aluminum:

In the spirit of being thrifty, I signed up for one of the (now notorious) Qu-bd extruders during their kickstarter campaign. So far I've managed to get the extruder to work with a couple of simple motifications. Some more hand fabricated mounts and I had the extruder attached:

For the electronics I am using one of the first generation Azteeg X1 boards. So far I've been very happy with it (save for an exciting wiring mistake where I had the polarity backwards). I opted to run the wires through the hollow MakerSlide extrusions whenever possible. Here is the nearly completed wiring, I really appreciated the zip-tie holes:

After a few days of tweaking and calibrating, I'm getting some nice results. Here are some action shots with the machine up and running:

ShapeOko CNC Mill

noreply@blogger.com (Will Winder) — Sat, 27 Apr 2013 22:20:00 +0000

April of 2012 I signed up for the first batch of ShapeOko kits from inventables.com. Unsure of how popular the kit would be, inventables had a kickstarter-style order of 150 (or so) kits. That number was reached handily and several more batches followed. Since then the ShapeOko has become a standard item in their store.

I've wanted a CNC mill for a long time, but could never justify the expense. Now there are products like the MakerSlide linear rail system that made it possible for low cost machines. The first round of kits were only $200 for the entire mechanical platform - add your electronics and a dremel tool and the machine can start cutting.

So thats what I did.

The stock kit plus motors after assembly:

One of the nice things about the ShapeOko is how hackable it is. For instance if you want to make the cutting area larger you can just replace the MakerSlide with longer rails. So I added longer rails, a second Y-axis motor, a torsion box to mount everything on, a bigger router and some woodworking T-slot to hold work. Here is a picture of the machine a few weeks ago while cutting a large aluminum part for another project:

The current cost of the machine is $663.75, that includes all components bought for the machine regardless of whether or not they were used and shipping. So far I've spent another $150.55 on endmill of various sizes for for cutting various materials.

Projects Past: The 2009 Automatic Cat Feeder

noreply@blogger.com (Will Winder) — Sat, 26 Jan 2013 16:27:00 +0000

In 2009 I took a week off from work to build an Automatic Cat Feeder to ration food to the overweight household cat. This was a project built from need and it had a deadline because my girlfriend and I were heading out of town for Thanksgiving.

The requirements were simple: It needed to be easy to refill and dispense the correct amount of food twice daily. Early on I had an idea for a large rotating drum to dispense the food, it needed to be easily removable for refilling and heavy enough that it could sit on a motorized wheel and cause enough friction to be turned.

Figuring out a latch mechanism that dropped the correct quantity of food after one rotation was the first tricky part. I didn't want any electronics on the drum so it had to be completely mechanical. In the end I hot glued a small box to the outside of the drum which filled during rotation, when it started moving back towards the top a tab would be hit to fling a door open and drop the food out. A magnet held the door shut and gravity would shut the door when the box made it back to the top. I'm extremely proud of this mechanism and it worked perfectly.

The dispenser mechanism! A marvel of engineering, hot glue, magnets and JB Quick Weld.

With all the details figured out, hacking it together turned out to be pretty easy. The hardware store patiently cut a sheet of 1/4" MDF into pieces for me which formed the box, and I was able to glue the pieces together with gorilla glue. A small stepper motor and some electronics were harvested from a 5 1/4" floppy drive and I was able to control it with an arduino. The program couldn't have been more than 50 lines, I figured out how many steps it took for 1 revolution of the drum through trial and error. Then coded it to do that every 12 hours.

The finished product. Seriously, we used it looking like this for several years.

The happy customer. My wife added a chute made from a mountain dew bottle.

I did make a couple upgrades after these shots were taken. The stepper motor died so I upgraded to sparkfun's cheapest stepper and an EasyDriver stepper driver. I also added a reed switch to the box and put a magnet on the drum so that I didn't have to count steps, this was needed because depending on how much food was in the drum it would sometimes slip and require a different number of steps to get back to the top.

Projects Past: The 2007 DIY Projector

noreply@blogger.com (Will Winder) — Sat, 26 Jan 2013 15:52:00 +0000

I've had many projects over the years, the most involved by far has got to be my DIY projector. It is also my first serious project. By the end of it I had vastly improved my knowledge of optics, electronics, woodworking, and even a little thermal dynamics.

Research started sometime mid 2007, the now defunct lumenlab forums had a whole community of people building DIY projectors. There is a lot of theory to learn, optics to figure out, components to buy, and bringing all the pieces together is no simple task. For such a complexe machine, at the core it has some very simple principles: A point light source radiates light into a fresnel lens, which straightens the light through the LCD screen, then another fresnel lens angles the light into a special lens which can focus the light on a screen. Here is an image from engadget which demonstrates the light path:

The major problem with these sorts of projectors is that they tended to be very large. 1080p monitors at the time tended to be at least 19" diagonal, which meant the entire box would need to be 3-4 feet long! Fortunately there were a few entrepreneurial types who started sourcing LCD screens that were smaller and ideal for DIY projection and selling them directly from the manufacturers. I managed to get my hands on one such monitor that was only 10.6" and had 720p resolution. With the LCD picked out I was able to begin designing the enclosure around December 2007.

Other than the LCD, the most important component is the light source. Some of the brightest bulbs consumed 400 watts of electricity or more, and had very impressive results. But with such high powered bulbs came a lot of heat that needed to be dealt with. I ended up going with a smaller 150 watt bulb.

The final pieces were the fresnel lenses and the projection lens. These were a fairly standard item and lumenlab took a lot of the guess work out of things by selling them from their store.

My final design to incorporate all these parts was a two chamber design. The top chamber had the optics. The bottom chamber contained all of the electronics: The LCD controller, the light bulb components (transformer and starting capacitor), and AC adapters and a heat sensitive trigger which would keep the fans running until the heat in the box dropped far enough.

The enclosure was designed to be simple to build and easy to assemble. The parts were cut on my dads table saw, and pieces were laminated together so that components could slide in and out of the enclosure during assembly. This design worked nicely.

In this picture you can see all the electronics (minus the thermal switch) and the various slots for the LCD, fresnel lenses, and chamber separators.

The LCD and Fresnel Lenses were built into frames that could easily slide into the slots:

The LCD frame had special protection for the delicate flexible cables of the monitor electronics. The frames for the fresnel lenses were similar.

The light source needs to be precisely positioned for the light to focus properly, so this box was built with rare earth magnets on the bottom. It attaches to a metal plate in the box which can be seen in the upper left of this photo.

The back was hinged so that adjustments could be made as needed, here you can also see the light box in place.

The triplet lens is another clever mechanism I came up with, by building the box a little loose I was able to pad it with felt so that the lens carriage could slide in/out but still be held snugly in place. The slot at the bottom was for a holding screw, but ended up being unnecessary.

Fully assembled, on the left is a front surface mirror which allows the box to be 11" deep instead of nearly 30.

The final result is big and ugly, but it worked and I used it for years. I'll never forget how surprised my girlfriend was when I plugged it in for the first time and an image, apparently she thought I was crazy for those 5 months. In the end it really did work nicely. Designing it was a lot of fun, but my craftsmanship really limited my options.

More pictures can be found on my flickr page: DIY video projector

XCode4: UITabBarController with UINavigationController using Interface Builder

noreply@blogger.com (Will Winder) — Thu, 19 May 2011 12:34:00 +0000

If this helped you, please consider downloading my FREE iPhone app "Hydrate Yourself"and leaving a review.

There are a whole bunch of tutorials to create a Navigation Controller inside a TabBarController with XCode3. But there are a number of small changes in XCode4 which can trip up someone not familiar with them. So this guide is essentially an updated version of those other guides, designed to create the same simple demo using the new tool.

1. Create a new Project
We are going to start with a Tab Bar Application, so go ahead and create a new project as usual. I'm calling mine Navigation Tab Bar.

2. Select the Tab Bar
Click MainWindow.xib and you should see the integrated Interface Builder. On the left is a section called Objects, select the Tab Bar Controller.

3. Add the Navigation Controller
On the right side of your screen click the Utilities box. This is new in XCode4 and should look familiar if you've been using XCode3. At the bottom find the Navigation Controller and drag it into the Tab Bar in the middle of your screen.

Here you can see I added it as the middle Tab

4. Configure new tab

Back in the Objects section you can now expand the Navigation Controller and select the View Controller object. Now on the right side you can select the Identity Inspector at the top and put in your UIViewController's class name. For the purpose of this demo I am calling it "YourNewViewController" which we will make at the end.

Once that is done switch to the Attribute Inspector and put in the name of your XIB file in the NIB Name section. Note: do not add the .xib extension, this will cause a runtime crash.

5. Create the new UI View Controller

This is probably familiar to you already, create file from the File menu and select a UIViewController subclass.

Be sure to select With XIB for user interface

Use the same file name as you used before.

6. Add an IBAction

We will be adding a button to push a view onto the navigation controller. Open up the header file and add an action.

I just added the buttonPushView line

Switch to the implementation file and add another few lines for a simple call to pushViewController.

This is the simplest pushViewController line I could come up with

7. Add a button

Select YourNewViewController.xib in the file menu. Adding a button is a simple matter of dragging/dropping from the bottom right Objects section.

Next you link the button to the new IBAction by right clicking the button and dragging the arrow to File's Owner, in the Placeholders section. Select buttonPushView.

8. You're done!

All thats left now is the build and run (Command-R). Select the second tab to see your new Navigation Controller and click the button to see it in action.

Vim Search for Tags File

noreply@blogger.com (Will Winder) — Tue, 05 Apr 2011 15:00:00 +0000

I use ctags with vim. This works great when the tags file is in the same directory as the files I'm working on, but is a chore when projects have nested directories.

Add this to your .vimrc file and the recursive search up the directory structure will be done for the tags file:

set tags=tags;/

Why Hydrate Yourself?

noreply@blogger.com (Will Winder) — Sat, 08 Jan 2011 01:57:00 +0000

This is a short essay about why I decided to create "Hydrate Yourself". It is a free app for the iPhone, iPod touch and iPod which tracks water intake and helps encourage a healthier lifestyle.

According to webmd.com [1] and the US National Library of Medicine [2], symptoms of dehydration include: Dizziness, Lethargy, Lightheadedness, Headaches, Fainting, Weakness and (naturally) a Dry mouth. Other symptoms even include heart palpitations (that feeling of your heart jumping or pounding).

So why is it that water intake is such a secondary concern to so many adults? All it takes is a little attention throughout the day, such as keeping track of how much water you are drinking or carrying a water bottle with you at all times. I strongly believe that drinking enough water is one of the easiest ways to improve your health. But how much water is enough?

Its hard to say, since so many foods we eat every day contain water. Common "rules of thumb" include 8-10 glasses a day, or your body weight divided by two. In reality it will always vary from person to person depending on their lifestyle. For example if you drink a lot of diuretics, like coffee or caffeinated sodas, you will retain some of that water but not all, which skews how much you need to drink. Another factor is your personal activity levels, anything which causes you to sweat causes you to lose more water which needs to be replenished.

One of the most accurate litmus tests for determining whether you should drink more water or not is the color of your urine [3]. The web page urinecolors.com has a color chart stating that if your urine is light orange or darker that you need to drink water immediately.

With all of that known it is easy to see why you should make an effort drink enough water. I spent the last three months of 2010 creating a program to help with just that. After being unsatisfied with the alternative water tracking applications I designed "Hydrate Yourself" from the ground up. There were three key features that needed to be done right: Easy to use multiple containers, multiple calculators for recommended intake, reminders throughout the day to drink more water.

I added two daily intake calculators and an option to enter a custom amount. These calculations can then be used or customized to fit your ideal intake amount. There is a simple calculator that is based off just your weight and basic activity levels (low or high), then there is a complex calculator which is based on your weight, daily minutes of activity, climate, illnesses and more. Of course you can also skip all that and enter a direct amount - maybe your doctor made a recommendation.

A problem with the alternatives is that it can be difficult to use multiple containers, or modify them on the fly. Taking this into account I added a configurable spot on the main page for three containers (such as a water bottle, your favorite glass, and a drinking fountain). From there you can just tap the icon and have it logged or press and hold to get extra options - add a partial serving (half a water bottle), change the default portion size (a new waterbottle holds 32oz instead of 20oz) or add a serving that you forgot to enter earlier (I often forget to add my first glass in the morning).

Perhaps the most useful feature is the ability to add customizable reminders. The hardest part about drinking water throughout the day is remembering to drink it at all. To help with that I added a reminder. It goes off a customizable amount of time after you add a portion, or finish using the app. If it goes off I know that its time to get myself some more water.

[1] http://www.webmd.com/a-to-z-guides/dehydration-adults
[2] http://www.nlm.nih.gov/medlineplus/ency/article/000982.htm
[3] http://www.urinecolors.com/dehydration.php

Fetching a large data set with Core-Data

noreply@blogger.com (Will Winder) — Thu, 02 Dec 2010 23:32:00 +0000

I needed loop over a potentially large set of data stored in Core Data, but did not want to force the system to load the entire data set in memory or have Core Data constantly process faults. There seem to be several ways to do this, some easier than others. One robust way would be to use an NSFetchedResultController, but it looked a little more complex than I wanted to deal with ( and involves implementing yet another protocol).

After poking around the NSFetchResult class page for a while I noticed these methods: setFetchLimit and setFetchOffset. It turns out that these are just like the LIMIT and OFFSET keywords in SQL, which is just what I was looking for.

Using this method is just a few extra lines of code. This is what it looks like:

Custom UISwitch to show YES / NO

noreply@blogger.com (Will Winder) — Thu, 18 Nov 2010 14:28:00 +0000

You can see this in action in my free water tracking application "Hydrate Yourself". If this helped you please consider downloading it and leaving a review.

One of the great things about iPhone development are that there are lots of user interface pieces that are easy to put together and customize. Unfortunately there are also a lot of pieces which don't have all the basic customizations that may people want. The UISwitch is one of those.

Here is the standard UISwitch:

Its size, colors, and ON / OFF text are not changeable. Fortunately there are several great re-implementations (and a few hacks that use restricted API calls) to modify all this. One that I am using is the RCSwitch class available here.

I made two small changes to this code, first I renamed his RCSwitchOnOff class to RCSwitchYesNo and changed the labels to YES / NO. Second I added an init method to the RCSwitch.m base class which called the usual init function but provided the default frame size. This works fine since I am positioning the x and y coordinates using the "center" property:

The final result looks and behaves just like the original UISwitch, but is actually completely rewritten using the UIControl class. My switches now look like this: