It’s really easy when one person or the other says “I really want <x>!” Or “I really do not want to do <y>.” That’s no big deal.

But, in real life, those things aren’t that common. There’s often this, kind-of, ‘meh’ reaction, a lot of the time. “Where do you want to eat?” “I dunno, where do you want to eat?” And that goes back and forth and you never get anywhere.

I started to find myself trying to guess what Alison wanted, and suggesting that. And she would try and guess what I wanted, and suggest that.

It ends up in a cycle, and it sounds a lot like the old saying about “Being married is just asking each other, ‘I dunno, what do you want to eat?’ over and over until you die.”

But we are big giant nerds who like to solve problems. So we’ve come up with a system. A system which I am now going to share with you, to fix all sources of martial strife.

The System

This mostly works with Boolean options – true/false, yes/no, go/stay, etc.

You come up with your ‘number’ – on a scale of 0-100, how much do you want the option that you want?

And the other person comes up with their number. And you compare. Whoever’s number is a larger absolute value from 50, their option wins. You take your W’s and your L’s, and the system just works.

I asked Alison if she wanted to go to the mall today. Me, personally, I could’ve gone or not gone – my number genuinely felt like a 50 (coin toss). She reports she’s at a 60 for no-go. Boom. Done. We don’t go to the mall.

What’s great about this is there’s no guilt. The system just works as designed. When you win, don’t back down from your win, just take it. When you lose, accept your L – the other person’s desire is stronger than yours, it needs to be respected.

There are always going to be some things like “never” or “always” – and that’s more out-of-scope from this. If you absolutely do not want to go to the mall today, then put your foot down and don’t go.

But, again, those are easy. It’s that “meh” line that’s harder to walk.

One thing that hasn’t come up since we started to use this is dealing with very high absolute values. Someone being at 90 versus someone being at 5. That’s going to be a lot harder to navigate, and losing after tossing out a 90 is going to really sting.

We haven’t yet had to make our votes ‘secret’ – for fear that someone might inflate or deflate their number to try to “just get along” or “go with the flow.” Or to “get your way.” But if we start to see that, we’ll probably do something where we write our number down in the Notes app on our phones or something. But that hasn’t happened yet.

Honestly, when I do ‘lose’ with this system I actually feel pretty good about it – if I’m 52% for something and she’s 44%, her feeling is stronger (2 points above 50, versus 6 points down from 50) – so long as I was honest at my 52, I’m definitely going to feel good that her choice was right for her and I can do it.

So far, for us, it hasn’t failed yet. The only time we’ve had problems with this was one time where we didn’t use it – I wanted to go to this “meet up” event here and she really didn’t. We should’ve put numbers on that and either she would’ve gotten dragged to the event, or I could’ve felt better about not going, knowing that she felt more strongly than I did.

So, you’re welcome. Enjoy a life of strife-free marriage.

So, as you may or may not know, I work at an Open Source company. We distribute our software (Snipe-IT) for free – and we support it, as best we can, also for free. But you can also pay us to host it for you, and we will be happy to do that. Or if you need to run it on your own infrastructure, you can pay us for support. This has worked out great for us. We do have some users who leave our paid services saying “moving to self-hosted!” – which, hey, less money for us, so not necessarily great news. But we have many, many more who migrate from their self-hosted installs to ours.

So open-source, for us, is not only an ethos that we believe in. But, just to be very, very clear – it absolutely is an ethos that we believe in. But it has a side effect of giving us a pool of potential customers – and they’re the best types of customers because they know the software already. And our hosted version, versus our “open source version” are both exactly the same.

Yes, we have to spend more resources supporting free users on GitHub, and Discord, and wherever else we might find them. That’s true. But our users will also happily support each other, and we don’t have to do any marketing at all in order to get a steady flow of customers. So, IMHO, you could say that this is a “cost of doing business” or maybe a “loss-leader,” from a business perspective. (Though, again, just to reiterate – we do Open Source because our entire careers have been built on open source. The language we write in is open source. The database we use is open source. The Operating System we use is open source. So this ‘thing’ that has given us so much, it makes sense for us to give back something. And, maybe, without open source I don’t know that we’d have jobs in tech at all?)

Background on WordPress:

WordPress has been, like, our “Big Sibling” since we began. They have a very similar model to ours – you can download WP for free and run it right now. In fact, this very blog runs on it. Or you can pay them to host it for you.

By doing that they’ve become a company that earns billions. Matt Mullenweg himself is reportedly worth somewhere in the neighborhood of $400 million dollars.

We’ve wanted to be just like them – never took VC money, never took any kind of investment, just built a good business from the ground up and employed a bunch of people, and provided a great product for free, and also, optionally, a paid-for product.

They got big enough that they do have competitors who sell the very same product that they sell. WPEngine being the pertinent example today.

WPEngine has a different angle on how they do WordPress. They enable some stuff. They turn off other stuff. They charge for some things, they don’t charge for other things.

And – and this is a bit of a dick move – they also don’t contribute much back to the open source community. The thing is, while I might call it a “dick move,” it’s also completely and totally permissible.

And that should be the end of the story, right there. WordPress is making billions, they have a competitor who’s also making hundreds of millions or billions, and that competitor doesn’t do a good job of contributing things upstream. And all of that is fine and everyone should be happy. Or at least happy-ish.

The state of affairs, as of a week or so ago

But Matt (the founder of WordPress) has fucking lost his fucking marbles. He changed the copyright on the “WP” and “WordPress” marks. All of a sudden. He’s blocked third-party access to updates and plug-ins for WordPress. He’s gone on tirades about WPEngine, his competitor. There are lawsuits and counter-suits flying around.

This is an absolute violation of not only the spirit of open source, but very much so the letter of it. You can’t say “my software is open source until you make at least a million bucks a year on it, and then it isn’t.” That’s not how it works.

Big names like DHH – a guy who I have taken as my personal goal to just “not be like” – has even come out swinging saying that WordPress is very, very much in the wrong here.

So. Cool, cool. Whatever. People on the internet are going to do weird Internet things. I have this blog, our https://snipe.pt blog, and we have a corporate blog – and I bet that Alison has a few more blogs I’m not remembering, all hosted on WordPress. This “drama” (a misnomer IMHO) is a nuisance, and sad, but shouldn’t affect us.

But it does and it will.

Because I’m sure that we’re going to have customers who start to ask, “how can I be sure you aren’t going to go full Matt on us?”

And that’s a big fucking problem. It hasn’t happened yet, thank God, but it could very well happen. And I don’t know what I might tell someone if it did.

So, even though this doesn’t directly affect us, it can possibly have an effect on us. So, now, we have to worry a little bit. No need to do anything, but, probably, have a little worry.

This is where I was as of last night. A little worried, but not losing sleep or anything.

The Latest

While what most of WordPress has done has been generally awful, this is the first time they’re doing something that is actually EVIL.

These custom fields folks made a nice plug-in for WordPress. They contributed it back to WP’s github, as a good open-source citizen should. And WP just stole it.

Now, mind you – the folks who wrote this plug-in are WPEngine. I suspected as much because I can’t imagine them just arbitrarily grabbing some third-party’s code and running with it. Though, hey, maybe they can do that next? The genie’s out of the bottle now.

This is the last straw, for me. I’m going to move this blog onto something else. Unfortunately, I hate everything else.

This is probably some of the worst destruction of value I’ve seen since Elon took over Twitter. Tons of goodwill – poof – destroyed in just a few shitty blog posts.

I’ll do my best to report back on whatever I happen to come up with for my new hosting platform. Honestly, this is more a symbolic move rather than a technological one – because I just simply cannot be a party to this type of shit-fuckery. Shit-fuckery that may even have echo effects on my actual business.

So for those of you who are my friends, none of this is news – but Alison and I are moving to Portugal. We don’t know when, and the process is really unpleasant and long, but as soon as our visas come through, things are really going to get moving. I’d be pretty shocked if we didn’t make it over there this year (unless we get flat-out rejected, and then who knows how long re-application would take). But I thought I might at least explain more about the why rather than the how – there are plenty of explanations about “how” that are already out there.

As to why – well, I’ll keep that brief. This country doesn’t feel like it once was, after 45 got in office – but also, it’s been feeling like that more and more with each Republican that ever got in office in my lifetime. And it’s not the former President that scares me so much – presidents come and go. It’s the people who vote for him. Those people actually really scare me, and make me not want to live here.

So it’s time to go. I want a functioning Democracy (even though I won’t be able to vote in it for a while). I want someplace warm. I will need some levels of creature comforts, because I am a bougie bitch. I want the general politics to line up with at least some of mine. And the culture to line up a little bit with mine (not interested in super-racist, super-sexist, etc.) Language need not be English (and, for Alison, she wanted it to definitely not be English). And the number one country we came up with was Portugal. (For what it’s worth, New Zealand would probably be second on the list, though I don’t think they’d ever actually have us).

Known Pros:

1. Functioning Parliamentary Democracy. The Socialist Party (PS) is pretty dominant, but they don’t always win. The do have a separate vote for President, but the President’s powers are pretty limited. (The current President is from the PS, and the “Assembly of the Republic,” or ”Parliament,” currently has an absolute majority with the PS – without having to form coalitions. That can, and will, change).
2. Some of the warmest weather in Europe. Can get chilly, especially in the North, but not too much so.
3. Lovely people who will leave you alone if you don’t feel chatty, but can be warm, charming, funny and even a little sarcastic if you let them in (and they let you in). Once you find a Portuguese friend they will move heaven and Earth for you.
4. Delicious food, from all around the world.
5. Solid internet service and cell service (in cities – in some rural areas, you’ll see the ’number of bars’ drop down to 0).
6. The Lisbon subway was prompt, with modern trains and great signage.
7. The full-size trains themselves took a bit of getting used to, but were also very pleasant to take throughout the country (modulo the Alfa Pendular, which I will get to later).
8. SOCIALIZED FUCKING MEDICINE. And that existing means that private healthcare is much, much cheaper (current advice is: use socialized medicine for severe things, like organ transplants and cancer, use your private healthcare insurance for normal everyday things, but then switch back to the socialized system for prescriptions)
9. Drug Decriminalization. We’re not big drug users or anything, but what I like here is that it means that the cops are more focused on real, like, crime-things, and not stupid things like drugs.
10. Prostitution is legal, pimping is not. Same reason as before – we don’t partake, but I’d rather cops focusing on other things. And love the idea that it’s not only that ”pimping ain’t easy” but also ”pimping ain’t LEGAL.” (though it wasn’t in the song, either, so, well, I digress).
11. Trans people seem pretty well-accepted, gay folks seem pretty well-accepted
12. Abortion is legal (though there have been attempts to change this)
13. Gay Marriage is legal!!!!!
14. Later lifestyle – we’re night-owls already so that works really well for us. It’s not hard to get food at 10:30 or 11:00.

Known Cons (and these are all ’nits’ – livable little annoyances)

1. Cripplingly horrific beauracracy, which is rather infamous. Apparently a lot of other European countries are similar, but theirs is really awful.
2. Some very old buildings, which means some poor insulation, for both sound as well as heat/cold.
3. Some things we’ve grown accustomed to we’re just not going to be able to get there. (No Ranch sauce! We’re San Diegans – how will we be able to cope??!?!)
4. Some things can be really antiquated – like, buying subway tickets in lisbon with a credit card you buy a big long number online and then when you get to the station you have to type it in. You have to pay cash sometimes – not as often as I feared, but more often than I’d want. (Since the pandemic, I’ve gotten into the habit of not taking out cash for months at a time).
5. The worst banking system I’ve seen. Nothing earns interest. You get nickle-and-dimed for everything. I’d do better to just keep my money under a mattress and would get a better return.
6. Some really nasty history – we are talking about the place that literally invented chattel slavery
7. There are gonna be shitty people there, just like there are shitty people here. We saw some graffiti saying ”COVID = NAZISMO = SOCIALISMO”, and we saw an ’anti-vax parade’.

Unexpected Pros

1. In bigger cities, the ’MultiBanco’ system (when they offer it) allows contactless payments, right with your Apple Watch or Phone. It’s really quite nice. And if you do use a credit card, it never leaves your hand – they don’t take it away then bring it back, you can just tap or insert it right there on the mobile terminal your server brings you.
2. Hooking up utilites and stuff – once you’ve got it going you just put in your IBAN number and it just takes the money out – easy-peasy.
3. They actually had a revolution to escape out of dictatorship. That’s pretty cool! (Yes, it’s embarassing that I didn’t know this. Blame American public schools)
4. And this is going to sound like a back-handed compliment, but it isn’t – European Portuguese is hard. But we very much do like a challenge
5. As our Portuguese gets a little better, we’re starting to understand more and more bits of other Romance languages – we were watching something in Romanian and both of us said, ”wait – why did I understand that?!” And something similar for French, and Italian. And the weird thing is that our Spanish is still better than our Portuguese (though not for long!). Basically some weird feature of Portuguese, I guess, that it’s closer to some of the ”Vulgar Latin” roots? I dunno, I read something on Wikipedia or something
6. Because of the way Parliament works, you can get little teeny parties that have different platforms, and if coalition-building happens, maybe they can get some of their stuff pushed through. There’s the Pessoas-Animais-Natureza party (PAN – “People, Animals, Nature”) that’s won a seat or two. The PCP (Partido Comunista Português) also has had a seat here and there (and it’s WEIRD to see sickle-and-hammers around on campaign posters. And it’s weirder to read what they’re saying and say, ”hrm, those are actually some decent ideas…”)
7. Some really amazing laws that we don’t have here. I haven’t vetted all of these but they include:
   1. You can’t take pictures of people without their permission
   2. It’s illegal to discriminate in housing to people with pets (though our real estate person said that sometimes people still do)
   3. Some strong privacy laws – I had to redact my social security number on my visa application because they didn’t want to see it. Same with some transaction details on my US-based bank.
   4. Every public building (school, hospital, prison, etc) must have at least one vegan option.
   5. AirBnB’s are regulated insanely hard – every (legal) one you see will have a completely identical plastic ”Alojamento Local” placard posted conspicuosly, and the licenses for those are hard to get.
8. The anti-vax parade was orderly, and had cops embedded in it to prevent the parade-goers from getting attacked by the regular folks, and vice-versa.
9. They’re pretty smoking-friendly, but that’s actually kinda good for us – as we both vape and most places now treat them the same.
10. Green wine is actually delicious!

Unexpected Cons (that did genuinely suck, but, not deal-breakers)

1. I went to get a haircut and shave and the douchey place I went to wouldn’t fix up Alison’s mohawk (I mean, just shave the sides is all!) because it wouldn’t ”Fit the aesthetic” they were going for. (I think they’re just sexist pieces of shit). Meh, that shit happens everywhere, but Portugal still isn’t some kind of Utopia.
2. The Alfa Pendular – a super-duper sophisticated train that uses ’tilting’ technology to be able to achieve very high speeds on relatively old tracks – can make you motion-sick. I got queasy, Alison got full-on horrifically sick.
3. My wife wants to live in the Algarve, and I’m more interested in Lisbon. SPOUSAL CONFLICT!
4. Driving is difficult, the streets are tiny, and weird little ’bollards’ will pop-up and prevent access to some areas.
5. Absolutely terrifying toilets. Honestly, Alison might start a blog about this once we get there.

So that’s why I want to move there. At this point, for me, it’s still more wanting to move there, rather than get away from here. Though the latter I must admit is true. I still can’t wait til we have us and our animals all safe in our apartment in Lisbon, wake up and walk the girls over near that one tree, then go out to the local cafe to have cafes pingados and some pasteis de nata, then relax for a bit with some portos, then wander around Lisbon for a bit until we go for a nice big lunch at noon, then it’s off to work!

The solution I went with was a simple software-based VPN using AWS Linux instances in either region. I went with IPSec as my encryption/tunneling mechanism, and ISAKMP IKE as my method of sharing keys. I selected Libreswan as my VPN software. I evaluated and discarded several other potential solutions, but this is what I actually got to work for me.

Prerequisites:

1. You need to know some intermediate networking. If you don’t, I don’t think I can help you.
2. You need to have non-overlapping network ranges in either region. If you don’t, you’ll need to evaluate some other kind of solution instead.
3. This isn’t a walkthrough; this is more of an explanation of my process, and I’m going to try and help give troubleshooting steps along the way.
4. You should probably be pretty good at AWS in general already. If not, you’re gonna have a bad time.

Instructions:

Stage 1: Information gathering

You should have a ‘here’ network, and a ‘there’ network. Launch your instances. Allocate an EIP for either side (you wouldn’t want IP addresses to change). Spin up your boxes, and make sure they’re on public-facing subnets. Make sure you can ssh into them. Make sure they can speak ISAKMP IKE (UDP Port 500) to each other. Make sure they can speak IPSec NAT-T (UDP Port 4500) to each other.

Take note of the private network and netmask on the ‘here’ side, and the private network and netmask on the ‘there’ side. Take note of your EIP’s for either side too.

Stage 2: ISAKMP IKE

yum install libreswan

on both boxes. On either one, go to /etc/ipsec.d

Create a file with the name of the other side of the network, with a .conf extension. Put this in there:

conn name-of-connection-that-you-like
  left=my.internal.ip.address
  right=their.EXTERNAL.ip.address
  rightid=their.INTERNAL.ip.address
  leftsubnet=my_entire_private_subnet/CIDR-netmask (e.g. 1.2.3.4/20)
  rightsubnet=their_entire_private_subnet/CIDR-netmask
  authby=secret
  auto=start

Do something similar (but not the same, as you can see) for the other side.

On both sides, create a file in that same directory with name-of-connection-that-you-like.secrets <— yes, that’s plural. Contents should be:

my.internal.ip.address their.internal.ip.address : PSK ""

In between the quotes put an extremely long random string, and make sure it matches on both sides of the network. Here’s an example way to get a nice long string for a PSK –

dd if=/dev/random bs=1 count=33|base64

The reason I don’t show it in there with the string already is I don’t want someone to copy-paste this and then have some stupid password already embedded.

On both sides, make sure to do:

/sbin/chkconfig ipsec on

Okay, now it’s time to see if we screwed something up. Don’t worry, you almost certainly did.

service ipsec start

(Do that on both sides). So, now we should try and see if we can at least have the tunnel up and running. Check out /var/log/secure to see if you see successful-looking things in there. It should say something about ‘connection established’ or something like that. Do a ping from one of your VPN boxes to the inside IP address on the other box. The do the same from the other side. Don’t expect to be able to ping further into the network either way yet. We haven’t done that bit.

If you can ping across both ways, then you have gotten over the worst hump. The rest is much easier. You can skip to stage 2z. But if it still doesn’t work, read on.

Stage 2a: Troubleshooting

So your pings across didn’t work. That sucks, and being in that sucky state would probably be around 75% of the time I spent working on this solution. It’s hard, because you don’t have very good troubleshooting tools.

First off, /var/log/secure – this was the one that clued me in to the rightid setting that I settled on, above. I was getting messages saying “No configuration found for (other-sides-private-ip).” Maybe if you’re lucky, you’ll see something glaring in that log.

Any time you make changes to your config or your secrets file, be on the safe side and do service ipsec restart

I tried a lot of ipsec status as well. That had a lot of gobbledygook, but sometimes gave me some decent clues.

Some other handy tools were ip xfrm help which was quite nice and led me to: ip xfrm monitor gives you a real-time-ish monitoring status of what the various IPSec things are doing.

And remember, all we should be able to do is ping from ‘here’ to the private IP address of ‘there’. Nothing more (yet).

Stage 2z: Your Future Self will thank you

Okay, now comes a sneaky, but important part: reboot both ends. Once they’re back up, make sure you can ping across just as before.

I caught that I had forgotten my chkconfig on both sides by doing this. I also learned about the auto=start setting that I recommend above. It’s better to do this NOW rather than have someone call you up at 5am and have you scrambling to remember how you set up all this VPN stuff months and months ago.

Stage 3: Routing

Okay, you can ping from your gateway across your tunnel, but not any further. Let’s see if we can get some routing going to make your VPN actually useful.

Stage 3a: One-and-a-half hops

From the gateway on this side of the tunnel, we want to ping across the tunnel and to some device on their side of the network.

First, in the AWS Console, there is a setting in EC2 that you will need to change on the far side gateway instance. It’s called “Enable src/dest checks”. Disable those checks; they make sense for normal servers but they won’t work on a router.

Next, on your far-side instance, go into /etc/sysctl.conf and enable routing:

net.ipv4.ip_forward = 1

And also turn this weird setting off:

net.ipv4.conf.default.rp_filter = 0

I don’t know what that second one means, but something told me to turn it off (ipsec verify maybe?) so I did. Safest bet is to reboot the instance after that, but there’s also some way of reloading the sysctl.conf – I just don’t know what that is.

Next, let’s check that you have a reasonable security group set on your far-side gateway. It should already be allowing the various VPN protocols across – and that’s nice. But it also needs to have the ability to talk to whatever far-side instance you want to talk to. Test by ssh’ing into your far-side gateway, and do a ping or a curl or whatever to your far-side instance. It should work. If it does, your security group situation is probably OK.

Last couple of steps now. In the VPC settings in the AWS Console for the far-side region, add a route for the near network, via the far-side gateway. Repeat this for any different routing tables you might have (I have private and public routing tables, for example).

Now, ping across to the far instance. If that works, congrats! It took me a little fiddling – I had missed a security group, and I had missed one of my routing tables. But I eventually got there. Hopefully, you will too. There just aren’t as many moving parts to deal with here, luckily.

Things to keep in mind: if you can ping across the tunnel, and if your far-gateway can ping to your far-instance, then the issue almost has to be routing-related. Either you messed something up in one of your routing tables, or you messed up the far instance and it’s not forwarding correctly.

Also, don’t try and jam everything all into one security group, I think that gets messy. I use one Security Group which allows the VPN protocols, and another security group that grants access to inside-ey instances. A little neater that way, IMHO.

Stage 3b: One and a half hops the other way

Now repeat the same process starting with the Far gateway, to a Near instance. Same rules apply. You should be able to ping, or even curl or whatever other protocol that you want to be able to use across the VPN. You’ll be checking security groups and adding routing tables and so on as before, but this time for the far network onto the near VPC routing tables.

Stage 3c: Two Full Hops

This one is mostly to get you to feel better. Try pinging from your near-instance to your far-instance. It should work. And vice-versa.

CONGRATS, YOU DID IT!

Stage ∞: Advanced

Resiliency

Some stuff to think about – you have an entire VPN hinging on two crappy little AWS boxes. You sure you’re ok with that? I don’t think I am. Consider maybe spinning up another pair of boxes, and having redundant routes in your routing tables. Hopefully the AWS magic will let the remaining route take over for the broken one, if there is a break. But I would test that. Two redundant routes is near-useless if every other packet goes over a dead link.

Routing

Once you have enough point-to-point connections that you’ve set up, trying to maintain routing tables everywhere starts to become a pain in the ass. Consider implementing some routing. I would probably stay away from RIP, even though it’s really easy, because I would fear its routing tables might not converge fast enough. But, it’s also easy – so maybe I would at least give it a try. OSPF is another option, but does have some complexity to it. IS-IS makes you sound like you’re on a watchlist, but is another interior routing protocol. BGP is the big daddy here, but my impression of it was that it was better at large routing tables – like, uh, the Internet.

But I suspect, if you were able to get any of these working, any of them would be OK-ish. One concern I might have is that you can’t influence the AWS Routing tables from your instance; I don’t know exactly how that would work. Would you have to route all traffic down to your own internal router instance? That would suck.

VPC VPN Gateway Service from AWS

Amazon has a VPN Gateway service, but it is really designed to hook the VPN device you have in your office to your Amazon network. Why they would be so strangely short-sighted, I have no idea. But a lot of my experimentation was trying to get this service set up in my ‘spoke’ networks, and then have a central network that is a pseudo-“Customer Gateway”. They support BGP and other stuff there, too. In trying to mess around with this, I had trouble because they really wanted some crazy 169.254.x.y/30 network to be either side of the gateway, and they want two tunnels per VPN Gateway device. Which is weird, because you only get one tunnel back as the ‘customer gateway’ but, whatever. Anyways, there might be something here – and if you go with routing this might be the only way for you to go.

Conclusion

The AWS Marketplace was really, really, really embarrassingly bad when it came to this. The Linux documentation on what tools to use here is a total shambles. There are apparently two competing sets of userland tools – the blah-swan stuff and ipsec-tools. I basically went with the one that had more examples written and better documentation. There was lots of weird stuff on the AWS Linux (which is really CentOS). Like, if you followed their (quite lovely) documentation, you would get some bizarre hard failures, and googling didn’t get me anywhere.

If this is anything like the last time I did all kinds of crazy networking work in AWS, I will end up finding out that Amazon releases “Amazon EZ VPN Interconnect Service” in a few weeks for like 2¢ a month or something. Oh well. Until then, we have the stuff I listed above.

Hope I at least saved you a bit of time, if nothing else.

I know a ton of languages. I know a decent number of frameworks. I have been working with Rails since like v1.0 or something.

I remember when I first started using it; it was amazing. All this stupid boring shit I had to do in PHP to get a database connection, grab results, put them on the screen, make forms around them – everything – could now get done in a ridiculously tiny fraction of the time. It was truly world-changing, and it made total sense that everyone was going nuts over it. As well they should.

Ruby, the language, is a bit slow. So what? Just run more servers. And I agree with that. I mean, if I can make my programmers just 2x as effective and make my server bill 10x as much, it’s still a steal. I have no problem with that math at all.

And there’s much to be said for the language and its elegance. Everything is an object. Classes are just more objects. Class methods and all kinds of other things are great too. Everything you would want to do to a string is a method in the String class. It’s really quite pleasant to read.

But I remember the first time I ran into something weird and magical in Rails that freaked me out. I was trying to troubleshoot why this one field kept outputting as a date, when it was actually a timestamp. I mean, the field’s name happened to be date but that was a legacy issue I just needed to live with. The name of a thing shouldn’t affect what type of thing it is, right? Eventually I renamed the field to fuck_you_rails_you_piece_of_shit and suddenly it began working. Too much magic.

But, still, I was far more productive with Rails than I was anywhere else. I’d looked into PHP frameworks at the time and they did not come remotely close. They seemed to just add complexity and not take anything away.

So I kept holding on to my toolset – Rails for anything with a decent amount of MVC-ish complexity; PHP for little one-or-two-pager apps that I need to bang out real quick. And that served me well.

That being said, Rails began to start to show some real problems for me. Upgrading an app I built from Rails 1 to Rails 2 was a pain in the butt, but I was able to do it once I spent enough time. Upgrading that app to Rails 3 was nearly impossible, and I had to just completely back out of the upgrade once I was around 6 hours in. The documentation was constantly changing. What was once rake console now suddenly became rails console and it’s not like the error messages were very clear about that either. A friend of mine tried to run through a Rails tutorial and had a horrible time of it, so he gave up. He wasn’t the best programmer in the world, but still. Keeping Rails running was also nightmarish, the best solution I had come up with at the time was Phusion Passenger, and that was OK. But getting code deployed was really difficult and generally very unpleasant. I was starting to feel like “Rails makes my developers 5 times more productive!” was still true, but conversely, “Rails makes my operations engineers 5 times less productive!” was also true.

And as Rails grew larger and more popular – and was still dog-slow – it started to have to add even more features around its miserable slowness. Rails got the Asset Pipeline. And support to push everything into CDN’s. Certainly, those are scaling issues that need to be handled, but Rails by default handled it sooner to try and make up some of that lost time. Now we have “Spring,” which means every command I type I have to preface with DISABLE_SPRING=1 or else I get hard-to-debug errors and lose countless amounts of time. Another innovation Rails has had to add is Turbolinks – which I actually think is spectacular technology – but is yet another way to hide or bury the generalized sloweousity of Rails. And the previous version completely shat up my apps; we pulled it pretty quick. (Caveat: the new Turbolinks 5 is excellent, and I’m using it in production and really enjoying it).

At some point, somehow, I got introduced to Laravel. And Laravel is all the good things of Rails and fewer of the bad things.

Yes, it’s written in PHP, and PHP isn’t a “pretty” language. Shut the fuck up, we’re not painting beautiful renaissance canvasses here; we’re putting together some bullshit web app for someone who is giving us money. Yes, there are gotchas in PHP. There are gotchas in Ruby. I spent an hour troubleshooting why some really basic, obvious comparison was failing only to eventually determine that one variable was a Symbol and one was a String. Or I don’t know if you’ve ever tried to use @@ class variables, but they’re confusing, weird, and I’m still not sure I understand them completely. Everything’s at least a little weird, everywhere. And another problem I keep running into (and this may be a community problem, not a language problem) is that everyone is so fucking busy jerking themselves off to how “Readable” and “Elegant” their code is that it slows them down – “Oh, should I write this as do_something! unless @mything.foo? or should I write it with an if…” and then they sit around with their thumbs up their asses trying to be these lovely hand-crafted artisan hipster douchewads. Sometimes too much syntactic sugar can lead to crushing decision-fatigue.

Laravel is fucking fast. I open up a Laravel console (called “Tinker”) on Snipeyhead‘s Snipe-IT project and it pops open in a split-second. If I wanted to be distracted – and believe you me, I really do seem like I want to be – I couldn’t be. It’s just already there. Whatever thing I wanted to dick around with on the console I can just bang right out. And in general everything seems to be fast like that. We don’t have a schema.rb file in Laravel, because you don’t need it; you can rip through a massive directory of migrations in a ridiculously tiny amount of time. So there’s no schema.rb file to get out of sync with your other developers or any other such kind of miserable shitshow.

There’s a lot of cool stuff that saves you time, but it’s not so crazily magical that you can’t figure out what it does. When you want to say that something hasMany of something else, you just create a method named appropriately, and have it return $this->hasMany('That').

Deployment is still harder than a boring old regular PHP app, but that’s mostly due to composer (PHP’s gem/bundler solution) – and I suspect that we gain much more from composer than we lose. I remember trying to grab PEAR libraries, and other ways of tracking down dependencies. Not fun. I can still deploy an app as complex as Snipe-IT in seconds, relative to the multiple-minutes-long-nail-biting that is the Rails apps that I work with.

So far my biggest pet peeve with Laravel is the namespacing. Everything has these god-awful backslashes everywhere. Every file you edit has like a bajillion use statements up front to load up all the namespacery. It’s annoying and ugly. And like in the early days of Rails, I still find myself occasionally digging through the Laravel source to try and figure out how something works, or what it does. But, for me, it still gets the job done much better, much quicker, and much less unpleasant than when I do something in Rails.

Oh, another Laravel peeve – I still don’t fully grok when I need to use ->get() or not. I generally try it one way until it fails, then I try it the other way. And “when to use parentheses on relations” also confuses me. But, generally, I try it one way then the other and it eventually works.

And, as a side note, I suspect you can pull together PHP programmers for way cheaper than Rails folks. So it might actually even pay off!

But that’s not why I’d choose Laravel; I’d choose it because it’s fast, saves me time, makes sense, and generally makes me feel good while I’m using it.

So, here’s how I did it for around $26, using two devices on Amazon:

This gets you from Thunderbolt 3 (via USB-C) to full-size Displayport.

This gets you from full-size Displayport to Mini Displayport.

I could’ve sworn that the USB-C-to-Displayport adapter I picked up listed Displayport Alternate mode and all that jazz, but apparently not. Either way, it seems to work for me (?).

Also wish it supplied power on its own – it doesn’t – but I guess I can live without that. Sucks though.

Here’s something I saw while playing Titanfall 2. First off, and I think this is a significant problem, they have a TON of datacenters, like around 8 or 10 within 100ms of my location. So I’m on at, like 1AM Pacific. There are a couple hundred people active on my closest-location data center (Salt Lake City). There are a couple hundred on in Oregon (GCE1?). A couple hundred more in Oregon-2 (GCE2). But we’re having matching problems, and it’s taking longer than it should to start games, and I keep seeing the same people. Logically, I can see that during peak times, with a crazy-high number of players logged-in, you want everyone to be as close as they can to their own data center, and you want as many damned data centers as you can throw money at to get. But when not? We’re splitting too many players across too many data centers, all for the sake of the difference between 45ms and 65ms. As it got later and later (I am a bit of a night owl) I started to check out other servers, farther away. In many cases, numbers as low as zero. Ugh. There’s a real concern that you might have some people logging in, seeing just a few users, and giving up. That’s enormously dangerous.

This is a really hard problem. Crazy hard. There are a lot of different ways to approach it, and as I was thinking about it, I thought I might’ve come up with a decent algorithm that might work. This is more of a thought experiment, me armchair-quarterbacking a problem that I’m sure the actual devs are already hard-at-work fixing. But anyways, here’s my idea:

• First off, instead of connecting to your closest data-center, you instead connect to one Grand Unified Centralized registration system. First come, first serve. And, yes, if you live 500ms away, your registration time will be 500ms older than someone else. Too bad, that’s life. A “registration consist of just an IP/serial/whatever, a version number, and the number of milliseconds of delay to every datacenter that exists.
• Now the server has a list of registrations, in time order. First, you grab the oldest registration. (Note the time, for keeping track of worst-case registration times). Grab the shortest datacenter from the list of datacenters. Walk through registrations in time order, trying to find (howevery many users you need for a game) users for whom that is also their closest datacenter.
  • Did that work? Then great. Shunt those ‘N’ players off to a game hosted in that data center. Then repeat with the next oldest registration.
  • Let’s say that didn’t work. Now, things get interesting. Of all the registrations you have, find the lowest-ranked first-choice. This, by the way, is the same method as Instant Runoff Voting. For anyone who had chosen that first choice, bump them to their second choice. That may include our candidate registration, itself.
  • Now, repeat the attempt to match up our candidate. Keep eliminating the least-popular data center until you can find a match.

So that’s it. Here are some of the practical advantages and disadvantages of this system, and some weird side-effects:

• If there’s, like, a 10 minute wait (which there oughtn’t ever to be), some people could beat that time by being at the end of the queue, but being in a region that needs players. I think that’s OK.
• I _think_ this might be hard to parallelize. You’d have high contention for the oldest items in the queue; you’d have to have some kind of locking mechanism, or something. This algorithm works best if running in series. Maybe there are some sneaky ways to do it otherwise, but I don’t know how off the top of my head. Maybe grab batches of 1000 registrations, and working that way?
• Super-duper single-point-of-failure. You lose the centralized registration server, game over. Maybe you fall back to the ‘old’ way? Or maybe you’re just down, and that’s it.
• I worry what might happen if you start getting to 10,000,000 players all online at the same time? Being unable to burn through those folks fast enough might become a problem. Maybe once your region gets >1000 users on it, you let them just hit their local region? I don’t know?
• The centralized registration system does have one simple function: match players and bounce them over to their regions. So it has exactly-one job, and I can’t imagine a single ‘matching’ taking more than a few hundred milliseconds or so?
• In terms of scaling and stuff, I would say that you should add capacity to a region when there are no free game servers available (or ‘game slots’). If new servers take a while to spin up, I’d be more liberal than that (maybe using an algorithm of “if there are less than ‘n’ slots available, then add a server”).
• You can go completely nuts spinning up data centers. Until you get into the, like 1000’s or so, this algorithm ought to keep working pretty well.
• When there are only ‘n’ people online, total – and they’re from all around the globe – they may end up having a bad time; high latency. That is life, too bad.
• There may be a point where you say, “Sorry, person, but you’re fucked. There’s no one close enough to you to play a game.” That would be a terrible result, but it could happen. It depends on where the barrier to ‘too many milliseconds’ is.
• I’ve thought about ways to globalize the registration system, with some kinds of distributed databases or whatever. I don’t think the costs are worth it, but, maybe, you could do some kind of distributed database-ey thing. I don’t know.
• The registration data is pretty dang ephemeral. You could maybe do this with something like Redis, though some of the queries we’d be talking about doing might not work there.
• I’d probably have it set up so that if the centralized service blows up and loses all its data, the various game clients will all try and reconnect. Especially if the registration service uses an ephemeral data-store, this could happen.

Of course, there are so many ‘gotchyas’ and caveats and potential failure modes, and all kinds of problems with networking and latency and who-knows-what. I don’t know if this system is harder to implement with those caveats or not.

And I’m just some guy, not a brilliant algorithms person or some distributed programming guru. This could all be a horrible, disastrous mistake of a system. I doubt most armchair quarterbacks actually call spectacular plays when they’re watching their various football games. (Look, sports reference! Yay!)

If you have something that’s simple and always synchronous, don’t use any of them. Just write a dumb function.

If you have a function that’s simple and only needs one asynchronous response – and there are no other potential responses – then a callback is fine.

If you have some kind of object that could have several different potential asynchronous responses – at various different points in lifecycle – and you might or might not want to listen to none, one, or more of them? Then use EventEmitters.

And finally, use Promises when:

• You have a collection of asynchronous functions, and you need to respond only when all of them have returned, or any one of them have returned.
• You’re doing mostly ‘imperative’ functions and don’t need to pass a lot of values around, you just need to chain together some callbacks in sequence.
• You have some functions that might be synchronous, and some that might not be, and you’re not sure which until runtime
• You have a collection of asynchronous events that are all firing, but the order that they must complete in is dependent upon some value determined only at runtime.
• (weaker argument) You are falling victim to callback-hell, and your code is steadily creeping rightward

Long version

Functions

function foo(param1,param2,param3) {
  return "something";
}

//usage:
var a=foo(1,2,3);

If you can do it this way your life will be better. Do it this way if at all possible.

Simple Callbacks

function foo(param1,param2,param3,callback) {
  process.nextTick(function () {
    callback("something");
  });
}

//usage:
foo(1,2,3,function (result) {
  console.warn("Yay, we got result! "+result);
});

If you find you’re passing “callback1, callback2, callback3” definitely don’t do this. But for small, simple asynchronous functions, with not much else going on, this is still fine. Still pretty easy to reason about. As functions grow larger, and nested callbacks grow deeper, it gets harder and harder to reason about, though. The invocations of your little function probably ought not to be more than just a few lines; if they are, you should consider the next option…

EventEmitters

I think 95% of the EventEmitters I create end up being ‘classes’ that extend the EventEmitter class, and I think that’s probably a good way to do it.

   /* ..... */
   this.emit("begin","something");
   /* ..... */
   this.emit("success","something");
   /* ...... */ 
   this.emit("failure","something");
   /* ...... */
   this.emit("complete","something",success === true);

What’s great about this model is that someone who’s consuming this object might only care about one particular event – in which case they can listen for just that one. I believe it’s ok, and actually good, to emit liberally, even events that are similar but not the same (in my example “success”/”failure” as well as “complete”).

Another nice side-effect is that all of your various listen events (.on(foo)) help document what the callback is actually for. E.g. –

on("complete",function (param) {
  /* see? Now we know this event handler fires when things are complete! */
});

If you’re not careful, you can absolutely slide into callback-hell here. But this is my personal favorite pattern to use. It’s pretty extensible.

Never do synchronous callbacks; ever. If you want to do something ‘immediate’ at least wrap it in a process.nextTick(function () {/* blah */}); block; that’ll effectively be immediate but allow for someone to use it in the way most EventEmitters are used.

Never throw errors; just emit “error” instead.

Promises

These are massively over-hyped as the solution to everything. While they are actually very, very cool; they definitely have some real drawbacks.

• They can get hard to debug
• They can be confusing
• missing something like a return – which is super-easy to do – may just cause silent code malfunctioning instead of issuing any kind of error
• Propagating data forward from previously-resolved promises into later promises looks and acts weird.
• You lose a lot of the benefits of ‘closing-over’ variables

But, when used properly – they can turn something nasty like this:

foo.on("bar",function (baz) {
  bif.on("blorgh",function (bling) {
    bloop.on("gloob",function (fweep) {
      /* .... */
    });
  });
});

Into something much prettier like this:

foo.bar().then(function (baz) {
  return "thing";
}).then(function(bling) {
  return "other_thing";
}).then(function (fweep) {
  return "last_thing";
});

Which, especially if you end up with a super-long list, can be helpful.

You can also use .catch() to grab any error in your list of actions – and that can be enormously useful.

Also, if you have an array of promises, you can do something like –

Promise.all(my_array_of_promises).then(function (results) {
  /* do something */
});

Which can be very, very handy.

Where it starts to get ugly is, if in that example I gave above foo.bar().... – if you need to treat an error condition for each of those steps slightly differently. Now you can throw various .catch statements after each .then statement, but I can imagine trying to visually read through that as being a nightmare.

The other huge thing here is that some promises can be fully synchronous – e.g. Promise.resolve(7) – that’s a promise that will resolve to the number 7. And some promises (well, probably most of them) are asynchronous. This is great, and the ability to unify these two modes together can be very helpful.

So, absolutely use them when they make sense. But my current thinking (which might change) is that you should use the simplest asynchronous mechanism that expresses what you need, without adding complexity. Step up the complexities of your tech as you need to, but not before.

Use the right tool for the job.

Why? Security.

Who ran Hillary Clinton’s Email server? What software did it run? Where was it hosted? How physically secure is that location? How secure is the ISP that serves data to that server? The administrators of that server – are they background-checked? Are we sure they are not in the service of any foreign entity, or even a domestic one? Is the server patched to the latest version? Are there any vulnerabilities on the version it is running? Are there any backdoors or rootkits or other such stuff on there? What’s the update schedule on that server? Is there any strong cryptography on it? What protocols are running?

And the answer in just about all of these cases is, “I have absolutely no idea.” And that is completely and totally unacceptable.

If you administer an email server, YOU CAN (usually) READ ANY EMAIL THAT IS ON IT. When it’s some regular schmoe somewhere, that’s less of a big deal (but still something to be concerned about). When it’s the Secretary of State of the United States of America, that’s a bit of a bigger deal.

If you are reading this and feeling like I’m not exactly right, go talk to your email admin. Ask them to read you your latest email – “just so you can see if your email software is working.” Let me know what you find.

I got it down to something simple and clean, finally. And it turns out the “recipes” section on the Gulp website pretty much had the answer already. I just needed to understand more about what I was trying to do in the first place.

Note: You need to install coffeeify to have it available as a transform!

var gulp        = require('gulp');
var browserify  = require('browserify');
var source      = require('vinyl-source-stream'); //to 'rename' your resulting file
var uglify      = require('gulp-uglify');
var buffer      = require('vinyl-buffer'); // to transform the browserify results into a 'stream'

var sourcemaps  = require('gulp-sourcemaps');

gulp.task("your_target",function() {
  browserify({
      entries: ["./sample.coffee"],
      debug: true,
      extensions: [".coffee"],
      transform: ["coffeeify"] // npm install --save-dev coffeeify
      })
    .bundle()
    .pipe(source('resulting_file_name.js'))
    .pipe(buffer())
    .pipe(sourcemaps.init({loadMaps: true,debug: true}))
    .pipe(uglify(/* {
          debug: true,
          options: {
            sourceMap: true,
          }
      }*/)).pipe(sourcemaps.write("./" /* optional second param here */))
    .pipe(gulp.dest('./build/'));

});

If you don’t want the sourcemap URL in your resulting JS file (which I didn’t, because I keep my sourcemap file private), add a second parameter ,{addComment: false} to the sourcemaps.write line near the end.

Edit: – the parameter to uglify isn’t even needed.