tag:blogger.com,1999:blog-65412664826185301582024-03-13T23:18:49.404-07:00The Life of KennethKenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comBlogger558125tag:blogger.com,1999:blog-6541266482618530158.post-28083917737245527042023-05-09T10:47:00.009-07:002023-05-09T19:48:23.587-07:00Building the Micro Mirror Free Software CDN<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlepHrsH5tLDDaw8Dj_kgotVdqcZyASLP4sGjPhkeTDQZAARnDhZf4aA8x4Jfv0a_KK2xyG0FBqv47EgjQHLKFSNnJKfjLF5SQua8YxkL-hurPSPV3SPLNG-YKap1W4CU-CMH5MlilRYVz2l-ezyTslQyJbM6uT88YH43-wfZ_C8dsGqAw3aQzERHJag/s4032/codingflyboy.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlepHrsH5tLDDaw8Dj_kgotVdqcZyASLP4sGjPhkeTDQZAARnDhZf4aA8x4Jfv0a_KK2xyG0FBqv47EgjQHLKFSNnJKfjLF5SQua8YxkL-hurPSPV3SPLNG-YKap1W4CU-CMH5MlilRYVz2l-ezyTslQyJbM6uT88YH43-wfZ_C8dsGqAw3aQzERHJag/w640-h480/codingflyboy.jpg" width="640" /></a></div><br /><div>As should surprise no one, based on my past projects of <a href="https://blog.thelifeofkenneth.com/2017/11/creating-autonomous-system-for-fun-and.html">running my own autonomous system</a>, building <a href="https://blog.thelifeofkenneth.com/2018/04/creating-internet-exchange-for-even.html">my own Internet Exchange Point</a>, and <a href="https://blog.thelifeofkenneth.com/2020/11/building-anycast-secondary-dns-service.html">building a global anycast DNS service</a>, I kind of enjoy building Internet infrastructure to make other people's experience online better. So that happened again, and like usual, this project got well out of hand. </div><div><br /></div><h2 style="text-align: left;">Linux updates</h2><div>You run apt update or dnf upgrade and your Linux system goes off and download updates from your distro and installs them. Most people think nothing of it, but serving all of those files to every Linux install in the world is a challenging problem, and it's made even harder because most Linux distributions are free and thus don't have a project budget to spin up a global CDN (Content Distribution Network) to have dozens or hundreds of servers dedicated to putting bits on the wire for clients over the Internet.</div><div><br /></div><div>How Linux distros get around this budget issue is that they host a single "golden" copy of all of their project files (Install media ISOs, packages, repository index files, etc) and volunteers around the world who are blessed with surplus bandwidth download a copy of the whole project directory, make it available from their own web server that they build and maintain themselves, and then register their mirror of the content back with the project. Each free software project then has a load balancer that directs clients to nearby mirrors of the content they're requesting while making sure that the volunteer mirrors are still online and up to date.</div><div><br /></div><div>At the beginning of 2022, one of my friends (John Hawley) and I were discussing the fact that the network who used to be operating a Linux mirror in the same datacenter as us had moved out of the building, and maybe it would be fun to build our own mirror to replace it.</div><div><br /></div><div>John: "Yeah... it would probably be fun to get back into mirroring since I used to run the mirrors.kernel.org mirrors" (world's largest and most prominent Linux mirror)</div><div>Me: "Wait... WHAT?!"</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieQqmbrVb-Y_kjeDZG4pxgCRR4ftUI41BLgtovIOcN5TwmQM-VTOaxVEYYdh-3X3w0brgCbTXFZ8kOQWB7faJkyJeMbxSHd-_IAgM5863W0RCl7uwzUS7LiYqMYjOjxpgm7tXV-On3XZ5Ko72dIypQRkZQXedntBbKe3U3WkAs7bb1pzxK7wsllqnjBQ/s4032/PXL_20220419_034344692.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieQqmbrVb-Y_kjeDZG4pxgCRR4ftUI41BLgtovIOcN5TwmQM-VTOaxVEYYdh-3X3w0brgCbTXFZ8kOQWB7faJkyJeMbxSHd-_IAgM5863W0RCl7uwzUS7LiYqMYjOjxpgm7tXV-On3XZ5Ko72dIypQRkZQXedntBbKe3U3WkAs7bb1pzxK7wsllqnjBQ/w640-h480/PXL_20220419_034344692.jpg" width="640" /></a></div><br /><div>So long story short, the two of us pooled our money together, and went and dropped $4500 on a SuperMicro chassis, stuffed it full of RAM (384GB) and hard drives (6x 16TB) and racked it below the Google Global Cache I'm hosting in my rack in Fremont.</div><div><br /></div><div>Like usual, I was posting about this as it was happening on Twitter (RIP) and several people on Twitter expressed interest in contributing to the project, so I posted a paypal link, and we came up with the offer that if you donated $320 to the project, you'd get your name on one of the hard drives inside the chassis in Fremont, since that's how much we were paying for each of the 16TB drives.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitJY0RUVofDd5JcUIRgPiWk5oBtR6FJJfvnnRpirY8-O8vurh936bv4nbu74av1K6t2x_1cR_bXK-FZzhDfSlwXHYEPVRDlMjRe790VDV3TUKGPwcrjsb29YIIFe0SO9Qpc98k-ku3ToyvjCLJL9lPUrFST-RLytKiDNO_C9KrGj_IWq9-9OPHC3G3Ow/s4032/PXL_20220409_032242219.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitJY0RUVofDd5JcUIRgPiWk5oBtR6FJJfvnnRpirY8-O8vurh936bv4nbu74av1K6t2x_1cR_bXK-FZzhDfSlwXHYEPVRDlMjRe790VDV3TUKGPwcrjsb29YIIFe0SO9Qpc98k-ku3ToyvjCLJL9lPUrFST-RLytKiDNO_C9KrGj_IWq9-9OPHC3G3Ow/w640-h480/PXL_20220409_032242219.jpg" width="640" /></a></div><br /><div>This "hard drive sponsor" tier also spawned what I think was one of the most hilarious conversations of this whole project, where one of my friends was trying to grasp why people were donating money to get their name on a piece of label tape, stuck to a hard drive, inside a server, inside a locked rack, inside of a data center, where there most certainly was no possibility of anyone ever actually seeing their name on the hard drive. A rather avant-garde concept, I will admit.</div><div><br /></div><div>The wild part was that we "sold out" on "Hard Drive Sponsor" tier donors, and enough people contributed to the project that we covered almost all of the hardware cost of the original mirror.fcix.net server!</div><div><br /></div><div>So long story short, we decided to spin up a Linux mirror, fifty of my friends on Twitter chipped in on the project, and we were off to the races trying to load 50TB of Linux distro and free software artifacts on the server to get it up and running. All well and good, and a very typical Linux mirror story.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhPTJN-jTOTAp1ZmiaQDzsfYlDAvxLfy67fbmPaNEx3YxjPkVrJ5Fw-Mk0rQ2F-RQNXNzLw-FiF-HwIYQzrgR3PqZRYrx1aMd97VOpeHfVuffz4gyYySwvJyX8eTInMbsDZag1BmBAgotkLfGkMzYzgOf0XLmDuuuFZ-7uGDWN2EBlxvWRzPk6u1Dm-Mw" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="428" data-original-width="1268" height="216" src="https://blogger.googleusercontent.com/img/a/AVvXsEhPTJN-jTOTAp1ZmiaQDzsfYlDAvxLfy67fbmPaNEx3YxjPkVrJ5Fw-Mk0rQ2F-RQNXNzLw-FiF-HwIYQzrgR3PqZRYrx1aMd97VOpeHfVuffz4gyYySwvJyX8eTInMbsDZag1BmBAgotkLfGkMzYzgOf0XLmDuuuFZ-7uGDWN2EBlxvWRzPk6u1Dm-Mw=w640-h216" width="640" /></a></div><br />Where things started to get out of hand is when John started building a Grafana dashboard to parse all of the Nginx logs coming out of our shiny new Linux mirror and analyzing the data as to how much of what projects we were actually serving. Pivoting the data by various metrics like project and release and file type, we came to the realization that while we were hosting 50TB worth of files for various projects, more than two thirds of our network traffic was coming from a very limited number of projects and only about 3TB of files on disk! And this is where the idea of the Micro Mirror began to take shape.</div><div><br /></div><h2 style="text-align: left;">The Micro Mirror Thesis</h2><div>If the majority of the network traffic on a Linux mirror is coming from a small slice of the assets hosted on the mirror, then it should be possible to build a very small and focused mirror that only hosts projects from that "hot working set" subset and while less effective than our full sized mirror, could be only half as effective as our full size mirror at 10% of the cost.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhsu6OWe0G0mvI4BKocs7VTKU55Lmio867IFYifNPuH0_LyXxrWIazlhOsGIiZ2ceaHQHvHbKyI1ncYAU7KYEw_maVxE2Pa-vqQiIOO1kH0XJPwu74f-i3y6yhUDSZQ4RDhRbZLQjQWehtPXoQOKQUJWs5rQ3sVAiuNH0_J3hRVyLjDTcyZb8RlRNRKw/s4032/codingflyboy.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhsu6OWe0G0mvI4BKocs7VTKU55Lmio867IFYifNPuH0_LyXxrWIazlhOsGIiZ2ceaHQHvHbKyI1ncYAU7KYEw_maVxE2Pa-vqQiIOO1kH0XJPwu74f-i3y6yhUDSZQ4RDhRbZLQjQWehtPXoQOKQUJWs5rQ3sVAiuNH0_J3hRVyLjDTcyZb8RlRNRKw/w640-h480/codingflyboy.jpg" width="640" /></a></div><div><br /></div><div>So we set ourselves the challenge of trying to design a tiny Linux mirror which could pump out a few TB of traffic a day (as opposed to the 12-15TB/day of traffic served from mirror.fcix.net) with a hardware cost less than the $320 that we spent on one of the hard drives in the main storage array. Thanks to eBay and my love for last gen enterprise thin clients, we settled on a design consisting of the following:</div><div><ul style="text-align: left;"><li><a href="https://www.parkytowers.me.uk/thin/hp/t620/">HP T620 thin client</a></li><li><a href="https://amzn.to/44EAw0R">2x4GB DIMMs</a></li><li><a href="https://amzn.to/3nNTOjF">2TB M.2 SSD</a></li></ul><div>This could all be had for less than $250 on eBay used, and conveniently fits nicely in a medium flat rate USPS box, so once we build it and find a random network in the US willing to plug this thing in for us, we can just drop it in the mail. </div></div><div><br /></div><div>We built the prototype and one of my other friends in Fremont offered to host it for us, since we're only using the 1G-baseT NIC on-board the thin client, and we were off to the races. Setting the tiny mirror up only hosting Ubuntu ISOs, Extra Packages for Enterprise Linux, and the CentOS repo for servers easily exceeded our design objective of >1TB/day of network traffic. Not a replacement for traditional "heavy iron" mirrors that can host a longer tail of projects, but this is 1TB of network traffic which we were able to peel off of those bigger mirrors so they could spend their resources serving the less popular content, which we wouldn't be able to fit on the single 2TB SSD inside this box.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-tFyoCzQPaYfG8jbogz3mUjkftkAufNYM4PbSB4uMOEdAFtx8lsG_xlKDIVwC-SbgTVQdWClxyRjJYHgc4Txt8elKx4J8VMtnyqsYcKwR5unBX4gs61g_xJMNYz7cADF6y2WV7fcHquP_PTe9NVF79-uLXojNe5NCbpNMY3qyKLfaRmdUgpzkVgvB2Q/s4032/micromirror-coresite-censor.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-tFyoCzQPaYfG8jbogz3mUjkftkAufNYM4PbSB4uMOEdAFtx8lsG_xlKDIVwC-SbgTVQdWClxyRjJYHgc4Txt8elKx4J8VMtnyqsYcKwR5unBX4gs61g_xJMNYz7cADF6y2WV7fcHquP_PTe9NVF79-uLXojNe5NCbpNMY3qyKLfaRmdUgpzkVgvB2Q/w640-h480/micromirror-coresite-censor.jpg" width="640" /></a></div><br /><div>Now it just became a question of "well, if one Micro Mirror was pretty successful, exactly how many MORE of these little guys could we stamp out and find homes for???"</div><div><br /></div><div>These Micro Mirrors have several very attractive features to them for the hosting network:</div><div><ul style="text-align: left;"><li>They are fully managed by us, so while many networks / service providers <i>want </i>to contribute back to the free software community, they don't have the spare engineering resources required to build and manage their own mirror server. So this fully managed appliance makes it possible for them to contribute their network bandwidth at no manpower cost.</li><li>They're very small and can fit just about anywhere inside a hosting network's rack.</li><li>They're low power (15W)</li><li>They're fault tolerant, since each project's load balancer performs health checks on the mirrors and if this mirror or the hosting network has an outage the load balancers will simply not send clients to our mirror until we get around to fixing the issue.</li></ul><div>Then it was just a question of scaling the idea up. Kickstart file so I can take the raw hardware and perform a completely hands-off provisioning of the server. <a href="https://github.com/PhirePhly/micromirrors">Ansible playbook</a> to take a short config file per node and fully provision the HTML header, project update scripts, and rsync config per server, and suddenly I can fully stamp out a new Micro Mirror server with less than 30 minutes worth of total work.</div></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw_1KGPgZ96D8L2VzUkM1qPjX4PqoCqkeuDPSgW3Ua82yENff2-sCagKDBq5hP8aTKw-XE6n6-1mdrFiRK-_DexxGK7cFim3phfivtyrGg6y3Ybs3i1x3yXrCffroFwSxTPh3cKTlfToGEoTHEt4TDNIOPiFoH7OWLEpKowXM9p0n3YmkwuTqGfklBOg/s4032/OhioIX_narrow.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw_1KGPgZ96D8L2VzUkM1qPjX4PqoCqkeuDPSgW3Ua82yENff2-sCagKDBq5hP8aTKw-XE6n6-1mdrFiRK-_DexxGK7cFim3phfivtyrGg6y3Ybs3i1x3yXrCffroFwSxTPh3cKTlfToGEoTHEt4TDNIOPiFoH7OWLEpKowXM9p0n3YmkwuTqGfklBOg/w640-h480/OhioIX_narrow.jpg" width="640" /></a></div><br /><div>Finding networks willing to host nodes turned out to be extremely easy. Between Twitter, Mastodon, and a few select Slack channels I hang out on, I was able to easily build a waiting list of hosts that surpassed the inventory of thin clients I had laying around. Then we just needed to figure out how to fund more hardware beyond what we were personally willing to buy. Enter <a href="https://liberapay.com/phirephly/">LiberaPay</a>, an open source service similar to Patreon where people can pledge donations to us to keep funding this long term.</div><div><br /></div><div>So now we have a continual (albeit very small) funding source, and a list of networks waiting for hardware, and it's mainly been a matter of waiting for enough donations to come in to fund another node, ordering the parts, provisioning the server, dropping it in the mail, and waiting for the hosting network to plug it in so we can run our Ansible playbook against it and get it registered with all the relevant projects.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTkWivV1vyOVEFVthjPUi0UW_Flr8a8cJ1wCvlBegMQENtsWyW5UENJJ2UGs7ewEJAQIP92PmfrvFLNLtxUmvOQcOrdbJq3Wcheo32cgFo9fzkuCEGv43p_0U0uGSccCWib4rOZYq07C-cgmxoIpsDw0z7aENeUKAgGtgvMde8XCYPz9Ia3uJ-LA91zQ/s2048/lesnet-moose.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1536" data-original-width="2048" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTkWivV1vyOVEFVthjPUi0UW_Flr8a8cJ1wCvlBegMQENtsWyW5UENJJ2UGs7ewEJAQIP92PmfrvFLNLtxUmvOQcOrdbJq3Wcheo32cgFo9fzkuCEGv43p_0U0uGSccCWib4rOZYq07C-cgmxoIpsDw0z7aENeUKAgGtgvMde8XCYPz9Ia3uJ-LA91zQ/w640-h480/lesnet-moose.jpg" width="640" /></a></div><br /><div>So now we had a solid pipeline set up, and we could start playing around with other hardware designs than the HP T620 thin client. The RTL8168 NIC on the T620s are far from ideal for pumping out a lot of traffic, and we actually got feedback from several hosting networks that they just don't have the ability to plug in baseT servers anymore, and they'd much prefer a 10Gbps SFP+ NIC handoff to the appliance. </div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjgcSr4pwjSgFRBZ1paOqILEyKEpjR5BRv-U4CiHpRbxobEVw45rNBsDCC0xEOzxinGEZU8fOVrwDSxjTAolEKjJ_K5EKQfkme_oa4A0J6_4B3cF-zCVfkhTTaR89HvSXFU7NCR_IaSfBdJWfWvpOBbQV146jMR7zNGAjoa7-s_gAm027dcSILuVxiTpA" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="1536" data-original-width="2048" height="480" src="https://blogger.googleusercontent.com/img/a/AVvXsEjgcSr4pwjSgFRBZ1paOqILEyKEpjR5BRv-U4CiHpRbxobEVw45rNBsDCC0xEOzxinGEZU8fOVrwDSxjTAolEKjJ_K5EKQfkme_oa4A0J6_4B3cF-zCVfkhTTaR89HvSXFU7NCR_IaSfBdJWfWvpOBbQV146jMR7zNGAjoa7-s_gAm027dcSILuVxiTpA=w640-h480" width="640" /></a></div><div><br /></div>The desire for 10G handoffs has been a bit of a challenge while still trying to stay within the $320 hardware budget goal we set for ourselves, but we have been doing some experiments with the HP T620 Plus thin client, which happens to have a PCIe slot that fits a Mellanox ConnectX3 NIC, and we also received a very generous donation of a pile of Dell R220 servers with 10G NICs from Arista Networks (Thanks Arista!)</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf3tH0j0byVIaOxQceGGS28yc9XVFGeX2RQMx2EO5IBajcQU91S0rYr3fB79rL8VehUvNQvsa3fKggL703p_IEM57T8f5WGqjp6kHenIzgHmCPfJCy-KYbkHgN_44kXmVsdOASH1aweB0hd-ocD8hyo0TNPAHXg1iBNk79TNeu3bcUzzcwaBphm0T5sA/s4032/PXL_20230428_215059412.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf3tH0j0byVIaOxQceGGS28yc9XVFGeX2RQMx2EO5IBajcQU91S0rYr3fB79rL8VehUvNQvsa3fKggL703p_IEM57T8f5WGqjp6kHenIzgHmCPfJCy-KYbkHgN_44kXmVsdOASH1aweB0hd-ocD8hyo0TNPAHXg1iBNk79TNeu3bcUzzcwaBphm0T5sA/w640-h480/PXL_20230428_215059412.jpg" width="640" /></a></div><br /><div>So now the project has very easily gotten out of hand. We have more than 25 Micro Mirror nodes of various permutations live in production, spanning not only North America but several of the nodes have been deployed internationally. Daily we serve roughly 60-90TB of Linux and other free software updates from these Micro Mirrors, with more than 150Gbps of port capacity. So while not making a profound difference to user experience downloading updates, each Micro Mirror we deploy has helped make a small incremental improvement in how fast users are able to download updates and new software releases.</div><div><br /></div><div>So if you've started noticing a bunch of *.mm.fcix.net mirrors for your favorite project, this is why. We hit a sweet spot with this managed appliance and have been stamping them out as resources permit.</div><div><br /></div><h2 style="text-align: left;">Interest in Helping?</h2><div><br /></div><div>The two major ways that someone can help us with this project is funding the new hardware and providing us locations to host the Micro Mirrors:</div><div><ul style="text-align: left;"><li>Cash contributions are best sent via my <a href="https://liberapay.com/phirephly/">LiberaPay account</a>.</li><li>Any service providers interested in hosting nodes in their data center network can reach out to <a href="mailto:mirror@fcix.net">mirror@fcix.net</a> to contact us and get on our wait list.</li></ul><div>We are not interested in deploying these nodes off of any residential ISP connections, so even if you have a 1Gbps Internet connection from your ISP, we want to limit the deployment of these servers to wholesale transit contexts in data centers where we can work directly with the ISP's NOC.</div></div><div><br /></div><div>Of course, nothing is preventing anyone from going out and setting up your own Linux mirror. Ultimately having more mirror admins out there running their own mirrors is better than growing this Micro Mirror project for the sake of diversity. If you're looking to spin up your own mirror and have any specific questions on the process, feel free to reach out to us for that as well.</div><div><br /></div><div>I also regularly <a href="https://social.afront.org/tags/MicroMirror">post about this project on Mastodon</a>, if you want to follow along real time.</div>Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-14006373403126386632021-05-05T10:00:00.002-07:002021-05-05T11:07:11.516-07:00Unlocking Third Party Transceivers on Older Arista Switches<p> Like many switch vendors, Arista switches running EOS only accept Arista branded optics by default. They accept any brand of passive direct attach copper cables, but since third party optics tend to cause a prohibitive number of support cases, it's needed to force the user to make a deliberate decision to enable third party optics so they can appreciate that they're venturing out into unsupported territory. </p><p>The thing is, if you're out buying an older generation of Arista switch on eBay to try and learn EOS, you're probably also not able to directly buy Arista branded optics, so the unlock procedure on older switches would be of interest to you. </p><p>EOS has two different methods for unlocking third party transceivers, depending on what hardware you're trying to unlock. It's not dependent on what version of EOS you're running on the switch, so all switches which originally supported the old "magic file" unlock method always supported the old method, and newer switches have always required the customer unlock code. </p><p></p><ol style="text-align: left;"><li>The "magic file" method is used on the earlier switches, and depends on EOS checking for a file named "enable3px" in the flash: partition. There doesn't need to be anything in this file; it just needs to exist, so an easy way to create this file from the EOS command line is "bash touch /mnt/flash/enable3px"</li><li>The newer "customer unlock code" method instead relies on each customer having a conversation with their account team to get a cryptographic key for their specific account, which is then visible in their running-config as a "service unsupported-transceiver CUSTOMERNAME LICENSEKEY" line</li></ol><div>Once you apply one of these two unlock methods, save your running config and reload the switch to have it unlock all your transceivers and accept all future optics you install. </div><div><br /></div><div>So if you're looking at a switch on the second hand market and trying to figure out if you can unlock it using the old magic file method, here is a list of the switches which predate the newer customer license key method and work with the empty "enable3px" file:</div><div><ul style="text-align: left;"><li>DCS-7120T-4S</li><ul><li>20x 10GbaseT</li><li>4x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7140T-8S</li><ul><li>40x 10GbaseT</li><li>8x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7048T-4S</li><ul><li>48x 1GbaseT</li><li>4x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7048T-A</li><ul><li>48x 1GbaseT</li><li>4x SFP+</li><li>Last EOS version 4.15.10M</li></ul><li>DCS-7124S</li><ul><li>24x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7124SX</li><ul><li>24x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7124FX</li><ul><li>24x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7148S</li><ul><li>48x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7148SX</li><ul><li>48x SFP+</li><li>Last EOS version 4.13.16M</li></ul><li>DCS-7050T-36</li><ul><li>32x 10GbaseT</li><li>4x SFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7050T-52</li><ul><li>48x 10GbaseT</li><li>4x SFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7050T-64</li><ul><li>48x 10GbaseT</li><li>4x QSFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7050S-52</li><ul><li>52x SFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7050S-64</li><ul><li>48x SFP+</li><li>4x QSFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7050Q-16</li><ul><li>8x SFP+</li><li>16x QSFP+</li><li>Last EOS version 4.18.11M-2G</li></ul><li>DCS-7150S-24</li><ul><li>24x SFP+</li><li>Last EOS version 4.23.x-2G (still an active train)</li></ul><li>DCS-7150S-52</li><ul><li>52x SFP+</li><li>Last EOS version 4.23.x-2G (still an active train)</li></ul><li>DCS-7150S-64</li><ul><li>48x SFP+</li><li>4x QSFP+</li><li>Last EOS version 4.23.x-2G (still an active train)</li></ul><li>7548S-LC line cards on an original 7504/7508</li><ul><li>48x SFP+ line cards</li><li>Last EOS version 4.15.10M</li><li>None of the E series or R series line cards support the magic file method</li></ul></ul><div>So hopefully you find that list helpful.</div></div><div><br /></div><div>If you're looking to unlock transceivers on newer switches, I'm afraid I'm not able to help there. You may try filling out the "contact sales" form on the Arista website and have a conversation with your account rep about what you're trying to do. Don't contact TAC or support; they're never authorized to hand out unlock keys and will only refer you to your account team. </div><p></p>Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-63316895670398152222020-11-18T21:00:00.001-08:002020-11-18T21:25:42.399-08:00Building an Anycast Secondary DNS Service<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5QlVZH1CwbE5k8dB0KlJJ-wokB0bIP-V9Qo1AmKu9QcYeBF0-knzfun6qTUvhan1KrBpBl7NTo8F5ozE9pSYodg1_OFt1K7o0KrrCBHfMgHh13cYSXlk93IjK-BnrvKD_7iMpi4zsDvJq/s854/V4_20201108.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="410" data-original-width="854" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5QlVZH1CwbE5k8dB0KlJJ-wokB0bIP-V9Qo1AmKu9QcYeBF0-knzfun6qTUvhan1KrBpBl7NTo8F5ozE9pSYodg1_OFt1K7o0KrrCBHfMgHh13cYSXlk93IjK-BnrvKD_7iMpi4zsDvJq/w640-h308/V4_20201108.png" width="640" /></a></div>Regular readers will remember how one of my friends offered to hold my beer while I went and <a href="https://blog.thelifeofkenneth.com/2017/11/creating-autonomous-system-for-fun-and.html">started my own autonomous system</a>, and then the two of us and some of our friends thought it would be funny to <a href="https://blog.thelifeofkenneth.com/2018/04/creating-internet-exchange-for-even.html">start an Internet Exchange</a>, and <a href="https://fcix.net/">that joke got wildly out of hand</a>... Well, my lovely friend Javier has done it again.<p></p><p>About 18 months ago, Javier and I were hanging out having dinner, and he idly observed in my direction that he had built a DNS service back in 2009 that was still <i>mostly</i> running, but it had fallen into some disrepair and needed a bit of a refresh. Javier is really good at giving me fun little projects to work on... that turn out to be a lot of work. But I think that's what hobbies are supposed to look like? Maybe? So here we are.</p><p>With Javier's help, I went and built what I'm calling "Version Two" of Javier's <a href="https://ns-global.zone/">NS-Global DNS Service</a>.</p><h2 style="text-align: left;">Understanding Secondary DNS</h2><p style="text-align: left;">Before we dig into NS-Global, let's talk about DNS for a few minutes. Most people's interaction with DNS is that they configure a DNS resolver like 1.1.1.1 or 9.9.9.9 on their network, and everything starts working better, because DNS is really important and a keystone to the rest of the Internet working.</p><div>The thing is, DNS resolvers are just the last piece in the chain of DNS. From the user's perspective, you fire off a question to your DNS resolver, like "what is the IP address for twitter.com?" and the resolver spends some time thinking about it, and eventually comes back with an answer for what IP address your computer should talk to for you to continue your doom scrolling. </div><p style="text-align: left;">And these resolving DNS services aren't what we built. Those are really kind of a pain to manage, and Cloudflare will always do a better job than us at marketing since they were able to get a pretty memorable IP address for it. But the resolving DNS services need somewhere to go to get answers for their user's questions, and that's where authoritative DNS servers come in.</p><p style="text-align: left;">When you ask a recursive DNS resolver for a resource record (like "A/IPv4 address for www.twitter.com") they very well may know nothing about Twitter, and need to start asking around for answers. The first place the resolver will go is one of the 13 root DNS servers (<a href="https://blog.thelifeofkenneth.com/2018/06/peering-with-root-dns-servers.html">which I've talked about before</a>), and ask them "Hey, do you know the A record for www.twitter.com?", and the answer back from the root DNS server is going to be "No, I don't know, but the name servers for <b>com</b> are over there". And that's all the root servers can do to help you; point you in the right direction for the top level domain name servers. You then ask one of the name servers for com (which happen to be run by Verisign) if <i>they</i> know the answer to your question, and they'll say no too, but they will know where the name servers for twitter.com are, so they'll point you in that direction and you can try asking the twitter.com name servers about IN/A,www.twitter.com, and they'll probably come back with the answer you were looking for the whole time. </p><p style="text-align: left;">So recursive DNS resolvers start at the 13 root servers which can give you an answer for the very last piece of your query, and you work your down the DNS hierarchy moving right to left in the hostname until you finally reach an authoritative DNS server with the answer you seek. And this is really an amazing system that is so scalable, because a resolver only critically depends on knowing the IP addresses for a few root servers, and can bootstrap itself into the practically infinitely large database that is DNS from there. Granted, needing to go back to the root servers to start every query would suck, so resolvers will also have a local cache for answers, so once one query yields the NS records for "com" from a root, those will sit in the local cache for a while and save the trip back to the root servers for every following query for anything ending in ".com".</p><p style="text-align: left;">But we didn't build a resolver. We build an authoritative DNS service, which sits down at the bottom of the hierarchy with a little database of answers (called a zone file) waiting for some higher level DNS server to refer a resolver to us for a query only we can answer.</p><p style="text-align: left;">Most people's experience with authoritative DNS is that when they register their website with Google Domains or GoDaddy, they have some kind of web interface where you can enter IP addresses for your web server, configure where your email for that domain gets sent, etc, and that's fine. They run authoritative DNS servers that you can use, but for various reasons, people may opt to not use their registrar's DNS servers, and instead just tell the registrar "I don't want to use your DNS service, but put these NS records in the top level domain zone file for me so people know where to look for their questions about my domain"</p><h2 style="text-align: left;">Primary vs Secondary Authoritative</h2><p style="text-align: left;">So authoritative DNS servers locally store their own little chunk of the enormous DNS database, and every other DNS server which sends them a query either gets the answer out of the local database, or a referral to a lower level DNS server, or some kind of error message. Just what the network admin for a domain name is looking for!</p><p style="text-align: left;">But DNS is also clever in the fact that it has the concept of a primary authoritative server and secondary authoritative servers. From the resolver's perspective, they're all the same, but if you only had one DNS server with the zone file for your domain and it went offline, that's the end of anyone in the whole world being able to find any IP addresses or other resource records for anything in your domain, so you really probably want to recruit someone else to also host your zone to keep it available while you're having some technical difficulties. You could stand up a second server and make the same edits to your zone database on both, but that'd also be a pain, so DNS even supports the concept of an authoritative transfer (AXFR) which copies the database from one server to another to be available in multiple places.</p><p style="text-align: left;">This secondary service is the only thing that NS-Global provides; you create your zone file however you like, hosted on your own DNS server somewhere else, and then you allow us to use the DNS AXFR protocol to transfer your whole zone file into NS-Global, and we answer questions about your zone from the file if anyone asks us. Then you can go back to your DNS registrar and add us to your list of name servers for your zone, and the availability/performance of your zone improves by the fact that it's no longer depending on your single DNS server.</p><h2 style="text-align: left;">The Performance of Anycast</h2><p style="text-align: left;">Building a secondary DNS service where we run a server somewhere, and then act as a second DNS server for zones is already pretty cool, but not <i>stupid</i> enough to satisfy my usual readers, and this is where <a href="https://en.wikipedia.org/wiki/Anycast">anycast</a> comes in.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWtyGFj-oLq5onhJBrB3Mz9hXEwvOFj_1M_b_p8S6FiUVtkTLkB5LOSYl4Yl9QDaJ39fm8esdXD_y8-fUDx62jJVUq5sR1p9KSJ7g3YSDTU-rbMHAUwyG6JD1cpersglLyNtomdMi93Mni/s1078/RipeAtlas-SinglePOP-v4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="516" data-original-width="1078" height="306" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWtyGFj-oLq5onhJBrB3Mz9hXEwvOFj_1M_b_p8S6FiUVtkTLkB5LOSYl4Yl9QDaJ39fm8esdXD_y8-fUDx62jJVUq5sR1p9KSJ7g3YSDTU-rbMHAUwyG6JD1cpersglLyNtomdMi93Mni/w640-h306/RipeAtlas-SinglePOP-v4.png" width="640" /></a></div>Anycast overcomes three problems with us running NS-Global as just one server in Fremont:<div><ol style="text-align: left;"><li>The speed of light is a whole hell of a lot slower than we'd really like</li><li>Our network in Fremont might go offline for some reason</li><li>With enough users (or someone attacking our DNS server) we might get more DNS queries than we can answer from a single server</li></ol><p style="text-align: left;">And anycast to the rescue! Anycast is where you have multiple servers world-wide all configured essentially the same, with the same local IP address, then use the Border Gateway Protocol to have them all announce that same IP address into the Internet routing table. By having all the servers announce the same IP address, to the rest of the Internet they all look like they're the same server, which just happens to have a lot of possible options for how to route to it, so every network in the Internet will choose the anycast server closest to them, and route traffic to NS-Global to that server.</p><p style="text-align: left;">Looking at the map above, I used the RIPE Atlas sensor network (<a href="https://blog.thelifeofkenneth.com/2018/07/measuring-anycast-dns-networking-using.html">which I've also written about before</a>) to measure the latency it takes to send a DNS query to a server in Fremont, California from various places around the world. Broadly speaking, you can see that sensors on the west coast are green (lower latency) in the 0-30ms range. As you move east across the North American continent, the latency gets progressively worst, and as soon as you need to send a query to Fremont from another continent, things get MUCH worse, with Europe seeing 100-200ms of latency, and places like Africa feeling even more pain with query times in the 200-400ms range.</p><p style="text-align: left;">And the wild part is that this is a function of the speed of light. DNS queries in Europe just <i>take longer</i> to make it to Fremont and back than a query from the United States.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBWV9B9zCfV7HJqK48QpFwmYIwH_A3duI34eXZruhv-RFr_HfyKAlA70O3AFeGGO6Wo3l8JNYLDATSYoKGryitGHp1o1qOR3cEBYlMK77qeQja1NvmQ7er7Y26QquxKk6zyBICpDO7dVlT/s854/V4_20201108.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="410" data-original-width="854" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBWV9B9zCfV7HJqK48QpFwmYIwH_A3duI34eXZruhv-RFr_HfyKAlA70O3AFeGGO6Wo3l8JNYLDATSYoKGryitGHp1o1qOR3cEBYlMK77qeQja1NvmQ7er7Y26QquxKk6zyBICpDO7dVlT/w640-h308/V4_20201108.png" width="640" /></a></div>But if we instead deploy a dozen servers worldwide, and have all of them locally store all the DNS answers to questions we might get, and have them all claim to be the same IP address, we can beat this speed of light problem.</div><div><p style="text-align: left;">Networks on the west coast can route to our Fremont or Salt Lake City servers, and that's fine, but east coast networks can decide that it's less work to route traffic to our Reston, Virginia or New York City locations, and their traffic doesn't need to travel all the way across the continent and back.</p><p style="text-align: left;">In Europe, they can route their queries to our servers in Frankfurt or London, and to them it seems like NS-Global is a Europe-based service because there's no way they could have sent their DNS query to the US and gotten an answer back as soon as they did (because physics!) We even managed to get servers in San Palo, Brazil; Johannesburg, South Africa; and Tokyo, Japan.</p><p style="text-align: left;">So now NS-Global is pretty well global, and in the aggregate we generally tend to beat the speed of light vs a single DNS server regardless of how well connected it is in one location. Since we're also using the same address and the BGP routing protocol in all of these locations, if one of our sites falls off the face of the Internet, it... really isn't a big deal. When a site goes offline, the BGP announcement from that location disappears, but there's a dozen other sites also announcing the same block, so maybe latency goes up a little bit, but the Internet just routes queries to the next closest NS-Global server and things carry on like nothing happened.</p><p style="text-align: left;">Is it always perfect? No. Tuning an anycast network is a little like balancing an egg on its end. It's possible, but not easy, and easy to perturb. Routing on the Internet unfortunately isn't based on the "shortest" path, but on how far away the destination is based on a few different metrics, so when we turn on a new location in somewhere like Tokyo, because our transit provider there happens to be well connected, suddenly half the traffic in Europe starts getting sent to Tokyo until we turn some knobs and try to make Frankfurt look more attractive to Europe than Tokyo again. The balancing act is that every time we add a new location, or even if we don't do anything and the topology of the rest of the Internet just changes, traffic might not get divided between sites like we'd really like it to, and we need to start reshuffling settings again.</p><h2 style="text-align: left;">Using NS-Global</h2><p style="text-align: left;">Historically, to use NS-Global, you needed to know Javier, and email him to add your zones to NS-Global. But that was a lot of work for Javier, answering emails, and typing, and stuff, so we decided to automate it! I created a <a href="https://ns-global.zone/signup/">sign-up form for NS-Global</a>, so anything hosting their own zone who want to use us as a secondary service can just fill in their domain name and what IP addresses we should pull the zone from. </p><p style="text-align: left;">The particularly clever part (in my opinion) is that things like usernames or logging in or authentication with cookies seem like a lot of work, so I decided that NS-Global wouldn't have usernames or login screens. You just show up, enter your domain, and press submit. </p><p style="text-align: left;">But people on the Internet are assholes, and we can't trust the Internet people to just let us have nice things, so we still needed some way to authenticate users as being rightfully in control of a domain before we would add or update their zone in our database; and that's where the RNAME field in the Start of Authority record comes in. Every DNS zone starts with a special record called the SOA record, which includes a field which is an encoded email address of the zone admin. So when someone submits a new zone to NS-Global, we go off and look up the existing SOA for that zone, and send an email to the RNAME email address with a confirmation link to click if they agree with whomever filled out our form about a desire to use NS-Global. Done.</p><p style="text-align: left;">This blog post is getting a little long, so I will probably leave other details like how we push new zones out to all the identical anycast servers in a few seconds or what BGP traffic engineering looks like to their own posts in the future. Until then, if you run your own DNS server and want to add a second name server as a backup, feel free to <a href="https://ns-global.zone/signup/">sign up at our website</a>!</p></div>Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-68904830646094888802020-04-16T08:00:00.000-07:002020-04-16T13:52:27.453-07:00Validating BGP Prefixes using RPKI on Arista SwitchesI've written another <a href="https://eos.arista.com/bgp-origin-validation-rpki/">blog post for the Arista EOS blog</a>. This time, it's on how to use the brand new feature in EOS where it now supports RPKI via RTR. I walk through a simple example of how to enable Docker in EOS and spin up a <a href="https://www.nlnetlabs.nl/projects/rpki/routinator/">Routinator</a> instance on the switches' loopback interface to use RPKI to validate prefixes from BGP peers.<br />
<br />
Unfortunately, Arista does require that you create a free EOS Central account to read blog posts...Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-70691298550310858792020-04-01T09:00:00.000-07:002020-04-01T09:43:35.263-07:00A Simple Quality of Service Design Example<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiME2A0v9Q8g22sQlR2vAp8tkn09qt1eUogoIzoyW6yvNmmc2c8CsS3rzVkcgf61Dbbz9QDS7YwpxQdEYUFPyACGdo_C0tQ2L4a8Mm64G6SBovFJ1TmHyXYKrI4xzfbdB0TIeFlSZaqQlbV/s1600/WRR.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="481" data-original-width="1003" height="191" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiME2A0v9Q8g22sQlR2vAp8tkn09qt1eUogoIzoyW6yvNmmc2c8CsS3rzVkcgf61Dbbz9QDS7YwpxQdEYUFPyACGdo_C0tQ2L4a8Mm64G6SBovFJ1TmHyXYKrI4xzfbdB0TIeFlSZaqQlbV/s400/WRR.png" width="400" /></a></div>
<br />
I've written another blog post for the Arista corporate blog! This time, it's on <a href="https://eos.arista.com/simple-qos-design-example/">the basics of how to turn on the Quality of Service feature in EOS</a> and start picking out various classes of traffic to treat them better/worse than normal.<br />
<br />
EOS Central does require that you create a free login to read blog articles, unfortunately. I'm a little biased, but I'd suggest it might be worth the effort!Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-72486562590579058532020-03-29T06:00:00.000-07:002024-03-13T14:44:52.002-07:00Booting Linux Over HTTP<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsX18n4IPaFH12dosaONqfzeLJYtILeDCqBBB3rQRaJ_H4W8XhvvgnVGIq5fesUtGyha-kFhIfhS9LyeQmI4k3YdUaL8l1J0wOA7d9G4EWxZL_az04dmCKq7GcUW2D-4QTQWOJqDxMs94x/s1600/small.IMG_20200319_204908.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsX18n4IPaFH12dosaONqfzeLJYtILeDCqBBB3rQRaJ_H4W8XhvvgnVGIq5fesUtGyha-kFhIfhS9LyeQmI4k3YdUaL8l1J0wOA7d9G4EWxZL_az04dmCKq7GcUW2D-4QTQWOJqDxMs94x/s400/small.IMG_20200319_204908.jpg" width="300" /></a></div>
A couple years ago, one of my friends gave me a big pile of little Dell FX160 thin clients, which are cute little computers which have low power Atom 230 processors in them with the ability to support 3GB of RAM. Being thin clients means they were originally meant to be diskless nodes that could boot a remote desktop application to essentially act as remote graphical consoles to applications running on a beefier server somewhere else.<br />
<br />
That being said, they're great as low power Linux boxes, and I've been deploying them in various projects over the years when I need a Linux box somewhere but want/need something a little more substantial than a Raspberry Pi.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX8mSbmn_1Wz6_Xa7rnrMV3Qd_L_t35WlOKjEYqE-3HwMVVR_j-ignyUuIL6uZCToBYRJOkaicaE_nmAkWi2ucE-Qgu3J7HPEEuMuWDUERZlK1pV9I4rVVjoq5wzzNNiPT4DFIUJBXHl1B/s1600/small.IMG_20200319_204729.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX8mSbmn_1Wz6_Xa7rnrMV3Qd_L_t35WlOKjEYqE-3HwMVVR_j-ignyUuIL6uZCToBYRJOkaicaE_nmAkWi2ucE-Qgu3J7HPEEuMuWDUERZlK1pV9I4rVVjoq5wzzNNiPT4DFIUJBXHl1B/s400/small.IMG_20200319_204729.jpg" width="400" /></a></div>
The one big problem with them is that they didn't come with the 2.5" hard disk bracket, so I needed to source those drive sled kits on eBay to add more storage than the 1GB embedded SATA drive they all came with. Which is nominally fine; I bought a few of the kits for about $10 a piece, and for that to be the only expense to be able to deploy a 1TB 2.5" drive somewhere has been handy a few times.<br />
<br />
But it always left me thinking about what I could do with the original 1GB drive in these things. Obviously, with enough effort and hand wringing, you can get Linux installed on a 1G partition, but that feels like it's been done before, and these are thin clients! They're meant to depend on the network to boot!<br />
<br />
Fast forward to this year, and thanks to one of their network engineers hearing <a href="https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan" target="_blank">my interview for On the Metal</a>, I've been working with <a href="https://www.gandi.net/">Gandi.net</a> to help deploy one of their DNS anycast nodes in Fremont as part of the <a href="https://fcix.net/">Fremont Cabal Internet Exchange</a>. The thing is, how they designed their anycast DNS nodes is awesome! <a href="https://news.gandi.net/en/2019/03/booting-an-anycast-dns-network/">They have a 10,000 foot view blog post about it</a>, but the tl;dr is that they don't deploy their remote DNS nodes with an OS image on them. Each server gets deployed with a USB key plugged into them with a custom build of <a href="https://ipxe.org/">iPXE</a>, which gives the server enough smarts to, over authenticated HTTPS, download the OS image for their central servers and run the service entirely from RAM.<br />
<br />
Operationally, this is awesome because it means that when they want to update software on one of their anycast nodes, they can build the new image in advance on their provisioning server centrally, and just tell the server to reboot. When it reboots, it automatically downloads the new image from the provisioning servers, and you're up to date. If something goes terribly wrong and the OS on a node becomes unresponsive? Open a remote hands ticket with the data center "please power cycle our server" and the iPXE ROM will once again download a fresh copy of the OS image to run in RAM.<br />
<br />
Granted, they've got all sorts of awesome extra engineering involved in their system; cryptographic authentication of their boot images, local SSDs so while the OS is stateless, their nodes don't need to perform an entire DNS zone transfer from scratch every time it reboots, etc, etc. Which is all well and good, but this iPXE netbooting an entire OS image over the wide Internet using HTTP is just the sort of kind-of-silly, kind-of-awesome sort of project I've been looking to do with these thin clients I've got sitting around in my apartment.<br />
<h2>
Understanding The Boot Process</h2>
This left me with a few problems:<br />
<br />
<ol>
<li>The Gandi blog post regarding their DNS system was a 10,000 foot view conceptual overview, so they rightfully-so glossed over some of the technical specifics that weren't important to their blog post's message but really important for actually making it work.</li>
<li>I have been blissfully ignorant up until now of most of the mechanics involved with Linux booting in the gap between "The BIOS runs the bootloader" and "The Linux kernel is running with your <a href="https://en.wikipedia.org/wiki/Init">Init</a> server running as PID 1 and your fstab mounted"</li>
<li>I'm trying to do something exceedingly weird here, where there are no additional file systems to mount while the system is booting. There's plenty of guides available on booting Linux with an NFS or iSCSI root file system, but I'm looking at even less than that; I want the entire system just running from local RAM.</li>
</ol>
<div>
So before talking about what I ended up with, let's talk about the journey and what I had to learn about the boot process on Linux.</div>
<div>
<br /></div>
<div>
On a typical traditional Linux host, when you power it on, the local BIOS has enough smarts to find local disks with boot sectors, and read that first sector from the disk and execute it in RAM. That small piece of machine code then has enough smarts to load a more sophisticated bootloader like GRUB from somewhere close on the disk, which then has enough smarts to do more complicated things like load a Linux kernel and init RAM disk to boot Linux, or give the user a user interface to select which Linux kernel to boot, etc. One of the reasons why many Unix systems had a separate /boot partition was because this chainloader between the BIOS and the full running kernel couldn't mount more sophisticated file systems so needed a smaller and simpler partition for just the bare minimum boot files needed to get the kernel running.</div>
<div>
<br /></div>
<div>
The kernel file plus init RAM disk (often called initrd) are the two files Linux really requires to boot, and the part where my understanding was lacking. Granted, my understanding is still pretty lacking, but the main insight I gained was that the initrd file is a packed SVR4 archive of the bare minimum of files that the Linux kernel needs to then go and mount the real root file system and switch to it to have a fully running system. These SVR4 archives can be created using the "cpio" command as the "newc" file format, and the Linux kernel is smart enough to decompress it using gzip before mounting the archive, so we can gzip the initrd file to save bandwidth when ultimately booting the system.</div>
<div>
<br /></div>
<div>
(Related aside; there's many different pathways from the BIOS to having the kernel and initrd files in RAM. One of the most popular "net booting" processes, which I have used quite a bit in the past, is PXE booting, where the BIOS boot ROM in the network card itself has <i>juuuust</i> enough smarts to send out a DHCP request for a lease which includes a TFTP server and file path for a file on that TFTP server as DHCP options, and the PXE ROM downloads this file and runs it. This file is usually pxelinux.0, which I think is another chainloader which then downloads the kernel and initrd files from the same TFTP server, and you're off to the races.)</div>
<div>
<br /></div>
<div>
The missing piece for me inside the initrd file is that the kernel immediately runs a shell script in the root of the filesystem named "/init". This shell script is what switches the root file system over to whatever you specified in your /etc/fstab file, and ultimately at the very end of the /init script is where it "exec /sbin/init" to replace itself with the regular <a href="https://en.wikipedia.org/wiki/Init">init daemon</a> which you're used to being PID 1 and being the parent of every other process on the system.</div>
<div>
<br /></div>
<div>
I had never seen this /init script before, which is understandable because it's normally not included in your actual "/" root file system! It's only included in the initrd archive's "/" file system (which you can actually unpack yourself using gunzip and cpio), and disappears when it remounts the actual root and exec's /sbin/init... So since I want to run Linux entirely from RAM, "all" I need to do is figure out how to create my own initrd file, generate one that is not a bare minimum to mount another file system but everything I need to run my application in Linux, and figure out a simpler /init script to package with it which doesn't need to mount any local volumes but only needs to mount all the required virtual file systems (like /proc, /sys, and /dev) and exec the real /sbin/init to start the rest of the system.</div>
<h2>
Generating My Own Initrd File</h2>
<div>
So the first step in this puzzle for me is figuring out how to generate my own initrd file including the ENTIRE contents of a Linux install instead of just the bare minimum to get it started. And to generate that initrd archive, I first need to create a minimal root file system that I can configure to do what I want to then pack as the initrd file we'll be booting.<br />
<br />
Thankfully, Debian has some really good <a href="https://www.debian.org/releases/stable/i386/apds03.en.html">documentation on using their debootstrap tool</a> to start with an empty folder on your computer and end up with a minimal system. The first section of that documentation talks about partitioning the disk you're installing Debian on, but we just need the file system, so I skipped that part and went straight to running debootstrap in an empty directory.<br />
<br />
<pre>$ sudo debootstrap buster /home/kenneth/tmp/rootfs http://ftp.us.debian.org/debian/</pre>
<br />
Remember that there's plenty of Debian mirrors, so feel free to <a href="https://www.debian.org/mirror/list">pick a closer one off their list</a>.<br />
<br />
Once debootstrap is done building the basic image, from a terminal we can jump into the new Linux system using chroot, which doesn't really boot this system, but jump the terminal into it like it was the root of the currently running system, so you can interact with it <i>like</i> it's running. This lets us edit config files like /etc/network/interfaces, apt install needed packages, etc etc. Pretty much just following the rest of the Debian debootstrap guide and then also doing the configuration work needed to set up whatever the system should actually be doing. (things like setting a root password, installing ssh, configuring network interfaces, etc etc)<br />
<br />
<pre>$ LANG=C.UTF-8 sudo chroot /home/kenneth/tmp/rootfs /bin/bash</pre>
<br />
Since we're not installing this system on an actual disk, we don't need to worry about installing the GRUB or LILO bootloader like the guide says, but I did install the Linux kernel package since it was the easiest way to grab a built Linux kernel to pair with the final initrd file we're creating. Apt install linux-image-amd64 and copy that vmlinuz file out of the .../boot/ directory in the new filesystem to somewhere handy.<br />
<br />
The next step is to place the much simpler /init script in this new file system, so when the kernel loads this entire folder as its initrd we don't go off and try and mount other file systems or anything. This is the part where my friend at Gandi.net was SUPER helpful, since trying to figure out each of the various virtual file systems that still need to be mounted on my own only yielded me a lot of kernel panics.<br />
<br />
So huge thanks to Arthur for giving me this chunk of shell code! Copy it into the root of the freshly debootstrapped system and mark it executable (chmod +x)<br />
<br />
<a href="https://gist.github.com/PhirePhly/3856815f0a490c7d19eceffd945d5bfb">Source for init</a>:<br />
<script src="https://gist.github.com/PhirePhly/3856815f0a490c7d19eceffd945d5bfb.js?file=init"></script><br />
<br />
At this point, we're ready to pack this filesystem into an initrd archive and give it a shot. To create the archive, I followed <a href="https://www.kernel.org/doc/html/latest/admin-guide/initrd.html">this guide</a>, which boils down to passing cpio a list of all the file names, and then piping the output of cpio to gzip to compress the image.<br />
<br />
<pre>$ cd /home/kenneth/tmp/rootfs
$ sudo find . | sudo cpio -H newc -o | gzip -9 -n >~/www/initrd</pre>
<br />
At this point, you should have this initrd file which is a few hundred MB compressed, and the vmlinuz file (vmlinuz being a compressed version of the usual vmlinux kernel file!) which you grabbed out of the /boot directory, and that *should* be everything you need for booting Linux on its own. Place both of those files on a handy HTTP server to be downloaded by the client later.<br />
<h2>
Netbooting This Linux Image</h2>
Given the initrd and kernel images, the next step is to somehow get the target system to actually load and boot these files. Aside from what I'm talking about here of using HTTP, you can use any of the more traditional booting methods like putting these files on some local storage media and installing GRUB, or using the PXE boot ROM in your computer's network interface to download these files from a TFTP server, etc.<br />
<br />
TFTP would probably be pretty cute since many computers can support it stock, but that depends on your target system being on a subnet with a DHCP server that can hand out the right DHCP options to tell it where to look for the TFTP server. I didn't want to depend on DHCP, and I wanted to use HTTP, so I instead opted to use <a href="https://ipxe.org/">iPXE</a>, which is a much more sophisticated boot ROM than the typical PXE ROMs you get.<br />
<br />
It is possible to directly install iPXE on the firmware flash of NICs, but that's often challenging and hardware specific, and a good point that Arthur pointed out was that since they boot iPXE from USB, if for some reason they need to swap the iPXE image remotely, it's *MUCH* easier to mail a USB flash drive and ask them to replace it than to try and walk someone else through how to reflash the firmware on a NIC over the phone... I'm not going to be using a USB drive, since these thin clients happen to have convenient 1GB SSDs in them already, but it's the same image. Instead of dd'ing the ipxe.usb image onto a flash drive, I just temporarily booted Linux on the thin clients and dd'ed the ipxe ROM onto the internal /dev/sda.<br />
<br />
The stock iPXE image is pretty generic, and like a normal PXE ROM sends out a DHCP request for a local network boot image to download. This isn't what we want here, so we're definitely going to need to build our own iPXE binary in the end, but I started with the stock ROM because it allows you to hit control-B during the boot process and interactively poke at the iPXE command line, and manually step through the entire process of configuring the network, downloading the Linux kernel, downloading the initrd file, and booting them.<br />
<br />
So before building my own custom ROM, I burned iPXE onto a USB flash drive and poked at the iPXE console with the following commands on my apartment network:<br />
<br />
<pre>dhcp
kernel http://example.com/vmlinux1
initrd http://example.com/init1
boot</pre>
<br />
And that was enough to start iterating on my initrd file to get it to what I wanted. Since I was still doing this in my apartment which has a DHCP server, I was able to ask iPXE to automatically configure the network with the "dhcp" command, then download a kernel and initrd file, and then finally boot with the two files it just downloaded.<br />
<br />
So at this point, I was able to boot the built Linux image interactively from the iPXE console, and had a fully running Linux system in RAM, which was kind of awesome, but I wanted to fully automate the iPXE booting process, which means I need to build a custom image with an embedded "iPXE script" which is essentially just a list of commands for iPXE to run to configure the network interface, download the boot files, and boot.<br />
<br />
<a href="https://gist.github.com/PhirePhly/3856815f0a490c7d19eceffd945d5bfb">iPXE Boot Script</a>:<br />
<script src="https://gist.github.com/PhirePhly/3856815f0a490c7d19eceffd945d5bfb.js?file=ipxe.script"></script><br />
So given that script, we follow the <a href="https://ipxe.org/download">iPXE instructions to download their source</a> using git, install their build dependencies (which I apparently already had on my system from past projects, so good luck...), and the key step is that when performing the final build, we pass make the path to our iPXE boot script file to embed it in the image as what to run.<br />
<br />
<pre>$ cd ~/src/ipxe/src
$ make EMBEDDED_IMAGE=./bootscript bin/ipxe.usb</pre>
<br />
And at this point in the ipxe/src/bin folder is the built image of ipxe.usb which has our custom boot script embedded in it! Since the internal SATA disk is close enough to a USB drive, from a booting perspective, that's the variant of ROM I'm using.<br />
<br />
So given this custom iPXE ROM, I manually booted a live Linux image on the thin client, used dd to write the ROM to /dev/sda which is the internal 1G SSD, and the box is ready to go!<br />
<br />
Now, when I power on the box, the BIOS sees that the internal 1G SSD is bootable, so it boots that, which is iPXE, which runs the embedded script we handed it, which configures the network interface, downloads our custom initrd file and the Linux kernel from my HTTP server, and boots those. Linux then unpacks our initrd file, and runs the /init script embedded in that, which just mounts the virtual file systems like /proc/, /sys/, and /dev, and then <i>doesn't</i> try and mount any other local file system, and finally our /init/ script exec's /sbin/init, which in the case of Debian happens to be systemd, and we're got a fully running system in RAM!<br />
<br />
<a href="https://www.youtube.com/watch?v=7ppdut16aYA">Video of generally what that looks like</a>:<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/7ppdut16aYA" width="560"></iframe><br />
<br />
So once again, thanks to Arthur from Gandi.net for the original idea and gentle nudges in the right direction when I got stuck.<br />
<br />
Of course, the next thing to do is start playing "disk space golf" with the OS image to see how small I can make the initrd file, since the smaller the initrd file, the more RAM that is left over for running the application in the end! And actually doing something useful with one of these boxes running iPXE... a topic for another blog post.<br />
<br />
Update: One thing to note is that this documentation is for the minimum viable "booting Linux over HTTP". iPXE does support crypto such as HTTPS, client TLS certificates for client authentication, and code signing. <a href="https://ipxe.org/crypto">More details can be found in their documentation</a>.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-26094131380904403142020-03-18T18:00:00.000-07:002020-03-18T22:09:54.981-07:00Building My Own 50Ah LiFePO4 Lithium Battery Pack<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUE5yTprDlKETFiIC0QFowwALDy_DmB-CevXfmbe8y4zYoo0KoxaR9iZrNefTjcVwJUepOnaL6XgOJuVxi_VKAdHKbLMQBbtQAHO5Y-6_Bic9d65CPLSY2MfemUVWDCa2PZdc0kMN02y00/s1600/small.IMG_20200225_215745.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUE5yTprDlKETFiIC0QFowwALDy_DmB-CevXfmbe8y4zYoo0KoxaR9iZrNefTjcVwJUepOnaL6XgOJuVxi_VKAdHKbLMQBbtQAHO5Y-6_Bic9d65CPLSY2MfemUVWDCa2PZdc0kMN02y00/s400/small.IMG_20200225_215745.jpg" width="400" /></a></div>
Several years ago, I had purchased a <a href="https://amzn.to/3917Qkn" target="_blank">20Ah 12V Lithium Iron battery pack from Bioenno</a> for my various 12VDC projects. To help protect it, I ultimately built it up into a 50cal ammo can with a dual panel-mount PowerPole connector on the outside, which has proven really nice as far as battery boxes go:<br />
<br />
<ul>
<li>20Ah is a decent battery capacity for a small load</li>
<li>The packaged Bioenno pack left some space inside the box to also store the charger it came with, some PowerPole accessories, etc.</li>
<li>The fact that you're able to close up the box and use the power connectors on the outside once you're using it is real nice.</li>
<li>Bioenno battery packs are well packaged and fairly idiot proof since the battery management/protection board is built into the pack so it's hard to really harm these.</li>
</ul>
<div>
So I really liked that project; cut out the mounting hole for the panel mount, build a small harness to plug the battery pack into, and use hard foam to give the pack further protection inside the box.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSYrdylgu9hXtNiTDKJibs5dJlR7eZjQD9Q5b_fcXwBeSH2bP6sW3kJ3eLz2NG_jcQqyGlWT8_IrULcDoyJYBox1dsGMIvGxwworfxuFU1xGlfShrknw1eGjOWN1Rfvus8DscMm6dK89l4/s1600/small.IMG_20200202_191728.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSYrdylgu9hXtNiTDKJibs5dJlR7eZjQD9Q5b_fcXwBeSH2bP6sW3kJ3eLz2NG_jcQqyGlWT8_IrULcDoyJYBox1dsGMIvGxwworfxuFU1xGlfShrknw1eGjOWN1Rfvus8DscMm6dK89l4/s400/small.IMG_20200202_191728.jpg" width="300" /></a></div>
<div>
I was recently hanging out with one of my electronics buddies while he was browsing eBay, and he found a bulk lot of eight raw 50 amp-hour LiFePO<sub>4</sub> cells from a seller for about $400. A good deal, particularly because the seller said these had been purchased for a project which ended up being canceled, so these were practically new off the shelf cells!</div>
<div>
<br /></div>
<div>
The problem was, my friend was only interested in building a 4S 12V pack (4 series cells x 3.5V ~ kind of equals 12V), so since my unsuspecting ass was sitting there, and he was so persuasive, surprise! I now own four raw prismatic 50Ah lithium battery cells.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi1gE3crrmfVI5D7vIO9lKApuqksBc1CuHucuqAAAeKpbUgzpMlWO3lYkcD7FsED16304d8KqtzCNnbSVE6kCfCjwSwjxj7eWBryNfjVhiZFyY0SpX8tyNEn7Weoi3pX7-NT6k2eK3ia95/s1600/small.IMG_20200202_192132.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi1gE3crrmfVI5D7vIO9lKApuqksBc1CuHucuqAAAeKpbUgzpMlWO3lYkcD7FsED16304d8KqtzCNnbSVE6kCfCjwSwjxj7eWBryNfjVhiZFyY0SpX8tyNEn7Weoi3pX7-NT6k2eK3ia95/s400/small.IMG_20200202_192132.jpg" width="400" /></a></div>
<div>
To give you a sense of scale, each of these cells is about the size of a paperback book, with two studs on the top for the positive and negative terminals. The eBay lot happened to come with the cell spacers and busbars, so all that was missing to build these into a full battery pack was a battery management system module and an enclosure. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjspjzEiIc0rdfK0Vjz5GBJWcuPjzEkNK_RHjIVWmZtHUX_8mcs7f0gcLfxY926IUQTbOQB1TGaO3zyh4aoTrgZjF34_TN8p9-3NbK7wohyEoPRYYBb7YRA7DLAC2UveSBT7bwF1rVHyRzG/s1600/small.IMG_20200212_221343.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjspjzEiIc0rdfK0Vjz5GBJWcuPjzEkNK_RHjIVWmZtHUX_8mcs7f0gcLfxY926IUQTbOQB1TGaO3zyh4aoTrgZjF34_TN8p9-3NbK7wohyEoPRYYBb7YRA7DLAC2UveSBT7bwF1rVHyRzG/s400/small.IMG_20200212_221343.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo26m3AxHNDSSZ6JsU4kV7IKeqCBynCf5WtgCEaiB2jK2Irk150adRKzoJ6CsCJnCZcmwsnNGjMzVbeq3vWWaggLixS_fppVB4fqVIxL1Lqqtuim6sSVezUfvFy4DUx7l7fASSDcnnUDJs/s1600/small.IMG_20200212_221339.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo26m3AxHNDSSZ6JsU4kV7IKeqCBynCf5WtgCEaiB2jK2Irk150adRKzoJ6CsCJnCZcmwsnNGjMzVbeq3vWWaggLixS_fppVB4fqVIxL1Lqqtuim6sSVezUfvFy4DUx7l7fASSDcnnUDJs/s400/small.IMG_20200212_221339.jpg" width="400" /></a></div>
<div>
What finally convinced me to buy into half of this lot was that, based on the measurements, it looked like I'd be able to exactly fit this homemade pack into another 50cal ammo can, so that solves the enclosure issue...</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjtfeulyjn0gsLHofKy7233ArnuUFoBy6BO1V5gDWMbHZDkeXyewfNVpqm9oYHQmiIi7wayatvQmteWMS4_otIlhelFY5DhRAzlIYFFAd5r0_cnB7F7y1P_E97HaZmN-VhBIK1F9SKrlOU/s1600/small.IMG_20200202_205825.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjtfeulyjn0gsLHofKy7233ArnuUFoBy6BO1V5gDWMbHZDkeXyewfNVpqm9oYHQmiIi7wayatvQmteWMS4_otIlhelFY5DhRAzlIYFFAd5r0_cnB7F7y1P_E97HaZmN-VhBIK1F9SKrlOU/s400/small.IMG_20200202_205825.jpg" width="300" /></a></div>
<div>
Well... almost. The cell spacers were literally millimeters too wide to fit in the can, so I did need to shave off the exterior dovetails to make them fit...</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0DZpJ0iDuBk7cVM_IBzhyphenhyphenffvkJTwpihGfZRZKW6Tc9-bzqaJyRrqyvrR791xoUjhbVW843zUilnQ9O8Lx1EHEa8ADwNUChW-_0SYIXt3IChpKUeP__hzeCK_kf4xbV0n3F_mW7nv9R11f/s1600/small.IMG_20200212_221925.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0DZpJ0iDuBk7cVM_IBzhyphenhyphenffvkJTwpihGfZRZKW6Tc9-bzqaJyRrqyvrR791xoUjhbVW843zUilnQ9O8Lx1EHEa8ADwNUChW-_0SYIXt3IChpKUeP__hzeCK_kf4xbV0n3F_mW7nv9R11f/s400/small.IMG_20200212_221925.jpg" width="300" /></a></div>
<div>
One lesson I did learn was that apparently when I bought these ammo cans <i>years</i> ago, I didn't realize that this one was missing several of the hinge pins for the lid. The two that were left were also loose enough that as I started drilling on the case, they fell out. So those had to get epoxied back in, and I now know to never buy ammo cans without checking to make sure all of this hardware is still there.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg35vxgK-uMtxQKkze5Dy6YANc_4C4Aq-LUJDHQzYBDq6Fb3jzvShMSE3RLFNzSwaI85vpNxx_45iLuh6Wu_76eL7U1PaPzVMjlx9ktxhQZIk2AhBPkTz98zuApooG-WcnNpsFWDDLRpsPY/s1600/small.IMG_20200216_195012.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg35vxgK-uMtxQKkze5Dy6YANc_4C4Aq-LUJDHQzYBDq6Fb3jzvShMSE3RLFNzSwaI85vpNxx_45iLuh6Wu_76eL7U1PaPzVMjlx9ktxhQZIk2AhBPkTz98zuApooG-WcnNpsFWDDLRpsPY/s400/small.IMG_20200216_195012.jpg" width="300" /></a></div>
<div>
So let's talk about the battery management system for a second, because that's the important part to understand when building a lithium battery pack out of raw cells.</div>
<div>
<br /></div>
<div>
The LiFePO<sub>4</sub> battery chemistry is awesome compared to the typical deep cycle lead-acid which I'd normally use for the same applications as this. Higher energy and power density, much higher cycle life, and lithium batteries tend to have a very flat discharge curve, so you get a useful ~13V out of a 4S pack right up until the moment it's entirely dead, where lead-acid has a long slow decline starting at 12.6V down to below 11V, so some of your DC equipment might start misbehaving even after you've only discharged half of a lead-acid pack's capacity.</div>
<div>
<br /></div>
<div>
The problem with LiFePO<sub>4</sub> compared to lead-acid is that it is also nowhere near as forgiving or abuse tolerant. Granted, LiFePO<sub>4</sub> is better than other lithium chemistries, but you still need to be much more worried about issues like over-discharging or over-charging lithium batteries. Abuse a LiFePO<sub>4</sub> cell, and it's much more prone to experience permanent damage compared to a lead-acid battery.</div>
<div>
<br /></div>
<div>
The battery management system is an electrical module which has a small wire run to both terminals on every cell, so that it can monitor each cell's individual voltage, and even if just a single cell's voltage falls outside of the acceptable range, the BMS can disconnect the whole pack to prevent further damage. <a href="https://www.ebay.com/itm/202934944816" target="_blank">The 45A Daly BMS</a> that I bought also supports balancing, so if any one cell has slightly more or slightly less capacity or resistance than the rest, instead of continuing to drift further and further out of sync with the rest of the pack with regards to their charge state, when this BMS sees a cell reach 3.5V before the others, it starts bleeding 30mA off the cell, to slowly pull it back into line with the rest of the pack.</div>
<div>
<br /></div>
<div>
To help understand why this is so important, remember that in something more traditional like a flooded lead-acid battery, when a cell reaches full charge, it can still pass current to charge other cells by splitting water into H<sub>2</sub> and O<sub>2</sub>. This depletes your electrolyte, but it means the rest of your cells can still come up to full charge. Lithium chemistries don't have any sort of non-catastrophic current leakage like this, so the only way to get current past the highest voltage cell to the others is to use an active device to bleed the current for you.</div>
<div>
<br /></div>
<div>
It also means that LiFePO<sub>4</sub> batteries don't have any concept of a higher voltage balancing charge stage like lead-acid batteries have, so as long as you fully charge the pack to the recommended 14.6V and hold it there long enough for the BMS to finish balancing the pack (Which it shouldn't even need to do as long as the cells start well matched) the pack should be fine. Different charge profiles for different battery chemistries and even different manufacturer recommendations inside the same chemistry is a whole rabbit hole in itself, so for this article I'm going to just leave it at "only use a battery charger specifically designed for LiFePO<sub>4</sub> to charge these" and gracefully move on.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEPnLPEcRrgSD0WhxY0IKk5dSlway9oK9NJqstPC07pzresqyMWteTJe6o4BsEr1OHC2DbVetvDejTpLFJ4BFihoSF3w3thllYDkW1b6M4iuRl-WBl3u6sdVrHBJ7UioU-2vrhH4w_MWuU/s1600/s-l1600.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1458" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEPnLPEcRrgSD0WhxY0IKk5dSlway9oK9NJqstPC07pzresqyMWteTJe6o4BsEr1OHC2DbVetvDejTpLFJ4BFihoSF3w3thllYDkW1b6M4iuRl-WBl3u6sdVrHBJ7UioU-2vrhH4w_MWuU/s400/s-l1600.jpg" width="400" /></a></div>
<div>
All of the small wires running to each terminal came with my BMS per-terminated on the small connector block which plugs into the BMS module. Since these are just for monitoring cell voltage and balancing cells, they don't need to carry much current. The two big 10AWG wires molded into the BMS are meant to go between the most negative cell terminal in the pack (blue wire) and the charger/load (black wire), so that the BMS is able to disconnect the pack from everything else if it decides that something is wrong. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYvbUdzGETRGUQsz3d8PdB88doN7nj6g64cJHVVKLX7KnuvyF8D-mk6WOXIwAVSpXNkG6WbFkXkkiN4GgsOwE59A3ALDcTJ8JNcIusI6qkWzkAumv0ltH0dcYGC962C65fHEZqB2AVRuGd/s1600/small.IMG_20200219_201119.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYvbUdzGETRGUQsz3d8PdB88doN7nj6g64cJHVVKLX7KnuvyF8D-mk6WOXIwAVSpXNkG6WbFkXkkiN4GgsOwE59A3ALDcTJ8JNcIusI6qkWzkAumv0ltH0dcYGC962C65fHEZqB2AVRuGd/s400/small.IMG_20200219_201119.jpg" width="400" /></a></div>
<div>
These high current lines on the BMS were nice. Definitely 10AWG, "ultraflex" so there was more tiny strands which helps with flexibility, and just enough length on them to be useful for what I wanted. The battery line got terminated with a crimp terminal and connected directly to the pack to try and limit resistance.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUT_umEbPOAxc2fEGJG_lg9AG2DdpAXe8fsf4VcBn6mrv5GTA9FPr0SBf6crDQbnp0jWsJ4cAkQyrtFTCT6llf0bewByaiuJIrmk1qi49omubaVngwrUbw3RrWnAQ9KUwG9Jbpo6bpsLV3/s1600/small.IMG_20200219_201658.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUT_umEbPOAxc2fEGJG_lg9AG2DdpAXe8fsf4VcBn6mrv5GTA9FPr0SBf6crDQbnp0jWsJ4cAkQyrtFTCT6llf0bewByaiuJIrmk1qi49omubaVngwrUbw3RrWnAQ9KUwG9Jbpo6bpsLV3/s400/small.IMG_20200219_201658.jpg" width="300" /></a></div>
Since the PowerPole panel connectors I'm using can each accept 10AWG themselves, I wanted to split this single 10AWG wire from the BMS to two wires for the load/charger connections.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6OugWDdbCNbzua4xhs4bkI5GjAe77RCgm-Nj7opwnsuK-B5805wSmLy8hsroDP4f_SdAA3YIObDYtTUH26eN5vCjriv3hOXjXfdjV6r3zvix2hYxpCJNVtINHQq5QNU2NNpLp70smmFSE/s1600/small.IMG_20200219_202258.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6OugWDdbCNbzua4xhs4bkI5GjAe77RCgm-Nj7opwnsuK-B5805wSmLy8hsroDP4f_SdAA3YIObDYtTUH26eN5vCjriv3hOXjXfdjV6r3zvix2hYxpCJNVtINHQq5QNU2NNpLp70smmFSE/s400/small.IMG_20200219_202258.jpg" width="300" /></a></div>
To do this, I used a trick I picked up from my training for the IPC 620 cable assemblies standard, which is when you're trying to butt solder wires together to use a fine gauge wire to lash the wires together first before trying to flow solder in.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8i6wy333lx9DQkHE05QnGhrmV2LgHogjBFGx9Iqo2jDvvVc3slKUtU9bSUlTbv5au7RdFhWIMzzd8L1PYmtNseH3cEbOIA7ZaY7vujlQhtDuEZ9J1K1hN3qRAskUiVyj5fEhkHfezu3LC/s1600/small.IMG_20200219_202823.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8i6wy333lx9DQkHE05QnGhrmV2LgHogjBFGx9Iqo2jDvvVc3slKUtU9bSUlTbv5au7RdFhWIMzzd8L1PYmtNseH3cEbOIA7ZaY7vujlQhtDuEZ9J1K1hN3qRAskUiVyj5fEhkHfezu3LC/s400/small.IMG_20200219_202823.jpg" width="300" /></a></div>
<div>
All of the strength and conductivity comes from the solder, but you're not trying to hold burning hot wire and molten metal together while soldering thanks to the lashing.</div>
<div>
<br /></div>
<div>
Don't mind the blood on the finer gauge wire... That was because I was an idiot while lashing these wires together and forgot that I was working right next to a fully charged bank of raw battery cells with exposed busbars. The 30AWG wire happened to brush the other end of the pack while I was winding it, and even though it was only 12V, the lack of any current limiting meant the wire instantly turned into a hot knife in my hands and left me with a pretty nasty burn.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG6f_SQpKuhhzLnpcVd9O9YeR6695sKsXujbfB_5KikUXMT4Rw4qg4nGsIrBERg39a18w59UAHde7GHwf7plhQzSUjYlxeBV0eXagTruQEr7obrITyZVmGP9Ws2keiViVnwA4Xrl0H49NE/s1600/small.IMG_20200219_205111.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG6f_SQpKuhhzLnpcVd9O9YeR6695sKsXujbfB_5KikUXMT4Rw4qg4nGsIrBERg39a18w59UAHde7GHwf7plhQzSUjYlxeBV0eXagTruQEr7obrITyZVmGP9Ws2keiViVnwA4Xrl0H49NE/s400/small.IMG_20200219_205111.jpg" width="300" /></a></div>
<div>
Anyways... two layers of heat shrink later, we've got an incredibly good electrical connection between the BMS and two more short pieces of 10AWG wire to come out to the panel connector.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbbuQoSkdTENzs6QloZMFhf_i5U6yDbm9zvYlrvJ3gGrFX5mr2-LXFhB2ZLVUuCGtQ2bAcAwVd6F_QzAvJAcTaYc7DgbMCGkp5-6kWf53FkZUnoSO-W2GNJAZyukHgdmxlsrdUNP-D5kK_/s1600/small.IMG_20200219_214802.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbbuQoSkdTENzs6QloZMFhf_i5U6yDbm9zvYlrvJ3gGrFX5mr2-LXFhB2ZLVUuCGtQ2bAcAwVd6F_QzAvJAcTaYc7DgbMCGkp5-6kWf53FkZUnoSO-W2GNJAZyukHgdmxlsrdUNP-D5kK_/s400/small.IMG_20200219_214802.jpg" width="400" /></a></div>
<div>
I made a point of keeping all of the wires short, but deliberately put stain relief loops in the routing so the pack could have some ability to move around in the enclosure while being handled without stressing anything, since I'm building this battery pack for field use instead of as a static install. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWgGjVR5YZ5Fb8vRWR27uyLD2KjvEiM-3h5F1D_FHnyw_VVAp7WiZy32h1mVdzs4QFNJCdCoAblayjYVWOwP2Nb2Hd5SKgNdV6R175f49Gn8Sc8_pPCRXrwbDn5Q-33Zzq6Vz3jaT1oTg-/s1600/small.IMG_20200219_214825.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWgGjVR5YZ5Fb8vRWR27uyLD2KjvEiM-3h5F1D_FHnyw_VVAp7WiZy32h1mVdzs4QFNJCdCoAblayjYVWOwP2Nb2Hd5SKgNdV6R175f49Gn8Sc8_pPCRXrwbDn5Q-33Zzq6Vz3jaT1oTg-/s400/small.IMG_20200219_214825.jpg" width="300" /></a></div>
<div>
Several pieces of hard foam to insulate, cushion, and immobilize the pack inside the metal can. Doesn't leave much of any room in this can for DC power accessories like in my first power can, but I figure when I'm using a 50Ah pack instead of my 20Ah pack, I'm probably running a more significant load so it would be worth putting some more planning into exactly what cabling I'll need (or just bring both and have all my accessories plus 70Ah of capacity.)</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVHZHu-4n5Rk2BxR-Y32iW-9bEjH7uDj_RGvOtoGiRwdXcCbwra7yMuj83wN6BhGuLwQKssR5co73XQiyvo3hcL_mFHqQFg4adL5dKEST1cRwk_22B2cAUFDvU3ekPaa6mwobWwXrFTynW/s1600/small.IMG_20200220_204337.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVHZHu-4n5Rk2BxR-Y32iW-9bEjH7uDj_RGvOtoGiRwdXcCbwra7yMuj83wN6BhGuLwQKssR5co73XQiyvo3hcL_mFHqQFg4adL5dKEST1cRwk_22B2cAUFDvU3ekPaa6mwobWwXrFTynW/s400/small.IMG_20200220_204337.jpg" width="300" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-QvcfY7saz9ENN9-WsmKR8YTdCE3i6PMJeAMutnsoceULYMz7Rj9E6FwRP7dkp0i8bGES_VnOD-zdEvTEX3cPeeCu-lqDjspHecK4SYpPAhHnZyrAHlJM2JhCNiSXnWZmOXjj4d4AZ0F1/s1600/small.IMG_20200220_204353.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-QvcfY7saz9ENN9-WsmKR8YTdCE3i6PMJeAMutnsoceULYMz7Rj9E6FwRP7dkp0i8bGES_VnOD-zdEvTEX3cPeeCu-lqDjspHecK4SYpPAhHnZyrAHlJM2JhCNiSXnWZmOXjj4d4AZ0F1/s400/small.IMG_20200220_204353.jpg" width="400" /></a></div>
<div>
Ultimately, I'm really happy with how this pack turned out, and it was only about $300 for all the raw components to build a pack which still has all of the protection and balancing features that I'd require in a portable field-use battery pack like the off-the-shelf one that I originally bought from Bioenno.</div>
<span id="goog_1485333422"></span>Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-75517828857097493542020-03-01T13:00:00.000-08:002020-03-02T11:45:13.200-08:00Building a Raspberry Pi Stratum 1 NTP Server<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj33tTUAYogKEInF4ZX6nhmootahfFl0LnCgqQ02VDbv3-2o0cApWCrLqztB7BRHnp5vpzC9OYeYE5ABTgP1addGCyWtA_wz2px2m1ueuN2IXrcxLuANdN9HJyZRi0aqsWpU7KA5QXNqz-/s1600/smallIMG_20200229_211606.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj33tTUAYogKEInF4ZX6nhmootahfFl0LnCgqQ02VDbv3-2o0cApWCrLqztB7BRHnp5vpzC9OYeYE5ABTgP1addGCyWtA_wz2px2m1ueuN2IXrcxLuANdN9HJyZRi0aqsWpU7KA5QXNqz-/s400/smallIMG_20200229_211606.jpg" width="400" /></a></div>
<br />
After the nice reception <a href="https://blog.thelifeofkenneth.com/2020/02/introduction-to-network-time-protocol.html" target="_blank">my last article on NTP</a> got, I decided that it was about time I pulled a project off the shelf from a few years ago and get a stratum 1 NTP server running on a Raspberry Pi again.<br />
<br />
So this article is going to be my notes on how I built one using a cheap GPS module off eBay as the reference time source and a Raspberry Pi. (a 2B in this case, but hopefully the differences for newer models will be minor; if there are any, they're probably around the behavior of the serial port since I think things changed on the Pi3 with bluetooth support?)<br />
<br />
So anyways, before we jump into it, let's spend some time talking about why you would and wouldn't want to do this:<br />
<br />
Pro: NTP is awesome, and this lets you get a stratum 1 time server on your local network for much lower cost that traditional COTS NTP time standards, which tend to be in the thousands of dollars price range.<br />
<br />
Con: This is a Raspberry Pi, booting off an SD card, with a loose PCB plugged into the top of it, so I wouldn't classify it as "a robust appliance". It isn't a packaged enterprise grade product, so I wouldn't rely on this NTP server on its own for anything too critical, but then again, I would say that about NTP in general; it's only meant to get clocks to within "pretty close" of real time. If you're chasing individual milliseconds, you should probably be using <a href="https://en.wikipedia.org/wiki/Precision_Time_Protocol" target="_blank">PTP (Precision Time Protocol)</a> instead of NTP... Totally depends on what you're doing. I'm just being a nerd.<br />
<br />
Pro: The pulse-per-second driver for the Raspberry Pi GPIO pins is pretty good, so once you get this working, the GPS receiver will set the Pi's clock extremely well!<br />
<br />
Con: The Raspberry Pi's Ethernet interface is actually hanging off of a USB hub which is hanging off of the USB interface on the SoC (system on chip) that powers the Raspberry Pi, so there is an intrinsically higher level of latency/jitter on the Pi's Ethernet interface vs a "real" PCIe NIC. This means that your bottleneck for getting crazy good time sync on all of your computers is ironically going to be getting the time off of the Raspberry Pi onto the network...<br />
<br />
Pro: I would expect this USB jitter to be on the order of 500 microseconds, which is still less than a millisecond, and remember what I said about chasing individual milliseconds in NTP? This should be fine for any reasonable user of NTP.<br />
<br />
<h2>
Conceptual Overview</h2>
So we are going to be building an NTP server, which sets its time off of a GPS receiver, which is setting its time off of the global constellation of GPS (et al) satellites, which have cesium atomic clocks on-board which the US government puts a lot of effort into setting correctly, so that is where our time is actually coming from.<br />
<br />
I'm specifically using a u-blox NEO-6M GPS receiver, which I got on eBay for a few dollars on a breakout board. This module isn't the "Precision timing" specific NEO-6T variant, but the 6M module is still completely sufficient for the level of time accuracy that we're looking for, and much cheaper/more available than the 6T module which is specifically designed for this sort of static time-keeping application (to the extent where the 6T will even lose GPS lock if you start moving it!)<br />
<br />
The NEO-6M GPS receiver, like many GPS receiver modules, has a USB interface, which we will not be using, a UART serial interface, which we will be using, and a "Pulse-per-second" PPS output that it asserts high at the very top of every second, which we will definitely be using!<br />
<br />
The reason that we will be using both the UART serial port and the PPS outputs on the GPS is because the UART tells us the current date, wall time, etc, but it does it over a 9600 baud serial port, so the latency between when the module starts sending that report and when we receive all of it will depend on how <i>long</i> the message is, and that depends on things like which NMEA sentences you have turned on, how many satellites the receiver see, etc, so while it usually takes about 130ms to receive this report, you can't be sure exactly how long it was.<br />
<br />
The PPS output, on the other hand, is <i>extremely</i> precise on telling us exactly when the top of the second is by taking a single pin and asserting it high right on the top of each second, but all it tells us is when the top of the second is, not <i>which</i> second, or hour, or even what day it is! We want to know both what second we're currently on in history, and exactly when that second starts, so we need both the UART and PPS inputs to correctly set the time.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQH0H776ar1pZqf5rPpe5dof9m3tBkhtqKlfnznNMqkeykWma3pjUWNN9ORzi31mXLhA06xgpE74g6_Q9OLvwC_JtdcKVWgDoSIDwQeb4w6tLFB3YHkk-o5uy1TVyMqDxkizznlI_Z0nk3/s1600/NTPTimeFlow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="322" data-original-width="1342" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQH0H776ar1pZqf5rPpe5dof9m3tBkhtqKlfnznNMqkeykWma3pjUWNN9ORzi31mXLhA06xgpE74g6_Q9OLvwC_JtdcKVWgDoSIDwQeb4w6tLFB3YHkk-o5uy1TVyMqDxkizznlI_Z0nk3/s640/NTPTimeFlow.png" width="640" /></a></div>
Looking at conceptually how we're going to use the UART and PPS outputs of the module, we're going to feed both of them into the Raspberry Pi's GPIO header (more on this in the hardware section), and do the following:<br />
<br />
<ul>
<li>We're going to use the gpsd daemon to parse the /dev/ttyAMA0 UART text stream, which is usually NMEA sentences, but gpsd might put the receiver into its proprietary binary mode if it can figure out what kind of GPS receiver it is. The gpsd developers have put a lot of work into writing a GPS data stream parser, so we want to benefit from it. </li>
<li>Gpsd will then write the time information it gets from the UART to a shared segment of memory, which the NTP daemon is able to connect to using its shared memory segment driver, to get a rough idea of what date/time it is.</li>
<li>The PPS output of the receiver will be fed into the Linux kernel PPS driver, which watches for the positive edges on the pin and time stamps them relative to the local system clock. The Linux kernel makes these events available for other applications via the /dev/pps0 device, which NTP will read using the NTP PPS Clock Discipline driver. </li>
</ul>
<h2>
Required Hardware</h2>
<div>
For this project, I used a generic GPS breakout board off eBay, which I soldered to a piece of 4x6cm perf prototype board, and soldered wires to make the required connections. That being said, there are several Raspberry Pi GPS "hats" available off the shelf, and it will likely be much easier for you to just buy the one part from someone like <a href="https://www.adafruit.com/product/2324" target="_blank">AdaFruit</a> instead of trying to build your own and chase both hardware issues and software issues at the same time when trying to get this working. But hey, do what you want; I'm not the boss of you. I just built my own because I happened to have all of these parts on hand already.</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2OchjdP94Cv9A4uN-Mu3LrYMlfU0-vQGYNWR031ayPqGXBz0_fLy9ysrfG1dOeqacAPJIv0ovN_OKZzZ_yw5mUXhSuqABPPl5nbgjFaHwJDGpp3e_xpx3jB1TqYKkPV3GKPEadahBR3gY/s1600/smallIMG_20200229_211624.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2OchjdP94Cv9A4uN-Mu3LrYMlfU0-vQGYNWR031ayPqGXBz0_fLy9ysrfG1dOeqacAPJIv0ovN_OKZzZ_yw5mUXhSuqABPPl5nbgjFaHwJDGpp3e_xpx3jB1TqYKkPV3GKPEadahBR3gY/s400/smallIMG_20200229_211624.jpg" width="400" /></a></div>
The one key difference depending on which GPS hat you use or how you build your own, is which of the many GPIO pins the GPS PPS pin is attached to. Popular PPS pins are GPIO4 (pin 7) and GPIO18 (pin 12), but I suspect that there's no reason most of the GPIO pins can't support the PPS input, so if you're building your own board, you can pick a different GPIO pin all together. The only thing you need to make sure is that the physical GPIO pin that you use matches the GPIO pin that you configure in the /boot/config.txt file when you're enabling the PPS overlay driver. I used GPIO18 on my build.<br />
<br />
Nevermind the LEDs on the side of the perf board; those were left over from the last project I did on this perf board, and I figured they might come in handy if I wanted to program some status indicators for this NTP server.</div>
<div>
<br /></div>
<div>
The Raspberry Pi UART is tied to pins 8 and 10 on the header, so the UART should work regardless of what HAT you buy. What might vary is the exact baud rate. My modules happened to run at 9600 baud, but I've worked with GPS receivers that by default use 4800, 19200, or even 115200. Check the documentation for your receiver, and GPSD does a good job of trying to figure these things out on its own as well so it might not even matter.<br />
<br />
So to summarize:<br />
<br />
<ul>
<li>5V on Pi header to Vcc on GPS module</li>
<li>GND on Pi header to GND on GPS module</li>
<li>TX on Pi header to RX on GPS module</li>
<li>RX on Pi header to TX on GPS module</li>
<li>GPIO18 on Pi header to PPS on GPS module</li>
</ul>
</div>
<div>
<br /></div>
<h2>
Initial Raspberry Pi Setup</h2>
<div>
I'm going to mostly gloss through this part, because setting up Raspbian on a Raspberry Pi is a pretty well documented process which greatly depends on what OS you're trying to do it on, and if you're trying to use something like NOOBS or write a raw IMG file directly onto an SD card.</div>
<div>
<br /></div>
<div>
I will just say that my preferred method of creating a new Raspbian SD card from my Linux desktop is using the "dd" command, so assuming that my SD card reader came up as /dev/sdf (check the last few lines in "dmesg" to check") I do the following:</div>
<div>
<br /></div>
<br />
<div>
<pre>kenneth@thor:~/data/operatingSystems/rasberrypi$ sudo umount /dev/sdf*
umount: /dev/sdf: not mounted
kenneth@thor:~/data/operatingSystems/rasberrypi$ sudo dd if=./2019-07-10-raspbian-buster-lite.img of=/dev/sdf
4292608+0 records in
4292608+0 records out
2197815296 bytes (2.2 GB, 2.0 GiB) copied, 843.348 s, 2.6 MB/s
kenneth@thor:~/data/operatingSystems/rasberrypi$ sync</pre>
</div>
<div>
<br /></div>
<div>
I should probably download a more recent image of Raspbian Lite... It all sorts itself out when I eventually run "sudo apt update; sudo apt upgrade -y".</div>
<div>
<br /></div>
<div>
Once you're created a Raspbian SD card, put it in your Pi, and boot it. I was having some difficulty getting the Pi to boot initially with the GPS hat plugged in (likely since by default the serial port is configured as a login console, and Linux didn't like the GPS receiver spewing a bunch of NMEA sentences into it), so I left the GPS unplugged until I got a chance to disable the login console on the serial port and reboot the Pi. </div>
<div>
<br /></div>
<div>
Install SD card in Pi, plug in monitor, keyboard, Ethernet, and power, and watch Raspbian automatically grow the filesystem to fill the card, and boot. Login with the default "user: pi" "password: raspberry" and do my usual new Raspbian host setup using the raspi-config tool (most of which probably isn't critical to getting NTP working):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqv1T5Zza-GN9ryOZWQRBre3tiGhY1GZPSizeNEJvcFj07lDR4riXMAr3zHTgfWagZjTZM9KmCUA-Dqlob_itihqlJ9734AjFlJskfO90uQYZGZN-SiiwmCoB_0tqA9VHMPLlFHeJHz1FK/s1600/Screenshot+from+2020-02-29+22-23-52.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="591" data-original-width="732" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqv1T5Zza-GN9ryOZWQRBre3tiGhY1GZPSizeNEJvcFj07lDR4riXMAr3zHTgfWagZjTZM9KmCUA-Dqlob_itihqlJ9734AjFlJskfO90uQYZGZN-SiiwmCoB_0tqA9VHMPLlFHeJHz1FK/s400/Screenshot+from+2020-02-29+22-23-52.png" width="400" /></a></div>
<div>
<br /></div>
<div>
From the command line, run "sudo raspi-config" to open the Raspbian config tool. Up/down arrows to navigate, enter to select, right arrow to jump down to the select/finish options.</div>
<div>
<br /></div>
<div>
1 Change User Password -- Pick a new password so you can later enable SSH</div>
<div>
2 Network Options - N1 Hostname -- Pick a meaningful hostname for the Pi, like... ntp1pi... or something...</div>
<div>
3 Boot Options - B2 Wait for Network at Boot -- I usually turn this off for all my projects, but it likely doesn't matter for a network time server...</div>
<div>
4 Localisation Options - I1 Change Locale -- I scroll down and turn off en_GB.UTF-8 and turn on en_US.UTF-8 with the space bar. Page up/down are your friend on this dialog. I pick C.UTF-8 as my default locale. This change takes some time while Raspbian generates the new locale. </div>
<div>
4 Localisation Options - I2 Change Timezone -- I go in and pick my local timezone.</div>
<div>
4 Localisation Options - I3 Change Keyboard Layout -- This one is actually pretty important! Mainly because the default GB keyboard layout moves around a few keys like the | and @ I think? The one issue I have is that if I try and open this menu now, it displays as gibberish, so I need to reboot the Pi again before making this change.</div>
<div>
4 Localisation Options - I4 Change Wi-Fi Country -- Doesn't matter on the Pi2, but on Pi3 and above, I set the WiFi to us.</div>
<div>
5 Interfacing Options - P2 SSH -- I enable SSH, because this initial set up is both the first and the last time I plan to have a monitor and keyboard plugged into the Pi. I'll log in remotely over SSH going forward.</div>
<div>
5 Interfacing Options - P6 Serial -- <b>This is the important setting to change in here for this project!</b> You would not like a login shell to be accessible over serial (answer no) but you would still like the serial port hardware to be enabled (answer yes)</div>
<div>
7 Advanced Options - A3 Memory Split -- Since I'm never going to run a GUI on this Pi, or even ever plan to plug a monitor into it, I like to turn the GPU RAM down to the minimum of 16MB to free up a little more RAM for Linux itself.</div>
<div>
<br /></div>
<div>
When you right arrow to finally select "finish", it will probably prompt you to reboot, and you probably should, particularly because you need to do so to come back in to raspi-config a second time to change the keymap.</div>
<div>
<br /></div>
<div>
SUMMARY: So this was a bunch of my usual quality of life settings on new Raspbian images, and the one crucial step of turning off the login shell on the serial port, but leaving the serial port itself enabled.</div>
<div>
<br /></div>
<h2>
Configuring GPSD</h2>
<div>
At this point, plug in the GPS hat. </div>
<div>
<br /></div>
<div>
The GPS receiver will take some time to gain a lock and start reporting valid location/time information (Ranging from 15 seconds to 20 minutes, depending on how recently the receiver last had a fix, how good its skyview it, etc), but most receivers will still spew out NMEA sentences, even when they don't have a lock. </div>
<div>
<br /></div>
<div>
To perform our first sanity check, I installed minicom (sudo apt install minicom) and used it to open the /dev/ttyAMA0 serial port at 9600 baud to confirm that I had the GPS wired up correctly and was getting at least something that looked like this:</div>
<div>
<br /></div>
<div>
<div>
<pre>$GPRMC,064605.00,V,,,,,,,,,,N*7C
$GPVTG,,,,,,,,,N*30
$GPGGA,064605.00,,,,,0,00,99.99,,,,,,*67
$GPGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99*30
$GPGSV,1,1,01,29,,,35*75
$GPGLL,,,,,064605.00,V,N*4B</pre>
</div>
<div>
<br /></div>
<div>
At this point, we can install gpsd and configure it to handle the /dev/ttyAMA0 serial port!</div>
<div>
<br /></div>
<div>
<pre>pi@ntp1pi:~ $ sudo apt install gpsd</pre>
</div>
<div>
<br /></div>
<div>
The GPSD daemon doesn't have a configuration file, but you pass it a few options via the /etc/defaults/gpsd file to tell it what you want.</div>
<div>
<br /></div>
<div>
<pre>pi@ntp1pi:~ $ sudo nano /etc/default/gpsd</pre>
<div>
<br /></div>
<div>
We need to change two things in this file: </div>
<div>
<ol>
<li>Add "/dev/ttyAMA0" to the empty DEVICES list</li>
<li>Add the "-n" flag to the GPSD_OPTIONS field so GPSD will always try and keep the GPS receiver running. When gpsd is running on something like a phone, it makes sense to try and minimize when the GPS is running, but we don't care. I think GPSD also doesn't count NTP as a client, since NTPd is using the shared memory segment to talk to GPSD, so GPSD will randomly just stop listening to the receiver if you don't add this flag.</li>
</ol>
<div>
This is what my /etc/defaults/gpsd file looks like in the end:</div>
</div>
<div>
<div>
<pre># Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
DEVICES="/dev/ttyAMA0"
# Other options you want to pass to gpsd
GPSD_OPTIONS="-n"</pre>
</div>
<div>
<br /></div>
<div>
To apply these changes, restart GPSD:</div>
<div>
<br /></div>
<div>
<pre>pi@ntp1pi:~ $ sudo service gpsd restart</pre>
</div>
<div>
<br /></div>
<div>
To perform a sanity check that this configuration is working, we can install the gpsd clients (sudo apt install gpsd-clients) and use one of them like "gpsmon" or "cgps" to see if GPSD is even talking to the receiver, and hopefully reading good time/location information if you've left it running long enough already. You can also run "sudo service gpsd status" and should see a green dot that indicates that systemd has successfully started gpsd. (q quits out of that display)</div>
<div>
<br /></div>
<h2>
Configuring the PPS Device</h2>
<div>
The next step is to tell the Linux kernel which GPIO pin we've attach the PPS output from the GPS receiver to, so Linux can capture the pulse events and timestamp them. This is done by editing the "/boot/config.txt" file, and adding a new line with the driver overlay for pps-gpio (I usually add it at the end of the file), and specifying which GPIO pin you used at the end of the line (in my case, 18):</div>
<div>
<br /></div>
<div>
<pre>dtoverlay=pps-gpio,gpiopin=18</pre>
</div>
<div>
<br /></div>
<div>
At this point, you will have to reboot again to apply this change to the /boot/config.txt file. </div>
<div>
<br /></div>
<div>
Once the Pi has rebooted, you should see a new /dev/pps0 device, and can test it using the ppstest tool (sudo apt install pps-tools):</div>
<div>
<br /></div>
<div>
<div>
<pre>pi@ntp1pi:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1583046422.047708052, sequence: 66 - clear 0.000000000, sequence: 0
source 0 - assert 1583046423.047205387, sequence: 67 - clear 0.000000000, sequence: 0
source 0 - assert 1583046424.046730740, sequence: 68 - clear 0.000000000, sequence: 0
source 0 - assert 1583046425.046282831, sequence: 69 - clear 0.000000000, sequence: 0
source 0 - assert 1583046426.045858285, sequence: 70 - clear 0.000000000, sequence: 0</pre>
</div>
<div>
<br /></div>
<div>
If you instead see error messages about timeouts, it's possible the GPS receiver hasn't gained GPS lock yet, since many won't output PPS until they have a lock, or you've got a problem with your connections between the GPS and your selected GPIO pin...</div>
<div>
<br /></div>
<div>
One thing to note about the output of ppstest is that it timestamps each PPS event with the local time down to the nanosecond resolution. If you notice in the output above, each pulse seems to be happening 500us sooner than the pulse before, which shows that the local system clock speed is <i>grossly</i> off, since pulse per second events should be happening... once per second... not once per every 0.9995 seconds! Once you get ntpd running and disciplining the local clock off of this PPS, you should see the assert timestamp get close to all zeros after the decimal point, which means that your system clock is well aligned to the actual top of the second. So once we're all done here, it should look something like this:<br />
<br /></div>
<pre>pi@ntp1pi:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1583098783.999996323, sequence: 46821 - clear 0.000000000, sequence: 0
source 0 - assert 1583098784.999995254, sequence: 46822 - clear 0.000000000, sequence: 0
source 0 - assert 1583098785.999995642, sequence: 46823 - clear 0.000000000, sequence: 0
source 0 - assert 1583098786.999994935, sequence: 46824 - clear 0.000000000, sequence: 0
source 0 - assert 1583098787.999994174, sequence: 46825 - clear 0.000000000, sequence: 0</pre>
<br />
Notice how the local time stamps relative to the PPS input is pretty dang extremely close to once per second, near the top of each second!</div>
<h2>
Configuring NTP</h2>
<div>
At this point, we should have both GPSD and the kernel PPS driver pulling information from the GPS receiver, so now we need to install the NTP server and edit its config file to tell it to use both of these time sources!</div>
<div>
<br /></div>
<div>
<pre>pi@ntp1pi:~ $ sudo apt install ntp</pre>
</div>
<div>
<br /></div>
<div>
One thing that is a little unusual about NTP is that for local time sources, it's about the only system I've ever seen that takes advantage of the fact that the IPv4 loopback address space is an entire /8 (127.0.0.0/8), so each different type of time source, and each instance of each time source, is actually defined by a different 127.127.x.x IP address!</div>
<div>
<br /></div>
<div>
Looking at the <a href="https://www.eecis.udel.edu/~mills/ntp/html/refclock.html" target="_blank">NTP documentation for time sources</a>, the two that we are interested in are the <a href="https://www.eecis.udel.edu/~mills/ntp/html/drivers/driver22.html" target="_blank">PPS clock discipline</a> (driver 22) and the <a href="https://www.eecis.udel.edu/~mills/ntp/html/drivers/driver28.html" target="_blank">shared memory driver</a> (driver 28). </div>
<div>
<br /></div>
<div>
Since we are interested in using the 0th PPS device (/dev/pps0), the server address we want for the PPS clock source is 127.127.22.0. Likewise, for the SHM(0) driver, we want address 127.127.28.0. Notice how the third octet in both of these IPv4 addresses correspond to the number of the driver we're using, and the fourth octet is the instance of that driver that we're using (the 0th/first for both). For example, if we were monitoring multiple PPS devices for some reason, we could configure multiple server addresses: 127.127.22.0 for /dev/pps0, 127.127.22.1 for /dev/pps1, 127.127.22.2 for /dev/pps2, etc. For this blog post, we're just looking at one of each...</div>
<div>
<br /></div>
<div>
We also need to configure a few flags for each of these time sources, so the new chunk of text we're adding to /etc/ntp.conf should look like this (<a href="https://web.archive.org/web/20141207104123/http://ntpi.openchaos.org/pps_pi/" target="_blank">thanks to this blog post for this snippet</a>):</div>
<div>
<br /></div>
<div>
<pre># pps-gpio on /dev/pps0
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS
fudge 127.127.22.0 flag3 1 # enable kernel PLL/FLL clock discipline
# gpsd shared memory clock
server 127.127.28.0 minpoll 4 maxpoll 4 prefer # PPS requires at least one preferred peer
fudge 127.127.28.0 refid GPS
fudge 127.127.28.0 time1 +0.130 # coarse offset due to the UART delay</pre>
<div>
<br /></div>
<div>
The first half configures the pulse per second device:</div>
<div>
<ul>
<li>The minpoll 4 maxpoll 4 options on the server line tell NTP to always poll the PPS device every 16 seconds (2 raised to the power of 4) instead of the default "start at 64 second intervals and back off to 1024 second intervals" that ntpd uses by default, since we're not sending queries to remote NTP servers here! We're just looking at events from a local piece of hardware.</li>
<li>The "fudge 127.127.22.0 refid PPS" line assigns a human readable identifier of ".PPS." to this time source. Again, if you were doing something squirrely like monitoring multiple PPS devices (i.e. "PPS1", "PPS2", etc), or just wanted to assigned a different name to this server than "PPS", you could change that here.</li>
<li>The "fudge 127.127.22.0 flag3 1" line enables the kernel Phase Locked Loop clock discipline... which is about all I can say about it... It sounds important!</li>
<li>Same thing for the "server 127.127.28.0 minpoll 4 maxpoll 4 prefer" line with regards to the minpoll/maxpoll options; query the shared memory driver every 16 seconds. The "prefer" option tells ntpd to prefer this time source, which according to the comment seems to be required, but I don't quite follow why, since this is a stratum 0 time source, so I'd expect ntpd to end up preferring it anyways.</li>
<li>Again, "fudge 127.127.28.0 refid GPS" is assigning a human readable refid to this time source, which in this case is ".GPS." to indicate that this is the NMEA data over the serial port, vs the PPS coming in over the GPIO pin.</li>
<li>The oddest line is probably "fudge 127.127.28.0 time1 +0.130" which adds a 130ms offset to the exact time reported from the UART. Remember how I said that the precise time from the UART tends to be pretty bad, since it takes a while to deliver the data over the serial port at 9600 baud, and the exact length of the message will vary second to second? This 130ms offset is a crude approximation of how long it takes to send the NMEA report on the second, so that this clock will at least not be <i>grossly</i> off from true time. You will still see a few ms offset, and plenty of jitter, but at least the offset won't be huge!</li>
</ul>
<div>
So given this chunk of configuration, we add that to /etc/ntp.conf. Granted, even though we're setting up a stratum 1 time server here, it will likely still be wise to leave some other NTP servers in the configuration, so in case our GPS receiver dies for some reason, our server can still get its time from another server and will simply increment its stratum from 1 to one more than whichever other server it has selected as its system peer. Why the stock Raspbian ntp.conf comes with four pools configured (the pool gets expanded into multiple servers, so I don't think you need four of them) is beyond me...</div>
<div>
<br /></div>
<div>
My /etc/ntp.conf file ends up looking like this!</div>
</div>
<div>
<pre># /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/lib/ntp/ntp.drift
leapfile /usr/share/zoneinfo/leap-seconds.list
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
pool 0.debian.pool.ntp.org
# pps-gpio on /dev/pps0
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS
fudge 127.127.22.0 flag3 1 # enable kernel PLL/FLL clock discipline
# gpsd shared memory clock
server 127.127.28.0 minpoll 4 maxpoll 4 prefer # PPS requires at least one preferred peer
fudge 127.127.28.0 refid GPS
fudge 127.127.28.0 time1 +0.130 # coarse offset due to the UART delay
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
# Needed for adding pool entries
restrict source notrap nomodify noquery</pre>
<div>
<br /></div>
<div>
To apply this new configuration, you need to tell Linux to restart ntp:</div>
<div>
<br /></div>
<div>
<pre>pi@ntp1pi:~ $ sudo service ntp restart</pre>
</div>
<div>
<br /></div>
<h2>
Checking Our Work</h2>
<div>
So now the BIG question is if this all actually worked. I didn't see any signs of life right away, so I did try rebooting the Pi, which might be required.</div>
<div>
<br /></div>
<div>
The tool you can use to interrogate the local NTP server with regards to what peers it's monitoring and what offsets it has calculated is "ntpq".</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHGqtT5HMFegj7B9nZfLQB6zxb-JEb33n_cDkdEWE68YGaQkgVnOnJPUflVP5OEz7z6Y6LHBvA35-_htdYGVh6b_PwPtLrVHyDE4EQFI3uPaf-RXkcEGvd-_Yo60mNqoQL0DwQXykpVAQn/s1600/Screenshot+from+2020-03-01+00-40-55.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="192" data-original-width="724" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHGqtT5HMFegj7B9nZfLQB6zxb-JEb33n_cDkdEWE68YGaQkgVnOnJPUflVP5OEz7z6Y6LHBvA35-_htdYGVh6b_PwPtLrVHyDE4EQFI3uPaf-RXkcEGvd-_Yo60mNqoQL0DwQXykpVAQn/s640/Screenshot+from+2020-03-01+00-40-55.png" width="640" /></a></div>
Here you can see the output of the "ntpq -p" command. Notice how the reach for the SHM(0) remote is no longer zero! This might take a few minutes, and once the NTP server can reach the shared memory segment, it will wait a few <i>more</i> minutes before it starts also polling the PPS, so don't freak out if you don't start seeing that reach value incrementing as well. It seems to typically take my server about 5-10 minutes of just monitoring the SHM(0) source before it starts also querying the PPS(0) source. If after 15-30 minutes you still see a 0 reach for both local clocks, you should investigate all of the sanity checks we did above (is the pps event being seen by the kernel, is gpsd running and happy, etc)<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVQfKkv72fPCNgt9CXA2LQRO4DdSG0JxbS9vLTsIfuyajY2R5Rc36u4ARF1hJIJcr83wZNcdEH-L-SegeC2awzQiPdUoebO-AY68U84Y21AwFqTDOo8Jde7BrTMOeYxqMBybyVZvGHYgwF/s1600/Screenshot+from+2020-03-01+13-00-59.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="239" data-original-width="722" height="209" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVQfKkv72fPCNgt9CXA2LQRO4DdSG0JxbS9vLTsIfuyajY2R5Rc36u4ARF1hJIJcr83wZNcdEH-L-SegeC2awzQiPdUoebO-AY68U84Y21AwFqTDOo8Jde7BrTMOeYxqMBybyVZvGHYgwF/s640/Screenshot+from+2020-03-01+13-00-59.png" width="640" /></a></div>
After allowing my NTP server to run for several hours, we can see how the PPS(0) offset and jitter have gone to essentially zero. The SHM(0) offset and jitter are to be expected, since like I said, that precise timing is based off how long it takes to read the data over the serial port.<br />
<br />
And with that, we have a working NTP server! The last time I built one of these, I didn't have access to a cabinet in a datacenter, so I will need to play around with this and see how it performs under load, but that's another project on its own...</div>
</div>
</div>
</div>
</div>
</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-57708263170234264862020-02-25T15:00:00.000-08:002020-02-29T20:20:49.260-08:00Introduction to Network Time Protocol<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjabOXj_S6PxFu1RnpcAMvPfoEWJQDz5-DqkI9R36OZs69LgGbj4YDQohUJIclriWpi4FpgU6xqqN-tOMCurWqDq6MERlHxtQtEhzLjr1vrkhbE6vAZyUgrtpSUIFOBcHFoN9WTF31aFjxS/s1600/Screenshot+from+2020-02-29+20-20-06.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="657" data-original-width="392" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjabOXj_S6PxFu1RnpcAMvPfoEWJQDz5-DqkI9R36OZs69LgGbj4YDQohUJIclriWpi4FpgU6xqqN-tOMCurWqDq6MERlHxtQtEhzLjr1vrkhbE6vAZyUgrtpSUIFOBcHFoN9WTF31aFjxS/s400/Screenshot+from+2020-02-29+20-20-06.png" width="237" /></a></div>
<br />
I guess before I start this, I should mention that I am no longer funemployed, and now work for Arista! Arista primarily sells Ethernet switches, and after working with their products so much for the last few years for my <a href="https://fcix.net/" target="_blank">Internet Exchange project</a>, it was a pretty easy sell for them to convince me to come over and join their customer engineering team, so I now get to spend my day solving customer problems.<br />
<br />
As part of that, I can now write blog posts for Arista's blog! Granted, you need a (free) EOS Central login to view it... but <a href="https://eos.arista.com/introduction-to-ntp/" target="_blank">my first article covering the basics of the Network Time Protocol is live</a>!Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-42997919186193344482020-01-09T13:00:00.000-08:002020-01-09T13:31:11.728-08:00Teardown of a Sonoff Basic WiFi Relay<a href="https://www.youtube.com/watch?v=o1KkrNoqlPM" target="_blank">Video</a>:<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/o1KkrNoqlPM" width="560"></iframe><br />
<br />
The Sonoff Basic is a low cost IoT WiFi connected relay which you can use to switch things like lights or small fans. The Sonoff is particularly hackable, since it brings out all the pins needed to reflash the firmware and add remote buttons/indicators. Third party firmware images like Tasmota (<a href="https://tasmota.github.io/docs/">https://tasmota.github.io/docs/</a>) allow you to directly control these using MQTT or as a Hue device.<br />
<br />
Sonoff Basics are available on Amazon (<a href="https://amzn.to/2uwKYea">https://amzn.to/2uwKYea</a>) as well as many other online retailers. Sonoff (<a href="https://sonoff.tech/">https://sonoff.tech/</a>) also has many other form factors of relays other than the Basic supported by Tasmota.Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-73863263512302795772020-01-09T10:00:00.000-08:002020-01-09T13:29:48.870-08:00How to power ANYTHING using USB-C Power Delivery<a href="https://www.youtube.com/watch?v=aIHj3qMRqqE" target="_blank">Video</a>:<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/aIHj3qMRqqE" width="560"></iframe><br />
<br />
A video on how to use ZY12PDN trigger boards to negotiate various voltages from USB-C ports. The USB-C Power Delivery Trigger boards are available on <a href="https://amzn.to/2T8tmPX" target="_blank">Amazon</a>.Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-33847088874078909452019-10-01T10:00:00.000-07:002019-10-01T10:00:05.920-07:00Building a Storage Shelf for All My Digital Storage<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5XWkFfpH-uXlLhJL0XRdGhhG66sj-6S0vNAcECN8zs3FOgTf3RLacYn60lcTlpzFTSndmualO-t2uYNjviwc_dJHBrEBp9leaJ5QO5NuyTBM-cAAintZIaa0EZg6mmdcpDpiQm7XnZYGB/s1600/small.IMG_20190904_200928.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5XWkFfpH-uXlLhJL0XRdGhhG66sj-6S0vNAcECN8zs3FOgTf3RLacYn60lcTlpzFTSndmualO-t2uYNjviwc_dJHBrEBp9leaJ5QO5NuyTBM-cAAintZIaa0EZg6mmdcpDpiQm7XnZYGB/s400/small.IMG_20190904_200928.jpg" width="400" /></a></div>
I've been using a Synology DS1813+ NAS (<a href="https://amzn.to/30eJkNd" target="_blank">modern equivalent is the DS1819</a>) in my apartment for the last few years since one of my friends gave it to me. It's a little pricey, but if this one rolled over and died, I would definitely replace it with another; once you stop wanting to spend your time mucking with the infrastructure in your life, the "just works" experience of the Synology is pretty attractive.<br />
<br />
Unfortunately, what hasn't been very attractive is the dusty corner it's been living in. I finally got tired of blowing the dust out of it regularly and at least stood it up on a milk crate, but this could be improved. At the same time, my apartment has been slowly collecting other spare hard drives on various horizontal surfaces, as leftovers from various different projects and the like.<br />
<br />
So I decided this would make for another good woodworking project this summer, so I decided to make a storage shelf... for my storage media... it's a <i>storage</i> shelf.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgehTvBOJDwlI0yuXNJO5VP2UPcS9filFRuj_XRjqE0IlZSpkklipwrX1lcZk7rQ9exAUKhkK3xUllHz0L-GjBfLD69IpeXx_i6DR3qliNuSN2USIOXG1UKa8AxD9E7Q5_iOhRcOPGEn_6Q/s1600/small.IMG_20190904_202050.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgehTvBOJDwlI0yuXNJO5VP2UPcS9filFRuj_XRjqE0IlZSpkklipwrX1lcZk7rQ9exAUKhkK3xUllHz0L-GjBfLD69IpeXx_i6DR3qliNuSN2USIOXG1UKa8AxD9E7Q5_iOhRcOPGEn_6Q/s400/small.IMG_20190904_202050.jpg" width="300" /></a></div>
Might I saw, this turned out pretty gorgeous. It was made from about a third of a sheet of 3/4" AB grade plywood with a red oak veneer on it. All of the joints were fairly regular <a href="https://en.wikipedia.org/wiki/Rabbet" target="_blank">rabbet</a> or <a href="https://en.wikipedia.org/wiki/Dado_(joinery)" target="_blank">dado</a> joints cut to fit the mating piece of plywood. The left shelf was specifically sized to fit my UPS, which I do regret a little since I didn't consider how warm it would get in such a tight space, and the right two shelves were sized specifically to hold two rows of 3.5" hard drives, so the shelf can hold 28 cold drives, in addition to the eight drives running in the Synology itself. (For those wondering, I'm running it with 7x4TB drives in RAID6, and a 128GB SSD read cache drive in the 8th bay)<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaBkEmJZbozj99FobE2VMocL1poIW12FSt8eyZhqWqWs-owjFW-SDXRXwFtQ7-gyfu5eeSIOxWGQAg9dXPsybAZ1nwW2Lt5rMU2xSBD8EWhd15M831Dj-Op3_5hQ1xSzwJ1M4RFRINOqjM/s1600/small.IMG_20190904_202321.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaBkEmJZbozj99FobE2VMocL1poIW12FSt8eyZhqWqWs-owjFW-SDXRXwFtQ7-gyfu5eeSIOxWGQAg9dXPsybAZ1nwW2Lt5rMU2xSBD8EWhd15M831Dj-Op3_5hQ1xSzwJ1M4RFRINOqjM/s400/small.IMG_20190904_202321.jpg" width="300" /></a></div>
The finish was three coats of semi-gloss polyurethane, which took longer than the actual wood cutting since you need to let each coat dry and re-sand it again with something fine like 400 grit for the next coat, and I could only work on this the few days I dropped by my parent's house to work in the woodshop and also wasn't creating dust.<br />
<br />
Over all, it ended up taking about six weeks to get it all the way from concept to finished. I was tweeting about it and my other various projects quite a bit on <a href="https://twitter.com/KWF" target="_blank">my Twitter</a>, which has really been my outlet for a lot of these sorts of posts lately.Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-3692282527128801122019-09-24T10:00:00.000-07:002019-09-24T10:00:06.467-07:00Organizing My Tubes of Integrated Circuits<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-UU97OvvOIp9rxBWiYI9jMa3kpO_LAMZGn-Wyk7K1Sb8FoG6WMDPVzs-1qAuzwZCop7c3Ee86Tn2B3hJh2ioihM-a0IRRDppknX-7bDrM_R55L67Z75dY9ax6JsGmo-rBNhdTYfdYCoVd/s1600/small.IMG_20190807_174146.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-UU97OvvOIp9rxBWiYI9jMa3kpO_LAMZGn-Wyk7K1Sb8FoG6WMDPVzs-1qAuzwZCop7c3Ee86Tn2B3hJh2ioihM-a0IRRDppknX-7bDrM_R55L67Z75dY9ax6JsGmo-rBNhdTYfdYCoVd/s400/small.IMG_20190807_174146.jpg" width="400" /></a></div>
Like many electronics hobbyists, I've been slowly collecting tubes of integrated circuits and LED displays that come in plastic tubes. Sadly, this collection has literally spent the last ten years as a slowly growing pile of tubes sitting on the floor of my lab. One of those problems that really sneaks up on you, unfortunately.<br />
<br />
I decided to try and get back into the woodworking hobby this summer, so I decided to make a small set of vertical cubbies for my IC tubes, and used it as an excuse to practice my <a href="https://en.wikipedia.org/wiki/Dovetail_joint" target="_blank">dovetail joinery</a>, since a storage rack made out a cheap piece of whitewood 1x12 board from Home Depot is pretty low stakes.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi2tjBgkV5ivzBo531rgNURqIVGXxMfr8z6fwpR8cYtEX_vDEm0TpeFDez6oeGARpydoZaTw7Ih7OJB4sfvMU756hdiX_MNCaZgjz6WA0_sr_jJNwArAJKptkrEob-M_e7aavPANspvafc/s1600/small.MVIMG_20190807_161500.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi2tjBgkV5ivzBo531rgNURqIVGXxMfr8z6fwpR8cYtEX_vDEm0TpeFDez6oeGARpydoZaTw7Ih7OJB4sfvMU756hdiX_MNCaZgjz6WA0_sr_jJNwArAJKptkrEob-M_e7aavPANspvafc/s400/small.MVIMG_20190807_161500.jpg" width="300" /></a></div>
The main body of the rack is all made from a piece of 1x12 pine, with dados cut on the inside before I put it together to accept two pieces of 1/4" plywood as dividers. For finish, I just used one coat of boiled linseed oil, which is pretty minimal, but enough considering the fact that this piece isn't meant to be at all attractive. I mean... the bar has been set awfully low for the last ten years at "random pile of debris"<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcCLAjG2PvtJeZ4xw6AWOyFRw8f8xFQ8E5KUmezyHz6_aUClFuCH_wNJTQ6-COUsSWdAGc3ItaZcEIXM8k3poPOoqXTr_iDonxi3ijGZXZJK2n07G-5i90k9XciZWfORgBd8w6vUDvMlqH/s1600/small.IMG_20190807_174447.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcCLAjG2PvtJeZ4xw6AWOyFRw8f8xFQ8E5KUmezyHz6_aUClFuCH_wNJTQ6-COUsSWdAGc3ItaZcEIXM8k3poPOoqXTr_iDonxi3ijGZXZJK2n07G-5i90k9XciZWfORgBd8w6vUDvMlqH/s400/small.IMG_20190807_174447.jpg" width="300" /></a></div>
I think it turned out pretty well for being my first attempt at dovetails. My biggest mistake was that since this board was too wide to fit in my jointer, I just went "eh; it's kind of flat enough..." -- It <i>was not</i> flat enough to accommodate fine joinery. I ended up chasing my tail quite a bit trying to get the dovetails to fit together, since every piece of board was slightly cupped one direction or the other.<br />
<br />
But hey, less than perfect or not, this was a step forward in the never ending battle on entropy in my life, so big win!Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-6532573935298919042019-09-20T21:00:00.000-07:002019-09-20T21:52:24.158-07:00Getting Android to Work in an IPv6-Only Network<a href="http://blog.thelifeofkenneth.com/2019/02/running-nat64-in-bgp-environment.html" target="_blank">As previously covered</a>, I recently set up NAT64 on my network so that I didn't need to assign IPv4 addresses to all of my subnets or hosts and they'd still have a usable connection to the Internet at large.<br />
<br />
Unfortunately, functional NAT64 is only most useful when hosts can auto-configure themselves with IPv6 addresses, default routes, and DNS64 name servers to resolve with. Figuring out how to get this to work, and then figuring out the workarounds required to get Android to work, took some time...<br />
<br />
<h3>
What Was Intended</h3>
<br />
Since my network is running off a Cisco 6506, which is a relatively ancient Ethernet switch, it isn't running what you might call the latest and greatest version of an operating system, so it took some historical research to figure out how IPv6 host configuration was even supposed to work at the time IPv6 was implemented on the 6500:<br />
<br />
<ul>
<li>A new host connects to the network, and sends out a <a href="https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol" target="_blank">neighbor discovery protocol/ICMPv6</a> router solicitation to discover the local IPv6 subnet address and the address of a default gateway to reach the rest of the Internet.</li>
<li>The router(s) respond with the local subnet and default gateway, but the ND protocol did not originally include any way to also configure DNS servers, so the router would set the "other-config" flag, which tells the ND client that there is <i>additional</i> information they need that they should query over stateless DHCPv6.</li>
<li>The client, now having a local IPv6 address, would then send out a stateless DHCPv6 query to get the addresses for some recursive DNS servers, which the router would answer.</li>
<li>The client would now have a <a href="https://howdoesinternetwork.com/2013/slaac-ipv6-stateless-address-autoconfiguration" target="_blank">self-configure SLAAC address</a>, default gateway, and RDNSS (recursive DNS server), which together enable it to usefully interact with the rest of the Internet.</li>
</ul>
<div>
Great! How do you implement this in IOS?<br />
<br />
First you need to define an IPv6 DHCP pool, which ironically isn't really an address pool at all, but just some DNS servers and a local domain name. Realistically, it could be a pool, since IOS does support prefix delegation, but we're not using that here, so we just define what DNS server addresses to use and a local domain name if we felt like it:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">ipv6 dhcp pool ipv6dns</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> dns-server 2001:4860:4860::6464</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> dns-server 2001:4860:4860::64</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> domain-name lan.example.com</span><br />
<div>
<br /></div>
<div>
Since this pool doesn't even really have any state to it, other than maybe defining a different domain-name per subnet, you can reuse the same pool on every subnet that you want DHCPv6 service on, which is what I'm doing on my router, since the domain-name doesn't really make any difference:</div>
<div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">interface Vlan43</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> description IPv6 only subnet</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> no ip address</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ipv6 address 2001:DB8:0:1::1/64</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ipv6 nd other-config-flag</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ipv6 dhcp server ipv6dns</span></div>
</div>
<div>
<br /></div>
<div>
IOS sends out router advertisements by default, so to enable handing out addresses and default gateways we just need to not disable that, but the "ipv6 nd other-config-flag" option is what sets the bit in the router advertisements to tell clients to come back over DHCPv6 to also ask for the DNS server addresses. </div>
<br />
Now, before I outline the pros and cons for this design, I will disclaim that this is my perspective on the issue, so I'm not speaking from a place of authority, having not yet graduated elementary school when all of this originally was designed... This back and forth of ND - DHCPv6 does have some upsides:</div>
<div>
<ul>
<li>Since both the ND and DHCPv6 queries are "stateless", all that the router/DHCP server is doing is handing out information it knows the answers to, and isn't adding any per-device information into any sort of database like a DHCP lease table, so a single router could now conceivably assign address to a metric TON of devices.</li>
<li>The separation of the DNS configuration from the IPv6 addressing configuration preserves the elegant <a href="https://en.wikipedia.org/wiki/Separation_of_concerns" target="_blank">separation of concerns</a> that protocol designers like to see because it makes them feel more righteous.</li>
</ul>
<div>
There are also some pretty serious downsides:</div>
</div>
<div>
<ul>
<li>Instead of just a DHCP server, you now need both a correctly configured router advertisement for the L3 addressing information and a correctly configured DHCPv6 server to hand out DNS information.</li>
<li>I still don't understand how hostname DNS resolution is supposed to work for this. In IPv4 land, you use something like dnsmasq which both hands out the DHCP leases and then resolves the hostnames back to those same IP addresses. Since all of this host configuration in IPv6 is stateless, by design DNS can't work... Maybe the presumption was that dynamic DNS wouldn't turn out to be a still-born feature?</li>
<li>The Android project, for reasons which defy understanding, refuses to implement DHCPv6. </li>
</ul>
<div>
That last point is a hugely serious issue for my network, since without DHCPv6, there is no mechanism for my Cisco 6506 to communicate to my phone what DNS servers to use over IPv6. My phone gets an IPv6 address and default gateway from my Cisco's router advertisement ICMPv6 packet, and then ignores the "other-config" flag, and is left without any way to resolve DNS records. </div>
</div>
<div>
<br /></div>
<div>
Making the network... you know... <b><i>useless</i></b>.</div>
<div>
<br /></div>
<div>
For the record, how Android is presumed to work is by utilizing a later addition to the ICMPv6 router advertisement format, <a href="https://tools.ietf.org/html/rfc6106" target="_blank">RFC6106</a>, which added a Recursive Domain Name Service Server (RDNSS) option to the router advertisement to allow DNS information to be included in the original RA broadcast along with the local subnet and default gateway addressing information. Unfortunately, since this addition to ICMPv6 was made about fifteen years late, RDNSS options aren't supported by the version of IOS I'm running on my Cisco 6506, so it would seem I'm pretty shit out of luck when it comes to running Android on my IPv6-only subnets of my network.</div>
<div>
<br /></div>
<h3>
My (Really Not) Beautiful Solution</h3>
<div>
<br /></div>
<div>
So we've got a router that doesn't support the RDNSS option in its router advertisements since it predates the concept of RA RDNSS, and we have one of the most popular consumer operating systems which refuses to support DHCPv6, leaving us as an impasse for configuring DNS servers. I actually spent a few weeks thinking about this one, including slowly digging deeper and deeper into the relevant protocols, before I eventually was reading the raw specifications for the ICMPv6 router advertisement packets (kill me), and realized that there was the ability to have a router broadcast Router Advertisement packets while indicating in the RA packet that they <i>shouldn't</i> be used for routing.</div>
<div>
So here's my solution, which admittedly even I think feels a little dirty, but an ugly solution that works... still works.</div>
<div>
<br /></div>
<div>
<ul>
<li>My Cisco 6506 sends out Router Advertisements specifying the local subnet, and that it should be used as a default gateway.</li>
<li>I then spun up a small Linux VM on the same subnet running <a href="http://www.litech.org/radvd/" target="_blank">radvd</a>, which advertises that this VM <i>shouldn't</i> be used as a default gateway, but <i>does</i> advertise an RDNSS option pointing to my DNS64 resolver as the DNS server to use, since radvd supports RDNSS.</li>
<li>Any normal functional IPv6 stack will receive the Cisco's RA packet, note the "other-config" flag, and send a DHCPv6 query to receive the address for a DNS server.</li>
<li>Android receives the Cisco's RA, configures its address and gateway, but ignores the "other-config" flag. The phone will then receive a <i>second</i> RA packet from the Linux server running radvd, which includes the RDNSS option which I'm not able to configure on my router, and Android maybe will merge the two RA configurations together to collectively generate a functional subnet, gateway, and DNS server configuration for the network.</li>
</ul>
<div>
Now let us be very clear; while I was setting this up, I was <b>very confident</b> that this was not going to work. Expecting Android to receive two different RAs from two different hosts with only parts of the information it needed, and combining those together, seems like an insane solution to this problem.</div>
</div>
<div>
<br />
The bad news is that this actually worked.<br />
<br />
So we spin up a VM, install radvd on it, and assign it one network interface on my WiFi vlan. The /etc/radvd.conf file is relatively short:<br />
<span style="font-family: Courier New, Courier, monospace;">kenneth@kwfradvd:/etc$ cat radvd.conf </span><br />
<span style="font-family: Courier New, Courier, monospace;">interface ens160 {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span style="white-space: pre;"> </span>AdvSendAdvert on;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span style="white-space: pre;"> </span>IgnoreIfMissing on;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span style="white-space: pre;"> </span>AdvDefaultLifetime 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span style="white-space: pre;"> </span>RDNSS 2001:4860:4860::6464 {};</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span style="white-space: pre;"> </span>RDNSS 2001:4860:4860::64 {};</span><br />
<span style="font-family: Courier New, Courier, monospace;">};</span><br />
<div>
<br /></div>
The network interface happens to be ens160, the "AdvDefaultLifetime 0;" parameter indicates that this router shouldn't be used as a default gateway, and the two RDNSS parameters specify which IP addresses to use for DNS resolution. Here I show it with both of Google's DNS64 servers, but in reality I'm running my own local DNS64 server, because I'm directly peered with two root DNS servers and b.gtld, so why not run my own resolver?<br />
<br />
I still feel a little dirty that this works, but it does! I then installed an access point in each of my data center cabinets on this vlan advertising the SSID "_Peer@FCIX" so we're both advertising my little Internet Exchange project and I've got nice fast WiFi right next to my cabinet.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-54299067137163107532019-09-19T21:00:00.000-07:002019-09-19T21:28:39.043-07:00Using Catalog Zones in BIND to Configure Slave DNS ServersI was recently asked to help one of my friends think about re-architecting his anycast DNS service, so I've been thinking about DNS a lot on the backburner for the last few months. As part of that, I was idly reading through the whole ISC knowledge base this week, since they've got a lot of really good documentation on BIND and DNS, and I stumbled across <a href="https://kb.isc.org/docs/aa-01401" target="_blank">a short article talking about catalog zones</a>.<br />
<br />
Catalog zones are this new concept (well, kind of new; there's speculative papers about the concept going back about a decade) where you use DNS and the existing AXFR transfer method to move configuration parameters from a master DNS server to its slaves.<br />
<br />
In the context of the anycast DNS service I'm working on now, this solves a pretty major issue which is the question on how to push new configuration changes to all of the nodes. This DNS service is a pretty typical secondary authoritative DNS service with a hidden master. This means that our users are expected to run their own DNS servers serving their zones, and we then have one hidden master which transfers zones from all these various users' servers, and then redistribute the zones to the handful of identical slave servers distributed worldwide which all listen on the same anycast prefix addresses.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxoO8ZGQ8QbrDOT40fj-Eb2Bl9qifhTHs5harMNIQklxtO-f3QiaZRCxqGguHUk85MPq42XVU0aZlmotVf1wSIOQ1D2hO35CYc7xH3tDiNdhSiHNq_fRbGwyb9KTkUrTfiU1ldF8P6eB-P/s1600/Hidden+Master.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="804" data-original-width="1002" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxoO8ZGQ8QbrDOT40fj-Eb2Bl9qifhTHs5harMNIQklxtO-f3QiaZRCxqGguHUk85MPq42XVU0aZlmotVf1wSIOQ1D2hO35CYc7xH3tDiNdhSiHNq_fRbGwyb9KTkUrTfiU1ldF8P6eB-P/s400/Hidden+Master.png" width="400" /></a></div>
Updates to existing zones is a standard part of the DNS protocol, where the customer updates their zone locally, and when they increment their SOA serial number, their server sends a notify to our hidden master, which initiates a zone transfer to get the latest version of their zone, and then sends a notify to the pool of anycast slaves to all update their copy of the customer zone from the hidden master. Thanks to the notify feature in DNS, pushing an updated zone to all of these slave authoritative servers happens pretty quickly, so the rest of the Internet sending queries to the anycast node nearest to them start seeing the new zone updates right away.<br />
<br />
The problem is when you want to add new zones to this whole pipeline. After our customer has created the new zone on their server and allowed transfers from our master, we need to configure our hidden master as a slave for that zone pointed to the customer server, and we need to configure <b>all</b> of the anycast nodes to be a slave for the zone pointed back at our hidden master. If we miss one, you start experiencing very hard to troubleshoot issues where it seems like we aren't correctly being authoritative for the new zone, but only in certain regions of the Internet depending on which specific anycast node you're hitting. Anycast systems <i>really</i> depend on all the instances seeming identical, so it doesn't matter which one you get routed to.<br />
<br />
There are, of course, hundreds of different ways to automate the provisioning of these new zones on all of the anycast nodes, so this isn't an unsolved issue, but the possible solutions range anywhere from a bash for loop calling rsync and ssh on each node to using provisioning frameworks like Ansible to reprovision all the nodes any time the set of customer zones changes.<br />
<br />
Catalog zones is a clever way to move this issue of configuring a slave DNS server for what zones it should be authoritative for into the DNS protocol itself, by having the slaves transfer a specially formatted zone from the master which lists PTR records for each of the other zones to be authoritative for. This means that adding a new zone to the slaves no longer involves changing any of the BIND configuration files on the slave nodes and reloading, but instead is a DNS notify from the master that the catalog zone has changed, an AXFR of the catalog zone, and then parsing this zone to configure all of the zones to also transfer from the master. DNS is already a really good protocol for moving dynamic lists of records around using the notify/AXFR/IXFR mechanism, so using it to also manage the dynamic list of zones to do this for is in my opinion <b>genius</b>.<br />
<br />
<h3>
Labbing It at Home</h3>
<br />
So after reading the <a href="https://kb.isc.org/docs/aa-01401" target="_blank">ISC article</a> on catalog zones, and also finding <a href="https://jpmens.net/2016/05/24/catalog-zones-are-coming-to-bind-9-11/" target="_blank">an article by Jan-Piet</a> on the matter, I decided to spin up two virtual machines and have a go at using this new feature available in BIND 9.11.<br />
<br />
A couple things to note before getting into it:<br />
<br />
<ul>
<li>Catalog zones are a pretty non-standard feature which is currently only supported by BIND. There's a <a href="https://tools.ietf.org/html/draft-muks-dnsop-dns-catalog-zones-04" target="_blank">draft RFC on catalog zones</a>, which has already moved past version 1 supported in BIND, so changes are likely for this feature in the future.</li>
<li>Both of the tutorials I read happened to use BIND for the master serving the catalog zone, and used rndc to dynamically add new zones to the running server, but this isn't required. Particularly since we're using a hidden master configuration, there's no downside to generating the catalog zone and corresponding zone configurations on the master using any provisioning system you like, and simply restarting or reloading that name server to pick up the changes and distribute them to the slaves, since the hidden master is only acting as a relay to collect all the zones from various customers and serve as a single place for all the anycast slaves to transfer zones from.</li>
<li>This catalog zone feature doesn't even depend on the master server running BIND. As far as the master is concerned, the catalog zone is just another DNS zone, which it serves just like any other zone. It's only the slaves which receive the catalog zone which need to be able to parse the catalog to dynamically add other zones based on what they receive.</li>
</ul>
<div>
We want to keep this exercise as simple as possible, so we're not doing anything involving anycast, hidden masters, or adding zones to running servers. We're only spinning up two servers, in this case both running Ubuntu 18.04, but any distro which includes BIND 9.11 should work:</div>
<div>
<ul>
<li>ns1.lan.thelifeofkenneth.com (10.44.1.228) - A standard authoritative server serving the zones "catalog.ns1.lan.thelifeofkenneth.com", "zone1.example.com", and "zone2.example.com". This server is acting as our source of zone transfers, so there's nothing special going on here except sending notify messages and zone transfers to our slave DNS server.</li>
<li>ns2.lan.thelifeofkenneth.com (10.44.1.234) - A slave DNS server running BIND 9.11 and configured to be a slave to ns1 for the zone "catalog.ns1.lan.thelifeofkenneth.com" and to use this zone as a catalog zone with ns1 (10.44.1.228) as the default master. Via this catalog zone, ns2 will add "zone1.example.com" and "zone2.example.com" and transfer those zones from ns1.</li>
</ul>
<div>
We first want to set up ns1, which is a normal authoritative DNS configuration, with the one addition that I added logging for transfers, since that's what we're playing with here. </div>
</div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns1_etc_bind_named.conf.options"></script></div>
<div>
<br /></div>
<div>
Nothing too unexpected there; turn off recursion service, and turn on logging.</div>
<div>
<br /></div>
<div>
The only particularly unusual thing about the definition of the zone files is that I'm explicitly listing the IP address for the slave server under "also-notify". I'm doing that here because I couldn't get it to work based on the NS records like I think it should, but that might also be because I'm using zones not actually delegated to me. </div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns1_etc_bind_named.conf.local"></script></div>
<div>
In my actual application, I'll need to to use also-notify anyways, because I need to send notify messages to every anycast node instance on their unicast addresses. In a real application I would also lock down the zone transfers to only allow my slaves to transfer zones from the master, since it's generally bad practice to allow anyone to download your whole zone file.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns1_etc_bind_zone1.example.com.db"></script></div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns1_etc_bind_zone2.example.com.db"></script></div>
<div>
The two example.com zone files are also pretty unremarkable. </div>
<div>
<br /></div>
<div>
Up until this point, you haven't actually seen anything relevant to the catalog zone, so this is where you should start paying attention! The last file on ns1 of importance is the catalog file itself, which we'll dig into next:</div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns1_etc_bind_catalog.db"></script></div>
<div>
<br /></div>
<div>
Ok, so that might look pretty hairy, so let us step through that line by line.</div>
<div>
<ul>
<li>Line 1: A standard start of authority record for the zone. A lot of the examples use dummy zones like "catalog.meta" or "catalog.example" for the catalog zone, but I don't like trying to come up with fake TLDs which I just have to hope isn't going to become a valid TLD later, so I named my catalog zone under my name server's hostname. In reality, the name of this zone does not matter, because no one should ever be sending queries against it; it's just a zone to be transferred to the slaves and processed there.</li>
<li>Line 3: Every zone needs an NS record, which again can be made a dummy record if you'd like, because no one should ever be querying this zone. </li>
<li>Line 5: To tell the slaves to parse this catalog zone as a version 1 catalog format, we create a TXT record for "version" with a value of "1". It's important to remember the importance of a trailing dot on record names! Since "version" doesn't end in a dot, the zone is implicitly appended to it, so you could also define this record as "version.catalog.ns1.lan.thelifeofkenneth.com." but that's a lot of typing to be repeated, so we don't.</li>
<li>Lines 7 and 8: This is where the actual magic happens, by defining unique PTR records with values for each of the zones which this catalog file is listing for the slaves to be authoritative for. This is somewhat of an abuse of the PTR record meaning, but adding new record types has proven impractical, so here we are. Each record is a [unique identifier].zones.catalog.... etc.</li>
</ul>
<div>
The one trick with the version 1 catalog zone that's implemented by BIND is that the value of the unique identifier per cataloged zone is pretty specific. It is the hexadecimal representation of the SHA1 sum of the on-the-wire format of the cataloged zone.</div>
</div>
<div>
<br /></div>
<div>
I've thought about it quite a bit, and while I can see some advantages to using a stable unique identifier like this per PTR record, I don't grasp why BIND should strictly require it, and reading the version 2 spec in the draft RFC, it looks like they might loosen this up in the future, but for now we need to generate the exact hostname expected for each zone. I did this by adding a python script to my local system based on Jan-Piet's code:</div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=dns-catalog-hash.py"></script></div>
<div>
<br /></div>
<div>
I needed to install the dnspython package (pip3 install dnspython), but I could then use this python script to calculate the hash for each zone, and add it to my catalog zone by appending ".zones" to it and adding it as a PTR record with the value of the zone itself. So looking back at line 7 of the catalog zone file, by running "<span style="font-family: Courier New, Courier, monospace;">dns-catalog-hash zone1.example.com</span>" the python script spit out the hash "<span style="font-family: Courier New, Courier, monospace;">ddb8c2c4b7c59a9a3344cc034ccb8637f89ff997</span>" which is why I used that for the record name.</div>
<div>
<br /></div>
<div>
Now before we talk about the slave name server, I want to again emphasize that we haven't utilized any unusual features yet. NS1 is just a normal DNS server serving normal DNS zones, so generate the catalog zone file any way you like, and ns1 can be running any DNS daemon which you like. Adding each new zone to ns1 involves adding it to the daemon config like usual, and the only additional step is also adding it as a PTR record to the catalog zone.</div>
<div>
<br /></div>
<div>
On to ns2! This is where things start to get exciting, because what I show you here will be the only change ever needed on ns2 to continue to serve any additional zones we like based on the catalog zone.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns2_etc_bind_named.conf.options"></script></div>
<div>
Again, we've turned off recursion, and turned on transfer logging to help us see what's going on, but the important addition to the BIND options config is the addition of the catalog-zones directive. This tells the slave to parse the named zone as a catalog zone. We do explicitly tell it to assume the master for each new zone should be 10.44.1.228, but the catalog zone format actually supports you explicitly defining per zone configuration directives like masters, etc. So just appreciate that we're using the <i>bare minimum</i> of the catalog zone feature here by just adding new zones to transfer from the default master.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns2_etc_bind_named.conf.local"></script></div>
<div>
This is the totally cool part about catalog zones right here; our local zones config file just tells the slave where to get the catalog from, and BIND takes it from there based on what it gets from the catalog.</div>
<div>
<br /></div>
<div>
If you fire up both of these daemons, with IP addresses and domain names changed to suit your lab environment, and watch the /var/cache/bind/zone_transfers log files, you should see:</div>
<div>
<ul>
<li>ns1 fires off notify messages for all the zones</li>
<li>ns2 start a transfer of catalog.ns1.lan.thelifeofkenneth.com and processes it</li>
<li>Based on that catalog file ns2 starts additional transfers for zone1.example.com and zone2.example.com</li>
<li>Both ns1 and ns2 are now authoritative for zone[12].example.com!</li>
</ul>
</div>
<div>
<br /></div>
<div>
To verify that ns2 is being authoritative like it should be, you can send it queries like "<span style="font-family: Courier New, Courier, monospace;">dig txt test.zone1.example.com @ns2.lan.thelifeofkenneth.com</span>" and get the appropriate answer back. You can also look in the ns2:/var/cache/bind/ directory to confirm that it has local caches for the catalog and example.com zones:</div>
<div>
<script src="https://gist.github.com/PhirePhly/3afff4838f4a8e31ed039d00737efb17.js?file=ns2_cachedir"></script></div>
<div>
The catalog file is cached in whatever filename you set for it in the named.conf.local file, but we never told it what filenames to use for each of the cataloged zones, so BIND came up with its own filenames for zone[12].example.com starting with "__catz__" and based on the catalog zone's name and each zone's name itself.</div>
<div>
<br /></div>
<h3>
Final thoughts</h3>
<div>
<br /></div>
<div>
I find this whole catalog zone concept really appealing since it's <i>such</i> an elegant solution to exactly the problem I've been pondering for quite a while.</div>
<div>
<br /></div>
<div>
It's important to note that this set of example configs aren't production worthy, since this was just me in a sandbox over two evenings. A couple problems off the top of my head:</div>
<div>
<ul>
<li>You should be locking down zone transfers so only the slaves can AXFR the catalog zone and all of your other zones, since otherwise someone could enumerate all of your zones and all the hosts on those zones.</li>
<li>You probably should disallow even any queries against the catalog zone. I didn't, since it made debugging the zone transfers easier, but I can see the downside to answering queries out of the catalog. It wouldn't help enumerate zones, since it'd be easier to guess zone names and query for their SOAs than guessing the SHA1 sums of the same zone names and asking for the PTR record for it out of the catalog, but if you start using more sophisticated features of the catalog zone like defining per-zone masters or other configuration parameters, you might not want to allow those to be available for query by the public.</li>
</ul>
</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-66927187899794911642019-09-19T10:00:00.000-07:002019-09-19T13:28:35.511-07:00Making a Walnut Guest WiFi Coaster<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgA3ng2ScuRmTGH_2gNQM0RDQ4KFBoGQ92wTJprB1A1zEjy8naFFwgfMd_IR-GVWbqm5X9xUTI1qaWneHHsskg07VTw7ELVmN6oJ9Vg9Ns65RY3_Avx76t6sJT5X2riPHLqjpEKniJIcaoc/s1600/IMG_20190612_143857_Small.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="1440" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgA3ng2ScuRmTGH_2gNQM0RDQ4KFBoGQ92wTJprB1A1zEjy8naFFwgfMd_IR-GVWbqm5X9xUTI1qaWneHHsskg07VTw7ELVmN6oJ9Vg9Ns65RY3_Avx76t6sJT5X2riPHLqjpEKniJIcaoc/s400/IMG_20190612_143857_Small.jpg" width="400" /></a></div>
<div>
<br /></div>
I was recently reading about the 13.56MHz NFC protocol and the standard tags you can write and read from your phone, when I realized that one of the features of NFC is that you can program tags with WiFi credentials, via the concept of NDEF records, which let you encode URLs, vCards, plain text, etc.<br />
<div>
<br /></div>
<div>
I thought this would be a good gift idea, so I bought some NFC tags on eBay, and then built a coaster around it using walnut and sand blasted glass.</div>
<div>
<br /></div>
<div>
<a href="https://www.youtube.com/watch?v=A7pLAahORrU" target="_blank">Video</a>:</div>
<div>
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/A7pLAahORrU" width="560"></iframe><br />
<br />
The main thing for building one of these is getting some NFS tags, which you can easily find on <a href="https://amzn.to/2LW8oyo" target="_blank">Amazon</a>, and writing your WiFi credentials to it as an NDEF record, which is possible using various phone apps, including the TagWritter app from NXP, which is surprisingly good for being a vendor app.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-86112341576089957942019-09-05T12:00:00.000-07:002019-09-05T12:19:04.308-07:00Adding Webseed URLs to Torrent FilesI was recently hanging out on a Slack discussing the deficiencies in the BitTorrent protocol for fast file distribution. A decade ago when Linux mirrors tended to melt down on release day, Bittorrent was seen as a boon for being able to distribute the relatively large ISO files to everyone trying to get it, and the peer-to-peer nature of the protocol meant that the swarm tended to scale with the popularity of the torrent, kind of by definition.<br />
<br />
There were a few important points raised during this discussion (helped by the fact that one of the participants had actually <a href="http://edgyu.excess.org/ols/2008/John%20Hawley%20-%20Issues%20in%20Linux%20Mirroring%3A%20Or%2C%20BitTorrent%20Considered%20Harmful..pdf" target="_blank">presented a paper on the topic</a>):<br />
<br />
<ol>
<li>HTTP-based content distribution networks have gotten VASTLY better in the last decade, so you tend not to see servers hugged to death anymore when the admins are expecting a lot of traffic.</li>
<li>Users tend to see slower downloads from the Bittorrent swarm than they do from single healthy HTTP servers, with a very wide deviation as a function of the countless knobs exposed to the user in Bittorrent clients.</li>
<li>Maintaining Bittorrent seedbox infrastructure in addition to the existing HTTP infrastructure is additional administrative overhead for the content creators, which tends to not be leveraged as well as the HTTP infrastructure for several reasons, including Bittorrent's hesitancy to really scale up traffic, its far from optimal access patterns across storage, the plethora of abstract knobs which seem to have a large impact on the utilization of seedboxes, etc.</li>
<li>The torrent trackers are still a central point of failure for distribution, and now the content creator is having to deal with a ton of requests against a stateful database instead of just serving read-only files from a cluster of HTTP servers which can trivially scale horizontally.</li>
<li>Torrent files are often treated as second class citizens since they aren't as user-friendly as an HTTP link, and may only be generated as part of releases to quiet the "hippies" who still think that Bittorrent is relevant in the age of big gun CDNs.</li>
<li>Torrent availability might be poor at the beginning and end of a torrent's life cycle, since seedboxes tend to limit how many torrents they're actively seeding. When a Linux distro drops fifteen different spins of their release, their seedbox will tend to only seed a few of them at a time and you'll see completely dead torrents several hours if not days into the release cycle. </li>
</ol>
<div>
As any good nerd discussion on Slack goes, we started digging into the finer details of the Bittorrent specification like the Distributed Hash Table that helped reduce the dependence on the central tracker, peer selection algorithms and their tradeoffs, and finally the concept of webseed.</div>
<div>
<br /></div>
<div>
Webseed is a pretty interesting concept which was a late addition to Bittorrent where you could include URLs to HTTP servers serving the torrent contents, to hopefully give you most of the benefits of both protocols; the modern bandwidth scalability of HTTP, and the distributed fault tolerance and inherent scaling of Bittorrent as a function of popularity.</div>
<div>
<br /></div>
<div>
I was aware of webseed, but haven't seen it actually used in <i>years</i>, so I decided to dig into it and see what I could learn about it and how it fits into the torrent file structure.</div>
<div>
<br /></div>
<div>
The torrent file, which is the small description database which you use to start downloading all of the actual content of a torrent, at the very least contains a list of the files in the torrent and checksums for each of the fixed-size chunks making up those files. Of course, instead of using a popular object serializer like XML or JSON (which I appreciate might not have really been as popular at the inception of Bittorrent), the torrent file uses a format I've never seen anywhere else called <a href="https://wiki.theory.org/index.php/BitTorrentSpecification#Bencoding" target="_blank">BEncoding</a>.</div>
<div>
<br /></div>
<div>
The BEncoding format is relatively simple; key-value pairs can be stored as byte strings or integers, and the file format supports dictionaries and lists, which can contain sets of further byte strings, integers, or even other lists/dictionaries. Bittorrent then uses this BEncoding format to create a dictionary named "info" which contains a list of the file names and chunk hashes which define the identity of a torrent swarm, but beyond this one dictionary in the file, you can modify anything else in the database without changing the identity of the swarm, including which tracker to use as "announce" byte-strings, or "announce-list" lists of byte-strings, comments, creation dates, etc.</div>
<div>
<br /></div>
<div>
Fortunately, the BEncoding format is relatively human readable, since length fields are encoded as ASCII integers, field delimiters are characters like ':', 'l', and 'i', but unfortunately this is all encoded as a single line with no breaks, so trying to edit this database by hand with a text editor might be a little hairy.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrxoxEoTCkLA31-1b10OE_6Mgkl-uJWgt1T6y65bBB5SMF43yTffgrUdtZxuA9I-onM1JFlF6Fa36vOGVLFnWFxvrEnoc6r9W9dtjvv2P3ciYJ5noo_DW1wHzzCYqcMZzjgRf5To7CCcrE/s1600/exodos_original.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="365" data-original-width="459" height="317" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrxoxEoTCkLA31-1b10OE_6Mgkl-uJWgt1T6y65bBB5SMF43yTffgrUdtZxuA9I-onM1JFlF6Fa36vOGVLFnWFxvrEnoc6r9W9dtjvv2P3ciYJ5noo_DW1wHzzCYqcMZzjgRf5To7CCcrE/s400/exodos_original.PNG" width="400" /></a></div>
<div>
I wasn't able to find a tremendous amount of tooling for interactively editing BEncode files; there exists a few online "torrent editors" which give you basic access to changing some of the fields which aren't part of the info dictionary, but none of them seemed to give the arbitrary key-value editing capabilities I needed to play with webseed, so I settled on a Windows tool called <a href="https://sites.google.com/site/ultimasites/bencode-editor" target="_blank">BEncode Editor</a>. The nice thing about this tool is that it's designed as an arbitrary BEncode editor, instead of specifically a torrent editor, so it has that authentic "no training wheels included" hacker feel to it. User beware. </div>
<div>
<br /></div>
<div>
As an example, I grabbed the torrent file for the eXoDOS v4 collection, which is a huge collection of 7000 DOS games with various builds of DOSBOX to make it all work on a modern system. Opening the torrent file in BEncode Editor, you can see the main info dictionary at the end of the main root dictionary, which is the part you don't want to touch since the info dictionary is what defines the identity of the torrent. In addition to that, you can see five other elements in the root dictionary, including a 43 byte byte string named "announce" which is a URI to a primary tracker to use to announce yourself to the rest of the swarm, a list of 20 elements named "announce-list" which is alternative trackers to use (the file likely contains both the single tracker and a list of trackers for backwards compatibility for Bittorrent clients which predate the concept of announce-lists?) and some byte strings labeled "comment", "created by", and an integer named "creation date", which looks like a Unix timestamp.</div>
<div>
<br /></div>
<div>
Cool! So at this point, we have an interactive tool to inspect and modify a BEncode database, and know which parts to not touch to avoid breaking things (The "info" dictionary).</div>
<div>
<br /></div>
<h3>
Now back to the original point of somehow adding webseed URLs to a torrent file</h3>
<div>
<br /></div>
<div>
Webseeding is defined in Bittorrent specification <a href="http://www.bittorrent.org/beps/bep_0019.html" target="_blank">BEP_0019</a>, which I didn't find particularly clear, but the main takeaway for me is that to enable webseeding, I just need to add a list to the torrent named "url-list", and then add byte-string elements to that list which are URLs to HTTP/FTP servers serving the same contents.</div>
<div>
<br /></div>
<div>
So first step, log into one of my web servers and download the torrent and throw the contents in an open directory. (In my case, https://mirror.thelifeofkenneth.com/lib/) For actual content creators, this should be part of their normal release workflow for HTTP hosting of the content, so this is only really needed for when you're retrofitting webseed into an existing torrent.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7vRDJs0avNCu051vXYbUHoBvJsoOGmNQTgjUwYC_Ow9ujU_5fNhp7JMr7wwQbHLPF-chx0vPuPjk8MR5_KtLu9E_qoVdF0Gest8YV9J8yK8Q3WkqRUvSkmAfvzcOCzW6V7SYNGGPLQ2fg/s1600/exodos_withwebseed.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="368" data-original-width="463" height="317" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7vRDJs0avNCu051vXYbUHoBvJsoOGmNQTgjUwYC_Ow9ujU_5fNhp7JMr7wwQbHLPF-chx0vPuPjk8MR5_KtLu9E_qoVdF0Gest8YV9J8yK8Q3WkqRUvSkmAfvzcOCzW6V7SYNGGPLQ2fg/s400/exodos_withwebseed.PNG" width="400" /></a></div>
<div>
Now we start editing the torrent file, by adding a "url-list" list to the root dictionary, and the part I found a little tricky was figuring out how to add the byte-string child to the list, which is done in BEncode Editor by clicking on the empty "url-list" list, and clicking "add" and specifying that the new element should be added as a "child" of the current element.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBOcvahBBdXz1qPgw40fVfGnuE6oodG2BQ9_0DHEtcD-_zFTWmQNZ4fcUE3DoF-iI2O2i8GVg2O5zePp7suiuapKS_DOlJ3fsCqjeJ_v0XFHnGQDL01sf-TFoljOEtIGVXXXeQ6zT9d42D/s1600/exodos_addingchildtolist.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="487" data-original-width="461" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBOcvahBBdXz1qPgw40fVfGnuE6oodG2BQ9_0DHEtcD-_zFTWmQNZ4fcUE3DoF-iI2O2i8GVg2O5zePp7suiuapKS_DOlJ3fsCqjeJ_v0XFHnGQDL01sf-TFoljOEtIGVXXXeQ6zT9d42D/s400/exodos_addingchildtolist.PNG" width="377" /></a></div>
<div>
Referring back to BEP_0019, if I end the URL with a forward slash, the client should append the info['name'] to the URL, so the binary string I'm adding as a child to the list is "https://mirror.thelifeofkenneth.com/lib/" such that the client will append "eXoDOS" to it, looking for the content at "https://mirror.thelifeofkenneth.com/lib/eXoDOS/", which is correct.</div>
<div>
<br /></div>
<div>
Save this file as a new .torrent file, and success! Now I have a version of the <a href="https://mirror.thelifeofkenneth.com/lib/eXoDOS/eXoDOS_v4.webseed.torrent" target="_blank">eXoDOS torrent with the swarm performance supplemented by my own HTTP server</a>! The same could be done for any other torrent where the exact same content is available via HTTP, and honestly I'm a little surprised that I don't tend to see Linux distros using this, since it reasonably removes the need for them to commit to maintaining torrent infrastructure since the torrent swarm can at least survive off of an HTTP server, which the content creator is clearly already running.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-13925886188709797002019-07-22T10:30:00.000-07:002019-07-22T10:40:45.647-07:00Building Your Own Bluetooth Speaker<a href="https://www.youtube.com/watch?v=K3iyVRHGuIY" target="_blank">Video</a>:<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/K3iyVRHGuIY" width="560"></iframe><br />
<br />
I recently found a nice looking unpowered speaker at a thrift shop, so I decided to turn it into a bluetooth speaker so I could use it in my apartment with my phone. The parts list is pretty short:<br />
<br />
<ul>
<li>Unpowered 8 ohm speaker from the thrift store</li>
<li><a href="https://amzn.to/2SvcO27" target="_blank">Bluetooth receiver and amplifier based on the TDA7492</a></li>
<li><a href="https://amzn.to/2JVBTQ9" target="_blank">12V power supply with 2.1x5.5mm barrel jack to match the amplifier</a></li>
</ul>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-50267820452196262692019-06-05T11:00:00.000-07:002019-06-05T11:19:43.558-07:00Using 0603 Surface Mount Components for Prototyping<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYbZ1RlHl9PIuSSZYlB4Z6dCqPWBot6eCGG0y9IQkSWQHhvtt0y-RjBEtKgdN5ph9bsPlw-eU4Y1yckuTcpAMGFW4_g6e5Gg12ArBtwYUY3GuT6E1_21MzEO0Ede286fHTPczNmd3McbCp/s1600/small.IMG_20190603_192547.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1298" data-original-width="1600" height="323" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYbZ1RlHl9PIuSSZYlB4Z6dCqPWBot6eCGG0y9IQkSWQHhvtt0y-RjBEtKgdN5ph9bsPlw-eU4Y1yckuTcpAMGFW4_g6e5Gg12ArBtwYUY3GuT6E1_21MzEO0Ede286fHTPczNmd3McbCp/s400/small.IMG_20190603_192547.jpg" width="400" /></a></div>
As a quick little tip, when I'm prototyping circuits on <a href="https://amzn.to/2Ik0LjD" target="_blank">0.1" perf board</a>, I like using 0603 surface mount components for all of my passives and LEDs, since they nicely fit between the pads. This way I don't need to bother with any wiring between LEDs and their resistors, since I can just use three pads in a row to mount the LED and their corresponding current limiting resistor and just need to wire the two ends to where they're going.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpnU2F68V0NoFH7ZsT5MkwAbpr1bWdYfZN_zPImYbMmE1dz0ko_JM4gbuhQrmSYZXok5cs0ya577H-KV-D7eLey2c5wfoORmYH8htKv0lW2PLSt8X3yEbztL3G_j4BYipn2k92NxJko_2O/s1600/small.IMG_20190605_102435.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1354" data-original-width="1600" height="337" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpnU2F68V0NoFH7ZsT5MkwAbpr1bWdYfZN_zPImYbMmE1dz0ko_JM4gbuhQrmSYZXok5cs0ya577H-KV-D7eLey2c5wfoORmYH8htKv0lW2PLSt8X3yEbztL3G_j4BYipn2k92NxJko_2O/s400/small.IMG_20190605_102435.jpg" width="400" /></a></div>
I also like using it as a way to put 0.1uF capacitors between adjacent pins on headers, so filtering pins doesn't take any additional space or wiring.<br />
<br />
The fact that this works makes sense, since "0603" means that the surface mount chips are 06 "hundredths of an inch" by 03 "hundredths of an inch", so they're 0.06" long, which fits nicely between 0.1" spaced pads.<br />
<br />
I definitely don't regret buying an <a href="https://amzn.to/31d4lVd" target="_blank">0603 SMT passives book kit</a>, which covers most of the mix of resistors and capacitors I need. I then buy a spool of anything that I manage to fully use up since it's obviously popular, so I eventually bought a spool of 0.1uF capacitors and 330 ohm resistors to restock my book when those strips ran out.Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-56581747295054178872019-05-20T22:00:00.000-07:002019-05-20T22:37:59.222-07:00Twitter-Connected Camera for Maker Faire<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwjXTXZnrvN6qcEHpNWQfuWn0IbDGN_S8P5zX4zPFCVfEHjs1HLqB6gV-vfegxawdFeLahr3V_u8eWJ56Fz1ebFs9kkeAhaeGcG2VdsZmlgSeJqD2kYSA1NtVgCXkK48KoUviWekXshP7o/s1600/20190520_212645.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwjXTXZnrvN6qcEHpNWQfuWn0IbDGN_S8P5zX4zPFCVfEHjs1HLqB6gV-vfegxawdFeLahr3V_u8eWJ56Fz1ebFs9kkeAhaeGcG2VdsZmlgSeJqD2kYSA1NtVgCXkK48KoUviWekXshP7o/s400/20190520_212645.jpg" width="400" /></a></div>
This last weekend was <a href="https://makerfaire.com/bay-area/" target="_blank">Maker Faire Bay Area</a>, which is easily one of my favorite events of the year. Part county fair, part show and tell, and a big part getting to see all my Internet friends in one place for a weekend.<br />
<br />
This year, I got struck with inspiration a few weeks before the event to build a camera that could immediately post every picture taken with it to Twitter. I don't like trying to live tweet events like that since I find it too distracting taking the photo, opening it in Twitter, adding text and hashtags, and then posting it. This camera would instead upload every picture with a configurable canned message, so every picture will be hashtagged, but I have a good excuse for not putting any effort into a caption for each picture: "this thing doesn't even have a keyboard"<br />
<br />
I went into this project with the following requirements:<br />
<br />
<ul>
<li>Simple and fast to use. Point, click, continue enjoying the Faire.</li>
<li>Robust delivery of tweets to Twitter tolerant of regularly losing Internet for extended periods of time when I walk inside buildings at Maker Faire. The camera would be tethered off my cell phone, and while cell service at Maker Faire has gotten pretty good, the 2.4GHz ISM band usually gets trashed in the main Zone 2 hall, so the camera will definitely be losing Internet</li>
<li>A very satisfying clicky shutter button.</li>
<li>A somewhat silly large size for the enclosure, to make the camera noticable and more entertaining.</li>
<li>Stretch goal: A toggle switch to put the camera in "timer delay" mode so I could place it somewhere, press the shutter button, and run into position.</li>
</ul>
<br />
The most interesting challenge for me was coming up with a robust way for the camera to be able to take one or multiple photos without Internet connectivity, and then when it regains Internet ensure that all of the pictures got delivered to Twitter. It would be possible to code some kind of job queue to add new photos to a list of pending tweets and retry until the tweet is successfully posted, but I only had a few weeks to build this whole thing, so chasing all the edge cases of coding a message queue that could even tolerate the camera getting power cycled sounded like a lot of work.<br />
<br />
I eventually realized that guaranteed message delivery for a few MBs of files while experiencing intermittent Internet is already actually a very well solved problem; email! This inspiration got me to where the camera will be running a local Python script to watch the shutter button and capture the photo, but instead of trying to get the picture to Twitter, instead attach it to an email and send it to a local mail server (i.e. Postfix) running on the Raspberry Pi. The mail server can then grapple with spotty Internet and getting restarted while trying to deliver the emails to some API gateway service to get the photos on Twitter, while the original Python script is immediately ready to take another photo and add it to the queue regardless of being online.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_HQylUrty-5YnIlEbi6_-OFNP7TZt_GZp5gid8V1tDkVWhDh1kN5zdob7uk1YMsvN11zxDoUFJz5itjEMYPhQgsy4siOYgiE_6LVsoV3zODuYySkeb6bXaMRcVWcJ_OQxKGqCzQAgMkVC/s1600/20190513_215044.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_HQylUrty-5YnIlEbi6_-OFNP7TZt_GZp5gid8V1tDkVWhDh1kN5zdob7uk1YMsvN11zxDoUFJz5itjEMYPhQgsy4siOYgiE_6LVsoV3zODuYySkeb6bXaMRcVWcJ_OQxKGqCzQAgMkVC/s400/20190513_215044.jpg" width="400" /></a></div>
<br />
<a href="https://www.youtube.com/watch?v=XK3Xbm9TuKA" target="_blank">YouTube Video</a>:<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/XK3Xbm9TuKA" width="560"></iframe><br />
<br />
<a href="https://github.com/PhirePhly/tweetcam-smtp" target="_blank">The Python script running on the Raspberry Pi is available on GitHub</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
The camera was really successful at Maker Faire! I enjoyed being able to snap photos of everything without needing to pull out my phone, and really loved the looks of amusement when people finally figured out what I was doing with a colorful box covered in stickers with a blinking LED on the front.<br />
<br />
I ended up taking several HUNDRED photos this weekend (<a href="https://cloud.thelifeofkenneth.com/index.php/s/oFBsXGzpfbmzycR" target="_blank">the good ones are available to browse here</a>). Unfortunately, this means I very quickly found out that IFTTT has a 100 tweet per day API limit, which they don't tell you about until you hit it. Talking to others, this is apparently a common complaint about IFTTT where it's great to prototype with, but as soon as you try and actually demo it, you hit some surprise limit and they cut you off. If I had known they were going to cut me off, I would have sat down and written my own email to Twitter API gateway, but by the time I realized this problem Friday of Maker Faire, sitting down to try and write an email parser and using Twitter's API directly wasn't an option anymore. GRRR. Two out of Five stars, would not recommend IFTTT.<br />
<br />
When I opened a ticket with IFTTT support, they said I was out of luck on the 100 tweet/day limit, and suggested I went and got a Buffer account, which is a paid social media queue management service, so my per day tweet limit would be higher, but we've now got one more moving part in that the photos are going Raspberry Pi - my mail server - IFTTT - Buffer - Twitter. UNFORTUNATELY, Something wasn't happy between IFTTT and Buffer, so only about 20% of the API calls to add my photos to my Buffer queue were successful. Buffer also sucked for what I was trying to do because it's more meant for uploading a weeks worth of content as a CSV to post three times a day on social media. To get Buffer to post all of my photos, I had to manually go in and schedule it to post the next photo in my queue every five minutes... all day... so I was sitting there on my phone selecting the next five minute increment and clicking "add to schedule" for quite a while. 1/5 stars, will definitely never use again.<br />
<br />
So the irony here is that the photo delivery from the Raspberry Pi in the field back to my mail server was rock solid all weekend, but getting it from my mail server to Twitter fell on its face pretty hard.<br />
<br />
The other notable problem that I ran into while stress testing the camera the week prior to Maker Faire was the fact that Postfix was getting its DNS settings when it started, and seemed to not expect the server to be roaming between various WiFi networks, so I needed to edit the /etc/dhcpcd.conf file to force the Raspberry Pi (and thus also Postfix) to just use some public DNS resolvers like 8.8.8.8 and 1.1.1.1 instead of trying to use my Phone's DNS resolver, which obviously wasn't available when my camera roamed to other WiFi networks like the one at Supply Frame's office.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ5pWVAJpGFMCBxSDtGHI0BaewvPlQGSyszQVhlWcX_VJMcIsWqin_M0ZL_4B2jbHbjsMUU9_acyhVB4e7mC_qxrb-GjtVOxOWEKvfueyRLPtE4lFXc79917MPRwLdVMSKEcmpnxTCafa5/s1600/20190520_212708.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ5pWVAJpGFMCBxSDtGHI0BaewvPlQGSyszQVhlWcX_VJMcIsWqin_M0ZL_4B2jbHbjsMUU9_acyhVB4e7mC_qxrb-GjtVOxOWEKvfueyRLPtE4lFXc79917MPRwLdVMSKEcmpnxTCafa5/s400/20190520_212708.jpg" width="400" /></a></div>
I also spent all weekend adding stickers to the camera's box, so by the end of the weekend the cardboard box was beautifully decorated.<br />
<br />
<br />
Material used for this build:<br />
<ul>
<li><a href="https://amzn.to/2DECfrQ" target="_blank">Raspberry Pi 3B</a></li>
<li><a href="https://amzn.to/2LbEZDh" target="_blank">32GB MicroSD card</a></li>
<li><a href="https://amzn.to/2XRLA7u" target="_blank">Raspberry Pi Camera module</a></li>
<li><a href="https://www.adafruit.com/product/1434" target="_blank">Pi Camera mount</a></li>
<li><a href="https://amzn.to/2XSNDIA" target="_blank">Anker PowerCore 10000</a></li>
<li><a href="https://amzn.to/2IWRlwu" target="_blank">Right Angle MicroUSB Cable</a></li>
<li>Box 3"x4"x7"</li>
<li>Velcro tape</li>
<li>Assorted wires, LEDs, resistors, buttons, solder perf board</li>
</ul>
<br />
Starting from a clean Raspbian Lite image, run raspi-config and<br />
<ul>
<li>Set a new user password and enable ssh</li>
<li>Set the locale and keyboard map to suit your needs</li>
<li>Set the WiFi country and configure your WiFi network credentials</li>
</ul>
<div>
Install the needed software and my prefered utilities</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt update</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt install postfix mutt vim dstat screen git</span></div>
<div>
<br /></div>
<div>
Starting with Postfix, we need it to be able to relay email out via a remote smarthost, since pretty much no consumer Internet connection allows outgoing connections on port 25 to send email. It's possible to use Gmail as your smarthost, so feel free to search for guides on how to specifically do that, but I just used a mail relay I have running for another project.</div>
<div>
<br /></div>
<div>
To do this, I first created a new file /etc/postfix/relay_passwd and added one line to it:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">smtp.example.com USERNAME:PASSWORD</span></div>
<div>
<br /></div>
<div>
This file gets compiled into a database (relay_passwd.db) that postfix uses to look up your username and password when it needs to log into your relay. Conceivably you could have multiple sets of credentials for different hosts in here, but I've only ever needed one for my relay.</div>
<div>
<br /></div>
<div>
I then changed the permissions on it so only root can read it, and generated the database file postfix actually uses to perform lookups against this host to username/password mapping.</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">chmod 600 /etc/postfix/relay_passwd</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">postmap /etc/postfix/relay_passwd</span></div>
<div>
<br /></div>
<div>
To configure Postfix to use this relay, I added these lines to my /etc/postfix/main.cf file:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">relayhost = [smtp.example.com]:587</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">smtp_use_tls=yes</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">smtp_sasl_auth_enable = yes</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">smtp_sasl_security_options =</span></div>
</div>
<div>
<br /></div>
<div>
At this point, you should be able to use a program like mutt or sendmail on the Raspberry Pi to send an email, and watch the respective /var/log/mail.log files to see the email flow out to the Internet or get stuck somewhere.<br />
<br />
Tweaking Postfix defer queue behavior - Since using Postfix as a message queue on a portable Raspberry Pi is a bit unusual compared to the typical Postfix application, which probably involves a datacenter, we expect the Internet connection to be quite a bit more flaky, so retries on messages should happen a lot more often than on typical mail delivery. I changed it to start retrying emails at 100 seconds and then Postfix doubles that up to the maximum of 15 minutes, which seems like a reasonable upper limit for the time my phone would go offline and then find signal again.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">queue_run_delay = 30s (300s default)</span><br />
<span style="font-family: Courier New, Courier, monospace;">minimal_backoff_time = 30s (300s default)</span><br />
<span style="font-family: Courier New, Courier, monospace;">maximal_backoff_time = 300s (4000s default)</span><br />
<br />
<a href="https://github.com/PhirePhly/tweetcam-smtp" target="_blank">The actual camera script is available on GitHub</a>, including the modified dhcpcd config file, a systemd service to start the Python script on boot, and an example copy of the configuration file that the tweetcam script looks for at /etc/tweetcam.conf which includes where to send the picture emails and what to put in the subject and body of the emails.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis40ektHK6rc5K6HGWSs3sNR0BaopLeD9nEmnSig6qdV_mxOjiRy_0IOPohqEcik_EUBj34whguA-MkLrkJ1lwppm4H-iSXvfuki8rve4y4yOismAZn29VY7DVuqm4fwNOrxa6c_KTkgEi/s1600/20190520_223348.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis40ektHK6rc5K6HGWSs3sNR0BaopLeD9nEmnSig6qdV_mxOjiRy_0IOPohqEcik_EUBj34whguA-MkLrkJ1lwppm4H-iSXvfuki8rve4y4yOismAZn29VY7DVuqm4fwNOrxa6c_KTkgEi/s400/20190520_223348.jpg" width="225" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsHl3HNAVT4GsEZlBFTP3XGFrA1n95RgQP4SAxbCmoihw1lp2XqT1QHoqypG14cCoXVu5GF3ZrVzGQItHT6jVIvycKv2flj7vnYkVpMXCwXHe6sZZXRgbbLy1daDAZxNTauxP9CsvILGCU/s1600/20190520_223427.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsHl3HNAVT4GsEZlBFTP3XGFrA1n95RgQP4SAxbCmoihw1lp2XqT1QHoqypG14cCoXVu5GF3ZrVzGQItHT6jVIvycKv2flj7vnYkVpMXCwXHe6sZZXRgbbLy1daDAZxNTauxP9CsvILGCU/s400/20190520_223427.jpg" width="225" /></a></div>
<br />
The hardware was a small square of perf board to act as an interface between the Raspberry Pi 40 pin header and the buttons / LEDs mounted in the box. For the inputs, the Pi's internal pull-up resistors were used so the button just needed to pull the GPIO pin to ground. A 0.1uF capacitor was included for debounce, which ended up not really mattering since it took a little more than a second for the python script to capture the photo and send the email.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0YnKfcT87Dy2gFXkIDDYytyzV4g41u4NUy7nJzsl0OBa88e-IzHZ6LUCbzR7w3AH_JgBeVlKZp-0KYW0_-PW_KrwCf5Kg7Je0vIznDot-HO1Abw_i5Kp74uztX8_NVd6XorusfJarSo8d/s1600/20190520_222340.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0YnKfcT87Dy2gFXkIDDYytyzV4g41u4NUy7nJzsl0OBa88e-IzHZ6LUCbzR7w3AH_JgBeVlKZp-0KYW0_-PW_KrwCf5Kg7Je0vIznDot-HO1Abw_i5Kp74uztX8_NVd6XorusfJarSo8d/s400/20190520_222340.jpg" width="225" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
I put an NPN transistor buffer on both of the LEDs using a 2N3904 transistor, which I probably could have done without, but I hadn't decided how big, bright, or how many LEDs I wanted to drive when I soldered the perf board.<br />
<br />
The camera module was plugged into the camera CSI port on the Pi.<br />
<br />
In the end, this project was a huge hit and I particularly appreciated the nice tweets I was getting from people who couldn't make it to Maker Faire and were enjoying getting to see ~500 photos of it tweeted live from me walking around. I suspect I'll end up continuing work on this project and try and fix some of the issues I encountered like the IFTTT Twitter API limits and how really underwhelming the image quality is coming out of the OEM Raspberry Pi Foundation camera module... </div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-51321880229796894322019-05-11T23:00:00.000-07:002019-05-11T23:05:20.498-07:00FCIX - Lighting Our First Dark Fiber<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwJ-sCZx0McDfWlZq4_XDWDv8DtUIwKjx9SibVAo5Grpy0Ws389sPRQDiW-EjfF6cc3xOyHQK2fvaRLP71BK0BFu7s9eDeVxfaC_PIJ33n48kwkWWRzfhvW6JLz5nKfxlMpVUnoqGrm5no/s1600/20190118_174252.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwJ-sCZx0McDfWlZq4_XDWDv8DtUIwKjx9SibVAo5Grpy0Ws389sPRQDiW-EjfF6cc3xOyHQK2fvaRLP71BK0BFu7s9eDeVxfaC_PIJ33n48kwkWWRzfhvW6JLz5nKfxlMpVUnoqGrm5no/s400/20190118_174252.jpg" width="400" /></a></div>
The Fremont Cabal Internet Exchange project continues!<br />
<br />
The <a href="http://blog.thelifeofkenneth.com/2018/12/fcix-state-of-exchange.html" target="_blank">first eight months of FCIX</a> was spent spinning up the exchange in the <a href="http://he.net/tour/fremont2.html" target="_blank">Hurricane Electric FMT2</a> building running out of my personal cabinet there. At the end of 2018, in the typical 51% mostly kidding style that is starting to become an integral part of the FCIX brand, we asked our data center host <a href="http://he.net/" target="_blank">Hurricane Electric</a> if they would be willing to sponsor us for a second cabinet in their other facility (<a href="http://he.net/tour/fremont1.html" target="_blank">HE FMT1</a>) and a pair of dark fiber between the two buildings.<br />
<br />
This felt like a pretty big ask for a few reasons:<br />
<br />
<ol>
<li>Hurricane Electric has already been a very significant sponsor in enabling FCIX to happen by their sponsoring of free cross connects for new members into the exchange so joining the exchange is a $0 invoice. </li>
<li>Asking for a whole free cabinet feels pretty big regardless of who you're talking to.</li>
<li>Hurricane Electric is in the layer 2 transport business, so us asking to become a multi-site IXP puts us in the somewhat tricky position of walking the line of being in the same business as one of our sponsors while using their donated services. </li>
</ol>
That last point is an interesting and possibly somewhat subtle one. It's also not an unusual awkward position that Internet Exchange Points are put in; when an Internet Exchange Point goes multi-site, their multiple sites are often connected by donated fiber or transport from a sponsor who's already in that business. This means that once an IXP starts connecting networks between the two facilities over the donated link, pairs of networks which might have already been leasing capacity between the two buildings might be motivated to drop their existing connection and move their traffic onto the exchange's link. Some IXPs will also enforce a policy that single networks can only connect to the IXP in one location, so members can't use the fiber donated to the IXP as internal backhaul for their own network, since that's a service they should be buying from the sponsor themselves, not enjoying the benefit of it being donated to the IXP.<br />
<br />
Do I think this is a major concern between HE FMT1 and HE FMT2 for FCIX? No. These two buildings are about 3km apart, and Hurricane Electric had made sure there is a generous bundle of fiber between the two buildings, so it is unlikely that HE is making a lot of money on transport between two buildings within walking distance.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEa6Y52N15V1BCqlibP9UnyKNBOlznqKmi8lF7m_j-mA-M4YS-Zz9CoWqGtOB4AFlidMoZs-owS5Mv-Lei_UZUH0MKUDbaVCp7KZM1clrrZtkHHPlQFf3RM3QhcELx_CrFUO881dfVWQff/s1600/Screenshot+from+2019-05-11+22-16-21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="626" data-original-width="584" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEa6Y52N15V1BCqlibP9UnyKNBOlznqKmi8lF7m_j-mA-M4YS-Zz9CoWqGtOB4AFlidMoZs-owS5Mv-Lei_UZUH0MKUDbaVCp7KZM1clrrZtkHHPlQFf3RM3QhcELx_CrFUO881dfVWQff/s400/Screenshot+from+2019-05-11+22-16-21.png" width="372" /></a></div>
So we asked, and Hurricane Electric said yes.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvoXnRIdkELMI1DS95umRovQqVP7avUBk5JV4oAslgpuTnozT7CK_TqUWpK7pYfVEMavpGIDxt22FbACUynfH09IplUN0H0NjqY-o2nlImNe9B-nU9XjgWI5wMZ4SI4Y2MQsHQBo85tetT/s1600/20190118_183451.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvoXnRIdkELMI1DS95umRovQqVP7avUBk5JV4oAslgpuTnozT7CK_TqUWpK7pYfVEMavpGIDxt22FbACUynfH09IplUN0H0NjqY-o2nlImNe9B-nU9XjgWI5wMZ4SI4Y2MQsHQBo85tetT/s400/20190118_183451.jpg" width="400" /></a></div>
At that point, we had an empty cabinet in FMT1, and a pair of dark fiber between the two buildings allocated for us, but a second site means we need a second peering switch...<br />
<br />
"Hey, Arista..."<br />
<br />
Arista was kind enough to give us a second 7050S-64 switch to load into FMT1, so we now have a matching <i>pair</i> of Ethernet switches to run FCIX on. Cool.<br />
<br />
The final design challenge was lighting this very long piece of glass between the two sites. Thankfully, 3km is a relatively "short" run in the realm of single mode fiber, so the design challenge of moving bits that far isn't too great; pretty much every single generation of fiber optic has off-the-shelf "long-reach" optics which are nominally rated for 10km of fiber, so we weren't going to need to get any kind of special long range fiber transponders or amplifiers to light the pair.<br />
<br />
In reality, they aren't really rated for 10km, so much as they're rated for a certain signal budget that usually works out well enough for 10km long links. For an example, lets take the <a href="https://www.flexoptix.net/en/transceiver/sfp-plus-lr-transceiver-10-gigabit-stm64-sm-1310nm-10km-8db-ddm-dom.html" target="_blank">Flexoptix 10G LR optic</a>, which has the following technical specifications:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbQcyZ5L8XMVaADfNBFx3K39Za7C5rAtwXPtQ-G3Wn1n8PeYW3nLJCbxe2tPzmZiBE7sbiU8xccirPlqVDVEeocKiai3dtPRA6bUJvRp3Q18IWgAhYtRVLe2BXD6PHfUzNbPv0KNbnW3O8/s1600/Screenshot+from+2019-05-11+22-23-49.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="452" data-original-width="1173" height="153" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbQcyZ5L8XMVaADfNBFx3K39Za7C5rAtwXPtQ-G3Wn1n8PeYW3nLJCbxe2tPzmZiBE7sbiU8xccirPlqVDVEeocKiai3dtPRA6bUJvRp3Q18IWgAhYtRVLe2BXD6PHfUzNbPv0KNbnW3O8/s400/Screenshot+from+2019-05-11+22-23-49.png" width="400" /></a></div>
The important numbers to focus on in this table are on the top right:<br />
<br />
<ul>
<li>Powerbudget (db): 6.2db</li>
<li>Minimum transmit power: -8.2dbm (db over a milliwatt)</li>
<li>Minimum receive power: -14.4dbm</li>
</ul>
<div>
The powerbudget is the acceptable amount of light that can be lost from end to end over the link, be it through linear km of fiber, connections, splices, attenuators, etc. So the 10km "distance" parameter is more a rule of thumb statement that a 10km link will typically have 6.2db of attenuation along it than the optic really being able to tell exactly how far it is from the other end of the fiber. </div>
<div>
<br /></div>
<div>
The minimum transmit power and minimum receive power are actually related to the powerbudget by the fact that the power budget is the difference between these two numbers. Usually, your powerbudget will be much better than this to begin with, because most optics will put out much more than -8.2dbm of power when they're new, but <i>some</i> of them <i>might</i> be that low, and the lasers will actually "cool" over their life-span, so even if an optic comes out of the box putting out -2dbm of light, as it ages that number will go down. </div>
<div>
<br /></div>
<div>
When it comes to serious <b>long</b> fiber links, it's likely you'll want to get a signal analysis done of the actual fiber you plan on using, and then using that information to plan your amplifiers accordingly. We, on the other hand, very carefully looked at how far the two buildings were on Google Maps, stroked our chins knowingly like we knew what the hell we were doing, and decided that a 10km optic would probably be good enough. Time to call another FCIX sponsor.</div>
<div>
<br /></div>
<div>
"Hey, Flexoptix..."</div>
<div>
<br /></div>
<div>
Given that we wanted to use 10km optics, we really had three choices for optics, given what our 7050S-64 switches could support: We could light it with a pair of 1G-LX optics, which would be pretty lame in this day and age, or we could light it with a pair of 10G-LR optics, which would probably be pretty reasonable given the amount of traffic we expect to be moving to/from this FMT1 extension, <b>OR</b>, we could ask Flexoptix for a pair of $400 <a href="https://www.flexoptix.net/en/transceiver/qsfp-plus-lr4-transceiver-40-gigabit-sm-4-waves-1270-1330nm-10km-4db-ddm-dom.html" target="_blank">40G LR4 optics</a> and use some of those QSFP+ ports on our Aristas... because <strike>why not?</strike> faster is better.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7puaNGDB8ni3Ny5BW4zxrHw3uB44hQ_4BZB4KytkDB63lTh4IP1x7q4hp1leEPzkPB03ekixDWU2fEObNHVgDKSgbAzhkuoagvmcBWDloAhs7j0ozDC-QnEoIgnXKcrmY1tbXZjPOQv6x/s1600/20190118_174252.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7puaNGDB8ni3Ny5BW4zxrHw3uB44hQ_4BZB4KytkDB63lTh4IP1x7q4hp1leEPzkPB03ekixDWU2fEObNHVgDKSgbAzhkuoagvmcBWDloAhs7j0ozDC-QnEoIgnXKcrmY1tbXZjPOQv6x/s400/20190118_174252.jpg" width="400" /></a></div>
<div>
So that's what we did. FCIX now has a 40G backbone between our two sites. </div>
<div>
<br /></div>
<div>
40G LR4 is actually a little mind blowing with how they get 40G across a single pair of fibers, because a single 40G transceiver isn't how they actually did it. 40G was really an extension of 10G by putting 4x10G in one optic, and there was two ways of then transporting these 4x10G streams to the other optic:</div>
<div>
<ol>
<li>PLR4, or "parallel" LR4, where you use an 8 fiber cable terminated with MPO connectors, so each 10G wavelength is on its own fiber.</li>
<li>LR4, which uses the same duplex LC fiber as 10G-LR, but uses four different wavelengths for the four 10G transceivers, and then integrates a <a href="https://en.wikipedia.org/wiki/Wavelength-division_multiplexing#Coarse_WDM" target="_blank">CWDM (coarse wave division multiplexing)</a> mux IN THE FREAKING OPTIC.</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUER5jSUwFseXiRiKkGAku9CRr_ffy08EL838fFTTLUbbtUvSYWiSz0oPSOTebAdzEkMSca-K4sbFpp6Bgyjo2rx5DfHMl3_ICL0VF9llgccyjzD7N0PL2GuWIAZNnoPObVEjNO9C1WJoX/s1600/CWDM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="310" data-original-width="980" height="126" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUER5jSUwFseXiRiKkGAku9CRr_ffy08EL838fFTTLUbbtUvSYWiSz0oPSOTebAdzEkMSca-K4sbFpp6Bgyjo2rx5DfHMl3_ICL0VF9llgccyjzD7N0PL2GuWIAZNnoPObVEjNO9C1WJoX/s400/CWDM.png" width="400" /></a></div>
<div>
<br /></div>
<div>
It's not entirely correct, but imagine a prism taking four different wavelengths of light and combining them into the single fiber, then a second prism on the receiver splitting them back out to the four receivers. Every time I think about it, it still blows my mind how awesome all of these Ethernet transceivers are once you dig into how they work.</div>
</div>
<div>
<br /></div>
<div>
So we now have 40G between our two sites, and like always, it wouldn't have been possible without all of our generous FCIX sponsors; they're pretty cool. If you happen to be an autonomous system running in either of those facilities, or want to talk to us about extending FCIX to another facility in the Silicon Valley, feel free to shoot us an email at contact@fcix.net.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-14416894145848095522019-02-03T22:00:00.000-08:002019-02-05T20:06:41.158-08:00Running NAT64 in a BGP EnvironmentIPv6 is one of those great networking technologies, like multicast, which makes your life as a network operator much easier, but is generally pretty useless outside of your own network if you try to expect it to work across the Internet.<br />
<br />
People like to focus on how the percentage of traffic going over IPv6 has slowly been creeping up, but unfortunately until that number reaches practically 100%, an Internet connection without IPv4 connectivity is effectively broken. Ideally, everyone runs dual-stack, but the whole shortage of IPv4 addresses was the original motivator for IPv6, so increasingly networks will need to run native IPv6 plus some form of translation layer for IPv4, be it carrier grade NAT on 100.64.0.0/10, 464XLAT, or as I'll cover today, NAT64.<br />
<br />
NAT64 is a pretty clever way to add IPv4 connectivity to an IPv6-only network, since it takes advantage of the fact that the entire IPv4 address space can actually be encoded in half of a single subnet of IPv6. Unfortunately, I also found it somewhat confusing since there are actually quite a few moving parts in a functional NAT64-enabled network, so I figured I'd write up my experience adding NAT64 to my own network.<br />
<br />
There are really two parts to NAT64, and to make it a little more confusing, I decided to add BGP to the mix because it's a Sunday and I'm bored.<br />
<br />
<ol>
<li>A DNS64 server, which is able to give a AAAA response for every website. For what few websites actually have AAAA records, it just returns those. For websites which are IPv4 only, it returns a synthetic AAAA record which is the A record packed into a special IPv6 prefix. The RFC recommended global prefix for this packing is 64:FF9B::/96, but there could possibly be reasons to use any /96 out of your own address space instead (I can't really think of any). I'm using the standard 64:FF9B::/96 prefix mainly because it means I can use <a href="https://developers.google.com/speed/public-dns/docs/dns64" target="_blank">Google's DNS64 servers</a> instead of running my own. </li>
<li>A NAT64 server, which translates each packet destined for an IPv4 address packed in this special IPv6 /96 prefix to an IPv4 packet from an IPv4 address in the NAT64's pool destined for the actual IPv4 address to be routed across the Internet. My NAT64 server goes a step further and NAT's all of these IPv6-mapped IPv4 addresses to one public address, since mapping IPv6 to IPv4 addresses one-for-one doesn't gain me much.</li>
<li>BGP to inject a route to 64:FF9B::/96 towards the NAT64 server into my core router. Realistically, you could just do this with a static route on your router pointed towards your NAT64's IPv6 address, but I want to be able to share my NAT64 server with friend's I'm peered with over <a href="http://fcix.net/" target="_blank">FCIX</a>, so since BGP is a routing policy protocol, it helps. If you're not running a public autonomous system, just ignore anything I say about BGP and point a static route instead.</li>
</ol>
<br />
<br />
So to do this, I'm using a software package called Tayga, along with these numbers (to make my specific example clearer):<br />
<br />
<ul>
<li>23.152.160.4/26 is the public IPv4 address of my NAT64 server. You would replace this address with whatever IPv4 address you wanted all your NATed traffic to be sourced from in the end.</li>
<li>2620:13B:0:1000::4/64 is the public IPv6 address of my NAT64, which is the gateway to the special /96 prefix containing all IPv4 addresses. This address again needs to be something specific to your network, and is mainly important because it's where your route to the /96 needs to point.</li>
<li>64:FF9B::/96 is the standard prefix for NAT64. There's no reason to change it to anything else.</li>
<li>100.65.0.0/16 is an arbitrary big chunk of local-use-only addresses I picked for this project. For how I have my NAT64 server configured, it really didn't matter what prefix of CGNAT or RFC1918 address space I picked here, since it's really only used as an intermediary between the client's actual IPv6 source address and the single public IPv4 address that they're all NATed to. A /16 is plenty of address space, since trying to NAT even 64k hosts to a single public IPv4 address is going to be a bad time. Theoretically there is ways to use CGNAT concepts to use a whole pool of IPv4 addresses on the public side, but one address is plenty for my needs.</li>
<li>AS4264646464 is an arbitrary private-use ASN I picked to be able to set up an eBGP peering session between my NAT64 server and my core AS7034 router.</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJcIU6hJ6fVBqNTXe79pMw_ytRbslMYRaO8Vv75yWwj5gRI2FFo8atGCdGeqI-5iFFk2_PAry24O91GOFcoE1yh83_8fSxZQVem4gPOVTUtx3UzTfOSL-6H8kWLkEISV7ojv2HyJ5ZdTOa/s1600/NAT64_Network.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="688" data-original-width="888" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJcIU6hJ6fVBqNTXe79pMw_ytRbslMYRaO8Vv75yWwj5gRI2FFo8atGCdGeqI-5iFFk2_PAry24O91GOFcoE1yh83_8fSxZQVem4gPOVTUtx3UzTfOSL-6H8kWLkEISV7ojv2HyJ5ZdTOa/s400/NAT64_Network.png" width="400" /></a></div>
<br />
The source IPv6-only host makes a DNS query against a DNS64 server, which encodes IPv4 A records into the bottom 32 bits of the 64:FF9B::/96 subnet as a AAAA record, which gets routed to the NAT64 server:<br />
<span style="font-family: "courier new" , "courier" , monospace;">[SOURCE IPv6 ADDRESS] > 64:FF9B:0:0:0:0:[DESTINATION IPv4 ADDRESS]</span><br />
Tayga allocates an address out of its 100.65.0.0/16 NAT pool for the IPv6 source address, and translates the packet on the nat64 loopback interface to:<br />
<span style="font-family: "courier new" , "courier" , monospace;">100.65.X.Y > [DESTINATION IPv4 ADDRESS]</span><br />
The iptables nat MANGLE rule set up by Tayga then NATs the local 100.65.0.0/16 subnet to the one public address and sends the packet on its way:<br />
<span style="font-family: "courier new" , "courier" , monospace;">23.152.160.4 > [DESTINATION IPv4 ADDRESS]</span><br />
And dramatically, the IPv6 packet has been translated into an IPv4 packet, with the NAT64 server holding two pieces of NAT state; one in Tayga for the IPv6 to IPv4 translation, then one in iptables for the CGNAT address to public address translation.<br />
<h3>
Setting up Tayga</h3>
<br />
Starting from a stock Ubuntu 18.04 system, <span style="font-family: "courier new" , "courier" , monospace;">sudo apt install tayga</span> gets us the needed NAT64 package, and edit the <a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-etc_netplan_01-netcfg-yaml" target="_blank">/etc/netplan/01-netcfg.yaml</a> file to match your network.<br />
<br />
The Tayga config (/etc/tayga.conf) is relatively short, ignoring all the comments. The main parameters to update are the "ipv4-addr" so it's inside your local use IPv4 block, "dynamic-pool" to match the ipv4-addr, "ipv6-addr" to your public IPv6 address on the server, and turn on the "prefix 64:ff9b::/96" option.<br />
<br />
<a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-etc_tayga-conf" target="_blank">Tayga config listing</a>:<br />
<script src="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767.js?file=etc_tayga.conf"></script><br />
<br />
The second file you need to edit to enable Tayga is the /etc/default/tayga file, which feeds parameters into the /etc/init.d/tayga RC file (which you don't need to touch).<br />
<br />
Key parameters to note in the defaults file is to change RUN to yes, and make sure both CONFIGURE_IFACE and CONFIGURE_NAT44 are turned on so the RC file will do all the iptables setup for us.<br />
<br />
<a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-etc_default_tayga" target="_blank">Tayga default listing</a>:<br />
<script src="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767.js?file=etc_default_tayga"></script><br />
<br />
At this point, you should be able to <span style="font-family: "courier new" , "courier" , monospace;">sudo service tayga start</span> and do a sanity check by looking at your IPv4 and IPv6 routing tables and seeing the prefixes referenced in the Tayga config added to a new nat64 tun network interface. <a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-ip_route_sanity_check" target="_blank">Listing</a>:<br />
<script src="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767.js?file=ip_route_sanity_check"></script><br />
<br />
<h3>
Enabling Forwarding in sysctl.conf</h3>
<br />
Unfortunately, while the Tayga RC file does a good job of setting up the tun interface and NAT iptable rules, the one thing it doesn't do is turn on IPv4 and IPv6 forwarding in general, which is needed to forward packets back and forth between the public interface and the nat64 tun device. Uncomment the two relevant lines in sysctl.conf and restart. <a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-etc_sysctl-conf" target="_blank">Listing</a>:<br />
<script src="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767.js?file=etc_sysctl.conf"></script><br />
<br />
<h3>
Setting up BGP to advertise the NAT64 prefix</h3>
<br />
For exporting the /96 of NAT64 prefix back into my networks core, I decided to use the Quagga BGP daemon. This is a pretty standard configuration for Quagga, exporting a directly attached prefix, but for completeness...<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt install quagga</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo cp /usr/share/doc/quagga-core/examples/zebra.conf.sample /etc/quagga/zebra.conf</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo touch /etc/quagga/bgpd.conf</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo service zebra start</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo service bgpd start</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo vtysh</span><br />
<br />
I then interactively configured BGP to peer with my core and issued the write mem command to save my work. There's plenty of Quagga BGP tutorials out there, and the exact details of this part is a little out of scope for getting NAT64 working.<br />
<br />
<a href="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767#file-etc_quagga_bgpd-conf" target="_blank">Quagga BGPd config listing</a>:<br />
<script src="https://gist.github.com/PhirePhly/71e0a03b993221db942d54dc92bbd767.js?file=etc_quagga_bgpd.conf"></script><br />
<br />
At this point, my network has NAT64 support, so any IPv6-only hosts which are configured with a DNS64 name server can successfully access IPv4 hosts. I wouldn't depend on this one VM to serve a whole ISP's worth of IPv6-only hosts, but for a single rack of VMs which just need to be able to reach GitHub (which is still amazingly IPv4 only) to clone my config scripts, this setup has been working great for me.Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-67943242372416274232019-01-25T19:00:00.003-08:002019-01-25T19:04:32.177-08:00Implementing BCP214 on Catalyst 6500When it comes to running an Internet Exchange, at its most basic level, you're a metro Ethernet provider with a range of as little as 19". The most basic IXP is a single rack mount Ethernet switch, that you plug in, power on, and start plugging customers into.<br />
<br />
Which is great, right up until you want to unplug or move any of those customers.<br />
<br />
The problem is that, unlike a private interconnect between two routers, with an IXP switch in the middle, if you just yank the cord on one of the routers, it may be able to see that the interface is down and start calculating better routes from other BGP peers than those on the IXP, but every other customer on the IXP sending traffic towards the poor sap who you just disconnected won't see any change. When unplugged customer A, customer B will continue to see their link to the IXP switch up, and will <i>continue</i> to send traffic towards customer A until the BGP session between them <i>eventually</i> times out, which can be on the order of minutes.<br />
<br />
So in an ideal world, before yanking the cord on an IXP peer, you'd like to be able to make it <i>seem</i> like you've yanked the cord (without actually doing it), give BGP the <i>minutes</i> it takes to reconverge around the soon-to-be-down link, and then finally unplug the physical cable only once all of the traffic has drained and doing so won't result in a minute or two of black-holing traffic.<br />
<br />
The simplest way to do this is to send an email to the customer under question the day/week/month before and say "hey, I'm going to be unplugging your port, so turn down all of your BGP sessions with others first" but that's a pretty unrealistic expectation to see that level of cooperation from another autonomous system, and wastes a lot of time on the customers part manually turning down all their peering sessions before, and then turning them back up after.<br />
<br />
A better way to do it is to actively force the BGP sessions to go down without disrupting any other traffic, then wait for the reconvergence that will happen because of that. This technique is called <a href="https://tools.ietf.org/html/bcp214" target="_blank">BCP214</a>. and basically involves using the IXP's switches' ability to filter traffic to specifically filter the IPv4 and IPv6 BGP packets going between peers on the exchange.<br />
<br />
I've been doing this "turn down to move the peer to another switch" action quite a bit in the last few months for <a href="http://fcix.net/" target="_blank">FCIX</a>, where we've been moving everyone off my Cisco 6506 to a much nicer Arista 7050S-64. The problem is that, while BCP214 helpfully provides some sample configs in the appendix to implement this technique on Cisco IOS, for some reason which is beyond my understanding of the history of IOS command syntax, the Cisco sample doesn't seem to work on my Cisco 6500 running IOS 15.1(2)SY11.<br />
<br />
It took some digging to figure out the exact syntax needed to implement the needed ACLs and then apply them to an interface on my 6506, so just in case anyone else needs these, enjoy:<br />
<br />
The first part of implementing BCP214 is permanently creating two ACLs for specifically dropping BGP traffic from one IXP address to another (one for IPv4 and the second for IPv6). It's important to appreciate why you want to be specific in filtering only BGP with IXP addresses on it; multi-hop BGP could be flowing over the IXP between two routers not connected to the IXP for some reason, and that traffic shouldn't be dropped. These example ACLs use the FCIX subnets of 206.80.238.0/24 and 2001:504:91::/64, but need to be modified accordingly to your own IXP subnets.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">ip access-list extended acl-v4-bcp214</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> deny tcp 206.80.238.0 0.0.0.255 eq bgp 206.80.238.0 0.0.0.255</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> deny tcp 206.80.238.0 0.0.0.255 206.80.238.0 0.0.0.255 eq bgp</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> permit ip any any</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">!</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ipv6 access-list acl-v6-bcp214</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> deny tcp 2001:504:91::/64 eq bgp 2001:504:91::/64</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> deny tcp 2001:504:91::/64 2001:504:91::/64 eq bgp</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> permit ipv6 any any</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">!</span><br />
<br />
There's two deny lines on each ACL because you don't know if this customer's router happened to be the initiator of the TCP connection for BGP or not, so the source TCP port might be port 179, or the destination port might be 179. You want to drop both of those.<br />
<br />
With those ACLs now part of the config, when you need to cause a port to drain its traffic, you temporarily apply those two ACLs to the interface's config and give it a few minutes for the BGP sessions to time out, the routers on both sides to re-converge, and the rest of the Internet to pick up the slack with no black-holed traffic when you <i>then</i> shutdown the interface.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">interface GigabitEthernet1/3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> description Peering: FCIX Peer</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> switchport</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> switchport access vlan 100</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> switchport mode access</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> ip access-group acl-v4-bcp214 in</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> ipv6 traffic-filter acl-v6-bcp214 in</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">!</span>Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-13655908492408677492018-12-31T11:00:00.000-08:002018-12-31T13:20:03.577-08:00FCIX - State of the Exchange<h3 style="text-align: center;">
The First Year of Running an Internet Exchange</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghMnXxQmkt8vRwvE4aOqYeTmTnpC1bS9cu7Xlks-S3cXiTZySwCmwX0a5sPugNHkfjihd_bdTwak_CEAOZW50u3wy_zxI8j2gC_G0VyisoQbK50Y9GbSiJaKHexuuYhVnRDXbkduzCiAYX/s1600/FCIX_switch.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1290" data-original-width="1512" height="341" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghMnXxQmkt8vRwvE4aOqYeTmTnpC1bS9cu7Xlks-S3cXiTZySwCmwX0a5sPugNHkfjihd_bdTwak_CEAOZW50u3wy_zxI8j2gC_G0VyisoQbK50Y9GbSiJaKHexuuYhVnRDXbkduzCiAYX/s400/FCIX_switch.jpg" width="400" /></a></div>
It has been a little over a year since one of my friends challenged me on a whim to <a href="http://blog.thelifeofkenneth.com/2017/11/creating-autonomous-system-for-fun-and.html" target="_blank">get an autonomous system number and stand up my own little corner of the Internet</a>, and what a long slippery slope that has been. One of the advantages to running your own autonomous system is that you can blend your own connection to the Internet via peering, so as we continued to make more friends in the Hurricane Electric FMT2 data center who we wanted to peer with, the number of desired cross connects started to get out of hand; particularly since they aren't free, and we have all of about zero budget for... just about everything.<br />
<br />
It's that quadratic growth of the number of interconnects in a full mesh that really get you.<br />
<br />
But that's exactly the problem that <a href="https://en.wikipedia.org/wiki/Internet_exchange_point" target="_blank">Internet Exchange Points</a> are meant to solve; I have N number of networks that want to interconnect in one place without running O(N<sup>2</sup>) Ethernet cables between each other, so everyone connects to one central Ethernet switch and it's just as effective at a much lower cost of entry.<br />
<br />
So eight months ago, we jokingly registered the domain <a href="http://fcix.net/">FCIX.net</a>, grabbed a spare /24 + /64 of address space we had laying around, and founded the Fremont Cabal Internet Exchange.<br />
<br />
We had a good laugh setting up a cheeky little website to make it look like we're a real Internet Exchange, which lasted all of about two weeks before the owner of the data center, Hurricane Electric, applied to our Internet Exchange, brought in the 75,000 prefixes from their customer cone, and put us on their advertising material for the building.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiT5i-FccxeBPJb4D9E6lbS9Rl0r4FGlK469YD9pKPJT0uvMEo6z5dWP8forlk3VQlSeOPzH-NRNNbyogbJOxhb0cKfZcvPYl8LgydImLp_SvQ96Nr30bjcOFf4jP8DrL00y4RXndtBKXlc/s1600/Screenshot+from+2018-11-14+16-55-43.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="357" data-original-width="792" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiT5i-FccxeBPJb4D9E6lbS9Rl0r4FGlK469YD9pKPJT0uvMEo6z5dWP8forlk3VQlSeOPzH-NRNNbyogbJOxhb0cKfZcvPYl8LgydImLp_SvQ96Nr30bjcOFf4jP8DrL00y4RXndtBKXlc/s400/Screenshot+from+2018-11-14+16-55-43.png" width="400" /></a></div>
<div style="text-align: center;">
<i>Well crap. That joke got out of hand rather quickly.</i></div>
<br />
<h3>
Membership Growth</h3>
For the first few months there, we were handling two or three new membership applications per week, so it became evident that we needed to get our act together rather quickly.<br />
<br />
The first point of order was dealing with the fact that we were running this no-longer-just-between-friends Internet Exchange on borrowed address space, so we needed to get the exchange its own ASN and /24 + /64, but that's $550 for the ASN and $250 for the resource service agreement for the address space... A bit of a problem when we have zero budget...<br />
<br />
But at that point we had the advantage of having ~15 current + pending members, so we passed around the hat and between our membership and some other very amused on-lookers, very quickly managed to scrape together the $800 needed to cover the registration costs of FCIX's resources.<br />
<br />
Not only did this enable us to re-number onto a real ASN and IXP address space, but getting such a concrete signal of support from our members was touching. This thing wasn't much of a joke anymore; we're actually providing a service to networks which they value enough to throw a few hundred bucks our way to make it happen.<br />
<br />
New membership applications have slowed down, but as of this writing we're up to 25 members, which I don't think is half bad for a less than one year old exchange in the east bay in a single site.<br />
<h3>
Sponsors</h3>
So after getting a round of donations to cover the start-up costs for the ARIN resources, the next question was how to handle getting networks actually connected to FCIX. Originally, we were just running FCIX as a VLAN on my Cisco 6506 which was powering my personal network (AS7034), but that suffered from a few issues, the largest of which is that the 6506 is so old that 10G Ethernet was a cutting edge feature at the time. At best, 10G line cards for the 6500 support 16 ports, but the 10G line card we had managed to scrounge for AS7034 only had four ports for XENPAKs, and burns 100W <i>per port</i>, so offering 10G for FCIX was going to be problematic at such a low port density, even ignoring the issue of sourcing line cards and such vintage optics.<br />
<br />
This is where <a href="https://www.arista.com/en/" target="_blank">Arista</a> stepped in and has contributed to FCIX in a huge way. I got a call from a long-time friend who works at Arista who liked what we were doing and was interested in getting us a real switch to run the exchange on. This means that we got a pair of Arista 7050S-64 switches, which have 48 SFP+ ports which can support either 1G or 10G optics, plus another four QSFP+ ports for 40G, because hey, maybe we'll need 40G at some point...<br />
<br />
This now only left the issue of optics. Every member port that we turn up needs an LX or LR optic, which even from third party vendors start to add up quickly (remember how we have zero budget?), so we were very rapidly tapping out our junk bins of left-over optics we all had laying around. So while we sat there brain-storming ways to work around this sustainability problem, we got an awesome direct message on Twitter from <a href="https://www.flexoptix.net/en/" target="_blank">Flexoptix</a>!<br />
<br />
Flexoptix is a third-party optical transceiver vendor who has the additional advantage that they sell what they call their "FLEXBOX" which allows you to insert one of their optics and over USB reprogram their optic for any vendor's switch which you need, so even though we've got that scrappy "we'll use whatever switch we can drum up" aesthetic to us, we only need to stock one tray of 1G and 10G optics to cover any possible switch we'd want to plug these optics into. Furthermore, as we moved from my Cisco 6506 to our shiny new Arista, we were able to simply reprogram the optics and reuse them, so already the <i>flex</i>ibility of their universal transceivers have borne fruit.<br />
<h3>
The Cost of Entry</h3>
Having been started originally as mostly a joke, we've been very against charging any kind of membership fee to join FCIX, for multiple reasons:<br />
<br />
<ol>
<li>There is already several established pay-to-play IXPs in the bay area, so trying to charge for ours when we have none of the valuable peers that existing exchanges have would be kind of silly.</li>
<li>We all have real day jobs, so if someone paid to join FCIX and it then stopped working during the day, they'd rightfully kind of expect us to get it working again ASAP. My day job boss probably wouldn't appreciate that, so we don't charge anything, and problems get fixed when we can get to them. Outages, of course, are refunded in full. (see what we did there?)</li>
</ol>
<div>
This sort of zero cost of entry and zero membership fee model definitely wouldn't have been possible without all the donations we've gotten, and particularly Flexoptix donating trays of optics so we can light our end of every new member's port without trying to deal with every new member somehow contributing an optic to FCIX to light their port.<br />
<br />
This has meant a few issues with people trying to abuse our free model, so we very quickly needed to institute an informal "one port per cabinet" rule, since we at one point got six applications from different people sharing one cabinet, and although Hurricane Electric is sponsoring all the cross connects, I'm not going to abuse that deal to run six pieces of fiber to one cabinet. Charging a one time turn-up fee like <a href="https://www.seattleix.net/" target="_blank">SIX</a> does probably would have been a good idea and prevented most of our issues with low quality membership applications.<br />
<h3>
Plans for Growth and Value Add</h3>
At this point the basic framework for the exchange has been set up. Adding new members is relatively painless and mainly involves generating them a letter of authorization to redeem for a free cross-connect from HE and adding them to a CSV which propagates to the website and route servers automatically.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPkZfJB5aix9M4rluxGGkxMC4Scyjwowv0H1tF5WcFPDqjkr-i4atU6zYvjABrpYLfEnBaB71-UAJ1S4ezcrgwliB-8rBARf6eVorz8YEvy_wCYTXBWTA5oWbg6n3zdI82X3SHHHsHVvsx/s1600/multisite.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="254" data-original-width="572" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPkZfJB5aix9M4rluxGGkxMC4Scyjwowv0H1tF5WcFPDqjkr-i4atU6zYvjABrpYLfEnBaB71-UAJ1S4ezcrgwliB-8rBARf6eVorz8YEvy_wCYTXBWTA5oWbg6n3zdI82X3SHHHsHVvsx/s400/multisite.png" width="400" /></a></div>
The most exciting piece of news with regards to growth is that Hurricane Electric has agreed to sponsor FCIX with a second cabinet in their other building, <a href="http://he.net/tour/fremont1.html" target="_blank">FMT1</a>. That, plus a pair of dark fiber between the two buildings (thanks HE!), plus a pair of LR4 40G optics (thanks Flexoptix!), plus a second switch (thanks Arista!), and FCIX will soon be multi-site! Membership applications from FMT1 are now open.<br />
<br />
The other challenge has been coming up with projects to focus on to help increase the value of the exchange to existing members. Adding new members is easy, but we have also been working on getting things like cache appliances and DNS servers on-net. Verisign was kind enough to contribute a J root + .com/.net DNS server to the exchange, so anyone running their own recursive DNS resolvers get to enjoy direct access to J root and B.gTLD over the exchange. Work on other value-add appliances is on-going.<br />
<br />
As we head into this new year, I couldn't be happier or more grateful with how far we've gotten with this project while keeping it sustainable. The annual expenses specific to the exchange are still below $500 between our ARIN fees and other misc fees, and contributions of hardware and money from sponsors and members have enabled us to grow much further than we would have been able to fund on our own.<br />
<br />
So thank you again to everyone involved in FCIX, and I wish all of you a lovely 2019.</div>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.comtag:blogger.com,1999:blog-6541266482618530158.post-51976970273080427892018-11-24T20:00:00.000-08:002019-02-03T13:36:19.253-08:00Inspiration for Date Nights<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheRAVmN08KLK6upuPR0z8HxwZ5hPYQT2Dn19B32wuyoXnLbQXBV4g-sfXpH3iO6FePoL-g6yT1kB2_LBmyi7G-JT7REQhBStU-mB5a8YzZyglM8tkTGWSxv_FYJ2WIfi4foysxLBi7yifL/s1600/ideadeck.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheRAVmN08KLK6upuPR0z8HxwZ5hPYQT2Dn19B32wuyoXnLbQXBV4g-sfXpH3iO6FePoL-g6yT1kB2_LBmyi7G-JT7REQhBStU-mB5a8YzZyglM8tkTGWSxv_FYJ2WIfi4foysxLBi7yifL/s400/ideadeck.jpg" width="400" /></a></div>
My previous romantic relationship was recently classified as "not successful," which is one of those things that generally sucks. A lot.<br />
<br />
After a few days of just being all around miserable about it, I moved on to the RCCA phase of me moping around after a relationship and tried to take a hard look at the Root Causes and Corrective Actions for my part in the relationship not being successful. One of the issues I identified was that my tendency as an introvert to not plan outings doesn't lend itself well to building healthy relationships, so I clearly need a more concrete framework for date night.<br />
<br />
Putting a few days into the idea during these holidays, this is what I came up with: trawl as many click-bait "top 100 date ideas" articles as I can stomach cherry picking any that I thought I'd enjoy with someone or wouldn't be a terrible way to push the envelope with someone, and write each one on a 2.5"x3" index card. Each card isn't an actionable plan, but a category of activity (i.e. "Go for a hike"), with space below it to fill in specifics. Whether the most specifics are something I do on my own or I sit down with my partner to spend an evening workshopping "exactly where would we be interested in hiking?" to fill in the rest of the space is unclear, but I'm interested to see if this proves helpful to me in the future.<br />
<br />
Since others have already asked for this list, I figured I could post my current collection of cards here. Some of them have pretty obvious "more specifics" to be filled in, but some of them do suffer from the difficulty of figuring out exactly how to find a place to do them:<br />
<br />
<ol>
<li>Visit a local tourist destination</li>
<li>Visit an aquarium/zoo</li>
<li>Go to a library together</li>
<li>Go shopping at a thrift shop together</li>
<li>Go shopping at a flea market together</li>
<li>Go on a local walking tour</li>
<li>Go to a movie</li>
<li>Go for a hike</li>
<li>Go camping</li>
<li>Night of coloring/doodling</li>
<li>Host a dinner party</li>
<li>Host a board game night</li>
<li>Go an art gallery</li>
<li>Visit a museum</li>
<li>Work on a home depot project/build something</li>
<li>Have a picnic</li>
<li>Go to a pottery class</li>
<li>Go fruit picking</li>
<li>Take a cooking class</li>
<li>Sample drinks at a bar/brewery/vineyard</li>
<li>Morning coffee date/brunch</li>
<li>Day at the beach</li>
<li>Go to a used book store</li>
<li>Make dinner together</li>
<li>Go to an amusement park</li>
<li>Go to a county fair</li>
<li>Take a boat/ferry ride</li>
<li>Go to a farmers market</li>
<li>Go play pool</li>
<li>Go out bowling</li>
<li>Go to an arcade</li>
<li>Go to a shooting range</li>
<li>Train for and run a race together</li>
<li>Brew beer together</li>
<li>Visit a local nursery for house plants</li>
<li>Read the same book</li>
<li>Visit a botanical garden</li>
<li>Work on a puzzle together</li>
<li>Go play miniature golf</li>
<li>Go garage/estate sale hopping</li>
</ol>
Kenneth Finneganhttp://www.blogger.com/profile/09597995268728038585noreply@blogger.com