Murray's FreeBSD Notes

Updated TCP Proposals and FreeBSD

2012-02-04T15:07:00.001-08:00

There are a number of proposals for improving TCP performance coming out of Google that have some implications for FreeBSD. These proposals have taken the form of a group of IETF proposals, RFCs, patches to the Linux kernel, and research publications. A nice summary of the different initiatives is available from Lets Make TCP Faster on the Google Code Blog.

TCP Fast Open by Radhakrishnan, Cheng, Chu, Jain, and Raghavan is based on the observation that modern web services are dominated by TCP flows so short that they terminate a few round trips after handshaking. This means that the 3-way TCP handshake is a signiﬁcant source of latency for such ﬂows, and they describe a new mechanism for secure data exchange during the initial handshake to reduce some of the round-trip network transmission and associated latency for such short TCP transfers. This work shares many goals and challenges with T/TCP, which was previously in FreeBSD but suffered from some security vulnerabilities.

David Malone posted some thoughts on my Google+ post about how FreeBSD could implement the various changes. Maybe we could have some Summer of Code students work in this area this summer?

Two New Videos: SuperPages and NanoBSD

2011-08-19T13:47:00.000-07:00

Thanks to Kirk McKusick, I'm happy to announce two new fully edited high quality videos from BSDCan 2011 in the BSD Conferences YouTube channel. I've also created a new playlist for the BSDCan 2011 videos.

The first talk is "Superpages in FreeBSD" by McKusick, and it describes the addition of superpage support to the FreeBSD 8 kernel on the Intel PC architecture. Superpages aggregate together standard-sized hardware pages into much larger "superpages". Each superpage requires only one entry in the page table replacing the numerous entries used by the standard-sized hardware pages.

The second talk is "Updates from NanoBSD: FreeNAS drives NanoBSD development" from Warner Losh, and it describes the basics of NanoBSD and how FreeNAS moved over to NanoBSD.

We now have 108 high-quality videos in the BSD Conferences channel. These videos have been watched in aggregate over 400,000 times, and our most popular video remains McKusick's FreeBSD Kernel Internals Lecture.

As a reminder, this channel was setup specifically for the BSD technical community and does not have the standard limitations on video size for other types of YouTube uploads. If you have additional video content from a conference, presentation, or class about BSD Unix please get in touch and I'd be happy to help you publish the content here.

Luigi Talks Netmap on Google Tech Talks Channel

2011-08-18T21:20:00.000-07:00

Last week, Luigi Rizzo visited Google and gave a talk on high-speed networking with Netmap.

This was Luigi's second talk at Google and the third talk about FreeBSD in the Google Tech Talks YouTube channel. We also have more than 100 videos available in the BSD Conferences channel.

FreeBSD at GSoC Mentor Summit

2010-10-24T21:31:00.000-07:00

As in previous years, Google held a "Mentor Summit" to bring together representatives from the open source organizations that participated in the Google Summer of Code to share experiences of what worked, what didn't, and generally learn from each other about shepherding students through the program. The mentor summit is always run Unconference-style and it is a great opportunity to meet, learn, and socialize with the many other open source organizations.

In addition to several hours of face to face FreeBSD-related catch-up with Brooks Davis over pizza and beer, I particularly enjoyed catching up with old colleagues and learning about the current state of a variety of other open source projects I use such as R, Boost, NTP, and Ganeti.

This weekend Brooks and I were the only FreeBSD representatives. Given that I'm local and Google sponsors the travel of 2 representatives from each open source organization it's quite unfortunate we couldn't get another FreeBSD mentor here this year. I would strongly encourage some of the other mentors that have never participated in this forum to volunteer to represent FreeBSD next year. This program has funded approximately 117 students to work on FreeBSD over the past 5 years and the mentor summit is best way I know of to improve the experience for students and open source projects next year.

Thanks again to all the FreeBSD mentors that worked with students this summer and hope to see some of you at the post-GSoC Mentor Summit next year...

FreeBSD Summer of Code Students Highlighted on Google Blog

2010-09-15T10:25:00.000-07:00

As in previous years, I've posted a summary of FreeBSD Project participation in Google Summer of Code on the Google Open Source Blog.

By my count we have now mentored at least 117 students on FreeBSD development through this program. As in previous years it was tough to identify a few student projects to highlight given how much cool work is going on here. My list is certainly not complete but at least a few other people mentioned that Efstratios Karatzas, Zheng Liu, and David Forsythe had done a lot of excellent work this summer. Hats off to them, all the students and mentors this summer, and Brooks and Robert for serving as administrators of this whole thing for us.

MeetBSD 2010 Videos in HD on YouTube

2010-08-20T08:08:00.000-07:00

Thanks to Tomasz Dudzisz we now have 14 videos from MeetBSD 2010 in Poland. Tomasz has created a playlist for the BSD Conferences YouTube channel, or you can select a video directly from the list below. These videos are all in HD quality and most are in English and a few are in Polish. Thanks Tomasz and the MeetBSD organizers for being so organized with the video recordings again this year.

Dru Lavigne - Update on BSD Certification

Hans Peter Selasky - The new USB stack in FreeBSD

Jakub Klama - FreeBSD on DaVinci DMSoC (polish)

Jan Srzednicki - What ideas can FreeBSD borrow from AIX?

Attilio Rao - The VFS/vnode interface in the FreeBSD kernel

Marko Zec - Network emulation using the virtualized network stack in FreeBSD

Paweł Jakub Dawidek - HAST -- Highly Available storage for FreeBSD (polish)

Paweł Jakub Dawidek - HAST -- Highly Available storage for FreeBSD (questions, polish)

Nikolay Aleksandrov - FreeBSD-based solution for Internet traffic management
Sławek Żak - NoSQL

Ramon Tancinco - meetBSD 2010 Welcome Intro

Martin Matuska - mfsBSD

Dmitri Epshtein - Advances in Embedded ARM processors, for performance driven applications

Warner Losh - Using FreeBSD in a Commercial Setting

BSDCan on Google's Open Source Blog

2010-08-10T23:28:00.000-07:00

A coworker of mine, Kirk Russell, just posted an excellent summary of BSDCan through the years on the Google Open Source Blog.

I wasn't able to make it to BSDCan this year due to family commitments, but I did make it to another open source conference later this summer that I also wrote about on Google's open source blog.

Kirk and I haven't worked closely together but we both do our best at evangelizing BSD and open source inside our respective corners of the company. It's great to see his post about all the excellent work happening in the BSD community on a corporate blog.

Kirk McKusick on Journaling Soft Updates in FreeBSD

2010-05-30T00:40:00.000-07:00

Dr. Kirk McKusick has produced a high quality recording of his talk on Journaled Soft-Updates at BSDCan 2010. This is the 92nd BSD conference video in the BSD Conferences YouTube channel.

AsiaBSDCon 2010 Videos

2010-05-29T23:38:00.000-07:00

The videos from AsiaBSDCon 2010 are now available on the BSD Conferences YouTube channel. The full list of 17 AsiaBSDCon videos includes:

Thanks Hiroki Sato and the other organizers of AsiaBSDCon for running a successful conference and uploading these videos. Some of these videos were previously available on ustream but are not currently accessible there. The YouTube channel provides automatic machine generated captions in ~50 languages, fast streaming, and a total of over 90 videos from conferences over the past ~3 years.

FreeBSD Tech Talk @ Google

2010-04-06T23:29:00.000-07:00

Long time FreeBSD developer Luigi Rizzo from the University of Pisa came to Google last week to visit with Sam Leffler and me, and he agreed to give a talk about some of his work on link emulation and packet scheduling.

This marks the second FreeBSD video in the Google Tech Talks channel in addition to the 70+ videos in the BSD Conferences channel. Enjoy.

Improved Conference Captions from Amazon Mechanical Turk (2)

2010-02-20T22:29:00.000-08:00

After my initial experiments last month, I applied to the FreeBSD Foundation for funds to pay for additional human editing of the YouTube machine generated transcripts. The screenshot on the left shows an example HIT (Human Intelligence Task) available on Amazon Mechanical Turk.

The task description on the left is based on a template I created with three variables: $VIDEO_URL, $VIDEO_TITLE, and $CAPTIONS_URL. New HITs are then created by uploading a CSV file with three columns for each of those variables, e.g.

VIDEO_URL,VIDEO_TITLE,CAPTIONS_URL
http://www.youtube.com/watch?v=mMmbjJI5su0,"BSD v. GPL, Jason Dixon, NYCBSDCon 2008",http://people.FreeBSD.org/~murray/improved-captions-bsdvsgpl.sbv
http://www.youtube.com/watch?v=Pe8LdJpBGJ4,"Isolating Cluster Jobs for Performance and Predictability, Brooks Davis (DCBSDCon 2009",http://people.FreeBSD.org/~murray/improved-captions-isolatingcluster.sbv

Using this method I created 12 HITs for the first pass of editing for which I offered between $9 and $14 per video. A slightly modified template with the same three variables was used to pay ~$7 per video for a second pass to further improve the transcripts improved in the first pass.

The template has gotten more detailed over the past month in response to all of the minor ways that workers submitted less than perfect transcripts. The actual SBV file format used by YouTube captions is not formally specified anywhere as far as I can tell, but the 60 character maximum width and simple format can be verified in submitted transcripts with a few emacs macros.

The transcript files have been checked into the FreeBSD Doc CVS Repository. The full list of videos with human-edited English language transcripts is:

Improved Conference Captions from Amazon Mechanical Turk

2010-01-10T17:38:00.000-08:00

Just wanted to send a quick note that three of the popular videos from the BSD Conferences YouTube channel have been updated with human-edited English language caption files. These offer a significant improvement over the machine generated captions I wrote about last month.

The following videos have been updated:

I've also posted three simple captions text files which provide the times and text in a very simple ascii format in case anyone wants to provide a diff to improve any remaining mistakes in the captions.

The transcriptions were done with the help of the industrious workers behind Amazon Mechanical Turk. The three transcripts above, representing at least 6 person hours of work, but easily twice that much time, were completed for less than $50 by leveraging the timing information from free machine generated captions and mechanical turk for the editing. This is less than 1/10th of the cost of a commercial transcription service.

What is the quality of these captions in other languages when automatically translated with YouTube? Are there any other videos for which captions would particularly be useful?

AsiaBSDCon is coming up in March, and I hope to have things streamlined by then such that videos with both Japanese and English captions can be added to the channel shortly after the conference.

Machine generated captions for BSD conference videos

2009-12-22T13:35:00.000-08:00

One of the most frequent requests I've received, since Launching the BSD Conferences YouTube channel last year, has been for captions in Spanish, Russian, Chinese, and other languages. I was excited last month when Google announced automatic captions for Youtube videos using machine translation. This feature is still highly experimental but I am happy to report that it has been enabled for the BSD Conferences channel. In combination with the much more mature automatic translation feature, this means that captions are now available in over 50 languages from Afrikaans to Vietnamese for most of the 73 videos in the BSD Conferences channel.

The automatic captions are still highly experimental and the quality of transcription for highly technical content spoken by a diverse set of international speakers is a significant challenge to get right. If you are interested in helping to correct any of the English transcripts I would be happy to provide you a simple text file of the transcription, with each line offering the start and end time for the caption to be displayed, and the caption text. One advantage of the machine translation is that the most time consuming part of manually creating captions, synchronizing the timing of the text with the speech, has been done automatically. Even when the technical words are mangled, the timing information in the automatic captions files can be leveraged to make the process of manually improving the captioning much easier.

The experimental automatic captions are only available directly from the video watch pages, and not from channel pages or other views. For example, visit www.youtube.com/watch?v=nwbqBdghh6E to see one of our most popular videos, Kirk McKusick speaking on FreeBSD Kernel Internals. Hover over the triangle at the bottom right of the video, then over the CC submenu and select "Transcribe Audio". You can then choose to "Translate Captions" into a different language as well.

The Slashdot Effect

2009-07-26T22:32:00.001-07:00

After 8 months, 66 videos uploaded, and 141,676 views, the BSD Conferences YouTube Channel was slashdotted for the first time last week. Specifically, Theo's OpenBSD Release Engineering talk was linked from this slashdot post. Views of the video spiked to nearly 8,000 a day after the Slashdot post, which dwarfs the previous highs of around 1,500 videos a day after I posted about Kirk McKusick's FreeBSD Kernel Internals lecture.

I think this is an excellent reminder of the power that forums like Slashdot still have in directing traffic among those seeking technical content online. I would encourage anyone interested in seeing more BSD related content online to install browser bookmarklets, toolbars, or other shortcuts to more easily share and promote FreeBSD content on Digg, Del.icio.us, Slashdot, etc..

FreeBSD Code Metrics Now on Ohloh.net

2009-07-18T23:28:00.000-07:00

I've written previously about Ohloh.net and how I'd like to see more of the dynamic code metrics calculated there available on the FreeBSD web site. I am happy to report that today I noticed after several years of attempts, the ohloh repository import servers have finally managed to get through the entire FreeBSD source repository. Their software setup previously had difficulties dealing with a project with as long of a history as FreeBSD.

You can now view the top level code metrics about FreeBSD from the FreeBSD Project Page on Ohloh.net. This page indicates that there are over 10 million lines of code, that more files are licensed under GPLv2 than any other license.

The committer totals do not quite match up with Peter's Commit Counters. Even after accounting for the fact that Peter's system could potentially double count a commit that touches both sys and non-sys parts of the source tree, the numbers from Ohloh are still lower for some committers. Unlike the numbers on cia.vc, the FreeBSD project on ohloh.net only contains the source repository. We are currently lacking the anonymous cvs access to our doc repository necessary to add the doc project to ohloh.net.

How do the numbers reported on ohloh.net compare to code metrics others have reported for FreeBSD? Does this match expectations or are there any major problems with this data? How can we use this information on our website? Would a badge on the front page showing "Last improvement made X minutes ago" be useful? A list of most active committers in the past week on one of the developer pages? Other ideas for utilizing the work the Ohloh project is doing?

Open Source in Recessions

2009-07-12T05:22:00.001-07:00

In general, recessions can be really good for open source. Large businesses look to cut back on IT budgets and this often involves re-evaluating whether proprietary software and maintenance contracts are necessary, given high quality open source alternatives. Companies may dedicate more internal resources to open source projects, and also the surplus of underemployed engineering talent in the market may be available for more open source development work.

Unfortunately, there are also some significant downsides, and one that I would like to highlight is the plight of the small open source shops around the world. We've seen pleas earlier this month to Save BSD Magazine, and in recent years some other smaller open source companies such as Daemonnews, and the Japanese publications FreeBSD Press and BSD Magazine have exited the business. In the current environment I would like to take the unusual step of plugging a company that has been selling, marketing, legally defending, and supporting FreeBSD from the very beginning.

FreeBSD Mall has been selling FreeBSD CDs since 1.0 in 1993 and is still selling and supporting CDs, DVDs, books, branded apparel, and more. The PC-BSD live-dvds make an excellent introduction to FreeBSD for new users and the complete FreeBSD DVD distributions are quite handy to have. Consider spending a few dollars at the FreeBSD Mall site, buy a BSD Magazine subscription, or otherwise spend some money to encourage the small commercial FreeBSD ecosystem and perhaps contribute to more funds being available to exploit the many disruptive opportunities (netbooks, cloud computing, etc..) that could be very good for open source and FreeBSD during this recession.

Remaining AsiaBSDCon 2009 Videos Posted

2009-05-17T23:56:00.000-07:00

The remaining 9 videos from AsiaBSDCon 2009 have been posted. The new videos include talks by Theo de Raadt, Eric Allman, Kris Moore, Mohamad Fauzie, Brooks Davis, Atillio Rao, A. Kantee, and the Works In Progress Sessions.

Thanks again to Hiroki Sato for posting the videos and organizing 3 consecutive years of successful AsiaBSDCons. Sato-san has also created two separate YouTube playlists for the AsiaBSDCon 2008 and AsiaBSDCon 2009 videos. These playlists make it easier to find the newest videos from among the 66 videos now in the BSDConferences YouTube channel.

First AsiaBSDCon 2009 Videos Posted

2009-05-10T21:33:00.000-07:00

Thanks to Hidetoshi Shimokawa and Hiroki Sato, we now have 7 videos from the recent AsiaBSDCon 2009 in Tokyo, with more on the way:

A. Zakharchenko, Mail system for distributed network

K. Dzonsons, Deprecating groff for BSD manual display

D. Gwynne, Active-Active Firewall Cluster Support in OpenBSD

R. Jaworowski, FreeBSD on hi-perf. multi-core embedded PPC

M. Warner Losh, An Overview of FreeBSD/mips, AsiaBSDCon2009

Constantine A. Murenin, OpenBSD Hardware Sensors Framework

Philip Paeps, Crypto Acceleration on FreeBSD, AsiaBSDCon2009

This makes 57 videos in the BSDConferences YouTube channel. We now have over 1,000 subscribers to the channel. Kudos to both the DCBSDCon and AsiaBSDCon organizers for getting the videos processed and uploaded shortly after the respective conferences when interest is highest. The videos from AsiaBSDCon 2008 were also made available earlier this year.

Nominations for Google-O'Reilly Open Source Awards

2009-04-22T13:33:00.000-07:00

Nominations for the 5th Annual Google-O'Reilly Open Source Awards are now being accepted. As in previous years the awards are open ended so nominations should also include the name of the award / achievement in addition to a nominated recipient (e.g. Best Lifetime Contributor, Community Builder, Tool Builder, Overall Hacker, etc..). Nominations are due by May 22, 2009 and will be judged by a committee before winners are presented with an award at OSCon 2009. (Via Google's Open Source Blog.)

50th BSD Video Posted: All DCBSDCon '09 Videos Live

2009-04-21T03:26:00.000-07:00

Jason Dixon has made available the last 4 videos from DCBSDCon 2009. The last video marks the 50th video uploaded to the BSDConferences YouTube channel. This channel was created less than 5 months ago and now has 924 subscribers from authenticated YouTube users, and the videos have been viewed over 76,000 times by users from around the world (includes partial views).

The newest 4 videos are :

The average number of daily views is around 500, with significant spikes above 1,500 in the days after a popular new video is announced:
And the top 10 videos sorted by views (biased towards older videos that have been available longer) are :

20 FreeBSD Summer of Code Students Announced

2009-04-20T19:26:00.001-07:00

Leslie Hawthorne from Google today announced the students selected to participate in this year's Google Summer of Code. Among the 1,000 students from 70 countries participating in this program, 20 will be working on FreeBSD projects, 11 on NetBSD projects, and 5 on DragonFly BSD projects.

Keep an eye on the FreeBSD wiki and mailing lists for more information about these projects over the summer. Congratulations to Robert Watson, Brooks Davis, and Tim Kientzle for pairing up mentors and students for this year's FreeBSD applications :

Student: alejandro pulver
Project: Ports license infrastructure (part 2: integration)
Mentor: Erwin Lansing

Student: Ana Kukec
Project: IPv6 Secure Neighbor Discovery - native kernel APIs for FreeBSD
Mentor: Bjoern A. Zeeb

Student: sylvestre gallon
Project: USB improvements under FreeBSD
Mentor: Philip Paeps

Student: David Forsythe
Project: Package tools rewrite via a new package library, with new features
Mentor: Tim Kientzle

Student: Fabio Checconi
Project: Geom-based Disk Schedulers
Mentor: Luigi Rizzo

Student: Fang Wang
Project: Implement TCP UTO
Mentor: Rui Paulo

Student: Gabor Kovesdan
Project: BSD-licensed libiconv in base system
Mentor: Xin LI

Student: Ilias Marinos
Project: Application-Specific Audit Trails
Mentor: Robert Watson

Student: marta carbone
Project: Ipfw and dummynet improvements
Mentor: Luigi Rizzo

Student: Prashant Vaibhav
Project: Reworking the callout scheme: towards a tickless kernel
Mentor: Ed Maste

Student: nikhil bysani
Project: Porting NetworkManager to FreeBSD
Mentor: Ed Schouten

Student: gabor janos pali
Project: Design and Implementation of Subsystem Support Libraries for Monitoring and Management
Mentor: Oleksandr Tymoshenko

Student: Satish Srinivasan
Project: TrustedBSD Audit: Developing BSD licensed tools for importing, exporting from/to Linux audit log format and BSM.
Mentor: Stacey Son

Student: tatsiana severyna
Project: puffs (pass-to-userspace framework file system) port for FreeBSD
Mentor: Konstantin Belousov

Student: edward napierala
Project: Hierarchical Resource Limits
Mentor: Brooks Davis

Student: aditya sarawgi
Project: Improving Second Extended File system (ext2fs) and making it GPL free.
Mentor: Ulf Lilleengen

Student: Zhao Shuai
Project: FIFO Optimizations
Mentor: John Baldwin

Student: Zachariah Riggle
Project: TCP\IP Regression Testing Suite
Mentor: George Neville-Neil

Student: gleb kurtsov
Project: In kernel stackable cryptographic filesystem (ownfs)
Mentor: Stanislav Sedov

Student: tatsiana elavaya
Project: ipfw ruleset optimization and highlevel rule definition language
Mentor: Diomidis Spinellis

Videos from DCBSDCon Posted

2009-04-19T15:48:00.000-07:00

Thanks to Jason Dixon and Will Backman, the first 8 videos from the first DCBSDCon are now available in the BSDConferences YouTube channel. The audio quality for these is better than many of the previous conference videos because Jason was able to sync the audio with a direct recording from the podium taken by Will. These videos were also made with pure open source tools such as avidemux, mplayer/mencoder and audacity. More information will be coming soon to the VideoProductionAndPublishing wiki.

In the mean time, enjoy these technical presentations from DCBSDCon 2009:

Cell phones and BSD

2009-03-26T18:57:00.000-07:00

Xin LI pointed me at BSDroid, a project which aims to provide native binaries for tools and make it possible to develop Android applications on FreeBSD. This reminded me of the underreported fact that the Android software for the G1 phone uses various bits of NetBSD and OpenBSD userland, and of course the iPhone 3g includes elements of the BSD subsystem. You can even get a terminal app for the Android. What other phones have significant BSD software in them?

All Videos from AsiaBSDCon 2008 Posted

2009-03-22T21:35:00.000-07:00

Thanks to Hiroki Sato, we have finished uploading all remaining videos from last year's AsiaBSDCon conference. The new videos are especially appreciated because they include some NetBSD and OpenBSD videos which have previously been under represented in the bsdconferences channel and it is nice to encourage cross-fertilization of ideas (including to more general open source conferences).

Sleeping Beauty - NetBSD on Modern Laptops, Jörg Sonnenb

OpenBSD Network Stack Internals, Claudio Jeker

Keynote: Hideki Sunahara - 25 years with BSD

A Portable iSCSI Initiator, Alistair Crooks

Logical Resource Isolation in the NetBSD Kernel, Kristaps Džonsons

BSD Implementations of XCAST6, Yuji Imai

Tracking FreeBSD in a Commercial Setting M. Warner Losh

These videos are in addition to the 6 other FreeBSD videos already posted from AsiaBSDCon 2008 as announced here and here.

Summer of Code 2009

2009-03-22T19:06:00.000-07:00

By this point I hope everyone has seen the announcement from Tim Kientzle that FreeBSD will be participating in the Google Summer of Code for the fifth consecutive summer. More general information is available from the Google Open Source Blog Post about GSoC 2009.

The list of potential project ideas has improved significantly in recent weeks and is still being updated. This year Robert and Manolis put together some posters that can be used on university campuses to raise awareness about this program. Please help forward the announcement mails on to student lists and if you have a university affiliation please consider posting the posters on department bulletin boards.