Funtoo by Daniel Robbins

"mini-manifesto"

2010-07-16T23:40:00.005-06:00

Over the last few days, I've incorporated configurable compression format support into Metro, and I am now creating Funtoo stages using the .xz compression format (these patches are in git, and not yet in an official Metro release.) On the mirrors, this is resulting in a very nice 40% size decrease over bzip2, with stage3's weighing in at around 95MB.

But it gets better. Our old git-based Portage snapshots compressed with bzip2 used to weigh in at a whopping 413MB. By migrating to mini-Manifests, xz compression and adding some additional Funtoo optimizations, our Portage tree snapshot files are now weighing in at a svelte 25MB :)

That's 6% of their original size, or about 16x smaller - more than an order of magnitude difference. This will make the snapshot download go a lot faster - and in the coming months I hope to be able to reduce the Portage snapshot size even further, both in .xz form and on-disk, uncompressed. Stay tuned.

Portage post-GLEP-55 - Moving Forward

2010-07-08T12:18:00.004-06:00

GLEP 55 support has now officially been removed from Portage-2.2_rc, and now I'm going to focus on working with Zac Medico to improve the much-in-need-of-love Portage codebase.

Zac and I already have some projects we are working on that should reduce the size of the on-disk Portage tree by an order of magnitude and be a great benefit for many users. Other spiffy improvements are also in the works.

I realize that there is at least one extremely vocal proponent of GLEP 55 who thinks that it is the best and only reasonable solution for the EAPI problem. I respect your opinion but think you are wrong in this case. Rather than engage in marginally productive debate, I'd rather just move forward with improvements to Portage.

If we can implement a solution to the EAPI issue, we can also implement solutions for other related areas of Portage as well. I don't see any problem overcoming the logistical and technical hurdles you have raised thus far.

Hello again... and Why GLEP 55 Is A Bad Idea

2010-07-06T00:12:00.006-06:00

Uh... hello there. I haven't posted to my blog lately. So what has been up in Funtoo land? Not much? Actually, quite a lot - tons - I've just been neglecting my blog. I'll try to catch you up in upcoming blog posts.

One very recent thing that I wanted to blog about is that I am going to be more active in helping to define the future of Portage. So I thought I'd start by telling you why GLEP 55 is a bad idea.

GLEP 55, currently in draft form, is somewhat supported in the current Portage 2.2_rc source code. It specifies a new ebuild file name extension that isn't just ".ebuild" - but ".ebuild-[number], where [number] specifies the EAPI version, which is the revision of the ebuild API that is supported by the ebuild.

The rationale is that Portage currently can't figure out the EAPI of the ebuild until it sources the ebuild using bash, and it needs to know the EAPI of the ebuild *before* it sources the ebuild, otherwise problems might occur when it sources the ebuild. So it can't use the current mechanism of sourcing the ebuild to find out what EAPI it uses. It can lead to problems. So far, this is true.

Another rationale for GLEP 55 is GLEP 54, which proposes the "-scm" extension to ebuilds. This extension is presumably going to be phased in in a future EAPI. The GLEP 55 proposal claims that Portage is going to need to figure out the EAPI of the ebuild even before it can try to grok the *version* specified in the ebuild filename, and therefore, the EAPI must be in the filename too...! Wow! Really? This certainly isn't true.

But before I go into more detail, let me digress for a moment and speak about a general observation I have regarding file formats. Did you know that there are actually many different versions of .jpg and .png files, but they all use the same file extension? This is true of many other different file types as well. Why is this? It appears that people have figured out a way to solve this problem in a more elegant way than GLEP 55 proposes. I'd like to also have Portage apply an elegant solution to this problem so that we do not need to uglify .ebuild filenames.

Here's another observation I have. UNIX already has a solution to this problem - used for shell scripts. You've probably noticed that the beginning of a shell script looks like this:

#!/bin/sh

...and likewise, the beginning of a Python script will often look like this:

#!/usr/bin/python

The GLEP 55 proposal suggests that using a mechanism like this to identify the proper interpreter for a file actually "hurts performance." Really? Well, in a general sense it does not, but I suppose if you think that you need to figure out the EAPI before you can properly parse the version string contained in the filename (which isn't true,) then maybe you might convince yourself that there are performance problems.

However, you don't need to figure out the EAPI before you parse the version string.

Let's take a look at a more elegant solution, shall we?

First, GLEP 54. The -scm extension should not be phased in via EAPI but as a Portage Repository-specific capability. In other words, a Portage repository-specific configuration file can define what capabilities are enabled in that particular Portage repository, and if GLEP 54 is used in the repository, a flag will be switched (in a file such as layout.conf, which contains repo-specific settings) to indicate that ebuilds can contain -scm extensions. Using this approach, we view the versions found in ebuild filenames, and the format that they use, as a *repository-specific* capability. GLEP 55 should really have focused on adding Portage Repository capabilities, which is a needed feature in Portage, and this would have paved the way for GLEP 54 without adding wacky ebuild filenames.

If we move the -scm extension in GLEP 54 so that it is a repository-specific capability, we can now figure out how to grok the version of the ebuild without having to know its EAPI. This solves the potential performance problem.

Now we can revisit the time-tested approach of storing the EAPI inside the filename, using the first few bytes of the file, something like:

#?ebb4?

This could mean "*EB*uild in *B*ash format, EAPI *4*." I am using a "?" instead of the more common "!" so that the following characters are not misinterpreted to be the path to an interpreter. The initial "#" is pretty much universally supported as a comment string. I bet you could probably think up an even classier-looking initial few bytes, or even a better, more feature-rich format for the first few bytes of the file that could be used for other things.

So there we go. No need to add the EAPI extension to each ".ebuild" file.

The only reason to add any additional data to a ".ebuild" filename is so that two files can exist independently on disk, next to each other. Since we're never going to need to have EAPI 3 and EAPI 4 versions of the same ebuild sitting alongside one another on disk, or in a git repo, there is no need to put EAPI data in the filename.

So that's why GLEP 55 is a bad idea. Portage Repository capabilities should be implemented instead.

Crazy weekend

2009-11-01T21:03:00.000-07:00

So, what I was going to do this weekend was re-work how our git branches are organized, creating a testing branch called "too", and work on getting the new Funtoo/OpenRC network scripts finalized.

What I *actually* did was try to reload one of my servers -- and ran into lots of problems. This process actually started the middle of last week, sucking up my after-work hours. I wanted to reload one of my Nehalem servers, and for some reason -- the thing just wouldn't boot. I tried rolling back firmware. I tried using known-good kernels. I tried using an exact replica of my *working* identical Nehalem server, so it was basically set up totally identically to my other box, and the thing still wouldn't boot. Everything I tried, it complained about an unknown block device and that it was unable to mount the root filesystem. But the strange thing was -- sysrescuecd could access the disk just fine. And I created a minimal initrd with "bash" and "mount" on it, and I was able to mount the root filesystem just fine with no extra modules required.

I found the solution around 2:30 AM last night - for some reason, the kernel auto-detection of what filesystem is on the partition was consistently failing. If I pass a "rootfstype=ext3" kernel boot option, everything works. But if I leave my "root=/dev/sda3" boot option to fend for itself, the thing won't boot. I've never ran into this behavior before.

But thinking about it, the "rootfstype" option makes a lot of sense. I don't want to rely on my kernel auto-detecting the type of my filesystem by itself - especially now that I've seen it fail so badly. Apparently some funky data on your block devices can cause this auto-detection to bite it pretty hard.

All that to say -- I had a crazy weekend. 20 hours of troubleshooting.

But there's some good things that are going to come out of it - I am going to fork the Gentoo GRUB package, which needs quite a bit of love. It's time to get GPT partitioning documented properly and grub-1.97 supported offiicially as it is in Ubuntu. So expect to see some things related to that soon.

Ruby 1.9.1 and Diakonos

2009-07-23T22:36:00.005-06:00

Tonight, I've gone ahead and done some work on dev-lang/ruby, and I've gotten ruby 1.9.1 into funtoo unstable. Along with the new ruby comes an updated eselect-ruby, and a transition from the "/usr/bin/ruby18" suffix to the "/usr/bin/ruby1.9" suffix (notice the dot.) It seems the dot is popular these days. And I also figured that python uses the dot, so why not ruby too? Seems worthwhile to be consistent.

I've also added the diakonos editor - written in ruby and dependent on ruby-1.9 - into the funtoo unstable tree. I also think it's worth mentioning that diakonos was first submitted to bugs.gentoo.org by Pistos, its author, on October 22, 2005. Three years, nine months and one day later, it's in funtoo. Better late than never.

Funtoo-dev Mailing List - Initscripts

2009-07-08T16:09:00.002-06:00

Funtoo now has a Google Groups mailing list called funtoo-dev, which can be used for Funtoo discussion, patch submissions or bug reports.

On the list, I've posted an overview of upcoming planned changes to Funtoo's init scripts. With dhcpcd-5, all that will be required to configure a new Funtoo install for wired DHCP ethernet is:

# rc-update add dhcpcd default

# rc

Pretty nice!

I'm looking for feedback on the new initscript plan. Please feel free to post feedback here as well as on the mailing list. Talk to you soon!

Apache/Slowloris DOS Mitigation Guide

2009-07-05T17:00:00.004-06:00

My friend Ryan Vick and I just finished up our Slowloris DOS Mitigiation Guide. We hope you find it useful. We detail various ways to protect against the Slowloris DOS, and have a few surprises along the way. Give it a read and let us know what you think!

Covered in the article: anti-slowloris.diff, iptables connlimit, proper hardware load balancer configuration, Cherokee web config

10th Anniversary of Gentoo

2009-06-30T15:57:00.006-06:00

NeddySeagoon and I have been trying to figure out the official 10th anniverary date of Gentoo, and here are the dates I've figured out so far:

July, 1997 - Started a new position at University of New Mexico, was using Debian 1.3
Nov, 1998 - Was using/developing for Stampede Linux at home, but had not (yet) started Enoch?
Apr, 1999 - Was working on Enoch - wrote xpak .tbz2 code that is still in Portage
May 18-27, 1999 - First version of Enoch released, according to LWN.net. My blurb for Enoch: "Enoch is an advanced GNU/Linux distribution for the x86 PC Architecture, designed to bring your Linux experience into a new dimension. Or something like that."
Aug, 1999 - My new dual Celeron mobo would not run Linux, went to FreeBSD, Achim Gottinger kept Enoch going
Late 1999 - Must have came back to Enoch and done the Gentoo name change right about now - the "Gentoo" name was Bob Mutch's idea - started incorporating some FreeBSD ideas into Enoch - Portage (as we know it today) was born.
July 26, 2000 - Gentoo 1.0 release "imminent", cvs online, rsync "coming soon"
Nov 3, 2000 - Gentoo 1.0 Release Candidate 2
Dec 11, 2000 - Gentoo 1.0 Release Candidate 3
Aug 14, 2001 - New Gentoo Logo/Web site debut - designed by me! Still in use today! :)
Feb 16-17, 2002 - Was in Brussels, Belgium to attend FOSDEM
Mar 31, 2002 - Gentoo Linux 1.0 Released !!!
May 10, 2002 - Gentoo Listed as one of the top 10 Linux distributions on DistroWatch
Apr 8, 2002 - Gentoo Linux 1.1a released :)
Jun 10, 2002 - Gentoo Linux 1.2 released :)

So, we're thinking of calling "late 2009" the official 10th anniversary of the birth of Gentoo - whaddya think? If it sounds good, we need to get the party planning committee together...

Metro 1.4.1 Released

2009-06-29T01:23:00.003-06:00

Metro 1.4.1 has been released! I have new documentation online, now hosted on funtoo.org:

Metro 1.4.1 Quick Start Tutorial

Metro 1.4.1 Data Model Reference

Enjoy!

Anti-Slowloris DOS patch in Funtoo

2009-06-27T16:44:00.003-06:00

I've committed an anti-slowloris patch to Funtoo, in apache-2.2.11-r1. I recommend that Apache users upgrade to this new version. (see http://bit.ly/WGSvD)

All still-affected MPMs (everything except prefork) have been masked. The intent here is to force you to use an Apache configuration that is not vulnerable to slowloris. If you are in a situation where slowloris cannot affect you, you can manually enable the other MPMs to gain access to them.

I tested the anti-slowloris patch myself on a Drupal installation I had on my machine. Without the patch, slowloris was able to make Apache unresponsive immediately. With the patch, Apache was a bit laggy but continued to run with no problems.

Apache Slowloris DoS is Nasty - Protection Guide in Works

2009-06-26T16:33:00.004-06:00

The Apache SlowLoris DoS is a pretty nasty thing. If you are running Apache (who isn't) then I strongly recommend you look carefully at the link above, learn about how this exploit works, and ensure that your infrastructure is safe.

If you are running Apache or IBM Web servers that are directly exposed to the Web, you are vulnerable. If you have a load balancer in front of your Web site (most of us don't) you may still be vulnerable. Your load balancer needs to be configured to protect against this DoS, many (including Cisco) need to be told to do so and do not protect by default. So test to ensure your infrastructure is protected.

From what I've read, if you have a squid proxy in front of Apache, you should be safe, but we have not confirmed this yet.

I'm planning to get a SlowLoris Protection Guide available early next week which will help to provide detailed information on how to protect against this particularly nasty DoS.

I also think that the Apache team's historical response to this very preventable issue has been horrendous - we all run Web servers in the real world, not some theoretical happy world, and it's Apache's job to ensure that it manages its own resources properly. For future Web-related efforts, I think I'm going to be avoiding Apache and looking at the Cherokee Web Server.

Initscripts - Keeping It Simple

2009-06-25T22:41:00.005-06:00

I've been reviewing Roy Marples' 0.5.0 release of OpenRC for inclusion in Funtoo (we're currently at 0.4.2,) and the big change with 0.5.0 is that Roy has ripped out the existing networking functionality, and replaced it with something a lot simpler.

I think this is a step in the right direction, but I'm leaning towards "going all the way" and ripping out networking support entirely, in favor of having Funtoo-supplied /etc/init.d/net.* templates that actually call the route/ifconfig/ip/vconfig/brctl commands directly, and users can modify to suit their needs.

Why do this? Here are some reasons:

WYSIWYG - no need to figure out some kind of distro-specific network configuration layer that is supposed to make things easier for you.
It encourages (but does not force) users to become familiar with the ifconfig/ip/route commands, which is a good thing.
It is the most flexible option, since it allows you do get as freaky with your network as you want.
It allows users to leverage the dependency-based functionality in the initscripts for their own purposes - the dependency functionality was designed to be used by end-users.
Theoretically faster.
Self-documenting.
Simpler.
Reduces footprint of the initscript code (reduced maintenance.)

Now, here are some negatives, along with how I hope to address them:

It's more complicated - Yes, slightly, initially, which means Funtoo network documentation needs to be written to help users. Several easy-to-use /etc/init.d/net.* samples need to be available for beginners. That will make things easy for networking newbies.
Configuration gets stored in /etc/init.d rather than /etc/conf.d (where some might say it "should" be) - This, I do not really agree with, so I don't see it as a significant negative. I don't think that /etc/conf.d should be a mandatory design rule for storing configuration. For networking, putting configuration in /etc/init.d is simpler - users who like /etc/conf.d can still use it, and I can have our templates be usable with /etc/init.d or /etc/conf.d for configuration storage. And designing complex network configuration scripts just for the sake of getting all network configuration stuffed inside variables in /etc/conf.d is not a winning strategy.
Not desktop-friendly - Some might say that Gentoo and OpenRC network configuration scripts isolate the user from the complexity underneath, and are thus better. In theory, this may be right - in reality, it creates another distro-specific configuration layer you need to learn, and developers need to maintain. And it's not significantly easier, really.

So, I think this is the right way to go. Let me know what you think.

I'm on Twitter

2009-06-24T17:35:00.002-06:00

In addition to keeping my blog more up-to-date, I've also decided to post tweets to my Twitter account (funtoo) at least daily.

I have to admit that I haven't been consumed by Twitter hysteria - if you're someone who really loves Twitter, and think I should totally love it too, please feel free to try to enlighten me about how awesome it is. :)

Until then, I'll be tweeting regularly, so that I am at least giving it a chance :) Please send any tips for newbies my way.

Quiet but busy

2009-06-24T13:43:00.004-06:00

I haven't blogged in a long time, but strangely there has been a ton of stuff for me to blog about. I've just been very busy and need to get back into the habit.

For one, I am now working full-time on Funtoo-related endeavors (no longer with E*TRADE) - I've been spending my extra time catching up on much-needed geeky things behind the scenes, which I hope to be able to share with you soon.

For Metro, even though I haven't made an official release in a while, there have been a ton -- a TON -- of changes in the git repo. Fortunately, it still mostly works the same and the quick start guide should still mostly be applicable. There is still much to do with Metro, and my focus over the next month or so will be on the parser and its next evolution. I am removing *all* existing conditional logic (i.e. "[when ...]" and ["collect foo when ..."]) from Metro itself, and replacing it with something that should work quite a bit better and be a lot more powerful. A Metro without conditionals! Crazy, huh? Turns out we don't need them.

I'll try to post more here soon.

-Daniel

Vserver, Opteron Funtoo Stages Now Available

2009-03-25T12:42:00.001-06:00

We now have Funtoo (unstable) stages available for the AMD Athlon 64 and Opteron processor. These stages differ from the generic “amd64” stages in that they have been optimized with -march=opteron, and are thus AMD-specific. They should run on AMD-based Opteron and Athlon64 systems. To download them, head over to http://www.funtoo.org and click on the “Opteron” link.

Also, I wanted to thank Benedikt Böhm for submitting patches to add Linux vserver support to Metro. I am now building Linux vserver templates for all Funtoo builds. You can find the vserver templates in the “vserver” subdirectory inside each stage directory. Likewise, you can find my OpenVZ templates in an “openvz” subdirectory inside the stage directory too. Thanks again, Benedikt, for the submission :)

Baselayout /etc/shadow local vulnerability

2009-03-02T12:27:00.002-07:00

On Friday, February 27, 2009, we discovered a local vulnerability in Funtoo caused by world-readable permissions on the /etc/shadow file. (credit: Piotr Karbowski)

This issue was due to a bug originating from Gentoo’s baselayout-2.0.0.ebuild that causes the template /usr/share/baselayout/shadow file to be installed to /etc with world-readable permissions.

The security implications of this bug are that local users will be able to retrieve password hashes for all other local users’ accounts, including root. This vulnerability is local, not remote, in nature. It likely affects all Funtoo stage3 tarballs and OpenVZ templates that have a build date earlier than “2009.02.27”.

To resolve this issue manually, type the following command as root:

# chmod 0600 /etc/shadow

To resolve this issue using a Funtoo package update, run the following commands:

# emerge --sync
# emerge baselayout

Ensure that baselayout-2.0.0-r2 is installed, which will ensure that /etc/shadow is readable only by root.

It is also recommended that you reset any critical passwords which may have been exposed to malicious local users.

This security vulnerability may also affect any non-Funtoo stages that were built from scratch using Gentoo’s baselayout-2.0.0.ebuild.

All stages and OpenVZ templates that were affected by this vulnerability have been removed from Funtoo’s mirrors. All Funtoo stages and OpenVZ templates with a build date of “2009.02.27” or later do not have this vulnerability.

This security vulnerability does not impact stable Gentoo stages as Gentoo is not (to our knowledge) currently using the unstable baselayout-2.0.0.ebuild for any of their stage builds.

This bug may affect any unstable Gentoo stages or Gentoo-based distributions that use baselayout-2.0.0.ebuild in their stage builds. If your /etc/shadow file was installed by baselayout-1.x, then it should have proper permissions. This vulnerability will impact you only if your /etc/shadow file was installed by baselayout-2.0.0.

Funtoo will be integrating security audit functionality into Metro in the coming weeks to ensure this specific issue as well as other issues like this are detected immediately, before they can impact users.

Moving Stuff Around and a Tip

2009-02-18T14:25:00.001-07:00

I’m in the process of changing the directory structure of all my builds on Funtoo.org, and everything is getting uploaded again right now. That means that stages won’t be available for several hours.

I am putting all the official Gentoo stuff in http://www.funtoo.org/linux/gentoo, and all the official Funtoo stuff in http://www.funtoo.org/linux/funtoo. I am also separating snapshots so they are in their own “snapshots” directories, under gentoo and funtoo respectively, and new snapshots (both gentoo and funtoo) will all have the “portage-“ prefix in their filename. I think this makes things a bit more straightforward, and also opens up the possibility of adding unstable Gentoo builds or stable Funtoo builds in the future. But the primary motivation is to just have a saner directory structure. I will also be committing changes to Metro so that its default fslayout is updated to follow this pattern.

EDIT: Also, in an effort to free up Web site real estate, you’ll notice that I’ve removed the OpenVZ section from http://www.funtoo.org. I am still building OpenVZ templates for all my funtoo builds, but I’m now storing them alongside the respective funtoo stage3 they were built from.

TIP: Some people have posted in the blog comments that they are having “emerge --sync” failures. If git is unable to update your tree, you can use the “git reset --hard” (run when you are inside the /usr/portage directory tree) to reset your tree to match the upstream funtoo.org tree. Any local changes will be lost, but “emerge --sync” should work again. A “git reset --hard” is typically needed when you abort a sync mid-stream, as it leaves your Portage tree in an inconsistent state. Hope this helps some people.

The Camel has Landed!

2009-01-30T00:44:00.001-07:00

Just wanted to let everyone know that Perl 5.10 is now enabled in the Funtoo Portage tree, and the Funtoo stage builds are now working with Perl 5.10.

Also, I’ve decided to return as official maintainer of the Funtoo Portage tree. I’m glad to be back!

If you’d like to submit patches or improvements to me, please email them to me at drobbins@funtoo.org. Patches inline in the email work best.

Best Regards,

Daniel

Portage is now Git-Friendly

2009-01-08T04:00:00.000-07:00

I’m pleased to let everyone know that Portage is now git-friendly. As of version 2.2_rc20, Portage will now operate at full efficiency with git-based Portage repositories. Here are details on the changes:

First - “emerge --sync” will now properly recognize an existing git-based Portage repository. When you run “emerge --sync” with an existing git-based repo, Portage will run “git pull” inside the repository to update it.

Second - due to cache fixes, dependency calculations and searches are now just as fast as they are with rsync-based trees, and eix works properly with no issues. To benefit from cached metadata (this is what makes things fast,) you must use “emerge --sync” to update your git-based Portage tree.

Many thanks to Zac Medico (Gentoo Portage developer) for integrating our fixes into Portage, as well as providing general guidance, support and an excellent and much-appreciated fix for the cache speed issue.

So, if you are using the Funtoo Portage tree, first upgrade to Portage 2.2_rc20 or later, and then start using “emerge --sync” to update your tree. This will allow Portage to work optimally for you.

Note that the one thing that Portage does not yet do for you is to create the initial git-based Portage tree for you. To create your git-based portage tree, you can grab one of the Funtoo snapshot tarballs, or use git clone. The only additional step required beyond this is to create and checkout the branch you want to use. From that point forward, “emerge --sync” will take care of everything.

And a word for developers and distro creators:

One of the goals of Funtoo is not just to give you goodies, but to give you all the tools that we use ourselves, so you can do exactly what we do. So, one thing I am hoping to package up soon is the Portage code that I use to generate all that cached metadata that we add to our git tree that makes Portage fast for you. This will allow you to create your own git-based Portage trees that are fast for your users. The code itself is written by Zac, and we are both looking at the best way to package these tools up and make the cache-creation process as simple as possible for everyone.

My LinuxCrazy Interview

2009-01-07T11:58:00.001-07:00

Before Christmas, I did my first-ever podcast Interview with David Abbott (comprookie2000) of LinuxCrazy.com.

In the interview, we talk about Funtoo, as well as a bunch of other things. Be sure to check it out – David is a great guy and I had a great time doing the interview.

Funtoo and Sunrise

2009-01-06T19:17:00.001-07:00

Just wanted to let everyone know that I’ve gone ahead and merged the Sunrise overlay into the Funtoo Portage tree.

Much thanks to Caleb Cushing (xenoterracide) who did all the hard merging work. It took him a few hours, and because of his work, the merge only took me a few minutes. We both thought that git made things a little too easy – very cool as well as a bit spooky at the same time.

For those who don’t know, Sunrise is one of the oldest and biggest Gentoo overlays, and is an official Gentoo project as well. It contains about 590 new packages in a variety of categories.

Caleb is going to be sending me regular updates to Sunrise so that we can keep Funtoo up-to-date. So if you want to learn more about Sunrise, please check out the Sunrise project page.

Metro 1.2 Released

2008-12-15T18:04:00.001-07:00

Metro 1.2, my tool for building Gentoo and Funtoo releases and custom builds, has been released. Learn about Metro here. Technically, this is a maintenance release, but looking at the ChangeLog tells me that this is more like a major release.

I’ve added a metro-1.2.ebuild to the Funtoo Portage tree.

Here are the changes in 1.2:

NEW target: "stage3-freshen" uses emerge -up world to update stage
NEW target: "stage3-quick" builds stage3->stage3, skipping stage1 and 2.
NEW target: "git-snapshot" creates a full .tar.bz2 snapshot of git repo with the .git directory intact, so it is a full working git repo when unpacked.
NEW target: "emerge" for adding packages to an existing stage3
NEW subarch: ~core2 for Intel Core 2 series processors. Supported with gcc 4.3.2 which is in the funtoo build.
Updated targets: "openvz" updated to support OpenRC 0.4.0.
stage3-freshen supports emerge/packages/force target, which is a list of packages to rebuild using "emerge <packages>". This is a great way to force a rebuild of packages that may need to be rebuilt but did not get a rev-bump (ie. openrc.:)
Metro now properly cleans /var/tmp/metro after it is done, unless the build aborted due to an error.
scripts/build.sh: now supports stage3-freshen and stage3-quick by allowing an extra "quick" or "freshen" option as the last argument.
scripts/build.sh: now creates a "current" symlink pointing to the most recent stage3 built. This "current" symlink exists in the subarch directory and the filename is "stage3-subarch-current.tar.bz2."
scripts/build.sh: now uses "git-snapshot" for funtoo builds, so funtoo snapshots are now double the size but are a full working and "git pull"-able git repo, simplifying funtoo install process.
stage3 targets: try to ensure that /etc/make.conf.example exists. Create a symlink from /etc/make.conf.example to /usr/share/portage/config/make.conf.example.
metro core: robust mount point detection using /proc/mounts instead of output from the mount command, which can be unreliable.
metro core: more robust umount algorithm.
metro core: local name support, so your custom stages don't need to be called "funtoo" anymore. Also quite useful with the "emerge" target for creating custom stage3's with different names.
metro core: ccache works correctly, but has been disabled by default as it may cause build failures when used with Portage parallel merges (--jobs) and we now have "stage3-freshen" and "stage3-quick" when we need a stage3 built quickly.
parser syntax: For conditionals, the "in" keyword is now supported as an equivalent for "is".
parser syntax: New "lax" expansion method, ie: $[foo:lax] will expand to foo if it is defined, otherwise it will not throw a parser error (as it normally would) and instead expand to a dummy value which may contain spaces.
Documentation: github wiki documentation updated to reflect new features and QuickStart Guide refined to be easier to understand.
Projects: new projects/ directory contains source code for funtoo.org Web site. Funtoo.org Web site redesigned to be more of a portal for new Funtoo builds. The funtoo Web site can be generated by running "pytext index.html" (sent to stdout.) Embedded python code automatically updates Web site symlinks to point to most recent versions of builds and list relevant build-related information. Support for displaying build dates in human-parseable form (ie. "Yesterday" or "2 Days Ago".)

What I’ve Been Up To – New site, etc.

2008-11-26T16:30:00.002-07:00

Here’s what I’ve been up to:

http://www.funtoo.org has been redesigned. It now has more of a portal design to get you to the latest Funtoo, Funtoo+OpenVZ and Gentoo builds. I hope you like it :)

In the Funtoo Portage tree, I’ve added a new build of OpenRC, a new udev ebuild (133 with some OpenRC-compatibility and other fixes), and I’m keeping Portage 2.2_rc* unmasked. The Gentoo Portage tree has masked 2.2_rc* to try to get more testing of the upcoming 2.1* release, which is fine, but we’re going to stick with the development branch.

Work is ongoing with Metro. The git version (pre-1.2) now has a few new targets: stage3-quick and stage3-freshen. In the past, Metro would build a new stage3 by going through these steps:

seed stage3->stage1->stage2->stage3

“stage3-quick” builds a new stage3 as follows:

seed stage3->stage3

It uses ROOT=/tmp/stage3root and emerge system to make this happen.

“stage3-freshen” is even faster, and builds a new stage3 as follows:

seed stage3->stage3

The difference here is that the “stage3-freshen” target uses the seed stage3 in-place and runs an emerge –u –deep system and emerge –u –deep <extrapackages>. This is a nice way to freshen slightly old stage3’s without doing a full rebuild.

So there you have it. I hope you enjoy the new site. I sure do, since it updates itself automatically, and it’s fun for me to look at every morning :)

Metro 1.1 Custom Builds

2008-11-05T14:17:00.003-07:00

Hey, I’ve just updated the Metro wiki with info on how to use Metro to perform Custom Builds. Armed with this info, you will be able to create your own customized versions of Gentoo. You can use everything from a customized Portage tree, to customized global and local USE variables and keywords, and custom package lists.

Dive in and let me know how Metro custom builds work for you! Post your feedback in the blog comments below.

Note: If you already have Metro installed, just run “git pull” in the /root/git/metro directory to automatically download all the latest Metro updates.

UPDATE: I've also updated the QuickStart Guide so you can see how I use fcron and build.sh to automate my daily builds. Armed with this info, you will be able to duplicate my daily build routine on your local workstation!

ANOTHER UPDATE: If you are using the Funtoo Portage tree, you will now find a dev-util/metro-1.1.ebuild that you can emerge to install Metro :)

Metro 1.1 Released

2008-11-04T17:12:00.001-07:00

I’ve just released version 1.1 of Metro and updated the QuickStart Guide to reflect this new version. Currently, the only way to install Metro is to use git – see the improved QuickStart Guide for more information. Here’s a list of all the improvements to Metro:

Reorganized /etc/metro directory
NEW: "Builds" defined in /etc/metro/builds/ - all key build information now in a single file for convenience.
NEW: "FSLayouts" defined in /etc/metro/fslayouts/ - all mirror-related file names and path structures now in a single file.
NEW: OpenVZ template support for Funtoo stages, enabled in scripts/build.sh
NEW: Reorganized file paths inside /usr/lib/metro.
NEW: Working gentoo/emerge target, when you just want to take a stage3 and create something new by emerging stuff in it.
NEW: Targets now have an OS prefix. What was target: snapshot is now target: gentoo/snapshot, etc. The path structure in /usr/lib/metro has also been improved so that people can easily create a /usr/lib/metro/targets/myos/ directory and start hacking.
NEW: Several parser bug fixes
FIX: Improvements to metro have allowed scripts/build.sh to be simplified.
NEW: Improved error and exception handling
NEW: Enhanced Portage git snapshot support - Metro will now keep a local cache of a git repository so it doesn't need to re-clone it every time, saving bandwidth. This cache location is defined in the path/cache/git variable in /etc/metro/metro.conf.
Embedded Python interpreter can no longer reference Metro metadata directly. This prevents weird side-effects from occurring.
NEW: Conditional expansion - a variable such as $[foo?] will get expanded to "yes" if defined and "no" if not defined, providing easy integration with scripts so you don't throw a parser exception by referencing an undefined variable.
* NEW: "zap" expansion - in a multi-line element, a variable such as $[foo:zap] will get expanded to the name of the variable if it is defined. If it is not defined, however, then the entire line will be deleted from the output. This allows very compact conditional lines in templates like this: CFLAGS=$[portage/CFLAGS:zap]
FIX: MAKEOPTS now properly applied to Gentoo builds, speeding up builds on multi-core systems significantly.
NEW: [options parse/lax] and [options parse/strict] to disable and enable strict parsing, respectively. When lax parsing is enabled, any referenced undefined variables in multi-line elements will get dummy values, but an exception will not be thrown.
NEW: conditonal sections - it's now possible to write:
[section foo when bar]
This is equivalent to the old:
[section foo]
[when bar]
NEW: conditional equality - it's now possible for all conditionals to test for equality, such as:
[section foo when bar is oni]
or
[when bar is oni]
NEW: conditional list-match - it's now possible to see if an element matches a list of values, such as:
[section foo when target/subarch is x86 amd64 pentium4] This section will be active if target/subarch has a value that matches one of the specified values.
CHANGE: multi-line elements that are referenced inside other multi-line elements, using the syntax $[[foo]], will only get expanded if they appear at the beginning of the line. This allows them to easily be commented out by prefixing them by a "#" or any other character. In this case, the $[[foo]] will not be expanded and will be passed to the output as-is.
CHANGE: path/mirror/srcstage and path/mirror/deststage have been renamed to /path/mirror/source and path/mirror/target respectively.

Enjoy!