I was one of THE first people to ever have a public GMail account, in 2004 (before the “GMail invitation market” was created in which people would buy and sell invitations to the service).  Since then, whenever there is a new Google service, I typically get invites.  I was among the first in my group of friends to get a Google Wave account, and I got a Google Music Beta invite before many of my friends.  Somewhat related, Google has approached me with employment more than once, over the years.  Somewhat even less related, I own 2 Android phones, and a Barnes & Noble Nook Color that I’ve rooted with Android 2.3.  (Have I made my Google resume’ clear yet?)

What I don’t understand is why I keep hearing about people using Google+, but I have not yet got an invite.  It isn’t like Google doesn’t know about me — granted, they know about everyone, but I would think they know about me more than the average Joe.

I signed up for an invitation on the very first day I heard about Google+ on their website, yet the status on that page remains the same:

Google+ is in limited Field Trial
Right now, we’re testing with a small number of people, but it won’t be long before the Google+ project is ready for everyone. Leave us your email address and we’ll make sure you’re the first to know when we’re ready to invite more people.

I’m not buying it. GMail was in “beta” for years. Five, to be precise.  Google Wave was in beta the entire time it was online (and still is, even though the entire Wave project has been scrapped).  Google Music is still in beta — officially named Music Beta by Google.

Google+, from my understanding, is that it is a social network service similar to Facebook.  Some have even called it “the Facebook Killer”.  As with any social service, success of the service is measured in how many people have subscribed to the service.  If I were Google, I would want as much exposure as possible…  by as many people that enjoy Google services as possible.  Why have a limited rollout of something so central to Google?  Why have a limited rollout of something, and then not invite your most loyal fans?  Is this their way of saying they don’t love me any more?

O, Google, why have you forsaken me?  I feel gypped.  I see you pushing Google+ heavily, redesigning GMail, Google Calendar to have a similar look and feel to Google+, and I can only feel like the 3rd cousin that has been forced to eat dinner in the garage because there are no more seats at the kids’ table.  While reading daily news (on Google Reader), I see tours, tutorials and helpful tips of how to use Google+, and can only think “wow, that seems really nice, and seems to solve what bothers me about Facebook… too bad I can’t try it out.”

What gives, Google?

I started when I was in the military, in a secure compartmented information facility (SCIF) communication center — basically a large multi-floor building with all the windows painted dark.  My coworkers would occasionally leave for what seemed like half an hour at a time, while I would be left routing messages, delivering messages, and troubleshooting errant messages.  Of course, this would happen multiple times during an 8 hour shift.  Being in the comm center, on the 3rd floor of a SCIF, it would take about 5 or 10 minutes to get to the designated smoking area within the compound outside the building, passing through checkpoints along the way.  I got tired of pulling more than my fair share, so I would sometimes go on a smokeless cigarette break that easily turned into a cigarette break, and then sometimes a multi-cigarette break since it took some effort to get to the right place to be able to take a break.  Cigarettes allowed me a certain freedom, to step away from what I’m doing so that I can congregate with the other “break takers”.

Over my 22 years of smoking, I’ve met many passersby, networked with fellow smokers, and made a few friends I would not have otherwise been able to if I had not taken a “smoke break”.  Of course, the nature of my work has changed in that 22 year span.  I no longer work in a secure communications center, luckily I also don’t work on the 3rd floor, and I don’t work in the military, but for most of the 22 years I’ve worked at a desk.  Sometimes the desk was in an office, sometimes it was in a 5-foot by 4-foot enclosed space inside a mobile comm center, and sometimes the desk was in a tent in the dirt and mud.  I’ve grown accustomed to being able to push my chair back, get up, and walk away from the desk, to think and smoke, for about 10 or 15 minutes at a time.  In fact, I can’t imagine being at a desk for a full 8 or 9 hour day without leaving, except to go to the bathroom or lunch.  I don’t think I can function being glued to a desk for more than a few hours.  I need to step away sometimes, for clarity and reflection.  I can say this with certainty now because of what happened 72 hours ago.  I quit smoking.

I don’t think I should need to explain the reason I quit smoking, because the health risks connected with smoking and tobacco products are not secret.  There are on the order of 4,000+ chemicals, additives, and carcinogens in a cigarette.  Some of them are consumed by flames, while some of them are blocked by a filter on the end of the cigarette, while others find their way into the mouth and lungs.  No one can say with any degree of authority exactly what is in a cigarette, but everyone knows they are bad for you.  This is not a tale about why I quit, but how.

One day while downloading things in Vuze, I was shown an advertisement for a new brand of electronic cigarette: Blu.  In the advertisement, it was explained briefly that the company had constructed a battery charger that recharged electronic cigarettes and looked almost exactly the same as a regular pack of tobacco cigarettes.  I thought this was genius.  No longer confined to smoking near an electrical source of power, the electronic cigarette (e-cig) could be put back into its “pack”, in my pocket, and be recharged the next time I pull it from my pocket.  I had casually looked into electronic cigarettes a few years ago, and concluded that it was just way too cost prohibitive, not practical, and would likely cause me to just stick to regular cigarettes.  That’s what I did, though I did make the switch from Lights to Ultra Lights.  I had looked into the gum, but seemed just as much or more expensive than cigarettes.  The patch scares me, and is also not cheap.  This time, it seemed that e-cig technology had advanced enough for them to actually be practical for me.  I began researching.

My first goal was to determine if electronic cigarettes were worth it.  Namely, would I get the same enjoyment and satisfaction out of e-cigs as I would a normal tobacco cigarette?  Would it feel like smoking, or would it feel like…  something else entirely?

I started searching the Internet. (God, I love Google.)  I found reviews, forums, videos, review websites, review videos, recommendations, e-cig makers, sellers, e-liquid makers and sellers, and entire “mod” underground community akin to Altoid Tin Hackers that modify electronic cigarettes to boost the amount of smoke or make the batteries last longer.  I researched for about 2 months, and learned several things.

Electronic cigarettes have actually come quite a way since first conceptualized.  The entire thing almost went away completely due to a rumor, but things bounced back, and the movement is growing.  E-cigs are well on their way to being a better alternative to tobacco cigarettes.

Being a type A personality but also a smoker looking to quit, yet being skeptical, rather than just reading good things about e-cigs, I tried to find reasons it wouldn’t work.  If e-cigs gave me reasons for it to fail for me, those reasons would linger for a while and might eventually be a self-fulfilling prophecy, and I would probably go back to smoking regular cigarettes.  So, the first question I asked the Internet:  What goes into an electronic cigarette, and are those ingredients bad for me?

Electronic cigarettes are comprised of 2 types:  3-piece, and 2-piece.  The older 3 piece design consists of the battery (the largest component), the atomizer (the middle piece), and the cartridge (the component that actually meets the lips, and holds the nicotine solution).  When in use, the user puffs on the cartridge end which draws a portion of the solution toward the atomizer, powered by the battery, and heats up the liquid to a vapor that is drawn into the mouth.  Typically, when this is going on an LED at the end of the battery will light up, to give the illusion of a regular cigarette’s fiery ball at the tip.  Through normal use, a cartridge will last about as long as a pack of regular cigarettes, depending on how strongly the user inhales.  The batteries of 3-piece e-cigs last about a day or so, again depending on use.  A 3-piece e-cig atomizer, however, typically lasts 5 or 6 months, needing to be replaced with a new one.  Individually, each component is rather cheap.  A battery costs about $30.  A cartridge, about $2.  An atomizer, about $20.  The newer technology 2-piece design, however, integrates the cartridge with the atomizer.  This means that replacing the atomizer separately is no longer needed, because when the cartridge is empty, a new cartridge is replaced with a completely fresh atomizer.  Cost is also somewhat reduced, because cartomizers (cartridge + atomizer) cost around $2-$3 each, and last about as long as a pack of cigarettes.  But, these are merely the components of the device.  What actually goes IN an electronic cigarette, and is it bad for me?

As it turns out, most e-cig makers tell you exactly what they put in them, since its only the “juice” that gives the taste and enters your body.  The typical ingredients for an e-cig:

• one of the following:
  • Propylene Glycol (PG),
  • Vegetable glycerine (VG), or
  • Polyethylene glycol 400 (PEG 400)
• Nicotine
• artificial flavor
• water

The most popular of the glycols seems to be PG, propylene glycol.  I realize that seems like a pretty scary name, even so much that it seems like its hiding behind the friendly acronym PG, but its pretty harmless.  PG is used quite a bit in:  fog machines, asthma inhalers, nebulizers, food coloring, moisturizers, hand creme, mouth wash, cosmetics, and massage and fragrance oils.  The only harmful (relatively speaking) ingredient in an electronic cigarette is nicotine.  This is comparable to the Nicotine Patch, or Nicotine gum, which also contains a similar amount of the substance.

My next Devil’s Advocate question:  Is it more expensive than regular cigarettes?

Short answer:  It depends.  Long answer:  Initially, yes, it will be a bit expensive, mainly because you are buying something you don’t already have –  batteries, chargers, etc.  If you purchased an expensive lighter at any point in your smoking career, this is a similar cost.  So, accessories aside, the actual cost needs to be compared in terms of use.  Since smokers typically measure their use in terms of packs smoked per day, that’s what I’ll use.  A pack of cigarettes, nowadays, costs around $5-$6, depending on where you buy them — a pack at the corner grocery costs less than a pack at a highway gas station, which costs less than a pack at the airport.  A cartomizer costs around $2-$3, depending on make, model, and “juice” flavor.  Also, you can purchase “blank” cartridges that you can fill yourself with liquid that you’ve bought through a third party, but “blanks” typically cost the same as pre-filled flavored cartomizers.  So…  e-cigs seem to be cheaper, at first glance.  Lets dig deeper.

A typical lithium-ion battery, which seems to be what most e-cig makers are using, degrades over time since they can be recharged a finite number of times (this varies depending on the quality of the battery, but ranges anywhere from 500 to 1000 times), and battery makers usually give a battery a much lower rating under 500 so that they can give a guarantee it will work at least that long.  So, lets say a typical battery will last a year.  That’s $30 per year.  This is a cost that cigarette smokers don’t pay — lighters are much cheaper annually.  So, now lets do some math.

I’ll assume I’m a pack-a-day smoker, which I’m not, but that makes the math much easier.

1 5-pack of cartridges:  $10  (this should last at least 5 days), so a year of cartridges would cost $730

2 batteries: $60 (each should last at least a year, but by the third year we may need to buy another)

That brings our total for e-cigs to $790 per year, for the next few years.

Cigarettes: $6 per pack.  365 times $6 = $2190.  That’s one year.

Ok, maybe that was too harsh.  $5 per pack, for 365 days = $1825.  Ok, so the actual cost is somewhere around two thousand dollars, given the fluctuating price of cigarettes, and taxes, and the fact that will only go up each year by some small amount.

That leaves me with a difference of over a thousand dollars per year.  Granted, I’m going to need chargers and accessories to work with whatever e-cig, but that will never approach the $1000 per year figure — and if it does, I’ll still break even with a large pile of accessories.

How different is it from regular tobacco cigarettes?

This was a difficult one to answer, since there is no website that can detail the experience without having a heavy commercial advertisement slant to it.  It seemed the answer for me depended largely on my own cigarette smoking style.  Some e-cig makers focus on making the “smoke” vapor look exactly like cigarette smoke, while others focus on creating the most amount of vapor per puff, while yet others focus on flavors.  I was beginning to see that since this was such a new thing I might have to purchase/try a bunch of e-cigs until I found one I liked.  Then, I saw this video.  This gentleman, after a short rant about why he no longer chats on a particular forum, begins to outline all the different electronic cigarettes he’s tried through his quest to find the perfect “vaping” device.  The video is pretty informative, since the guy is rather well-versed and considers himself an expert, but the thing that really got me…  what he said that galvanized things for me and convinced me I was on the right track…

“I’ve spent well into $2,000 on e-cig products, juices, but I have not one complaint about how much I’ve spent.  I don’t smoke, so that’s worth a million dollars.”

The gentleman seems really knowledgeable about what he doesn’t like.  I have no idea what he is talking about through most of the video, but one thing seems pretty clear:  He seems to really like “cartomizers”.  This is coming from a guy that appears to be using e-cigs since the beginning, and has formed a well-experienced opinion.

With all the answers telling me that I should make the switch, all that was left for me to do was decide WHICH electronic cigarette to get.  This sent me back onto the Internet in search of the best electronic cig for me — which meant I had to watch review after review after review. I had to add up all the pros and cons for each one that got a good review, then I had to decide on functionality and reliability versus cost.  This is not going to be a review of electronic cigarettes…  but this might turn into a review of e-cig review sites!  This site was pretty helpful.  Clearly, it recommends Green Smoke, so I checked that one out first.  While it seemed really expensive, the reviews were the best — almost as if they had nothing bad to say at all.  Second on their list was Blu, which is what brought me into this quest.  The review wasn’t that great, but I kept looking.  Third on their list was White Cloud.  1 word:  EXPENSIVE.  Even on the White Cloud website, they claim to be the most expensive electronic cigarette on the market (then they go on in attempts to justify the price).  The next on the list was V2, and then South Beach.  After reading all those reviews, I decided to go for another opinion.  So, I went here.  What an excellent site.  He has a clear and sensible methodology that I wish all reviewers of any product would have.  Given his information on that website, Green Smoke seemed to be the best.  So, then I went here.   Not a bad site, really, except she seems to rank all e-cigs pretty closely together.  Then, I noticed that for most reviews there is a coupon code.  After a bit of investigation I learned that once you buy into a particular electronic cigarette, the big brand name makers give buyers a coupon code, so that regular customers can refer their friends and get a discount.  That seemed like a biased approach to me, since all the reviews I’m interested in have a coupon code.  Then, I ended up here.  What I really like to see is real people, saying real things, with money they’ve spent from their pocket — really can’t get a better review than what a person thinks about how they’ve spent their own money.  That review eventually led me to this site in which the woman claims V2 is her favorite e-cig.

I was convinced. Not only was I sold on the concept of electronic cigarettes, I had a pretty good idea which one — either Green Smoke, or V2.

After thinking about it for a bit more, I settled on V2.  Green Smoke seems like a really good product, but… well…  They have very few flavors, and the cartridges are a tad more expensive.  V2 seemed like the better choice of the two since I get more options when dealing with them.

I ordered the “Travel Kit”, which included almost every accessory they offer, with 2 batteries (1 automatic that is activated by just puffing on the end, 1 manual that is activated by pressing a small button), and 15 cartridges of a flavor of my choosing.  I opted for 3 flavors; 5 of vanilla, 5 of cherry, and 5 of “red” which is supposed to be similar to Marlboro.  June 12, 2011, I ordered the kit with USPS Express shipping.  June 14, 2011, 10am was my last tobacco cigarette because at 11am the same morning my electronic cigarette was in my hand.  I’ve also re-ordered a ‘sampler’ pack in which they include all flavors they have, so I can try them all, and another 5-pack of cartridges.  I’ll try “menthol” this time, though I was not a frequent/regular menthol smoker.  The description of the flavor says its a mix between peppermint and cool mint, which doesn’t sound bad.  Their newest flavor, Cola, I don’t think I’ll try, but I’m waiting for their newly announced yet unreleased flavors, banana, and coconut.

I may even branch out into other cartomizers(cartridges) that are compatible with my e-cig, though I doubt I’ll go the route of buying “blank” cartridges and liquid to fill my own.

I’ll say that again… Email is not a storage mechanism.

By this, I mean that the purpose of electronic mail is not to store important files, information, or future reference material. It was never intended for that purpose, and even in today’s standards it still falls short of that use.  Of course, there is GMail today.  Of course, there are GMail extensions (like gDisk and GMail Drive Shell Extension) that allow you to store your MP3 collection, photos, etc.  That is a good example of what I’m referring to.  I’ll explain…

GMail, as most of us are aware, is not a typical electronic mail system.  It does not operate within the paradigm of traditional email systems.  Google Mail’s primary interface is via the web page in which email messages are only sorted by “threads” (“conversations” in GMail-speak).  But, more on GMail later.  Back to the point…

Email began as a way for users of a time-share system to communicate with one another, coordinating within the same closed system.  Soon thereafter, it became a method of communicating with users of other time-share systems, yet with serious limitations — namely, the sender of a message was required to know the path in which the message took to get to the intended recipient.  Instead of having addresses such as

poe@deadpoets.org

there were “addresses” such as

localhost!nextdoor!nextnextdoor!poe@deadpoets.org

which meant that the message had to travel from localhost, to nextdoor, to nextnextdoor, to finally deadpoets.org in order to reach user ‘poe’.  The machines did this in an automated way, as long as the route specified was correct.  If one of the machines along the message route was offline, or otherwise not accepting incoming mail, the sending machine held the message for a certain period of time until either the message was accepted on the receiving host or the sending machine effectively gave up — at which point, the message was lost forever.

Eventually, the machines connected to the network grew in number and a machine’s knowledge of other networked machines needed to scale as well.  Email needed to change with the new networking methodology, which is why we have user@somesystem.com today.  The sender of a message needs to know only the address(es) of the recipient(s), the subject of the message, and the message itself.

Notice in all the above explaination it reads “the message”, and not “the file”?  There is a reason for that.

Consider this example:

Alice wants to send Bob some files.  The total size of the files is 9.5MB. The contents don’t matter for the purpose of this example, so lets just say the email contains a few large photo images, and a large document.  In order for Alice to send these files, in an email to Bob, she would need to first specify Bob’s email address as the intended recipient.  Next, she will likely describe the contents in a few words in the Subject: field of the message, “The stuff I wanted to give you”.  Then, she sets about attaching each file she wants to send to Bob.  Each of these files becomes encoded in a very long set of letters and numbers, completely unreadable by any human, and inserted into the email message “envelope” so that each email system that handles the message will be aware that it is a message with a Subject: and multiple files attached of differing size and type, such not to get the files intermingled among each other, nor this specific message’s contents intermingled with any other message that might be handled.  Next, Alice presses “Send”.  It takes a moment for her computer to actually send it because most email systems aren’t expecting (or designed) to handle messages of that size… but it gets sent.  The message is then copied into Alice’s “Sent Mail” mail store (sometimes called “outbox”).  Bob does not see this message right away — this is not file sharing, nor is it Instant Messaging (IM).  Alice’s message is received from Alice’s computer, is copied onto Alice’s email server, which then needs to determine which machine handles Bob’s email.  Once that is determined, the message is sent again — to the machine listed after the ‘@’ in Bob’s email address.  That receiving machine typically makes attempts to verify that it is a message coming from an actual person (like Alice, and not a Spam robot), is destined for a person that it handles mail for (like Bob, and not Boob), and that the size of the message is within the system’s constraints for reasonable handling (typically 10MB).  After the message is accepted, it is written to Bob’s email server (this is the 3rd copy of the message) for delivery handling.  Assuming that Bob has not forwarded his mail elsewhere (which would further the process of sending/copying the message again), the message is then stored in a holding area on the server’s hard drive, to await Bob’s email client.  Once Bob’s email client connects to the email server, the message is copied yet another (4th) time to Bob’s computer.  The message will reside on both Bob’s email server, Bob’s computer, and Alice’s email server, and Alice’s computer (in her “Sent Mail”, remember?) until either Alice or Bob delete their respective copies of the message.  For a single 10MB message, it has taken multiple computers copying, and costing a total of at least 40MB of storage space.  This is not taking into account various spam/anti-virus systems, which also typically store each message for a short time.  More importantly, this is also not taking into account that had the message been addressed to more than one person (say Bob and Charles), the message would be stored 6 times — server and user’s computer, for each user — which would amount to a total of 60MB for the sender and two recipients.

Email systems treat messages as such. Sure, each message is a file, but a message to be delivered to user1 cannot/should not/will not be considered the same message as to be delivered to user2, even though it has the same file attached to the message and may contain the exact same contents.  Electronic mail is designed this way for privacy; not file-sharing.

“But, Mr. Linux Ninja Geek…  storage is cheap!”

Yes.  Storage is cheap.  However, transmission is not.  It takes relatively a small amount of effort for your computer to generate data, or even say copy data from your camera, and store it onto your computer.  It is much more effort to transmit that same data across the Internet to another computer, and have it stored there indefinitely.  Enter email into that equation and the effort is mulitplied by each computer the message travels through to get to the final destination.

“Ok.  So, I shouldn’t send files attachments in email at all??”

That is not what we’re talking about.  We’re talking about storing email messages indefinitely.  Consider that information in a typical message has a given lifetime.  Normally, this lifetime is very short, on the order of days or even weeks, possibly even months.  After this time passes, is the information in the message of the same importance, or has it become much less important?

To demonstrate this, let us employ an analogy…

In the old days, before email, people wrote correspondance — stone tablets, papyrus, handwritten, typewritten.  The message itself was carried, by another human, to its intended recipient, and either read aloud or delivered into their hands.  Once the information within the message was received, what happened to the message itself?  In the case of stone tablets, it was likely destroyed — or made into some type of monolith, depending on what the actual message was.  In the case of papyrus, the message was read aloud, retrieved, and kept for futher use — this is why the message was stored on a scroll, because it contained more than a single message for more than a single recipient.  In the case of handwritten or typewritten correspondance, either the letter was kept in a folder in a file, or it was discarded sometime after the message was understood.

That last part, concerning handwritten/typewritten letters, is probably the closest analogy to email.  After the letter was filed away, what was its disposition?  More often than not, the letter sat in the file for a long time, until someone either tossed it out with the trash, or it was framed for historical purposes.  Point:  a letter was hardly ever kept “in case I need it again”.  The physical letter’s disposition was certain, upon the moment of receipt, similarly to stone tablets and papyrus scrolls.  Why?  Because physical objects need space to be stored indefinitely.  The more physical objects that need to be stored, the more space required, of course.

Hard drive space is required to store electronic mail messages as well.  In all cases of message storage, the information contained within does not change after delivery.

Enter GMail.  GMail’s claim to fame was that, initially, the storage amount was enormous, compared to other offerings like Hotmail.  Leveraging Google’s search abilities, supposedly you could instantaneously find any email you ever received in the GMail system.  This goal is in line with having conversational correspondance with other people connected to the Internet, only in a different way.  GMail does not sort messages by date, subject, or even sender of the message like typical email client software.  The only sorting mechanism available is by “thread”, which makes GMail seem more like Usenet, or an online discussion forum.  This design structure does not seem to lend itself for file storage at all — much less attachment storage.  Sure, you can save a message (or entire conversations) indefinitely, for later review.  How easy and practical is it to do that?  How important is that email from years ago?  More importantly, how many other email systems are similar to GMail?  It doesn’t seem that GMail is a good gauge as to what an email system can or cannot do, since it seems to be a consensus that GMail is different from the rest, and since GMail is due to fail without warning among other technical limitations.

Given that a message’s information/content/meaning does not change after it is delivered, why is email kept for so long?

Not just that, but if a message is noticed to be lost (presumably a while after it was actually lost), why is it so important to have the message restored?  What could possibly be contained in a message, that wasn’t noticed to be missing, that has become critical this very moment?  Could the information not be resent from the sender?

More to the point:  Why are people storing information in email?

This term was coined in the mid-’90s when two guys at Yale wrote that lifestreaming is:

…a time-ordered stream of documents that functions as a diary of your electronic life; every document you create and every document other people send you is stored in your lifestream. The tail of your stream contains documents from the past (starting with your electronic birth certificate). Moving away from the tail and toward the present, your stream contains more recent documents — papers in progress or new electronic mail; other documents (pictures, correspondence, bills, movies, voice mail, software) are stored in between. Moving beyond the present and into the future, the stream contains documents you will need: reminders, calendar items, to-do lists.[1]

The Internet is attempting to capture lifestreams of its citizens, but doesn’t appear as inclusive as most companies want it to be — mostly due to the nature of the events, or documents, or because no one web company owns the rites to all of one person’s information.  …This seems to be a good thing.

There are, however, quite a few aggregators on the web, like Facebook, Tumblr, Posterous, Collectedin, and Flavors.me.  These do a fairly good job of aggregating in a social network context, meaning that they typically lifestream content from social networking websites (silos) that people post specific content to.  For example, if a person posts a photo to Flickr (a silo), the photo is on that website of course, but can also be retrieved from sites like Facebook (an aggregator, and a silo).  I say that Facebook is an aggregator and a silo because Facebook started as a silo (only allowing posts from its subscribers), then branched into reading feeds from other sites like Flickr and Twitter, but Facebook itself is difficult to integrate into a secondary aggregator (like an RSS reader) as it has changed its layout and hidden its RSS feeds numerous times.

I particularly like the look and layout of Flavors.me, as it attempts to present information in a person’s lifestream, yet also segregates the data from different sources, which doesn’t intuitively give the audience a flowing context of the lifestream itself.  In other words, personal Flavors.me sites look very nice, but are still somewhat disjointed.

I decided to task myself in creating my own lifestream on a website, but it had to meet my own specifications while maintaining a particular look and feel.  I began the process by researching all the different types of information on the web that one could aggregate — from Facebook posts, to Flickr photos; from blog entries to music tracks played recently.  I came upon a very nice and specific piece of software named Sweetcron.

Sweetcron, created by Jon “Yongfook” Cockle, was designed to be blog software that could capture and display a person’s lifestream.  Similar to Tumblr and Friendfeed, but could be installed and run on a personal server, eliminating the need to subscribe to yet another commercial service.  Sadly, I also found that Sweetcron, as wonderful as it might be, is no longer maintained by the original developer.  This is where I discovered a Sweetcron fork — derivative software, named Lifepress.

Lifepress seems wonderful, as it has the functionality to aggregate feeds from different sources, and also comes with a bunch of plugins to handle the sources that aren’t as intuitive.  It is also quite them-able, though there aren’t many Lifepress themes to be found.  Luckily, Sweetcron themes can be easily adapted to Lifepress, though there aren’t as many Sweetcron themes, either, compared to regular blogging software like WordPress.

As with most lifestream sites, they contain things like Flickr photos, blog posts, etc. that you can read/view in the lifestream itself, I decided that mine would only be an aggregator rather than a full-blown lifestream blog that contained comments, etc.  With mine, only linked posts from other sites can be read at the source if the link is followed.  This allows me ease of maintenance in that I don’t have to worry about comments on a blog post and comments on a aggregated blog post at the same time.  Obviously, this also prevents me from posting once and having software disperse the data — such as when people post to Twitter, and it magically appears on Facebook also.  If I were to do that with a lifestream, it seems it would likely be caught in an endless loop…  blog -> lifestream -> blog -> lifestream, ad infinitum.

So, after a few hours of installing and modifying PHP scripts and CSS stylesheets…even had to edit an image or two…

I have my own lifestream website.  I only have a few feeds added in, but I’m sure there will probably be more as I toy around with things.

[1] http://cs-www.cs.yale.edu/homes/freeman/lifestreams.html

For those not in the know, cracking passwords has been going on for decades.  Common rationale is that if the “good guys” can figure out your password before the “bad guys” do, then the “good guys” can inform you so that you can change your password before the “bad guys” do something horrible in your name.  That’s the thinking, anyway, though many systems administrators and security professionals don’t regularly attempt to crack passwords of their users.  Why?  Too much effort for little reward.  They figure “if I have to go through all this trouble to get a password, then there aren’t many people out there that will do the same”.

That is when the level of effort involved outweighs the risk involved.

Good guys’ view:  If the amount of effort to protect something outweighs the risk, it slowly becomes an acceptable risk.  In other words, if you have to put up barriers, maintain the barriers, and remind people to not tear down the barriers, all to protect one person’s Social Security Number, you are less apt to do so because that SSN is probably discoverable by other means.  If the level of risk is greater than the amount of effort to protect it, then new security measures are put into place.  The trick, here, is determining the level of effort.

Bad guy’s view:  If the target is greater than the level of effort to obtain the target, attempt to obtain the target as long as this holds true.  As time passes, the level of effort increases, or the value of the target decreases.  Specifically, if the target is a password of an account at a bank, the amount of effort to obtain the password is about the same as obtaining a password from just about anywhere else.  The value of the bank password is great, however, as time passes the value goes down because the owner of the password might change the password, and the game begins again from the beginning.

Game changer:  GPU.  Graphics Processing Units.  The difference between a CPU and a GPU is enormous.  We’ve talked about this before.  If you missed it, a very good visual conception of the difference between CPU and GPU is here: [flash http://www.youtube.com/v/fKK933KK6Gg]

Enter password cracking.  Historically, it was done on CPUs.

Password guessing applications attempt guess after guess.  1AAAAA, 2AAAAA, 3AAAAA  in what we call “brute force” attempts.  Another approach is to attempt guesses with the use of a dictionary.  Apple, Aardvark, AppleArdvark, ArdvarkApple, and so on until all the words in the dictionary were spent.  Keep in mind, here, that each change in letter represents a completely new attempt at the password.  A group of fast CPUs can crack a “weak” password in just under a few hours.  One CPU can obtain an obvious password in just a few minutes — but then, if it were obvious, you wouldn’t need a CPU to obtain it.  A good, or strong, password can take weeks or even months — if it is able at all.  The historical thinking was that the more CPUs you dedicated to the task, the faster you could crack the password, depending on how strong of a password and the type of algorithm was used to protect it.

It has recently become much easier.  For both the good guys and the bad guys.  Amazon EC2 is able to leverage GPUs very cheaply.  The level of effort to obtain passwords has hit rock bottom.  Your passwords need to be much stronger now.

Follow the link to learn how it was possible to crack multiple (weak) SHA1 passwords in under an hour using Amazon EC2 with CUDA. (For the cost of a few bucks and a few hours preparation.)  As a comparison, it would typically take an average desktop at least a day to accomplish similar results.

I’ve gone through this process quite often, and most times its pretty fun to see new things become available that solve particular eccentricities or minor flaws in the tools I use already.

Recently, I’ve been on the “Am I documenting things as efficiently as I can?” kick.

First of all…  I’m not a fan or wikis.  Actually, I dislike wikis.  Most of them are atrocious, with very few exceptions.  I find directly editing HTML easier than editing wikis, with their ridiculous markup language — come on,  two single-quotes for italics; i.e., ''italics'', three single-quotes for bold; i.e, '''bold''', and FIVE single-quotes for bold+italics; i.e., '''''bold+italics'''''?  Really?  That’s 10 mofreakin SINGLE quotes to make something italicized and bold!  HTML?  Much easier.  <i>italics</i>. <b>bold</b> and <b><i>bold+italics</i></b> Ok, maybe not so much easier, but definitely less confusing.  So, when creating documentation that typically lives on a webpage, are wikis actually easier?  I’m inclined to think they’re not.  But editing isn’t the only reason.

I have a philosophy on systems administration documentation, which I’ll get into later, but in essence it overcomes the chicken-and-egg problem.  Basically, if there is documentation for system administration about how to properly set up a wiki, where should this information be stored?  In a wiki?  Most wikis, like Mediawiki, store the information in a database.  Systems administrators need documentation about how to properly set up databases.  We just multiplied our chicken-and-egg problem by 2.

1. Set up a database as a prerequisite to have a wiki
2. Document how to set up a database properly
3. Set up a wiki to hold the documentation
4. Document proper set up of a wiki

Where does the information from #2 and #4 live?

HINT:  “Inside the wiki” is the wrong answer, and if this isn’t obvious to you, you should consider another career field because you are bound to have other more pressing problems than where to store your documentation.

Back to my periodic re-evaluation of tools…

In scouring the Interwebs for new and better tools for documentation, I found this abstract of a talk from a Linux conference in 2008.  It speaks directly to the chicken-and-egg problem I’ve been pondering:

Most professional teams will have some form of team documentation used as a reference by existing team members but also as a means of getting new staff up-to-speed. System administrators are no exception, but have their own set of additional requirements: lots of diverse, complex systems mean lots of procedures to recall. It’s also no good having procedures for repairing a database server stored in a database table.

I recently undertook a project to migrate Newcastle University UNIX team’s internal documentation from a simple filesystem-based scheme to a web-based wiki system using “mediawiki”, the software that powers Wikipedia. Once I had achieved this, it became apparent procedures and tools were necessary to help manage the continued review and updating of the documentation. I also became aware of several issues with the approach that impacted us specifically as systems administrators.

The talk itself was probably much more detailed, but the issue is relatively simple.  You need to be aware of any circular dependencies you’re creating, and try like the dickens to avoid them.  Also, everyone relying on the documentation needs to be well aware of the limitations of the documentation system.  Sure, you can devise a documentation system that doesn’t store its information in a database, but there are other dependencies as well — documentation about how to fix a broken web server might not be best suited inside a wiki, since all wikis rely on web servers to operate.  Granted, you may be able to quickly erect a web server, but this itself creates an unnecessary complexity that needs to be overcome in order to access procedures detailing how to fix a different problem.

Most things in systems administration deal with these types of issues, which I’m going to try not to get into except to say that if you can deal with these types of problems you will probably be a very good systems administrator and not paint yourself into a corner of dependencies.

Back to documentation.  In the abstract of the talk, it mentions a “wiki compiler” as a possible solution to the documentation chicken-and-egg problem.  The concept is that it isn’t strictly a wiki itself, but a method to construct a wiki dynamically based on files on disk.  If the web server explodes, you still have the files.  The files aren’t stored in a database, so that dependency is removed as well.  Also, the files can be under a version control system, so if an author makes a mistake somewhere this can be easily and quickly reverted.  One such “wiki compiler” is ikiwiki, apparently.  The more I read about this concept, the more I like it.  I’m still researching documentation tools, but this is definitely one that I will be exploring more, if only because it seems to make the documentation itself less complex.

My systems administration documentation philosophy is also considerably simple.  Systems documentation basically consists of only 3 types:  online, offline, and hybrid.

• Online systems documentation should only consist of procedures, concepts, ideas, and notes that have zero dependency on web servers, and not depend on any other part of an infrastructure needed to access the documentation itself.  A good example would be procedures about how to create a user account — creating a new user isn’t dependent on the wiki, the web server, or being able to access the documentation within.
• Offline documentation is system details and procedures needed that deal directly with portions of the infrastructure that may be offline or broken; e.g., disaster recovery procedures.  This type of documentation is offline primarily to simplify the review/update process needed to keep the documentation current.  If procedures to restore data from tape are online, of course copies can be printed to use when/if a critical file server needs to be restored.  However, if the data is moved to a different location or the restoration procedure is modified in some way, the documentation needs to be updated.  If this information is kept online, the new information should be added, and then printed again, making sure to destroy the previously printed copies to avoid confusion.  Keeping critical information online quickly becomes a complex procedure in itself because the information needs to be kept offline on paper in the event of an emergency, and updating the information in multiple locations is prone to error and confusion.
• Hybrid systems documentation is a mixture of offline and online.  Hybrid docs are normally created offline, converted or translated into online format, then added somewhere to be available online as well.  They are never edited online, to avoid having multiple conflicting documents that would cause confusion in an emergency.  DocBook is a good example of hybrid systems documentation:  a single source document is created from which other versions can be generated in HTML, PDF, PostScript, plain text, etc.

Anyone that puts an important file with an encrypted password on more than a dozen computers, with permissions of the file being world-readable, doesn’t really understand the ramifications involved.

Since then, I have been trying to “crack” the password through regular means.  Given their usage pattern, I’m fairly certain the password is 8 characters long, at least one capital letter, one number, and one special character.  Sadly, only that much information probably won’t be enough for me to crack it without throwing more compute power at it.  The tough part is that its a salted hash, so I can’t really use Rainbow Tables, and I’ve already tried dictionary attacks with Webster’s Dictionary.  So, I’ve resorted to brute-force automated guessing.  JTR seems pretty good at this, but even so it will probably take months to obtain the plaintext password.  It has been running for 11+ days so far.

The encrypted password in question is: {SSHA}KZhA0wzX4AThn9CkxBgYDmmy42pNY9ME

Salted SHA-1, of course.  If you know encryption algorithms, you already know what this is likely used for, but I won’t give that away.  I won’t tell you what its used for, or who it belongs to, or what you might do with it once you’ve cracked it, suffice to say the plaintext password in the wrong hands could cause some damage.

If you know of a quick way to crack such a password, other than what I’ve tried so far, drop me a line.

Recently, a Slashdot submission in this context was:

“Lately I’ve been rethinking my personal security practices. Should my laptop be stolen, having Firefox ‘fill in’ passwords automatically for me when I go to my bank’s site seems sub-optimal. Keeping passwords for all the varied sites on the computer in a plain-text file seems unwise as well. Keeping them in my brain is a prescription for disaster, as my brain is increasingly leaky. A paper notepad likewise has its disadvantages. I have looked at a number of password managers, password ‘vaults’ and so on. The number of tools out there is a bit overwhelming. Magic Password Generator add-in for Firefox seems competent, but it’s tied to Firefox, and I have other places and applications where I want passwords. And I might be accessing my sites from other computers that don’t have it installed. The ideal tool in my mind should be something that is independent of any application, browser, or computer; something that is easily carried, but which if lost poses no risk of compromise. What does the Slashdot crowd like in password tools?”

The response is filled with witty replies and interesting views and suggestions as per usual.  Nothing really new usually surfaces when someone asks this on Slashdot, since it seems the capacity to have passwords for online banking, social networking, work computers, home computers, blogs and whatever else grows and evolves faster than the ability to keep track of them all efficiently (and securely).

A while back, I wrote about my solution to this problem after having tried to solve it different ways.  In that post, I detailed my evolution from a Java application on a USB keychain to a website called Clipperz.

Well, I have been using Clipperz for almost 2 years now.  It is immensely useful and efficient.  I have had ZERO problems.  Yes, none, nada. NO problems whatsoever.  How many things can you say that about?

Clipperz does seem to be growing in popularity, since the last time I remembered the question asked on Slashdot, hardly anyone recommended Clipperz.  This time, a few people mentioned Clipperz on Slashdot.  However, its been 2 years and Clipperz still has the “beta” status.  Granted, Google Mail was in beta for years until they became “production”, but still…

Not just oh-we’ll-find-a-way-to-fix-or-deal-with-it bad, we’re talking a downward spiral that slopes deeper the further we decline until we have reached “terminal velocity”.  The bottom is not yet in sight.

Why?  Well, I’ll tell ya why…  in a minute.  First, I’ll put things into perspective by shedding a bright historical light on the subject.  This is not to mean that the history of email is dark or bad — but the present state of email certainly is, compared to its early days.

Email (not E-mail, since words that are introduced into the English language are often comprised of multiple words that stand on their own, separated by hyphens, normally lose their hyphens as the new words gain wider acceptance) as we know it today, was originally created in the early 1970′s, purely as an experiment, though in a slightly different form.  To put this into proper context, we’ll go back just a bit further. Email (at this time E-mail, or “electronic mail”) only existed in self-contained systems.  People would log into one specific machine (a time-sharing device, which was basically a big expensive computer that a group of people shared at different times) to perform their work, and would occasionally leave messages for one another to read whenever the next person would log in again.  This concept of “self contained” email would eventually evolve into other implementations of the same use — such as Microsoft Mail, which was designed as a central system, namely in an office building, that people would use to talk to only other people in the same office.  I digress…  but, even in its first use case, Email (and E-mail) was used as a convenience.  Some would say, a luxury tool — to save people from leaving yellow sticky notes somewhere, or picking up a phone to talk to someone that may not have time to talk to you.  In tech-geek-speak, email is asynchronous communication:  I can talk to you as much as I like, and you can reply back to me, but it is pure coincidence if we happen to talk to each other at the same time  (there is a variable delay between one person talking and the other person replying).

From being an easy way to leave messages for other people sharing the computer, it turned into a way of leaving messages for people using other computers — no longer “self contained” email, but networked email.  At this point, email diverged into two uses:  local “self contained” messaging on one computer, and “networked” messaging.  The two remained distinct for quite a while, as there were people sharing central computers that had very little need to communicate with people sharing other computers, yet there were people that had a valid need for such distant communication even if “distant” meant “the computer right next to mine in the same room”.  Still, it was viewed as leaving an electronic sticky note on the screen for whenever that person logged in again.  As such with StickyNotes, eventually the glue on the paper dries and at that point it no longer sticks to anything, falling off the surface to become lost when the cleaning lady vacuums the floor.  This was the expectation for early email — “Joe, I left you a quick message about the widget, if you have concerns just give me a call.”  If the email message was lost, deleted accidentally, or was never delivered, it was no big deal because the communication was eventually going to take place in person anyway, and there was no guarantee the intended person would ever read the message in the first place.

As the novelty of communicating with other people on other computers evolved, so did the implementation of email.  To send an email message to someone outside the shared computer, a person needed to know *which* other computer the recipient used.  The @ was born, since that seemed like the most logical delimeter to distinguish “user” from “computer”, and since neither could contain the @ symbol.  For similar computers, the method was “user@computer”, to properly address an email message.  For different systems, it wasn’t so clear.  In fact, it became downright complicated and confusing.  If a person needed to send a message to a distant computer, but the distant system could not accept “user@computer” (possibly it used the @ for something other than a delimeter), the sender of the message needed to know not only who to address the message to, and *which* computer that user used, the sender also needed to know the path the message would take when it was sent from computer to computer to computer.  UUCP (Unix-to-Unix-Copy) was born.  Imagine instead of smith@accounting it was  !cenvax!westnode!accounting!smith.  Gateways from one type of email system to another type had to be erected, to handle the messages and translate one address into another.  Yet, even then, email was still viewed as “fire and forget” in the sense that whenever the recipient got the message, IF they got the message, they will eventually acknowledge by replying in some fashion as courtesy.

Back in my early days of email, I worked in the military in the computer support office.  Then, email was more a novelty than a necessity.  I vividly remember a sergeant I worked with would get daily phone calls after creating a new email account for someone.  Someone would normally call him up to complain “its been 3 days since you created my email account, and I haven’t got any email yet.  I think its broken.”  He would always reply with the same thing:  “you have to send email to get email”, which basically was his smartypants way of saying “it isn’t broken because you didn’t get anything.  You probably didn’t get any email because no one knows you have an email address, or they have nothing to say to you, or all the people you want to talk to don’t have email themselves.”   He would hang up the phone and we would have a chuckle, then I would joke about how the first person in the world with a fax machine probably wondered why he invested so much money in a device that strangely never prints out any faxes.

Slowly, email became the “killer app”.

[For the uninitiated, a "killer app" is an application (a program or function) that is just so utterly cool and awesome it is NEEDED so much that the purchase of an expensive device is justified, simply to use the application.  The other programs and software are bonus, and not needed as much, compared to THE reason the computer was purchased.]

Everyone seemed fascinated with the ability to talk to ANYONE (as long as they were “on email too”) for FREE.  Its better than long distance calling!!  No more busy signals or answering machines!  And its FREE!!

“Move over word processor, I’m going to communicate with the world!!  Shrink yourself into a microscopic icon, Mr. Spreadsheet, EMAIL is the real reason I have a computer!  Now, if only I knew what to say, and who to talk to.  Maybe someone will figure out how to contact me, so we can send messages back and forth.”

Today, no one really needs to know the path a message takes to reach its intended recipient (in some instances, even the recipient need not be known) because we address email to “user@something.somethingelse.com” and we trust the system to do the Right Thing to deliver the message.  To the right person.  At the right time.  “When it absolutely positively needs to be there…”  within the next 15 seconds else I’m going to wonder what the HELL is taking so long, and why haven’t they replied yet because I just got a message that says they’ve read it and it better not have been marked as spam because it wasn’t spam!!

Email has become the primary method of daily communication.  No longer do you “need to send mail to get mail”.  If your email address is on a web page, business card, or if you have ever used your email address to log into a website, YOU’VE GOT MAIL.   Whether you want it or not.  We email each other about meetings, to talk about email.  We email appointments, contact information, political opinions, love letters, chain-messages, advertisements.  The type of content goes on and on.  The problem is no longer about how we communicate with the right person on the right computer, but how to silence the noise to get to the legitimate messages that we need to read.

In the past, whether it was “self contained” or sent from the other side of the continent, each message was read and discarded soon thereafter.  Lately, email is received and almost immediately archived for “safe-keeping”, sometimes without it even being read.   It seems the focus now is not the immediate meaning of each message, but that a potential need might arise in the future where we might need to re-read the message.  Email used to consist of one file, appended to whenever new messages arrived — older mail was at the top of the file and newer mail was at the bottom/end.  Email now has folders, sorting, searching, tagging, categorizing, filtering, and archiving of all types.  We rarely, if ever, delete email that we’ve read.  Sure, it was really nifty when Google unleashed GMail to the world with its “2GB and growing” size limit on the amount of email one person could have, but if we’re only talking about purely text-based messages it amounts to billions of messages. (By the way, it is no longer only 2GB — its more like 7 or 8GB now.)

Email is no longer just the “killer app” in the sense of being able to communicate with anyone.  It is a presentation moniker; an address with @gmail.com is not as prestigious as it once was, but an address with @yourreallastname.com is.  It is a storage mechanism; people have figured out a way to use free online web email accounts to store documents, MP3s, and photos.  It is a calendar; if you’re using a particular email system that is tied into a shared calendar, you can send/receive appointments, and reminders of upcoming events.  It is a ToDo list; some people have an email folder with messages they have sent to themselves containing the errands they need to perform in the course of a day.  It is a webpage; modern email software will accept HTML in the body of an email message and interpret the language of webpages, even in the sense that images need not be attachments to the email but can be referenced to elsewhere on the Internet.  It is submissible legal evidence; there is legal precedence where email messages are a form of evidence, able to be subpoenaed by a court of law.

How did we get this way?  What changed so radically that “e-mail” could come from an experiment on the ARPANET (a solution looking for a problem), to “email”, a common term of the layman’s vernacular so much that it is no longer a privilege but a rite?   How could a function of computer networking change the way we communicate, yet itself change so little?

How is it that email is no longer a novelty method of asynchronous communication, but is now a basic human necessity in the modern world, measured not in its content of communication, but in cosmetic appeal of its address and in its storage size limit?

I haven’t even got to the bad part yet.

SMTP, or Simple Mail Transfer Protocol, was basically an afterthought in the broad historical map of the creation of the experimental networks that were the grandfathers of the Internet we know today.  SMTP is the most widely accepted and “standardized” method of sending and receiving email.  It was essentially created to bridge the gap between unlike electronic messaging systems, back when “e-mail” was growing in popularity and usefulness.  The unfortunate part of the story, though, is that SMTP was created back when there was no real malicious threat or intent proliferating through the networks.  Users basically trusted other users in the sense that everyone followed the same rules because that was what it meant to “be connected”.  After all, if you behaved badly on the network, people would want to network with you less, until eventually you would be partitioned from everyone else in such a way that you gain a decreasing benefit from being part of the network.  It was a self-governing system, yet relatively unofficial.  “Netiquette” dictated good form and respectable practices toward other network-citizens, which mainly consisted of college students and faculty among connected higher education organizations.  SMTP was very trusting back then, and still is.

To this day, anyone can still send email as anyone else — so easily that specialized software is of little concern.  Simply connecting to a mail server with a bare terminal (Telnet), typing the correct sequence of commands and syntax, and voila!  You just forged an email message.  If you’re lucky, someone will believe they’re talking to whom you pretended to be.

What does all this mean, then?  Put together all what I’ve said so far, and it paints a rather dark and confusing portrait.  Email is *everything*, yet flimsy in it being unreliably verifiable.  Email messages pass from machine to machine across the room, or across the hemisphere, and yet they are “essential communications”.  They are submissible in a court of law, yet easily forged.  Messages are quickly and easily created and more easily deleted, yet we archive them for years or even decades with the possibility that we might need them later even though we already know what each message means, resulting in a liability if they are ever subpoenaed, and requiring constantly increasing storage.

How do we end this accelerating downward spiral, or at least slow it down so we might recognize and begin to approach the problem?

When will added functionality, storage space, and guarantees of quality be enough for this old and simple luxury of slow and insecure communication?  When will we finally realize that we have already outlived email’s usefulness and begin using the next electronic communication “killer app”?

If anyone reading this knows the answer to any of the above, drop me an email. 

[3 Oct 2009 Edit: I JUST found out about Google Wave!!  Go here, here, or here to learn more about it.  It is in closed invitation beta right now, but I hear its going to be released this year.]

Anyway…

I’m actually still in the initial stages, in the overall context of the project.  Basically, to create a Linux distribution (from scratch, as I am doing), it begins similar to a chicken-and-egg problem.  You have to use an existing system to create building blocks, and use the building blocks to create a complete toolset.  Then, you use the toolset in a limited environment, which makes zero assumptions (a “chroot” environment, for geeks out there), to begin construction of the target system.

While doing this is actually pretty straightforward, usually, the difficulty and complexity enters the stage when thoughts about the future begin to crop up.  Issues such as future maintenance and automation complicate matters because while there might be a working toolset to construct a target system, no one wants to have to manually create the target system each time something changes.  Also, if something changes within the toolset itself, the component will need to be rebuilt, or worse, the entire toolset will require rebuilding such that the whole thing remains cohesive.

This is similar to around the time when man first thought about hunting and killing for food.  He realized he could fashion a tool to make a weapon.  Next, he used the tool to create the weapon, which was then used to hunt and kill for food.  As evolution continued, he created more efficient tools, to in turn create better weapons that would enable him to hunt and kill more exotic food.  The same general principle exists when creating a Linux distribution — though, we’re not depending on the need to eat in our software toolchain, we are depending on the ability to make more software available, which will comprise a Linux distro.  The tricky part is to know how to manufacture tools in such a way that when a component in the process breaks, it is trivial to fix it so much to the point that if a tool very deep in the toolset breaks there is less need to re-manufacture everything that was built with it.  Henry Ford had a very good idea, with small replaceable parts.

At this point, I’m at the point that I have the toolchain built.  Before I go much further, I need to figure out a sensible way to maintain this toolchain so that if/when something becomes stale or breaks, I will spend less time repairing the toolchain and more time tending to the actual Linux software in the distribution.  Also, since the toolchain itself was build manually and painstakingly, I also need a system to churn out software using the toolset I have.  Perhaps a framework/system could be used for both maintaining the toolset, and maintaining the distro?

I’m quite sure this is not entirely a brand new problem — major Linux distros would have had to flesh this out very early on.  The trouble is, though, whatever they’re using to handle this type of framework, they’re not very up front in saying exactly what they use, what it does, or where it came from.

I know what I want my system to do.  I have a general idea how it should fit all together.  My dilemma is that I’m splitting my time between searching the Internet for any existing software to satisfy the need, and planning how I might create my own toolchain/distro maintenance framework/system.  There are, actually, existing systems that do something along these lines, but they were designed for very specific purposes, and implemented with specific needs in mind, which resulted in assumptions that require specific platforms.  Not very useful in my case, because I’m creating from scratch with no assumptions.

Eventually, I will have an automated system that will:

• institute modularity wherever practical
• automatically rebuild (with dependent software if necessary) anything that needs updating/fixing
• handle both the toolchain itself, and the target Linux software