I was just looking through my github repositories and I thought it would be cool to post about the simple logging mechanism I created for tweeter.js. Although not the coolest pattern I used in the project (which probably goes to mixins), I like how simple the solution is. Check out the code for other cool features..

Logging

The cool thing about this logging framework is that it allows you to write out logs to different streams. For instance, suppose you want to write all logging information other than errors to log.txt, but you want to keep errors in errors.txt.

The way I implemented this was to create an object of streams on the Log object. I used getters/setters to provide validation that an object being assigned to one of the properties is actually a stream. Although I didn't bother with it, this object could possibly be sealed or frozen.

// An object with safe getters/setters for each stream.
// Streams are:
// debug, info, error, warn
Log.prototype.streams = {
    get none() { return { write: function() { }}; },
    get debug() { return this._debug; },
    set debug(val) {
        requiresStream(val);
        this._debug = val;
    },
    get info() { return this._info; },
    set info(val) {
        requiresStream(val);
        this._info = val;
    },
    get warn() { return this._warn; },
    set warn(val) {
        requiresStream(val);
        this._warn = val;
    },
    get error() { return this._error; },
    set error(val) {
        requiresStream(val);
        this._error = val;
    }
}

Because this is written for node.js, requiresStream() verifies the value is an instance of EventEmitter with a write function.

Defaults for these streams are set in the constructor with a default logLevel of -1 (disabled).

var Log = module.exports = exports = function Log(level) {
    // default error to stderr and the rest to stdout
    this.streams.debug = process.stdout;
    this.streams.info = process.stdout;
    this.streams.warn = process.stdout;
    this.streams.error = process.stderr;

    // default the logLevel to disabled
    this.logLevel = level || -1;
}

An interesting thing about this logger is how I make the calling convention flexible. I use .NET regularly and I would call a logging function in .NET with an enum value to specify the logLevel. I wanted a similar functionality in this logger, so I created a lookup object to map a key (i.e. DEBUG) to a logLevel (0) and a stream (debug).

// [DISABLED] else [DEBUG > INFO > WARN > ERROR]
Log.prototype.logLevels = {
    DISABLED:   { value: -1, stream: 'none' },
    DEBUG:      { value: 0, stream: 'debug' },
    INFO:       { value: 1, stream: 'info' },
    WARN:       { value: 2, stream: 'warn' },
    ERROR:      { value: 3, stream: 'error' }
}

I considered accessing these on the streams object directly, but did it this way for readability and flexibility. A major deciding factor in choosing this route was the ability to reorder logging levels in a single place in the code.

// ## log(level msg);
// logs the output with a nice colorful indicator of the stream.
// `msg` may be a string or a formattable string and options.
Log.prototype.log = function(level, msg) {
    var original = level;
    try {
        if(typeof level === 'string'){
            level = this.logLevels[level.toUpperCase()];
        }
        // get logLevel value
        if( (this.logLevel > -1) && (level.value >= this.logLevel) ) {
            var stream = this.streams[level.stream];
            stream.write( util.format.call(this, '[\033[%sm%s\033[39m]\t', colors[level.value], level.stream) );
            stream.write( util.format.apply(this, slice.call(arguments,1) ) + '\n');
        }
    } catch (err) {
        console.error("logging is improperly configured!\nIs %j a supported logLevel?\nError:\n%j\n",original, err);
    }
}

An example

I've uploaded an example to github.

In this example, I write out logs for each of the 4 supported levels. The default is to write these out to stdout. After writing to the defaults, I set

var Log = require('./log.js'),
    fs = require('fs');

var level = process.argv[2] || 1;
var logger = new Log(level),
    level = logger.logLevels;

logger.log(level.DEBUG, "This is a debug message");
logger.log(level.INFO, "This is an info message");
logger.log(level.WARN, "This is a warning message");
logger.log(level.ERROR, "This is an error message");

var debugStream = fs.createWriteStream('./debug.txt');
var infoStream = fs.createWriteStream('./info.txt');
var warnStream = fs.createWriteStream('./warn.txt');
var errorStream = fs.createWriteStream('./error.txt');

logger.streams.debug = debugStream;
logger.streams.info = infoStream;
logger.streams.warn = warnStream;
logger.streams.error = errorStream;

logger.log(level.DEBUG, "This is a debug message");
logger.log(level.INFO, "This is an info message");
logger.log(level.WARN, "This is a warning message");
logger.log(level.ERROR, "This is an error message");

debugStream.end();
infoStream.end();
warnStream.end();
errorStream.end();

This provides the following input when passing a log level in via the command line:

+1 for teaching me something new today thanks. -kobe

Being one of my favorite answers, I thought I should discuss it on my blog a little more in-depth than just posting the SO answer. I'd like to briefly discuss what ASP.NET really is (in the context of IIS), why AppDomains are needed, and lastly what shadow-copying does for an application. There is no code associated with this post, and it is driven more by contemplation than by a specific resolution to a problem. So, I apologize in advance if it seems disjointed at times.

What is an ASP.NET application?

When I first started using the .NET Framework, I was somewhat confused about how applications actually ran considering an assembly is a PE format with some CLR headers and CLR data. What this means (and it makes a lot of sense) is that the CLR is really just a COM application hosting a runtime environment (the CLR).

I usually consider programming languages like C# to be compiled into a binary, but the CLR really only operates on IL. This means C# et al. are simply abstractions to execute "just-in time"-compiled code in a compartmentalized sandbox within an instance of the CLR. The compartmentalization is further secured using AppDomains to prevent (or attempt to prevent) applications from destroying data in other AppDomains. It makes sense, but I've never really seen it put so simply before. Even though I knew this to be the case and understood how it all worked, my thoughts of the CLR were also compartmentalized. When I stepped back and looked at it, I thought to myself, "Wait, what? That's it?" Why I thought the CLR was some sort of voodoo is beyond me.

An ASP.NET application is .NET code written against a framework (ASP.NET), executing in a runtime (CLR), which is in turn hosted in a (COM) process (IIS). That's all there is to it, there's no magic. It's been a few years since I learned this, but I seem to still encounter developers with the same years of experience or more who don't see how these pieces are integrated. It really only takes a little exploration on your system to get a better picture of the whole environment.

IIS

If you take a look at an IIS assembly in FileAlyzer, you might see CorBindToRuntimeEx or CLRCreateInstance listed as PE Imports. This is because IIS manages the creation of multiple CLR instances and multiple AppDomains per instance. ASP.NET is implemented as an ISAPI extension within IIS.

In fact, %WinDir%\Microsoft.NET\Framework\[Version]\aspnet_regiis.exe imports RegisterAspNetEx from webengine.dll. Running

dumpbin /imports webengine.dll

will output all header information, in which you will see mscoree.dll is delay-loaded and provides the CLR creation methods (from .NET 2.0):

mscoree.dll
              00000001 Characteristics
              6A301BB8 Address of HMODULE
              6A2FE1C4 Import Address Table
              6A2FB5D4 Import Name Table
              00000000 Bound Import Name Table
              00000000 Unload Import Name Table
                     0 time date stamp

          6A2FAC25               0 CorBindToRuntimeEx
          6A2B292E               0 CorBindToRuntimeHost
          6A2B21C6               0 ClrCreateManagedInstance

Here we've found CorBindToRuntimeEx! This library is part of the COM application responsible for running the CLR instances that host ASP.NET AppDomains inside IIS. Cool, huh? That's the magic.

IIS is the hosting environment responsible for running the ASP.NET ISAPI filter. Once ASP.NET is registered with IIS, webengine.dll handles the creation of CLR instances within Application Pools. Inside those Application Pools, one or more AppDomains may exist.

There is a lot that has to go on under the hood in an ASP.NET application, but I'd like to discuss the management of application resources-- specifically shadow copying.

Shadow Copying

Shadow copying is a technique used by ASP.NET to allow resources to remain available continuously without interfering with the execution of code in an AppDomain. Conceptually, it is a very simple idea. Unfortunately, most developers don't even know it is happening under the covers and I've seen production site get bitten by it in some weird configuration and file modification scenarios.

When shadow copying is enabled, ASP.NET copies certain resources (all assemblies) from the application directory into a temporary folder and those are the resources which remain in use during the life of an AppDomain. This cache is the same one used for downloaded assemblies and is cleaned by the CLR when the assemblies are no longer in use (akin to Garbage Collection of runtime resources). By default, the location for these cached assemblies will be something like

C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\demo1\9b9144a7\8665ac07

As you can see, the directories are hashed in what (I assume) is similar to the hashing algorithm used for the GAC. If you like, you can set the AppDomainSetup.CachePath property to shadow copy assemblies to a different location. You must also set the AppDomainSetup.ApplicationName property, otherwise CachePath will be ignored. One downside to this is that you'd have to create a new AppDomain to initialize these values and the management of assembly resolution may become cumbersome. You may also attempt setting the cache path as described here:

protected const string ApplicationAssembliesFolder = "~/Assemblies";

    protected void Application_Start(object sender, EventArgs e)
    {
        var assembliesPath = Server.MapPath(ApplicationAssembliesFolder);

        AppDomain.CurrentDomain.SetShadowCopyPath(
            AppDomain.CurrentDomain.SetupInformation.ShadowCopyDirectories +
            Path.PathSeparator + assembliesPath);

        Assembly.LoadFrom(
            Path.Combine(assembliesPath, "Example.dll"));
    }

When changing the default location of shadow copied assemblies, be aware that ASP.NET no longer handles the clean up of the assemblies-- it is up to the application to maintain these assemblies.

The shadow copying process

The shadow copying process is kicked off any time an AppDomain recycle occurs. Tess Ferrandez (who has a killer blog, you should check it out) lists some reasons why the AppDomain may be recycled in her post, ASP.NET Case Study: Lost session variables and appdomain recycles:

• Machine.Config, Web.Config or Global.asax are modified
• The bin directory or its contents is modified
• The number of re-compilations (aspx, ascx or asax) exceeds the limit specified by the <compilation
  numRecompilesBeforeAppRestart=/> setting in machine.config or web.config (by default this is set to 15)
• The physical path of the virtual directory is modified
• The CAS policy is modified
• The web service is restarted
• (2.0 only) Application Sub-Directories are deleted (see Todd’s blog http://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx for more info)

My rule of thumb is to assume that anytime something happens which could change the execution of an application, the AppDomain will be recycled.

When an AppDomain is recycled, resources are shadow copied to a new location and any connections to the old AppDomain are 'drained', meaning they are allowed to finish on the old AppDomain but subsequent requests are handled by the new AppDomain. This allows the framework to create a fairly seamless experience, but is also part of the reason why many developers don't know about shadow copying.

How shadow copied files are updated

When using .NET 4, shadow copying assemblies in an application for which assemblies rarely ever change has improved. In previous versions of ASP.NET, there was often a noticeable delay in application startup time while assemblies were being shadow copied. Now, the framework checks the file date/time of an application's assemblies and compares that with the file date/time of any shadow copied assemblies. If they are the same, the shadow copying process does not occur. This causes the shadow copying process to kick off only if an assembly has been physically modified.

Now, if these things are happening a lot and you either have many assemblies or your assemblies change often, .NET 4 is actually adding one extra step to the shadow copying process (checking file times). You can disable this extra step by adding the following configuration to your web.config:

<configuration>
   <runtime>
      <shadowCopyVerifyByTimestamp enabled="false" />
   </runtime>
</configuration>

The process itself includes the application location, a temporary location, and the shadow cache location. The process would look something like this for each assembly:

1. Copy assembly from application location to temporary location
2. Open assembly
3. Verify assembly name
4. Validate strong name
5. Compare update to current cached assembly
6. Copy to shadow copy location (if newer)
7. Remove assembly from temporary location

That is a pretty hefty process if your application contains a lot of assemblies. You can surely see why the file time comparison introduced in .NET 4 is beneficial. This is also why applications with many, many resources take longer to "start up". The framework isn't executing the code directly from the application directory.

Disabling shadow copying

Shadow copying is important if you are modifying assemblies directly in a live application. If you are working with a load balancer in which you're able to drain connections from a server and stop IIS for deployment, you may benefit from disabling shadow copying altogether.

<hostingEnvironment shadowCopyBinAssemblies="false" />

Alternative configuration: Wait-Change Notification

One modification I would suggest if you are swapping out assemblies regularly is to set the maxWaitChangeNotification and waitChangeNotification attributes on the httpRuntime element. Both of these attributes take an Int32 value that, when combined, determine when the AppDomain recycling occurs because of a file change.

The waitChangeNotification attribute will modify how long the framework waits between file changes. In other words, if File A finishes copying and waitChangeNotification is set to 2 seconds, the framework will wait 2 seconds for another file modification to occur before spawning a new AppDomain. If File B is modified within that time, it will again wait 2 seconds for another file to be modified.

If maxWaitChangeNotification is also set, the framework no longer chains the wait time. There is an absolute time from when the first file modification starts. For instance, if maxWaitChangeNotification is set to 5 seconds, it doesn't matter if a file is in the process of being modified, a new AppDomain will be spawned based on the File A modification (operation 1). The File B modification will begin the wait even of operation 2, after which another AppDomain will be spawned. I've created the image below as an attempt to clarify what these settings do. These file copy operations represent changes to the application directory. The "AppDomain restart" indicators are the points at which shadow copying begins. Files that are being written are locked and not copied to the temporary location.

Further Reading

• Hosting Overview
• CLR Inside Out: Improving Application Startup Time
• What is an ISAPI extension?
• Inside IIS – Enregistrement d’ASP.Net dans IIS
• CLR Hosting APIs
• Modules, Assemblies, Headers in CLR
• Shadow Copying Assemblies

In CLR via C# by Jeffrey Richter, a much quicker solution is presented (leaving my naive solution to be only dirty). This solution uses a Semaphore to count references to a resource (the application) as it will be managed by the operating system.

Here is a simple console application to display how this would work:

    class Program
    {
        const string Id = "af49d266-e4f4-4a63-b73c-f62c1144b584";
        static void Main(string[] args)
        {
            bool thisInstance;
            using (var semaphore = new Semaphore(0, 1, Id, out thisInstance))
            {
                if (thisInstance)
                {
                    Console.WriteLine("This is the first instance!");
                    Console.ReadLine();
                }
                else
                {
                    Console.WriteLine("There is another instance running.");
                    Console.ReadLine();
                }
            }
        }
    }

A semaphore is designed to limit access to a given resource. If you read the Wikipedia - Semaphore article, you'll see that a semaphore is described as a librarian handing out passes to study rooms. The librarian doesn't care who is in which room, only whether there are rooms available. There could be 10 rooms, there could be 100. In this single application example, we have 1 available resource which is the application identified by a constant Guid - "af49d266-e4f4-4a63-b73c-f62c1144b584". Because the operating system will use this string to uniquely identify our application, I've chosen a Guid over something simple like "MySemaphore" because the chances of a Guid being duplicated are theoretically nil.

Using this Semaphore constructor:

public Semaphore(
	int initialCount,
	int maximumCount,
	string name,
	out bool createdNew
)

we can create a binary semaphore.

A binary semaphore allows us to have an on/off switch for running our application -- is the resource available or not? When we have 1 resource available, and the first application starts up, it decrements the count of available resources to 0. If this decrement occurs, the createdNew out parameter will be true. On the other hand, if an application starts up and the semaphore's available resource count is 0, createdNew will be false.

Notice in the code above that Console.ReadLine() is used in each block of the if statement to block execution. Why is this not located at the end of the Main method?

It's because Semaphore is a type derived from WaitHandle. WaitHandle implements the IDisposable interface, which is why the Semaphore is wrapped in a using block. When the code exits this using block, semaphore.Dispose() executes semaphore.Close() and the currently-held resource is released. This increments the available resources back to the maximum value of 1. In a real application, the first block would call the Application's Run() method and the else block would either display a message, terminate silently, or switch to the first process.

It is also possible to have any subsequent processes wait for the semaphore to release its lock on the resource. This would be accomplished by calling semaphore.WaitOne() in the else block:

    class Program
    {
        const string Id = "af49d266-e4f4-4a63-b73c-f62c1144b584";
        static void Main(string[] args)
        {
            bool thisInstance;
            using (var semaphore = new Semaphore(0, 1, Id, out thisInstance))
            {
                if (thisInstance)
                {
                    Console.WriteLine("This is the first instance!");
                    Console.ReadLine();

                    // Release resource
                    semaphore.Release();
                }
                else
                {
                    Console.WriteLine("There is another instance running.");

                    // Wait for resource
                    semaphore.WaitOne();
                    Console.WriteLine("There is now the only instance.");
                    Console.ReadLine();
                }
            }
        }
    }

Wait a second. Didn't I say that wrapping the Semaphore object in a using block would call Close()? Why did I have to explicitly call semaphore.Release()?

The answer to this is Garbage Collection. Sure, Close() will be called (and thus, Release() is called and signals all waiting threads that the resource is available) when the using block exits, but this will only happen when the object is disposed. If you plan to queue up applications (or windows, dialogs, or any other resource), note this signaling behavior.

Code

I've decided to start making code from my blog posts available from a single github repository.

https://github.com/jimschubert/blogs

References

• The General Semaphore, original
• Semaphore Objects

c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\string.h(105) :
see declaration of 'strcpy' SomeFile.cpp(###):
warning C4996: 'strcpy': This function or variable may be unsafe.
Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS.
See online help for details.

I looked for the online help, which doesn't really exist for this specific warning. I did find the Security Features in the CRT article, but that was useless. Because I'm working on a shared project, I didn't want to manually add this #define to every project. The workaround is to add it to the *.user.props file.

Add the ItemDefinitionGroup node below to C:\Users\Jschubert\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.Win32.user.props:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0"
         xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemDefinitionGroup>
    <ClCompile>
      <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <ResourceCompile>
      <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ResourceCompile>
  </ItemDefinitionGroup>
</Project>

And there you have thousands of annoying warnings disappear.

Because MSBuild always includes %(PreprocessorDefinitions) when chaining PreprocessorDefinitions, you can add any number of defines here to exist globally on your machine only.

To get wireshark to show up in the 'Open With...' dialog's choice of applications, I followed the instructions on AskUbuntu.com. The instructions are for vim, so I've modified them for wireshark.

Save the following as ~/.local/share/applications/wireshark.desktop:

[Desktop Entry]
Encoding=UTF-8
Name=Wireshark
Comment=Wireshark packet capturing
Exec=wireshark %u
Terminal=false
Type=Application
Icon=/usr/share/pixmaps/hi48-app-wireshark.png
Categories=Application;Utility;
StartupNotify=true
MimeType=application/octet-stream
NoDisplay=false

For more information about .desktop files, check out the Desktop File Specification.

MSDN defines the null-coalescing operator as

The ?? operator is called the null-coalescing operator and is used to define a default value for nullable value types or reference types. It returns the left-hand operand if the operand is not null; otherwise it returns the right operand.

Although this is somewhat similar to a conditional operator (sometimes called 'ternary operator'), it is much more concise.

The null-coalescing operator works with nullable value types and reference types. When I first started with the null-coalescing operator, I often underestimated the usage with empty strings. I think this is very important to discuss, hence this blog post.

Strings

For example:

// conditional operator
string example = null;
string first = example == null ? "default" : example;

// null-coalescing operator
string second = example ?? "default";

In both of these cases, the resulting string will be "default".

The problem with strings is that the null-coalescing operator only checks against nulls. Code may often return String.Empty instead of nulls. Meaning the following example is not the same.

// conditional operator
string example = string.Empty;
string first = string.IsNullOrEmpty(example) ? "default" : example;

// null-coalescing operator
string second = example ?? "default";

In this example, 'first' will be "default" while 'second' will be the empty string. This is a situation in which the null-coalescing operator is useful only when a string must not be null but string.Empty is still perfectly valid.

Nullable Value Types

When working with nullable types, the conditional and null-coalescing operators have another alternative when retrieving the underlying value: GetValueOrDefault(). The problem here is that the method returns the value type, not another nullable type.

For example:

int? example = null;

// Conditional operator
int? first = example == null ? 100 : example;
int firstValue = first.Value;

// null-coalescing operator
int? second = example ?? 100;
int secondValue = second.Value;

// GetValueOrDefault()
int third = example.GetValueOrDefault(); // third = 0
int fourth = example.GetValueOrDefault(100); // fourth = 100

Notice the catch in the GetValueOrDefault() example... you don't have to reassign the nullable and then get the value. This is useful when you only need the value type. Sometimes, you need a default value and the nullable type. In these cases, I still prefer the null-coalescing operator.

Reference types

The null-coalescing operator can be used with reference types to guarantee that properties or methods do not return nulls.

For example:

public List<Person> People
{
    get
    {
        return _people ?? new List<Person>();
    }
    set { /* something */ }
}

This is definitely a design/coding preference. I've experienced environments where developers insist on the importance of the NullReferenceException being thrown in production code. I've also experience plenty of environments where exceptional code is properly handled and this construct would rarely, if ever, be useful.

Maybe it is not a well-known operator, or most developers would like code to be lengthier and explicit. In a web development team, I think the null-coalescing operator is very similar to the common JavaScript example of providing a default value using the || operator:

var process = function(val) {
    val = val || 100;
    /* etc. */
}

Originally submitted at O'Reilly

Best Practices for Creating Universally Usable Content

Accessible EPUB 3

Good content about a new standard

Pros: Accurate, Concise

Cons: "however"

Best Uses: Novice, Student

Describe Yourself: Software Engineer

I reviewed Accessible EPUB 3 as part of the O'Reilly Blogger program. This book is actually an excerpt from the upcoming EPUB 3 Best Practices.

The content of this excerpt is undeniably rich. The first 25 pages or so may be 'review' material if you're already pretty familiar with HTML5. If so, skim these pages for the EPUB-specific additions.

Knowing very little about the changes in EPUB 3, I was very intrigued by the audio and video capabilities that mirror those of HTML5. I would have liked to see more discussion about the support of these elements in readers on the market. There were a few places where the author warned about new features not being fully available in many readers, but the lack of statistics made me wonder if this is merely speculation or if any readers currently support the new features.

I thought the Text-to-Speech section was very informative. The concept of having a lexicon as a pronunciation index coupled with inline SSML will make EPUB 3 very usable on readers. I sometimes turn on a screen reader just to see how elements are read aloud, laughing at mispronounced words or phrases. If authors and publishers put in the effort to add these features of EPUB 3, text-to-speech engines will make only a fraction of the pronunciation mistakes.

I would have given this excerpt five stars, but the language was occasionally difficult to read. For instance, the author really enjoys ending sentences with "however" as if it is a filler word. After encountering this a few times, I ended up reading over sentences (sometimes spanning 2-3 lines of a paragraph) and replacing the ending "however" with synonyms to see if it made sense. Mostly, it didn't. I think this really detracted from the overall content, because I found myself retracing so much text to understand how the disjointed conjunctive adverb made sense that the content ended up feeling choppy.

Since the end result is still in production, I would imagine this text has already gone through a few rounds of editing.

(legalese)

The fixes, in short:

$ git fsck
$ git log --raw --all --full-history | grep SHA1-HERE
$ git hash-object -w OBJECT-PATH-HERE
$ git push

Here is the error and a walk-through of coming up with the fixes above:

jim at schubert in /media/16GB/projects/school on master
$ git push
Password:
Counting objects: 1945, done.
error: unable to find 2978ec4d75abb8c6bab225d8adfbd2bef064338a
error: unable to unpack bddbd13afd698e5ba7d572c9270e52bcac862661 header
error: inflateEnd: failed
Delta compression using up to 2 threads.
Compressing objects: 100% (1854/1854), done.
fatal: unable to read 2978ec4d75abb8c6bab225d8adfbd2bef064338a
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
fatal: write error: Bad file descriptor

After running git fsck, I found that I had two missing blobs:

jim at schubert in /media/16GB/projects/school on master*
$ git fsck
dangling tree dbe9172996edbb7df517b0305c38891d78b72f66
dangling tree fbf7d8336b5f2347da23eb8a3938de5ab18f783c
missing blob 2978ec4d75abb8c6bab225d8adfbd2bef064338a
missing blob bddbd13afd698e5ba7d572c9270e52bcac862661

To fix this, I had to get the filenames of these blobs and write them back into the repository:

jim at schubert in /media/16GB/projects/school on master*
$ git log --raw --all --full-history | grep bddbd13
:000000 100644 0000000... bddbd13... A	INFO 465/Project2/UseCase/Diagrams/Leader - Time & Mileage.vsd

jim at schubert in /media/16GB/projects/school on master*
$ git log --raw --all --full-history | grep 2978ec4
:000000 100644 0000000... 2978ec4... A	INFO 465/Project2/Prototype/WebPrototype/WebPrototype/bin/WebPrototype.dll

Writing these files back into the repository, the push was successful. To write these back, do the following:

jim at schubert in /media/16GB/projects/school on master
$ git hash-object -w INFO\ 465/Project2/UseCase/Diagrams/Leader\ -\ Time\ \&\ Mileage.vsd
bddbd13afd698e5ba7d572c9270e52bcac862661
jim at schubert in /media/16GB/projects/school on master
$ git hash-object -w INFO\ 465/Project2/Prototype/WebPrototype/WebPrototype/bin/WebPrototype.dll
2978ec4d75abb8c6bab225d8adfbd2bef064338a

Originally submitted at O'Reilly

Making Easy Things Easy and Hard Things Possible

Learning Perl, 5th Edition

Great intro to Perl

Pros: Helpful examples, Easy to understand, Concise, Well-written, Accurate

Best Uses: Novice, Student

Describe Yourself: Software Engineer

I picked up this book on a whim to learn Perl. I was extremely impressed, especially by the three chapters on Regular Expressions.

If you have used Perl, this book may be fairly basic. There are whole chapters on Scalars, Arrays, Hashes, etc. I have a lot of experience in a lot of languages and technologies, and I can tell you this book is well-written and well-organized to ease you into the Perl mindset.

First of all, most languages are like mathematics. 1+1=2 is a rule. If that rule didn't work, math wouldn't work. Similarly, 'int x = 1;' in C-style languages will always point the variable 'x' to the memory location storing the value of '1'. Rules are rules. However, Perl is contextual. When you do one thing in Perl, you may have a different result depending on the context of the code. For example (extremely-generalized example), hashes can be considered an array of values in which every even index is the key and every odd index is the value. So, an array of ["first", "1st", "second", "2nd" ] could easily become a hash with the keys and values related as expected. Weird? Yes. Basic, not really. I think that can be a very difficult concept for a lot of people... too meta.

The book does a great job of easing a lot of the Perl concepts onto the reader. Interestingly, the part of the book I found the most useful are the three Regular Expressions chapters. I have recommended this book to students in the past solely for the three Regular Expressions chapters. They are clearly and concisely written in a way that presents the Perl syntax very well.

If you're familiar with Perl already, this book probably isn't for you. You would probably be better off with the Intermediate Perl book also available by O'Reilly. Even after reading both books and working a bit with the Catalyst framework, I found myself coming back to the last chapter of this book to fully understand eval blocks.

(legalese)

Originally submitted at O'Reilly

CLR via C#, 3rd Edition

CLR via C#, 3rd Edition

One of the best C# books available.

Pros: Helpful examples, Accurate, Concise, Easy to understand, Well-written

Best Uses: Expert, Intermediate, Student

Describe Yourself: Software Engineer

It has been a while since I read this book. I never reviewed it because I didn't realize Microsoft Press was sold on the O'Reilly site.

I can not tell you how many times I have been asked an interview question which I answered quickly and concisely thanks to this book. It covers a lot of material. When I say "a lot", I don't mean the average amount of material in a book this length, I mean the average amount of material in a book that is three times the size of this book. Seriously, it goes over some topics that other books don't touch on: like why strings are interned the way they are, string comparison optimizations, globalization gotchas, assembly structure, and a lot more.

I recommended this book to a friend a few days ago. I would gladly recommend it to anyone who is a professional .NET developer or a student using .NET technologies. It thoroughly discusses the how, why, and when the CLR performs the way it does.

A note for beginners: I think this book is geared more toward intermediate-to-advanced developers. If you find some of the content too difficult, skim it and come back to it later. You won't be let down.

(legalese)

Originally submitted at O'Reilly

Writing Server-Side JavaScript Applications

Node for Front-End Developers

Or... Prototyping with Node.js

Pros: Well-written, Easy to understand, Concise, Accurate, Helpful examples

Cons: Too short

Best Uses: Student, Novice

Describe Yourself: Software Engineer

I've reviewed "Node for Front-End Developers" for the O'Reilly blogger program. Because I've been contributing to an open-source Node.js ebook, I was very excited to read this book. I must warn you-- this book is less about Node.js than it is about prototyping web applications using modules available in Node.js. If you begin reading this from that perspective and you have little knowledge of Node.js in general, you will gain a lot from reading this. If, however, you have experience with Node.js and have used frameworks such as Express and Connect, there is not much more covered in these 58 pages.

Garann Means has written this book in a great way. She doesn't start off explaining the Express framework like most introductions to Node.js would have done. Instead, she shows how to create a minimal http server, add templating and middleware, and then discusses Express and separating concerns into an MVC or MVC-like pattern.

There are a few times where I think the terminology could be misleading. For example, at the beginning she talks about 'scaffolding'. Scaffolding in general implies code generation and the only bit of true scaffolding discussed in the book occurs on one line in one of the last few chapters when the Express framework generates an empty application. I would have liked to see some true 'scaffolding' examples, either with code specific to the book or with third-party modules. Aside from one or two other terminology issues (which is probably just me being picky), the content is spot-on.

Garann does talk about asynchronous operations a little, but this is a huge area of concern for front-end and back-end developers when migrating to Node.js because the asynchronous and event-driven nature of common tasks (files, database) can often increase the complexity of a simple task. Had she expanded on the asynchronous nature of Node.js more, I would have given this book five stars. She does offer a caveat in the Postscript that this book doesn't cover all there is to know about Node.js, and she's right. Before or after reading this, you should read the introduction in the Node.js documentation and the Express framework's documentation.

In conclusion, this is an excellent introduction to prototyping web applications using Node.js. I would recommend this book to people who are familiar with JavaScript programming and want to being prototyping without the need to learn additional languages and frameworks like Ruby on Rails, Catalyst, ASP.NET MVC, etc.

(legalese)

Originally submitted at O'Reilly

McCullough and Berglund on Mastering Advanced Git

McCullough and Berglund on Mastering Advanced Git

Great demonstration of advanced Git

Pros: Accurate, Easy to understand, Concise, Helpful examples

Best Uses: Novice, Student, Expert, Intermediate

Describe Yourself: Sys Admin, Maker, Developer, Educator, Designer

At just under 4 hours long, this video course packs in a lot of content. Each video is about 30 minutes long, so I didn't have to watch it all in one sitting.

The structure of the demonstration is extremely useful because Tim and Matthew ask and answer questions conversationally while showing a split-view Git workspace. I think the hardest thing for some people to grasp in general about git is what 'distributed' means. Although the presenters are working from the same machine, the setup is done in a way to demonstrate this distributed nature.

I've been using git for a while, but I have learned a lot from these videos. I've only somewhat dug into the files inside the .git directory on my own because I was afraid to 'ruin' my repository. This course does a great job of alleviating a lot of those concerns by explaining the organizational structure of the .git contents.

If you're like me, you'll probably add quite a few of the concepts you learn from these videos to your repositories immediately after each video.

(legalese)

var Uuid = require('./uuid.node').Uuid;
var uuid = new Uuid();
var myId = uuid.generate();

The project files referenced in the following text can be downloaded from the repo: jimschubert/masteringnode

FunctionTemplate

In v8, a FunctionTemplate is used to create the equivalent to:

var template = function() { }

The function at this point is an object and not an instance of the function.

As an example, we will use the linux package uuid to generate a uuid. We will define the header for this addon as:

// addons/uuid/v0.1/uuid.h
#ifndef __node_uuid_h__
#define __node_uuid_h__

#include <string>
#include <v8.h>
#include <node.h>
#include "uuid/uuid.h"

using namespace v8;
using namespace node;

class Uuid : public node::ObjectWrap {
    public:
        Uuid() { }
        static Persistent<FunctionTemplate> constructor;
        static void Init(Handle<Object> target);
        static Handle<Value> New(const Arguments &args);
        static Handle<Value> Generate(const Arguments &args);
        static Handle<Value> GenerateRandom(const Arguments &args);
        static Handle<Value> GenerateTime(const Arguments &args);
    private:
        ~Uuid();
        static std::string GetString(uuid_t id);
};

#endif // __node_uuid_h__

This addon will showcase three methods, Generate, GenerateRandom, and GenerateTime. It will also include a trivial private GetString method to demonstrate how to Unwrap a node::ObjectWrap object and interact with C++ code that is not specific to node or v8.

A lot of the public function definitions should look similar to the Echo example. One notable difference is that instead of using a macro and hiding the FunctionTemplate method, we are defining static Persistent constructor;. The Persistent type is used "when you need to keep a reference to an object for more than one function call, or when handle lifetimes do not correspond to C++ scopes." source. Since we'd expect our object's constructor to last longer than a single function, we declare it separately and as a persistent handle. Another point to notice is that all of the method we're pulling from uuid.h have the signature static Handle Method(const Arguments &args) even though we will plan to call it as uuid.generate(). This is because we will be accessing the scope of the call via args.This().

Although more methods are implemented in uuid.cc, we will look at three:

Uuid::Init(Handle<Object> target)
Handle<Value> Uuid::New(const Arguments &args)
Handle<Value> Uuid::Generate(const Arguments &args)

Just as before, node expects to find a signature of extern "C" void init(Handle<Object> target) in order to initialize the addon. Inside this method, we may set parameters such as the version number from the previous example. We may also pass-through initialization to any modules within our node addon. In this example, our addon will be uuid.node and contain a single module, Uuid. There is no reason we can't later add Uuid2 which, instead of returning a normalized string value might return a Buffer object. To initialize the Uuid module, we pass the target object along to Uuid::Init and add the module definition to target:

// addons/uuid/v0.1/uuid.cc
void Uuid::Init(Handle<Object> target) {
    HandleScope scope;

    // Setup the constructor
    constructor = Persistent<FunctionTemplate>::New(FunctionTemplate::New(Uuid::New));
    constructor->InstanceTemplate()->SetInternalFieldCount(1); // for constructors
    constructor->SetClassName(String::NewSymbol("Uuid"));

    // Setup the prototype
    NODE_SET_PROTOTYPE_METHOD(constructor, "generate", Generate);
    NODE_SET_PROTOTYPE_METHOD(constructor, "generateRandom", GenerateRandom);
    NODE_SET_PROTOTYPE_METHOD(constructor, "generateTime", GenerateTime);

    target->Set(String::NewSymbol("Uuid"), constructor->GetFunction());
}

In this scope, we are instantiating the constructor using Uuid::New as a new FunctionTemplate. We then call InstanceTemplate() and on that object we call SetInternalFieldCount(1). This tells v8 that this object holds a reference to one object.

Next, we setup the prototype using another macro provided by node. These calls say, for instance, "Add a method called generate to the constructor function which executes the native method Generate".

Lastly, we have to create a "Uuid" module on the object returned by the call to require(). Here, Uuid will point to a function (constructor) which returns a function that internally executes Uuid::New. In other words, we have created something akin to:

var Uuid = function() { };
Uuid.constructor = function() {
    return function() {
        // Uuid::New executes here.
    }
}

Although the above is not exactly what we have done, it may provide a better view for some to understand FunctionTemplate references and why we assign one to the constructor object in such a way.

The Uuid::New method is defined as:

// addons/uuid/v0.1/uuid.cc
Handle<Value> Uuid::New(const Arguments &amp;args) {
    HandleScope scope;
    // no args are checked
    Uuid *uuid = new Uuid();
    uuid->Wrap(args.This());
    return args.This();
}

As you would expect, calling the constructor function multiple times will create newly-scoped Uuid objects on the heap. In this method, we wrap the parameter (scoped object) by setting a reference to Uuid in the args as a contextual scope (i.e. this) and then returns this.

Within the Generate method, we will want to unwrap the contextual Uuid object and call the private method GetString.

// addons/uuid/v0.1/uuid.cc
Handle<Value> Uuid::Generate(const Arguments &amp;args) {
    HandleScope scope;
    uuid_t id;
    uuid_generate(id);

    Uuid *uuid = ObjectWrap::Unwrap<Uuid>(args.This());
    return scope.Close(String::New(uuid->GetString(id).c_str()));
}

As with any JavaScript function call, we have to ensure functional scope. Scoped methods should create a HandleScope object at the start and call scope.Close() at the end. HandleScope will get rid of temporary handles when the scope is closed.

Within each of the Generate* methods, we will create a uuid_t type and call the corresponding method defined in /usr/includes/uuid/uuid.h (location may vary per system). To demonstrate accessing the pointer to our original Uuid object, we unwrap the contextual scope of this function using ObjectWrap::Unwrap(args.This()). From this pointer, we can access any private methods such as GetString. Be careful with your returned values, though. String::New in the v8 library does not take std::string in any of its signatures. Simply enough, std::string provides a c_str() method to return a const char* which String::New does accept.

uuid.node demo

Navigate to addons/uuid/v0.1/ and execute:

$ node-waf configure build

If there are build errors and the func.cc example from before built successfully, check that you have the uuid-dev package installed and rerun the above command. Then, navigate to build/default and try out the Uuid addon:

$ node
> var Uuid = require('./uuid.node').Uuid;
> var uuid = new Uuid();
> uuid.generate();
'83475e0c-212b-402c-bdc7-b81ebb7b34f8'
> uuid.generateRandom();
'4d597bda-8f5f-4c3c-b2fa-1cd6cd4a6903'
> uuid.generateTime();
'25a0dd30-5076-11e1-96be-0022fb93b24c'
> var util = require('util');
> util.inspect(uuid);
'{}'
> util.inspect(Uuid);
'[Function: Uuid]'
>

The above output may surprise you. Firstly, where is the version option?! It's at the required module level: require('./uuid.node').version;. Secondly, if we can access uuid.generate() and others, why don't they display when inspecting the object? That's because we defined those methods on the prototype:

> uuid.__proto__
{ generate: [Function: generate],
  generateRandom: [Function: generateRandom],
  generateTime: [Function: generateTime] }
>

Thirdly, you may have noticed that I didn't say anything about constructor->SetClassName(String::NewSymbol("Uuid")); in Uuid::Init. You may have also wondered where SetClassName actually sets a class name, considering JavaScript is a prototypal language. That string value is what is displayed when you call inspect and get the value '[Function: Uuid]'. Just as you would expect, Uuid is the constructor and it is named Uuid.

Now, if you've played around with this a bit, you may have noticed that uuid.__proto__ gives us our three functions but uuid.prototype is empty. Why is that? That's because uuid.__proto__ really is uuid.constructor.prototype, which is also really Uuid.prototype. This is the essence of prototypal inheritance. If this concept is foreign or difficult to grasp, be sure to check out the excellent explanation on JavaScript Garden.

Logically, the next step would be to understand how to declare a prototype of our own.

Creating a script may be considered cheating if you're in it for money. If you're in it for fun, script away. That's what I always say.

Here's the gist of it:

#!/usr/bin/env perl
# words.pl: Find all possible slogan words from a single sentence.
use strict; $|++;

@ARGV == 2 or die "usage: $0 input_file output_file 'sentence'\n";
my ($infile, $outfile, $sentence) = @ARGV;
$sentence = $sentence || 'how much wood could a woodchuck chuck';

open INPUT, "< $infile" or die $!;
open OUTPUT, "> $outfile" or die $!;

my $stdout = select STDOUT;
$| = 1;
select $stdout;

my %sentence_letters;
my $stmp = $sentence;
$sentence_letters{$&}++ while($stmp =~ s/[a-z]//);

print "Using the sentence '$sentence'\n";
print "Found the following letters:\n";
print "\t$_ - ". $sentence_letters{$_} ."\n" foreach(sort(keys %sentence_letters));
print "Processing $infile for slogan words\n";

my $count = 0;
my @indicators = qw{\ / | .};
LINE: while(<INPUT>) {
    my $word = $_;
    my $tmp = $word;
    next LINE if($word =~ /['\&\d]/);
    my %word_letters;
    $word_letters{$&}++ while($tmp =~ s/[a-z]//);

    foreach(keys %word_letters) {
        next LINE if ($word_letters{$_} > $sentence_letters{$_});
    }
    print OUTPUT $word;

    my $word_len = length($word);
    open WORD_LEN_OUTPUT, ">> $outfile.$word_len";
    print WORD_LEN_OUTPUT $word;

    print $indicators[++$count % 4], "\r";
}

print "\nDone.\nView $outfile.* for words\n";

When I wrote this, I had only recently started using Perl. Please go easy on me if it's poorly written.

The script takes an input file, an output file format (e.g. words.txt will be words.txt.20 for words of 20 characters), and an optional sentence to parse.

It gets a set of letters in the sentence, then runs through the list of words to see if the word can be made from any combination of letters.

For instance, if your 'sentence' is "baby cakes", the script will create a hash of those letters and their counts. Conceptually, this looks like:

// hash is an array
hash['a'] = 2
hash['b'] = 2
hash['c'] = 1
hash['e'] = 1
hash['k'] = 1
hash['s'] = 1
hash['y'] = 1

If, while walking line-by-line through your list of words, the script sees 'abracadabra', the loop will return false because (conceptually):

word['a'] = 5
word['a'] <= hash['a'] == false

The script also employs some interesting stdout manipulation. This allows the script to output "spinner text" and update the current line when the terminating character is a line-feed.

To run the script in a linux-based environment, you may do:

mkdir ~/projects && cd ~/projects
git clone git://gist.github.com/1733871.git gist-1733871
cd gist-gist-1733871
perl words.pl /usr/share/dict/words generated.txt 'Good goly, Miss Molly'

You should see output similar to:

jim at schubert in ~/projects/gist-1733871 on master*
$ tree .
.
├── generated.txt
├── generated.txt.1
├── generated.txt.2
├── generated.txt.3
├── generated.txt.4
├── generated.txt.5
├── generated.txt.6
├── generated.txt.7
├── generated.txt.8
└── words.pl

0 directories, 10 files

If you look at generated.txt.7, you will probably see something similar to:

Hollis
Osgood
glossy
goodly
idylls
igloos
solids

Originally submitted at O'Reilly

Android Open Conference 2011: Complete Video Compilation

Android Open Conference 2011: Complete Video Compilation

Hours of Android enjoyment

Pros: Helpful examples, Easy to understand, Accurate, Concise

Best Uses: Intermediate, Student, Expert

Describe Yourself: Developer

There are a lot of videos here. If you're only planning to write apps for the market, you'll most likely not be interested in the Embedded Android talk or Scala as a Java replacement. If you don't work on iOS and never plan to, you can probably scan through Nick Farina's talk (sorry, Nick). I found many of the videos worthwhile, whether they related to actual app development or the business of apps.

I also really enjoyed the keynotes. Tim O'Reilly's talk is probably my favorite because I've been saying the same things about many of Tim's comparisons since I bought my first DROID a couple of years ago.

Even if you're familiar with Android development, you should definitely watch the videos by Ken Jones. He shows the start of a Twitter client. Although he didn't get very far in the actual coding, he will point you to the finished code. One aspect I loved about his talks is that he discusses a lot of best practices. Many tutorials you find online either don't discuss best practices or don't incorporate them.

Be sure to watch the Arduino videos. I hadn't heard about Arduino before this video and I was very impressed. In fact, I watched the keynote video twice.

Honestly, there are some videos here that I'm not that interested in. That doesn't detract from the overall usefulness of this video collection. If you're 25% of the way through a video and it hasn't hooked you yet, move onto the next because there is a plethora of information.

(legalese)