beefycode.com

ConvertFrom-PDF PowerShell Cmdlet

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Wed, 08 Jul 2009 06:14:00 -1300

I hate PDFs.

And now I need to search through several hundred of them, ranging from 30 to 300 pages in length, for cross-references and personnel names which ... um ... well, let's just say they no longer apply. Sure reader has the search feature built-in, so does explorer, but that's so 1980's. And I sure don't want to do each one manually...

I poked around the 'net for a few minutes to find a way to read PDFs in powershell, but no donut. So I rolled my own cmdlet around the iTextSharp library and Zollor's PDF to Text converter project.

There isn't much to the cmdlet code, given that all of the hard work of extracting the PDF text is done in the PDFParser class of the converter project:

using System;
using System.IO;
using System.Management.Automation;
namespace PowerShell.PDF
{
    [Cmdlet( VerbsData.ConvertFrom, "PDF" )]
    public class ConvertFromPDF : Cmdlet
    {
        [Parameter( ValueFromPipeline = true, Mandatory = true )]
        public string PDFFile { get; set; }
        
        protected override void ProcessRecord()
        {
            var parser = new PDFParser();
            using( Stream s = new MemoryStream() )
            {
                if( ! parser.ExtractText(File.OpenRead(PDFFile), s) )
                {
                    WriteError( 
                        new ErrorRecord(
                            new ApplicationException(),
                            "failed to extract text from pdf",
                            ErrorCategory.ReadError,
                            PDFFile
                        )    
                    );
                    return;
                }
                s.Position = 0;
                using( StreamReader reader = new StreamReader( s ) )
                {
                    WriteObject( reader.ReadToEnd() );
                }
            }
        }
    }
}

The code accepts a file path as input; it runs the conversion on the PDF data and writes the text content of the file to the pipeline. Not pretty, but done.

Usage

Here is the simple case of transforming a single file:

> convertfrom-pdf -pdf my.pdf

> my.pdf | convertfrom-pdf

More complex processing can be accomplished using PowerShell's built-in features; e.g., to convert an entire directory of PDFs to text files:

> dir *.pdf | %{ $_ | convertfrom-pdf | out-file "$_.txt" }

More relevant to my current situation would be something along these lines:

> dir *.pdf | ?{ ( $_ | convertfrom-pdf ) -match "ex-employee name" }

Download the source: PowerShell.PDF.zip (1.10 mb)

Enjoy!

Creating a PowerShell Provider pt 3: Getting Items

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 07 Jul 2009 04:51:00 -1300

Now that the drive object of the ASP.NET Membership PowerShell provider is fully functional, it's time to extend the PowerShell provider to fetch users from the membership store.

The PowerShell cmdlet used to retrieve one or more items from a provider is get-item, or gi for you alias monkeys out there. get-item works against any provider that support item retrieval; e.g., it works for files and folders:

> get-item c:\temp
    Directory: Microsoft.PowerShell.Core\FileSystem::C:\
Mode           LastWriteTime       Length Name
----           -------------       ------ ----
d----     3/12/2009  2:23 PM        <DIR> temp

It works for environment variables:

> get-item env:PROGRAMFILES
Name                           Value
----                           -----
ProgramFiles                   C:\Program Files

It works for certificates:

> get-item cert:/CurrentUser/My/E0ADE6F1340FDF59B63452D067B91FFFA09A621F
    Directory: Microsoft.PowerShell.Security\Certificate::CurrentUser\My
Thumbprint                                Subject
----------                                -------
E0ADE6F1340FDF59B63452D067B91FFFA09A621F  E=jimbo@null.com, CN=127.0.0.1, OU=sec, O=ptek, L=here, S=nc, C=US

So, how does PowerShell know which provider to invoke when it processes a get-item cmdlet?

PowerShell Paths

The item's full path contains all of information necessary to locate the appropriate provider. In the examples above, the drive portion of the path indicates which provider should be used to process the item request; you can see the mapping of drives to providers using the get-psdrive cmdlet:

> get-psdrive
Name       Provider      Root
----       --------      ----       
Alias      Alias
C          FileSystem    C:\   
cert       Certificate   \
Env        Environment
Function   Function
Gac        AssemblyCache Gac
HKCU       Registry      HKEY_CURRENT_USER
HKLM       Registry      HKEY_LOCAL_MACHINE
Variable   Variable

Each PowerShell drive is directly associated with a provider type: the C: drive maps to the FileSystem provider, the cert: drive to the Certificate provider, and the env: drive to the Environment provider.

PowerShell recognizes several forms of path syntax, all of them designed to allow for provider discovery; providers are expected to support these path formats as appropriate:

Drive-qualified: this is equivalent to a fully-qualified file path. The drive is explicitly specified at the start of the path, as in "c:/temp/junk.txt" or "cert:/localmachine".
Provider-direct: starts with "\\" (or "//"); the provider for the current location (i.e., the value of $pwd) is assumed to be the provider of the path. This syntax is often used to identify resources on other machines, such as UNC paths or remote registry hives.
Provider-qualified: the provider name is prepended to the drive-qualified or provider-direct item path, delimited by '::'. E.g., "FileSystem::c:/temp/junk.txt", or "FileSystem::\\share\temp\junk.txt". This format is used when the appropriate provider must be explicity stated.
Provider-internal: this is the portion of the provider-qualified path following the '::' delimiter.

Of the four supported path formats, the ASP.NET Membership PowerShell provider will support these three:

Drive-qualified: users:/username
Provider-qualified: ASPNETMembership::users:/username
Provider-internal: this is idential to the drive-qualified path syntax

Provider-direct paths and UNC-style provider-internal paths will not be supported by the ASP.NET Membership PowerShell provider.

Knowing the path formats to expect, it's time to implement support for the get-item cmdlet.

Enabling Get-Item

Enabling item cmdlet support for the ASP.NET Membership PowerShell provider begins with deriving the provider from ContainerCmdletProvider:

using System.Management.Automation;
using System.Management.Automation.Provider;
namespace ASPNETMembership
{
    [CmdletProvider( "ASPNETMembership", ProviderCapabilities.None )]
    public class Provider : ContainerCmdletProvider
    {
        // ...
    }
}

Deriving from ContainerCmdletProvider adds many item-related methods to the provider. To enabling the get-item cmdlet, at least two of these methods must be overridden.

GetItem

The first required override is the GetItem method, which is called to process a get-item invocation at runtime:

protected override void GetItem( string path )
{
    var user = GetUserFromPath(path);
    if( null != user )
    {
        WriteItemObject( user, path, false );
    }
}

The GetItem override delegates almost all of the work to the GetUserFromPath utility method; if GetUserFromPath returns a valid object reference, it is written back to the current pipeline using the WriteItemObject method of the provider object's base (line 6).

GetUserFromPath uses the provider's custom drive object to access the ASP.NET Membership provider. The drive object for the path is available in the PSDriveInfo property; PowerShell conveniently sets this value to the appropriate DriveInfo object for the item's path before calling GetItem:

MembershipUser GetUserFromPath( string path )
{
    var drive = this.PSDriveInfo as MembershipDriveInfo;
    var username = ExtractUserNameFromPath( path );
    return drive.MembershipProvider.GetUser( username, false );
}
static string ExtractUserNameFromPath( string path )
{
    if( String.IsNullOrEmpty( path ) )
    {
        return path;
    }
    // this regex matches all supported powershell path syntaxes:
    //  drive-qualified - users:/username
    //  provider-qualified - membership::users:/username
    //  provider-internal - users:/username
    var match = Regex.Match( path, @"(?:membership::)?(?:\w+:[\\/])?(?<username>[-a-z0-9_]*)$", RegexOptions.IgnoreCase );
    if( match.Success )
    {
        return match.Groups[ "username" ].Value;
    }
    return String.Empty;
}

The custom drive object exposes the ASP.NET Membership Provider, which offers a GetUser method that returns the MembershipUser object for a valid username (line 5). The username is extracted from the path string using a simple regular expression that matches the three path formats supported by the PowerShell provider (line 17).

ItemExists

The second required override is the ItemExists method, which is called by PowerShell to determine if the provider contains an item at a specified path (e.g., by the test-path cmdlet).

PowerShell calls ItemExists before the GetItem method when processing get-item; if ItemExists returns false, GetItem is not called and a "cannot find path" error is reported on the pipeline. The ASP.NET Membership provider reuses the GetUserFromPath utility method to ascertain whether the path contains a valid username:

protected override bool ItemExists( string path )
{
    return null != GetUserFromPath( path );
}

With these two overrides and their supporting utility methods, our provider can support the get-item cmdlet.

Testing Get-Item

Build and run; in the PowerShell console, create the users drive as follows:

> new-psdrive -psp aspnetmembership -root "" -name users -server localhost -catalog awesomewebsitedb; 
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
users      ASPNETMemb...

Once the drive is created, you can use get-item to fetch MembershipUser objects from the ASP.NET Membership user store:

> get-item users:/testuser
PSPath                  : ASPNETMembership::testuser
PSDrive                 : users
PSProvider              : ASPNETMembership
PSIsContainer           : False
UserName                : testuser
ProviderUserKey         : 09a9c356-a400-4cff-825d-231207946c94
Email                   : user@hotmail.com
PasswordQuestion        : what is your favorite color?
Comment                 :
IsApproved              : True
IsLockedOut             : False
LastLockoutDate         : 12/31/1753 7:00:00 PM
CreationDate            : 6/11/2009 12:59:45 PM
LastLoginDate           : 6/11/2009 12:59:45 PM
LastActivityDate        : 6/11/2009 12:59:45 PM
LastPasswordChangedDate : 6/11/2009 12:59:45 PM
IsOnline                : False
ProviderName            : AspNetSqlMembershipProvider

At this point, a whole new world of complexity is available from our provider:

> ( get-item users:/testuser ).ResetPassword( 'asdf1234' )
^PlpmNMON@7A]w

We can also leverage some of the built-in goodies of PowerShell against our ASP.NET Membership store in a natural way:

$u = get-item users:/testuser;
if( $u.IsLockedOut ) 
{ 
    $u.UnlockUser(); 
}

Pretty cool.

Coming Up

Discovery is a big part of PowerShell, and in the post I'll extend the ASP.Net Membership PowerShell provider to support the get-childitem (alias dir or ls) cmdlet, to enable listing of all users in the store. I'll also add support for the set-location (alias cd) cmdlet, which will allow operators to set the shell's current location to our custom users drive.

The code for this post is available here: ASPNETMembership_GetItem.zip (5.55 kb)

As always, thanks for reading, and if you liked this post, please Kick It, Shout It, trackback, tweet it, and comment using the clicky thingies below!

Twitter Appender for Log4Net

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 07 Jul 2009 04:13:00 -1300

Casey Watson just published a post describing how to create a Twitter appender for log4net! Not only is the idea creative and fun, but it's also great sample code if you want to build your own custom appenders.

Please check it out here: http://caseywatson.com/2009/07/07/log4net-twitter-awesome/

NICE WORK CASEY!

Death by Logging Mantra #1 - Logs Consume Space

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Mon, 15 Jun 2009 18:03:00 -1300

As much as I love and lean on logging, it's not immune from problems. In fact, it can be the source of some serious headaches. A recent snafu at work prompted me to write about some situations where logging can bring your application performance to a screeching halt, or crash your application altogether.

Here's what happened...

An incredibly complex system of hardware and software had been running smoothly for months; as part of an instrumentation layer we opted to change the rolling log strategy from 50 10MB text files:

...
<appender name="TextFile" type="log4net.Appender.RollingFileAppender">
    <file value="logs\log.txt" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="50" />
    <maximumFileSize value="10MB" />
    <layout type="log4net.Layout.XMLLayout">
      <prefix value="" />
    </layout>
</appender>
...

to 500 1MB xml files:

...
<appender name="XmlFile" type="log4net.Appender.RollingFileAppender">
    <file value="logs\log.xml" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="500" />
    <maximumFileSize value="1MB" />
    <staticLogFileName value="false" />
    <countDirection value="1" />
    <layout type="log4net.Layout.XMLLayout">
      <prefix value="" />
    </layout>
</appender>
...

As an ardent log4net user, I am aware of the performance impact of rolling a large number of files - if the CountDirection setting of the RollingFileAppender is less than 0 (which it is by default), the system will rename every log file each time the log rolls over. This is costly, and in our product configuration that would mean up to 500 file renames on each roll.

"Good thing I know what I'm doing...."

Several hours after firing up the new configuration a college asked me to come look at the device. It had slowed to a wounded crawl. I went to dig into the logs - I popped over to the log drive and started to find the most recent one.

... but there were 2000 log files, not 500. The 2GB drive dedicated to the log was completely full. And the application was still trying to write new log entries, which meant a slew of IO Exceptions were being continuously thrown by multiple threads.

"Um, I think I may have found the problem..."

I removed the oldest 1999 log files and the device immediately recovered.

So what happened?

The configuration XML is 100% correct. The problem was that I accidentally deployed the software to the device with an old beta version of log4net 2.1.9; that particular version contains a bug in the RollingFileAppender code that prevents the MaxSizeRollBackups from being honored when CountDirection was >= 0. Without the logs being limited in number, the software eventually filled up the hard disk with log entries.

Which bring me to my first death-by-logging mantra...

Logs Consume Space

It sounds silly I know, but this is the single most prevalent antipattern I see with any logging implementation. There is a finite amount of storage space, and you need to make sure your logging activity doesn't consume more than its share.

I frequently see this when apps use FileAppender - this beast has no chains and, as I've stated elsewhere, you should never ever use it. Ever. Even in "little" applications it can cause massive trouble because the log accumulates over process time or application runs with no checks. I've seen a 1KB application with 3GB of logs spanning almost a year of activity.

But don't think the problem is limited to the file-based appenders. Remember....

memory is finite...
event logs fill up more often than you think...
a database can be configured to boundlessly expand as needed...
mail servers generally put caps on the size of an inbox...

Whichever appender strategies you choose, you should carefully consider the following:

... How much persistent storage are you allocating to the log? Your answer should be a firm number, like "500MB", and not "the rest of the disk". If you can, base this on the amount of information you need to have access to. If a typical run of your application results in 10MB of logs, you can base the allocated size on the number of runs you want to persist. If your application runs continuously - like a web site or windows service - you can plot out the logging activity over time, then gauge the allocation size from the time span of activity you want to persist.

... How will you cope when the allocated storage is used up? Some appenders, like the RollingFileAppender, can handle this for you by freeing up space used by old log entries. Others, like the EventLogAppender or the AdoNetAppender, blithely log without regard to the amount of space being consumed, and it's up to you to manage the size of the log in other ways. E.g., I've seen SQL jobs dedicated to removing log records older than N hours, or truncating the log table to the N newest records.

... What happens when you log to a full repository? Plan for success, but understand the causes of failure. As I recently learned, our application slows down significantly when the log drive is full, so now checking the free space of the log drive is now included in our troubleshooting guide as a checklist item under "Application Performance Issues". Test your application under limited logging resources to understand how it will behave.

The most important thing to remember is that logging, like any other subsystem of your application, needs to be planned, tested, and verified.

Creating a PowerShell Provider pt 2: Parameters and Default Drives

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Thu, 11 Jun 2009 16:26:00 -1300

My previous post in this series discussed the first major step in developing a PowerShell provider: creating a custom drive object. This post rounds out the discussion by making the drive configurable with custom initialization parameters. In addition, I'll demonstrate how a provider can create one or more "default" drives without user intervention.

This post builds on code from the previous post, which you can download from here. The completed code for this post is also available.

Parameterizing the Drive

At the moment, the ASP.NET Membership provider configuration lives in code; that should really change so this drive can be used with other providers or for multiple sites. Such drive configurability is gained by supporting the NewDriveDynamicParameters method of the PowerShell provider.

The NewDriveDynamicParameters method is called each time the new-psdrive cmdlet is invoked to allow the provider to add drive creation parameters. Although the MSDN documentation of this method isn't terribly helpful, implementing it is pretty simple. All you need to do is return an object containing properties or fields for each custom parameter.

First, we need to know what parameters are needed to create the ASP.NET Membership provider. Take a quick look at the configuration items hard-coded in the drive creation logic:

// DriveInfo.cs
public MembershipDriveInfo( PSDriveInfo driveInfo )
    : base( driveInfo )
{
    var connectionStrings = ConfigurationManager.ConnectionStrings;
 
    var fi = typeof( ConfigurationElementCollection )
        .GetField( "bReadOnly", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic );
    fi.SetValue( connectionStrings, false );
 
    connectionStrings.Add(
        new ConnectionStringSettings(
            "ProviderConnectionString",
            "data source=localhost;Integrated Security=SSPI;Initial Catalog=AwesomeWebsiteDB"
        )
    );
 
    provider = new SqlMembershipProvider();
    var nvc = new System.Collections.Specialized.NameValueCollection
    {
        { "connectionStringName", "ProviderConnectionString" },
        { "enablePasswordRetrieval", "false" },
        { "enablePasswordReset", "true" },
        { "requiresQuestionAndAnswer", "false" },
        { "requiresUniqueEmail", "false" },
        { "passwordFormat", "Hashed" },
        { "maxInvalidPasswordAttempts", "5" },
        { "minRequiredPasswordLength", "6" },
        { "minRequiredNonalphanumericCharacters", "0" },
        { "passwordAttemptWindow", "10" },
        { "passwordStrengthRegularExpression", "" },
        { "applicationName", "/" },
    };
    provider.Initialize( "AspNetSqlMembershipProvider", nvc );
}

There are a number of parameters used to initialize the ASP.NET membership provider. Based on the code, I would want to configure any of the following facets of the ASP.NET Membership provider:

the Connection String server and databse name;
EnablePasswordRetrieval;
EnablePasswordReset;
RequiresQuestionAndAnswer;
RequiresUniqueEmail;
PasswordFormat;
MaxInvalidPassordAttempts;
MinRequiredPasswordLength;
MinRequiredNonalphanumericCharacters;
PasswordAttemptWindow;
PasswordStrengthRegularExpression;
ApplicationName.

While all of these should be configurable, assumptions can be made for most of them.

The first step towards implementing the NewDriveDynamicParameters method is to define an object containing read/write properties for each configurable parameter:

// DriveParameters.cs
using System;
using System.Management.Automation;
using System.Web.Security;
namespace ASPNETMembership
{
    public class DriveParameters
    {
        public DriveParameters()
        {
            EnablePasswordReset = true;
            EnablePasswordRetrieval = false;
            RequiresQuestionAndAnswer = false;
            RequiresUniqueEmail = false;
            MaxInvalidPasswordAttempts = 5;
            MinRequiredNonalphanumericCharacters = 0;
            MinRequiredPasswordLength = 6;
            PasswordAttemptWindow = 10;
            PasswordStrengthRegularExpression = String.Empty;
            ApplicationName = "/";
            PasswordFormat = MembershipPasswordFormat.Hashed;
        }
 
        [Parameter( Mandatory=true )]       
        public string Server { get; set; }        
 
        [Parameter( Mandatory = true )]
        public string Catalog { get; set; }
 
        public bool EnablePasswordRetrieval { get; set; }
        public bool EnablePasswordReset { get; set; }
        public bool RequiresQuestionAndAnswer { get; set; }
        public bool RequiresUniqueEmail { get; set; }
        public MembershipPasswordFormat PasswordFormat { get; set; }
        public int MaxInvalidPasswordAttempts { get; set; }
        public int MinRequiredPasswordLength { get; set; }
        public int MinRequiredNonalphanumericCharacters { get; set; }
        public int PasswordAttemptWindow { get; set; }
        public string PasswordStrengthRegularExpression { get; set; }
        public string ApplicationName { get; set; }
    }
}

Note the use of the System.Management.Automation.ParameterAttribute on lines 24 and 27 to identify the Server and Catalog parameters as required (Mandatory=true). The remaining optional parameters are initialized to default values in the constructor.

With the DriveParameters type defined, the NewDriveDyanmicParameters method of the ASP.NET Membership provider can now be implemented:

// Provider.cs
protected override object NewDriveDynamicParameters()
{
    return new DriveParameters();
}

PowerShell will call this method each time the new-psdrive cmdlet is invoked for this provider. The object returned to PowerShell is filled with parameter values and passed back to the provider's NewDrive method via the DynamicParameters property; in the Membership PowerShell provider, NewDrive simply passes the parameters to the MembershipDriveInfo constructor:

// Provider.cs
protected override System.Management.Automation.PSDriveInfo NewDrive( PSDriveInfo drive )
{
    var driveParams = this.DynamicParameters as DriveParameters;
    return new MembershipDriveInfo(drive, driveParams);
}

which must then act on the parameters when initializing the ASP.NET Membership provider:

// DriveInfo.cs
using System;
using System.Configuration;
using System.Management.Automation;
using System.Web.Security;
namespace ASPNETMembership
{
    public class MembershipDriveInfo : PSDriveInfo
    {
        MembershipProvider provider;
        public MembershipDriveInfo( PSDriveInfo driveInfo, DriveParameters parameters )
            : base( driveInfo )
        {
            var connectionStrings = ConfigurationManager.ConnectionStrings;
 
            var fi = typeof( ConfigurationElementCollection )
                .GetField( "bReadOnly", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic );
            fi.SetValue( connectionStrings, false );
 
            var connectionString = String.Format(
                "data source={0};Integrated Security=SSPI;Initial Catalog={1}",
                parameters.Server,
                parameters.Catalog
                );
 
            var moniker = Guid.NewGuid().ToString("N");
            var connectionStringName = "cxn_" + moniker;
            var providerName = "pvdr_" + moniker;
 
            connectionStrings.Add(
                new ConnectionStringSettings(
                    connectionStringName,
                    connectionString
                )
            );
 
            provider = new SqlMembershipProvider();
            var nvc = new System.Collections.Specialized.NameValueCollection
            {
                { "connectionStringName", connectionStringName },
                { "enablePasswordRetrieval", parameters.EnablePasswordRetrieval.ToString() },
                { "enablePasswordReset", parameters.EnablePasswordReset.ToString() },
                { "requiresQuestionAndAnswer", parameters.RequiresQuestionAndAnswer.ToString() },
                { "requiresUniqueEmail", parameters.RequiresUniqueEmail.ToString() },
                { "passwordFormat", parameters.PasswordFormat.ToString() },
                { "maxInvalidPasswordAttempts", parameters.MaxInvalidPasswordAttempts.ToString() },
                { "minRequiredPasswordLength", parameters.MinRequiredPasswordLength.ToString() },
                { "minRequiredNonalphanumericCharacters", parameters.MinRequiredNonalphanumericCharacters.ToString() },
                { "passwordAttemptWindow", parameters.PasswordAttemptWindow.ToString() },
                { "passwordStrengthRegularExpression", parameters.PasswordStrengthRegularExpression },
                { "applicationName", parameters.ApplicationName },
            };
            provider.Initialize( providerName, nvc );
        }
        
        public MembershipProvider MembershipProvider
        {
            get
            {
                return this.provider;
            }
        }
    }
}

Not a lot of new code here; the constructor now accepts an additional DriveParameters argument that is used to initialize the connection string and membership provider configuration. Using dynamically-generated connection string and provider names prevents naming collisions if more than one drive is initialized by the provider (lines 26-28).

Using the Drive Parameters

Build and run. The new dynamic drive parameters can be specified directly to the new-psdrive cmdlet just like any other cmdlet parameter:

> new-psdrive -psprovider aspnetmembership -name users -root '' 
    -server localhost -catalog AwesomeWebsiteDB 
    -MinRequiredPasswordLength 8 -PasswordAttemptWindow 4 
 
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
users      ASPNETMemb...

Note that PowerShell prompts the user for the any required parameters if they aren't supplied:

> remove-psdrive users
> new-psdrive -psprovider aspnetmembership -name users -root ''
cmdlet New-PSDrive at command pipeline position 1
Supply values for the following parameters:
Catalog: 
> AwesomeWebsiteDB
Server:
> localhost
 
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
users      ASPNETMemb...

Creating Default Drives

Another feature available to PowerShell providers is the ability to initialize a set of default drives when the provider assembly is first loaded. This is accomplished by overriding the provider's InitializeDefaultDrives method to return a list of PSDriveInfo objects describing the drives to create:

// Provider.cs
 
protected override Collection<PSDriveInfo> InitializeDefaultDrives()
{
    var driveInfo = new PSDriveInfo(
        "users",
        this.ProviderInfo,
        "",
        "Default ASP.NET Membership Drive",
        null
        );
    
    var parameters = new DriveParameters
                     {
                         Catalog = "AwesomeWebsiteDB",
                         Server = "localhost"
                     };
 
    return new Collection<PSDriveInfo>
           {
               new MembershipDriveInfo(
                   driveInfo,
                   parameters
                   )
           };
}

Each PSDriveInfo object in the returned collection will then be passed to the NewDrive method of the provider to complete the drive creation (the NewDriveDynamicParameters method is not called in this case). As such, the ASP.NET Membership PowerShell provider's NewDrive method must be modified to acommodate pre-initialized drives:

protected override PSDriveInfo NewDrive( PSDriveInfo drive )
{
    // see if the drive has already been initialized
    //    e.g., via InitializeDefaultDrives()
    if( drive is MembershipDriveInfo )
    {
        return drive;
    }
 
    var driveParams = this.DynamicParameters as DriveParameters;
    return new MembershipDriveInfo(drive, driveParams);
}

Testing Default Drive Creation

Build and run; you can verify that the default drive was created using the get-psdrive cmdlet:

> get-psdrive
 
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
...
users      ASPNETMemb...

No need to use the new-psdrive cmdlet - the users drive is created automagically as PowerShell loads the provider assembly.

Coming Up

That's about it for drives; for here on in, the focus will be on making the provider work with all the features available in PowerShell. The next post will add support for the get-item cmdlet and discuss the various flavors of item paths the provider must support.

The code for this post is available here:

ASPNETMembership_pt1_drives.zip (4.26 kb)

As always, thanks for reading, and if you liked this post, please Kick It, Shout It, trackback, tweet it, and comment using the clicky thingies below!

Creating a PowerShell Provider pt 1: Creating the Drive

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 09 Jun 2009 14:06:00 -1300

The first step to creating a full-featured PowerShell provider is to be able to create a new drive. The goal of this post is to enable the most basic functional drive that provides access to the ASP.NET Membership user store. Before you begin, make sure you have the project set up properly as I describe here, or you can download the project pre-coded (3.24 kb).

We'll start by creating a minimal provider implementation. Add a class to the project named Provider and fill it in like so:

// Provider.cs
using System.Management.Automation;
using System.Management.Automation.Provider;
namespace ASPNETMembership
{
    [CmdletProvider( "ASPNETMembership", ProviderCapabilities.None )]
    public class Provider : DriveCmdletProvider
    {
        protected override PSDriveInfo NewDrive( PSDriveInfo drive )
        {
            return base.NewDrive( drive );
        }
        protected override object NewDriveDynamicParameters()
        {
            return base.NewDriveDynamicParameters();
        }
    }
}

A few items of note:

the Provider class derives from System.Automation.Provider.DriveCmdletProvider. All PowerShell providers derive from one of the abstract *CmdletProvider base classes. I'll discuss these classes a bit in the next post, but for now know that DriveCmdletProvider is the most basic base class you can use to create a provider.
the class is decorated with a System.Management.Automation.Provider.CmdletProviderAttribute. This attribute is required to get PowerShell to recognize your class as a provider when the assembly is loaded. In addition, the attribute identifies some of the capabilities the provider supports; more on this in a later post.

The class overrides two methods related to drive creation - at the moment they are simply placeholders and defer to the DriveCmdletProvider implementations. When PowerShell is presented with a new-psdrive cmdlet specifying the ASPNETMembership provider, it will ferret the call to our provider's NewDrive method to create the drive.

Compile and run; in the PowerShell console that opens, enter the following command (note that my convention is to mark commands you type in with leading >'s):

> new-psdrive -PsProvider ASPNETMembership -Name users -Root "";

This will invoke the NewDrive method of the ASPNETMembership provider and create a new drive named "users" in the PowerShell session. To verify the drive exists, use the get-psdrive cmdlet. You should see output similar to the following :

> get-psdrive
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
C          FileSystem    C:\                                 ...ensions\Profile
cert       Certificate   \
Env        Environment
Function   Function
Gac        AssemblyCache Gac
HKCU       Registry      HKEY_CURRENT_USER
HKLM       Registry      HKEY_LOCAL_MACHINE
users      ASPNETMemb...

Custom Drive Objects

Providers are transient: PowerShell will create many instances of this provider in response to different cmdlets. So it's not possible to persist any state in a provider object. Instead, state needs to be stored in a custom drive object that is returned from the NewDrive method of the provider. PowerShell caches this instance and makes it available to the provider via the DriveCmdletProvider.PsDriveInfo property.

The only real requirement for a custom drive object is that it derive from System.Automation.PSDriveInfo. This class is the most basic implementation of a custom drive object possible; go ahead and add it to the project:

// DriveInfo.cs
using System.Management.Automation;
namespace ASPNETMembership
{
    public class MembershipDriveInfo : PSDriveInfo
    {
        public MembershipDriveInfo( PSDriveInfo driveInfo )
            : base( driveInfo )
        {
        }      
    }
}

Then modify the Provider class to return a new instance of this drive object in the NewDrive method:

// Provider.cs
// ... 
        protected override PSDriveInfo NewDrive( PSDriveInfo drive )
        {
            return new MembershipDriveInfo( drive );
        }
// ...

Build and run; in the console, enter the following commands:

> new-psdrive -psp aspnetmembership -name users -root ""
Name       Provider      Root                                   CurrentLocation
----       --------      ----                                   ---------------
users      ASPNETMemb...
> ( get-psdrive users ).gettype().fullname
ASPNETMembership.MembershipDriveInfo

Note that the type of the users drive is now our custom drive object type - at this point, we have our first soild hook into the powershell provider system. Now it's time to fold ASP.NET Membership into the custom drive.... but there's a small problem to deal with first.

Configuring ASP.NET Membership Programatically

Configuring ASP.NET Membership in a web application is pretty easy. You just pop in a little config section into the web.config and everything just works; e.g., this web.config sets up the SqlMembershipProvider for a website:

<configuration>
... 
    <connectionStrings>
        <add name="MembershipProviderServices" 
                 connectionString="data source=localhost;Integrated Security=SSPI;Initial Catalog=AwesomeWebsiteDB"/>
    </connectionStrings>
... 
        <membership>
            <providers>
                <clear/>
                <add name="AspNetSqlMembershipProvider" 
                         type="System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" 
                         connectionStringName="MembershipProviderServices" 
                         enablePasswordRetrieval="false" 
                         enablePasswordReset="true" 
                         requiresQuestionAndAnswer="false" 
                         requiresUniqueEmail="false" 
                         passwordFormat="Hashed" 
                         maxInvalidPasswordAttempts="5" 
                         minRequiredPasswordLength="6" 
                         minRequiredNonalphanumericCharacters="0" 
                         passwordAttemptWindow="10" 
                         passwordStrengthRegularExpression="" 
                         applicationName="/" />
            </providers>
        </membership>
...
</configuration>

In PowerShell, there is no equivalent to a web.config or app.config we can use to configure the membership provider. Instead, we have to configure the memberhip provider via code. Not a big deal; this short snippit of code results in the same membership provider configuration defined by the web.config example:

var provider = new SqlMembershipProvider();
var nvc = new System.Collections.Specialized.NameValueCollection
{
    { "connectionStringName", "MembershipProviderServices" },
    { "enablePasswordRetrieval", "false" },
    { "enablePasswordReset", "true" },
    { "requiresQuestionAndAnswer", "false" },
    { "requiresUniqueEmail", "false" },
    { "passwordFormat", "Hashed" },
    { "maxInvalidPasswordAttempts", "5" },
    { "minRequiredPasswordLength", "6" },
    { "minRequiredNonalphanumericCharacters", "0" },
    { "passwordAttemptWindow", "10" },
    { "passwordStrengthRegularExpression", "" },
    { "applicationName", "/" },
};
provider.Initialize( "AspNetSqlMembershipProvider", nvc );

The only wrinkle is on line 4, where we reference a connection string by its name. The ConfigurationManager.ConnectionStrings collection is read-only, so how can we add a new connection string to it? Simple, we cheat and make the collection writeable (many thanks to David Gardiner for posting a similar solution on his blog!):

using System.Configuration;
var connectionStrings = ConfigurationManager.ConnectionStrings;
// find the private bReadOnly field in the connection strings base type
var fi = typeof( ConfigurationElementCollection );
    .GetField( "bReadOnly", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic );
// change the field value to false, making the object writeable
fi.SetValue( connectionStrings, false );
// add a new connection string
connectionStrings.Add(
    new ConnectionStringSettings(
        "MembershipProviderServices",
        "data source=localhost;Integrated Security=SSPI;Initial Catalog=AwesomeWebsiteDB"
        )
    );

Now with the ability to add connection strings to the configuration manager programmatically, we can add ASP.NET Membership services to our custom drive object.

Adding ASP.NET Membership Features to the Drive

Modify the custom drive class with our ASP.NET Membership configuration code as follows:

// DriveInfo.cs
using System.Configuration;
using System.Management.Automation;
using System.Web.Security;
namespace ASPNETMembership
{
    public class MembershipDriveInfo : PSDriveInfo
    {
        MembershipProvider provider;
        public MembershipDriveInfo( PSDriveInfo driveInfo )
            : base( driveInfo )
        {
            var connectionStrings = ConfigurationManager.ConnectionStrings;
            var fi = typeof( ConfigurationElementCollection )
                .GetField( "bReadOnly", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic );
            fi.SetValue( connectionStrings, false );
            connectionStrings.Add(
                new ConnectionStringSettings(
                    "ProviderConnectionString",
                    "data source=localhost;Integrated Security=SSPI;Initial Catalog=AwesomeWebsiteDB"
                )
            );
            provider = new SqlMembershipProvider();
            var nvc = new System.Collections.Specialized.NameValueCollection
            {
                { "connectionStringName", "ProviderConnectionString" },
                { "enablePasswordRetrieval", "false" },
                { "enablePasswordReset", "true" },
                { "requiresQuestionAndAnswer", "false" },
                { "requiresUniqueEmail", "false" },
                { "passwordFormat", "Hashed" },
                { "maxInvalidPasswordAttempts", "5" },
                { "minRequiredPasswordLength", "6" },
                { "minRequiredNonalphanumericCharacters", "0" },
                { "passwordAttemptWindow", "10" },
                { "passwordStrengthRegularExpression", "" },
                { "applicationName", "/" },
            };
            provider.Initialize( "AspNetSqlMembershipProvider", nvc );
        }
        
        public MembershipProvider MembershipProvider
        {
            get
            {
                return this.provider;
            }
        }
    }
}

Note that the custom drive holds a reference to the ASP.NET Membership provider that is initialized in the constructor - since PowerShell caches the drive object, it's safe to save the provider reference. The drive also exposes the membership provider through the public MembershipProvider property.

Build and run. Now the fun begins - first, let's create get a reference to our drive:

> $d = new-psdrive -P
sProvider ASPNETMembership -Name users -Root ""
> $d.MembershipProvider | get-member
   TypeName: System.Web.Security.SqlMembershipProvider
Name                                 MemberType Definition
----                                 ---------- ----------
ValidatingPassword                   Event      System.Web.Security.Membersh...
ChangePassword                       Method     System.Boolean ChangePasswor...
ChangePasswordQuestionAndAnswer      Method     System.Boolean ChangePasswor...
CreateUser                           Method     System.Web.Security.Membersh...
DeleteUser                           Method     System.Boolean DeleteUser(St...
Equals                               Method     System.Boolean Equals(Object...
FindUsersByEmail                     Method     System.Web.Security.Membersh...
FindUsersByName                      Method     System.Web.Security.Membersh...
GeneratePassword                     Method     System.String GeneratePasswo...
GetAllUsers                          Method     System.Web.Security.Membersh...
GetHashCode                          Method     System.Int32 GetHashCode()
GetNumberOfUsersOnline               Method     System.Int32 GetNumberOfUser...
GetPassword                          Method     System.String GetPassword(St...
GetType                              Method     System.Type GetType()
GetUser                              Method     System.Web.Security.Membersh...
GetUserNameByEmail                   Method     System.String GetUserNameByE...
Initialize                           Method     System.Void Initialize(Strin...
ResetPassword                        Method     System.String ResetPassword(...
ToString                             Method     System.String ToString()
UnlockUser                           Method     System.Boolean UnlockUser(St...
UpdateUser                           Method     System.Void UpdateUser(Membe...
ValidateUser                         Method     System.Boolean ValidateUser(...
ApplicationName                      Property   System.String ApplicationNam...
Description                          Property   System.String Description {g...
EnablePasswordReset                  Property   System.Boolean EnablePasswor...
EnablePasswordRetrieval              Property   System.Boolean EnablePasswor...
MaxInvalidPasswordAttempts           Property   System.Int32 MaxInvalidPassw...
MinRequiredNonAlphanumericCharacters Property   System.Int32 MinRequiredNonA...
MinRequiredPasswordLength            Property   System.Int32 MinRequiredPass...
Name                                 Property   System.String Name {get;}
PasswordAttemptWindow                Property   System.Int32 PasswordAttempt...
PasswordFormat                       Property   System.Web.Security.Membersh...
PasswordStrengthRegularExpression    Property   System.String PasswordStreng...
RequiresQuestionAndAnswer            Property   System.Boolean RequiresQuest...
RequiresUniqueEmail                  Property   System.Boolean RequiresUniqu...

As you can see, the membership provider is fully accessible from PowerShell via the drive object. We can create users:

> $status = 0;
> $d.MembershipProvider.CreateUser( "myuser", "mypassword1234", "myuser@hotmail.com", "what is your favorite color?", "plaid", $true, [Guid]::NewGuid(), ([ref]$status))
UserName                : myuser
ProviderUserKey         : b6ebc2a4-2ff8-4cf8-928e-969416edd704
Email                   : myuser@hotmail.com
PasswordQuestion        : what is your favorite color?
Comment                 :
IsApproved              : True
IsLockedOut             : False
LastLockoutDate         : 1/1/1754 12:00:00 AM
CreationDate            : 6/11/2009 1:13:50 PM
LastLoginDate           : 6/11/2009 1:13:50 PM
LastActivityDate        : 6/11/2009 1:13:50 PM
LastPasswordChangedDate : 6/11/2009 1:13:50 PM
IsOnline                : True
ProviderName            : AspNetSqlMembershipProvider

We can get existing users:

> $d.MembershipProvider.GetUser( "myuser", $false )
UserName                : myuser
ProviderUserKey         : b6ebc2a4-2ff8-4cf8-928e-969416edd704
Email                   : myuser@hotmail.com
PasswordQuestion        : what is your favorite color?
Comment                 :
IsApproved              : True
IsLockedOut             : False
LastLockoutDate         : 12/31/1753 7:00:00 PM
CreationDate            : 6/11/2009 1:13:50 PM
LastLoginDate           : 6/11/2009 1:13:50 PM
LastActivityDate        : 6/11/2009 1:13:50 PM
LastPasswordChangedDate : 6/11/2009 1:13:50 PM
IsOnline                : True
ProviderName            : AspNetSqlMembershipProvider

We can even remove users:

> $d.MembershipProvider.DeleteUser( "myuser", $true )
True
> $d.MembershipProvider.GetUser( "myuser",$false ) 
#note that no output is returned this time

Using the drive directly to manage users may work, but it is a bit clumsy. You may be asking yourself why we implemented the drive at all, given that PowerShell could create the MembershipProvider object directly. The drive is the basis on which all PowerShell provider functionality will be based; many standard cmdlets leverage the data stored in a drive object. So things may look cumbersome at the moment, but soon we'll be working with cmdlets, pipes, and sets of user objects - all thanks to our custom drive object.

Coming Up

At the moment, the ASP.NET Membership provider configuration lives code; that should really change so this drive can be used with other providers or for multiple sites. The NewDriveDynamicParameters method of the PowerShell provider object allows the provider to request additional drive creation parameters from the new-psdrive cmdlet. My next post will describe how to properly implement this method.

In upcoming posts, I'll show you how to implement support for different core cmdlets, such as get-item, remove-item, new-item, and get-childitem (or 'dir' for you alias users). In addition, I'll dig deep into the various path syntaxes that PowerShell expects your provider to support.

The code for this post is available here:

ASPNETMembership_pt1.zip (3.24 kb)

As always, thanks for reading, and if you liked this post, please Kick It, Shout It, trackback, or comment.

Log4Net Tutorials and Resources

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Sun, 07 Jun 2009 03:57:00 -1300

I've been meaning to create a landing page for all the log4net traffic on my blog. A friend of mine recently told me how difficult it is to drill through all the tutorials, so I decided to organize access to all the log4net stuff in one place. I'll keep this page updated as the blog grows.

If there are any log4net resources you think I should add to this list or blog about, please add them in the comments or send them to me via the contact link.

Log4Net Tutorials

If you've never used log4net before, you should begin with the getting started and configuration topics. You'll want to review the appender, layout, and patterns tutorials next, and use the others as you see fit.

Getting Started with Log4Net - never used log4net before, but need to get running quickly?
Basic XML Configuration - describes how to isolate log4net configuration to an XML file for easier modification.
Appenders - a discussion of the catalog of appenders available in the log4net distribution. Need to figure out how to log to the console? Or a text file? Or a database? This tutorial is for you.
Layouts and Patterns - want more control over the way your log looks?
Using Logger Objects - learn to use logger objects to organize your logging code into independently configurable parts.
Log Event Context - need some extra data in that log message? Maybe a call stack label, user identifier, or session moniker?
Filters - learn how to tame that log output and filter out the stuff you don't want or need.
Lossy Logging - learn about an advanced technique of optimizing your logging activity. Lossy logging gives you the benefit of a verbose log with the performance of a minimal logging scenario.

Recomended Practices

Your Code - learn what you should do in your code to make the most of logging.
Isolating Bugs - see how logging can be used to isolate production bugs.

Death by Logging Mantras

These mantras discuss specific scenarios where logging can cripple your application or its performance. These are in-process and links will become available as I write the posts.

Mantra #1: Logs Consume Space
Mantra #2: Logging Code is Still Code
Mantra #3: Logging Consumes Resources
Mantra #4: Noise is not a Log

Resources

Extension Methods for Deferred Log4Net Message Formatting - a set of extensions methods that make deferring log message formatting easier.
A Look at the Live Labs Simple Logging Framework - a quick overview of the Live Labs SLF, and some pros and cons it has with regard to log4net.
Correlating and Grepping Logs with PowerShell - an example of how to leverage the log4net XML log output to slurp and correlate a log against the system event log.

MSDN Southern Fried Road Show Charlotte

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 02 Jun 2009 17:14:00 -1300

Many, many thanks to Brian Hitney, Chad Brooks, and Glen Gordon for letting me give a shotgun PowerShell presentation at the MSDN RoadShow in Charlotte today. I appreciate the opportunity, guys!

As promised, here is a quick summary and some links to PowerShell tutorials and developer blogs...

The Big Three Commands

Remember, you don't need to remember all 200+ commands available in PowerShell, because PowerShell provides built-in cheat sheets!

Get-Command

Example 1: list all available commands

Get-command

Example 2: list available commands pertaining to processes

Get-command *process*

Get-help

Example 1: general powershell help

Get-Help

Example 2: list help topics matching a term

Get-help *object*

Example 3: get “short” help for a specific command

Get-help get-process

get-process -?

Example 3: get “full” help (details, examples, etc) for a specific command

Get-help get-process -full

Get-Member

Example 1: list members of a local variable

$variable | get-member

Get-member –input $variable

Example 2: list members of a pipeline result

Get-process | get-member

Links

Download Powershell v2 CTP3

Community Resources

PowerShellCommunity.org - aggregator of scripts, blogs, and resources. A great place to start searching for powershell packages.
PowerScripting podcast – weekly podcast of all things powershell. Generally targeted at sys admins and DBAs, but often they discuss targeted technologies in powershell, such as sharepoint, sql server, exchange, VMWare, etc.
Nivot Ink – Oisin Grehan's blog containing an insane amount of detail concerning v2 CTP features, including creating modules, eventing, and remoting.
huddledmasses.org - Joel Bennet’s blog, author of the PowerBoots WPF extensions for powershell, and massive powershell development online presence.
PowerShell Team Blog – the horse’s mouth, so to speak.
PowerGUI - an awesome MMC-style GUI PowerShell host, very flexible and endlessly useful.

Open-Source Projects

PowerShell Community Extensions – targeted at powershell v1, the functionality offered here is largely absorbed in v2; however, this codebase is still a great resource for learning more about how powershell works.
SQL Server PowerShell Extensions – powershell support for managing SQL 2000 and 2005; note that SQL 2008 has powershell integration out of the box.
PolyMon – an open-source system monitoring solution with powershell support.

Module, Training, & Tool Vendors

Quest AD – active directory support for powershell. PowerGUI.org – Extensible and generic MMC-type UI front-end for powershell.
SAPIEN – makers of PrimalScript scripting IDE, designed specifically for sys admins; training courses in powershell, scripting, etc.
Quest – authors of free PowerGUI (http://www.powergui.org), a UI front-end for powershell, as well as the ActiveRoles management shell for AD and many, many other management products.
SoftwareFX – vendor of .NET and Java instrumentation controls, as well as PowerGadgets, a tool that enables you to create vista side-bar gadgets based on PowerShell scripts - it's BIG FUN!

Creating a PowerShell Provider pt 0: Gearing Up

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 26 May 2009 01:11:00 -1300

Last month I gave a talk at ALT.NET Charlotte on using PowerShell as a platform for support tools. The majority of the talk revolved around the extensible provider layer available in PowerShell, and how this enables support engineers to accomplish many things by learning a few simple commands. Creating providers isn't hard, but there is a bit of "black art" to it. Based on the lack of adequate examples and documentation available, I thought I'd fill the void with a few posts derived from my talk.

In these posts, I will be creating a PowerShell provider around the ASP.NET Membership framework that enables the management of user accounts (for more information on ASP.NET Membership, start with this post from Scott Gu's blog) . Basically, this provider will allow you to treat your ASP.NET Membership user repository like a file system, allowing you to perform heroic feats that are simply impossible in the canned ASP.NET Web Site Management Tool.

Like what you ask? Consider this:

dir users: | 
   where { !( $_.isApproved ) -and $_.creationDate.Date -eq ( ( get-date ).Date ) }

which lists all unapproved users created today. Or this:

dir users: | where{ $_.isLockedOut } | foreach{ $_.unlockUser() }

which unlocks all locked users. Or this:

dir users: | 
   where { ( (get-date) - $_.lastLoginDate ).TotalDays -gt 365 } |
   remove-item

which removes users who haven't authenticated in the past year. These are things you simply can't do using the clumsy built-in tools. With a simple PowerShell provider around the user store, you can accommodate these situations and many others you don't even know you need yet.

This first post is aimed at getting a project building, with the appropriate references in place, and with the debugger invoking powershell properly. The next few posts will focus on implementing the proper extensibility points to have PowerShell interact with your provider. Future posts will focus on implementing specific features of the provider, such as listing, adding, or removing users. Eventually I will discuss packaging the provider for distribution and using the provider in freely available tools, such as PowerGUI.

Prerequisites

You'll need the latest PowerShell v2 CTP; at the time of this writing, the latest was CTP3 and it was available here. Since we're working with a CTP, you should expect some things to change before the RTM; however, the PowerShell provider layer is unchanged from v1.0 in most respects. The only major difference is with regards to deployment and installation, which I'll discuss at the appropriate time.

If you are currently using PowerShell v1.0, you will need to uninstall it. Instructions are available here and here.

The .NET Framework 3.5, although not required to run PowerShell v2 CTP3, is required for some of the tools that accompany the v2 release.

Finally, I suggest you fire up PowerShell on its own at least once, to verify it installed correctly and to loosen up the script execution policy. To save some grief, one of the first commands I execute in a new PowerShell install is this one:

set-executionPolicy RemoteSigned

This will enable the execution of locally-created script files, but require downloaded script files be signed by a trusted authority before allowing their execution.

Setting Up the Project

Create a new Class Library project in Visual Studio named "ASPNETMembership". This project will eventually contain the PowerShell provider code.

You will need to add the following references to the project:

System.Web - this assembly contains the MembershipUser type we will be exposing in our PowerShell provider;
System.Configuration - we'll need to access the configuration subsystem to properly set up our membership provider;
System.Management.Automation - this contains the PowerShell type definitions we need to create our provider. You may have to hunt for this assembly; try here: C:\Program Files\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0.

Now that the necessary references are configured, it's time to configure debugging appropriately.

Open the project properties dialog and select the Debug tab. Under Start Action, select the "Start external program" option. In the "Start external program" textbox, enter the fully qualified path to the PowerShell executable: c:\windows\system32\windowspowershell\v1.0\powershell.exe.

Yes, even though you're running PowerShell v2, it still lives in the v1.0 directory.

In the "Command line arguments textbox, enter the following:

-noexit -command "[reflection.assembly]::loadFrom( 'ASPNETMembership.dll' ) | import-module"

The -command argument instructs PowerShell to execute the supplied command pipe string as if it were typed in at the console. It may not be obvious what the command pipe string is doing. In a nutshell, the first command in the pipe loads the ASPNETMembership project output dll into the AppDomain. The second command in the pipe causes PowerShell to load any cmdlets or providers implemented in the dll. I'll touch on the import-module command more in a future post.

The -noexit argument prevents PowerShell from exiting once the command has been run, which enables us to type in commands and interact with the console while debugging.

Test the Project Setup

Build and Run the empty project. A PowerShell console should launch.

In the console, run the following command, which lists all assemblies loaded in the currently executing AppDomain whose FullName property matches 'aspnet':

[appdomain]::currentDomain.GetAssemblies() | 
   where { $_.FullName -match 'aspnet' } | 
   select fullname

Verify that the ASPNETMembership assembly is listed in the output of this command:

If it isn't, double-check the command-line arguments specified in the project properties -> debug tab. Specifically, ensure that:

the command arguments are entered exactly as specified above, and
the working directory is set to the default (empty) value

That's it for now - next post we will begin implementing the MembershipUsers PowerShell provider and enable the most basic pieces necessary to list the users in the membership store.

Expanding-File Resource for Spring.NET

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Wed, 13 May 2009 03:48:00 -1300

I just had a fun emergency - the crew decided to redirect all configuration sources using environment variables, and while Log4Net supports this quite easily, the spring.net context resource-from-a-URI abstraction does not. E.g., this will not work:

<spring>
    <context>
            <resource uri="file://%PROGRAMFILES%/myapp/ioc.config" />
    </context>
    ...

Knowing that spring.net can leverage custom resource implementations, I set out to extend the file system resource to expand environment variables. Turns out to be easy-peasy-lemon-squeezie:

public class ExpandableFileSystemResource : Spring.Core.IO.FileSystemResource
{
    public ExpandableFileSystemResource()
        : base()
    {
    }
    public ExpandableFileSystemResource( string resourceName )
        : base( Environment.ExpandEnvironmentVariables( resourceName ?? String.Empty ) )
    {
    }
    public ExpandableFileSystemResource( string resourceName, bool suppressInitialize )
        : base( Environment.ExpandEnvironmentVariables( resourceName ?? String.Empty ), suppressInitialize )
    {
    }
}

<spring>
    <resourceHandlers>
      <handler protocol="filex" type="MyApp.ExpandableFileSystemResource, MyApp"/>
    </resourceHandlers>
    <context>
            <resource uri="filex://%PROGRAMFILES%/myapp/ioc.config" />
    </context>    
    ...

And it just works. Enjoy!

ALT.NET Charlotte Presentation

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Thu, 07 May 2009 14:16:00 -1300

I just got back from the second meeting of ALT.NET Charlotte, where I presented a talk on PowerShell as a Tools Platform. I think it went well. The demos centered around building tools in PowerShell to manage the ASP.NET Membership store for a website, and they seemed to drive the point home. Try doing this:

dir users: | where { $_.lastLoginDate –lt ( ( get-date ).addyear( -1 ) ) } | remove-item;

in the ASP.NET Website Management tool!

As well as I feel it went, I'm still hoping to get some feedback from the group. Feel free to leave a comment here if you wish, or use the contact link at the top of this page.

And a very special thank you to those organizations and companies that provided their support for this presentation:

and those who sposor our kick-ass group:

Thanks to everyone who came out to support the group and hear my talk!

Andy Hunt's Agile Carolinas Talk

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 28 Apr 2009 14:11:00 -1300

I just got back from hearing Andy Hunt discuss his new book, Pragmatic Thinking and Learning. I hadn't heard him speak before, and I have to admit I was not sure what to expect.

On the one hand, I hold many of this books in very high regard. In particular, the following books were career-altering reads for me:

And his keystone role in the Agile movement has earned my utmost respect. However, when I saw the title of this latest book I was a bit worried. I have a master's degree in cognitive psychology, and I know a lot about learning, memory, and perception. Most learning books outside of the field are crap, so honestly my first instinct was that this was going to be a cash-grab treatise of self-help porch psychology fuzzy-feel-goods for software developers.

After listening to Andy's presentation, I am happy to say my instincts were way off the mark.

First of all, the book (and Andy's presentation) remains true to the pragmatic banner. His recommendations are both practical and effective. For example, a recurring theme during this talk was to write things down. Carry around a small notebook and jot down every idea that pops into your head. Maintain a personal wiki. Mindmap as you read or think. Solidify your thoughts into something concrete. The point is not to be able to refer back to them later, but instead to force your brain to keep working, keep producing, keep processing. To summarize one of his points, that's the first step to an accomplishment.

Second, a lot of what he was saying is supported by academic research. Granted, Andy takes some license with his metaphors but his points hold water. E.g., what Andy refers to as the "memory bus" being shared between "dual cores" of the brain is probably more of an attention effect; however, the behavioral effect cannot be denied - the serial and holistic parts of your mind compete when trying to solve a problem.

Based on the presentation tonight, I wouldn't recommend the book to my former psychology colleges - it's too macro to be useful to them. However, for my fellow geeks, this is actually a useful introduction to becoming a more effective learner.

It was a great talk, a fun evening, and I plan to pick up the book next time I'm at the bookstore. Oh, and I had a short chat with Andy just before the talk, and I have to say how awesome it is to meet someone you hold as a guru and detect no ego. Awesome.

Log4Net Recommended Practices pt 2: Isolating Bugs

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Tue, 21 Apr 2009 04:56:00 -1300

I realize it's been a while since I've written about logging, but my experiences this morning compelled me to share. The muse for this post was a very simple bug located in a rat's nest of complexity.

The code in question drives the I/O layer in a device that is filled with proprietary hardware. There are about three dozen channels of internal communication when this thing is running, which doesn't include any of its networking. So lots of asynchronous I/O and threading. All supporting an awesome graphics and physical interface layer that is equally complex on its own.

Except that today, it was a paperweight. In fact, you couldn't even use the operating system effectively. Eventually task manager would pop up and show that something in the application was suffocating the CPU. I was not looking forward to diagnosing this problem given the level of complexity involved. I expected to spend the morning on it, but in all it took about 10 minutes to find and fix the problem.

I had worked with the application to know what pattern of messages to expect in the log files, but when I opened them I found some of the messages to be missing. Specifically, the messages pertaining to just one of those three dozen communication channels I mentioned earlier. After some initial testing, I had isolated the problem to a case where that channel was misconfigured. I re-ran the application and grabbed the fresh logs.

Since I mirror my class structure in logger objects, isolating the relevant log entries was easy using logparser. The list was surprisingly short, ending with this message:

initiating PSU polling thread

I quickly found this message in my code:

void PollForReports()
{
    Log.Info( "initiating PSU polling thread" );
    while( 1 == Interlocked.Read( ref suspendInterlock ) )
    {
        continue;
    }
 
    RequestSerialNumber();
    Log.Debug( "requested PSU serial number ..." );
        
    // ...

As you can see, a "requested PSU serial number..." log message should have been written almost immediately after the last message I found in the log. Something in the code after the "initiating PSU polling thread" message is written and before the "requested PSU serial number ..." message is locking the CPU.

Do you see it - the improperly implemented spin lock? It's pretty obvious when you know where to look, and logging can make it a lot easier to know where to look.

Delegates and Native API Callbacks - Answer

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Thu, 16 Apr 2009 16:35:00 -1300

A while back I posted a little puzzle about an exception I was hitting after passing a delegate to a native API call. In a nutshell, my application was passing an anonymous delegate to an unmanaged library call:

Result Code result = ExternalLibraryAPI.Open(
    deviceHande,
    delegate( IntPtr handle, int deviceId )
    {
        // ...
    }
);
VerifyResult( result );

The call returns immediately, and the unmanaged library would eventually invoke the callback in response to a user pressing a button on a device. After a semi-random period, pressing the button would yield an exception in my application. The questions I asked were:

What's the exception?
How do you avoid it?

I've waited a bit to post the answers to see if anyone besides Zach would chime in. Zach correctly identified the nut of the problem - the delegate is being garbage collected because there is no outstanding reference on the anonymous delegate once the unmanaged call returns. With no one referencing the delegate object, the garbage collector is free to reclaim it. When the device button is pushed, the native library invokes the callback, which no longer exists in memory. So, to answer the first question, the specific exception that is raised is a CallbackOnCollectedDelegate:

CallbackOnCollectedDelegate was detected.
Message: A callback was made on a garbage collected delegate of type 'Device.Interop!Device.Interop.ButtonPressCallback::Invoke'. This may cause application crashes, corruption and data loss. When passing delegates to unmanaged code, they must be kept alive by the managed application until it is guaranteed that they will never be called.

The verbage in this exception message answers my second question. To avoid the exception, you need to hold a reference to any delegate you pass to unmanaged code for as long as you expect the delegate to be invoked. You need to use an intermediary reference on the delegate to maintain it's life.

Based on this, any of these examples are doomed to fail eventually, because none of them maintain a reference on the delegate object being passed to the unmanaged library:

ExternalLibraryAPI.Open(
    deviceHande,
    delegate( IntPtr handle, int deviceId )
    {
        // ...
    }
);
 
ExternalLibraryAPI.Open(
    deviceHande,
    new ButtonPressCallback( this.OnButtonPress )
);
 
ExternalLibraryAPI.Open(
    deviceHande,
    this.OnButtonPress
);

The correct way to avoid the problem is to hold an explicit reference to the specific delegate instance being passed to unmanaged code:

ButtonPressCallback buttonPressCallback = this.OnButtonPress;
ExternalLibraryAPI.Open(
    deviceHande,
    buttonPressCallback
);
// hold the reference until we're sure no
// further callbacks will be made on the
// delegate, then we can release the
// reference and allow it to be GC'ed
buttonPressCallback = null;

At first I thought Zach's pinning solution was correct; however, you can only pin blittable types, of which delegates are not, so "pinning a delegate" isn't even possible or necessary. If you're interested, the details of how delegates are marshalled across the managed/unmanaged boundary are quite interesting, as I found out from Chris Brumme's blog:

Along the same lines, managed Delegates can be marshaled to unmanaged code, where they are exposed as unmanaged function pointers. Calls on those pointers will perform an unmanaged to managed transition; a change in calling convention; entry into the correct AppDomain; and any necessary argument marshaling. Clearly the unmanaged function pointer must refer to a fixed address. It would be a disaster if the GC were relocating that! This leads many applications to create a pinning handle for the delegate. This is completely unnecessary. The unmanaged function pointer actually refers to a native code stub that we dynamically generate to perform the transition & marshaling. This stub exists in fixed memory outside of the GC heap.

However, the application is responsible for somehow extending the lifetime of the delegate until no more calls will occur from unmanaged code. The lifetime of the native code stub is directly related to the lifetime of the delegate. Once the delegate is collected, subsequent calls via the unmanaged function pointer will crash or otherwise corrupt the process.

Thanks again Zach, and to everyone who reads my blog!

Load-Balancing the Build Farm with CruiseControl.NET

wheresthebeef.nospam@nospam.beefycode.com (jimbo) — Mon, 06 Apr 2009 02:25:00 -1300

Our CI system isn't terribly complicated, thankfully. It evolved from a batch file on a single machine to a farm of VMs running CruiseControl.NET and a single Subversion repository. The triggering mechanism hasn't changed during this transition though: it's a custom beast, consisting of a post-commit hook on our repository logs revisions into a database, and a custom sourcecontrol task is used to poll the database for revisions that haven't been built yet. It works fine and naturally creates a balanced build farm: several VMs can be configured to build the same project, and the database sourcecontrol task prevents more than one VM from building the same revision.

As well as it works, it has some problems. First, the post-commit hook relies heavily on SVN properties, which are a pain to maintain and make it impossible to simply "add a project to the build" without going through a lot of unnecessary configuration. Moreover, the hook is starting to seriously hold up commits, sometimes as long as 20 seconds.

Second, and more irritating, the system builds every individual commit. By that, I mean it builds each and every revision committed to the repository - which may not sound like a bad thing, except that it includes every revision, even those that occurred before the project was added to the CI farm - all the way back to the revision that created the project or branch. I have to manually fudge the database, adding fake build results to prevent the builds from occurring. It's not hard, but it is a pain in the ass. And with the team's overuse of branching I'm finding myself having to fudge more and more.

I'm really trying to move the system towards what "everyone else does," by which I mean trigger builds by polling source control for changes. No more database, no more post-commit hook, no more SVN property configuration, just Subversion and CruiseControl.NET. It would be easy enough to do - simply change our CC.NET project configurations to use the standard SVN source control task. The problem is that without the database, the farm is no longer automagically load-balancing - every VM in the farm would end up building the same source, which defeats the purpose of the farm.

I figured that I could recoup the load-balancing if I had an "edge" server between Subversion and the build farm. This server could monitor source control and when necessary trigger a build on one of the VMs in the build farm. So instead of each farm server determining when a build should occur, there is a single edge server making that decision.

CC.NET ships with the ability to split the build across machines - that is, for a build on one machine (like the edge server) to trigger a build on another machine (like a farm server); however, there is no load-balancing logic available. So I made some of my own...

Edge Server CC.NET Plugin

The edge server plugin operates on a very simple algorithm:

Get a list of farm servers from configuration;
Determine which of farm servers are not currently building the project;
Fail the build if no farm server is available for the project;
Force a build of the project on the first available farm server.

If all you want is the project source code, here it is: ccnet.edgeserver.zip (607.87 kb)

Take a look at the configuration of the plugin; I think it will make the code easier to digest.

Edge Server Configuration

The edge server consists of little more than a source control trigger and a list of farm servers:

<cruisecontrol>
  <project name="MyProject">
    <triggers>
      <intervalTrigger seconds="10" />
    </triggers>
    <sourcecontrol type="svn">
      <trunkUrl>svn://sourcecontrol/Trunk</trunkUrl>
      <autoGetSource>false</autoGetSource>
    </sourcecontrol>
    <labeller type="lastChangeLabeller" prefix="MyProject_"/>
    <tasks>
      <farmBuild>
        <farmServers>
          <farmServer priority="1" uri="tcp://build-vm-1:21234/CruiseManager.rem" />
          <farmServer priority="2" uri="tcp://build-vm-2:21234/CruiseManager.rem" />
          <farmServer priority="3" uri="tcp://build-vm-3:21234/CruiseManager.rem" />
        </farmServers>
      </farmBuild>
    </tasks>
    <publishers>
      <nullTask />
    </publishers>
  </project>
</cruisecontrol>

When CC.NET is run with this configuration, it will monitor the subversion repository for changes to the "MyProject" trunk (lines 6-9); note that since autoGetSource is false, no checkout will occur. The edge server will never have a working copy of the source.

The load-balancing is configured in lines 12-18; in this example, three farm servers are configured in the farm for "MyProject", with build-vm-1 having the highest priority for the build (meaning it will be used first when all three servers are available). When a change is committed to the repository, the edge server will choose one of these servers based on its availability and priority, and then force it to build the project.

Farm Server Configuration

The farm server is configured just as a normal CC.NET build, except for two key differences: first, it is configured with no trigger; second, a remoteProjectLabeller is used to label the build. Here's a sample configuration, with mundane build tasks omitted for brevity:

<cruisecontrol>
  <project name="MyProject">
    
    <triggers/>
    <sourcecontrol type="svn">
      <trunkUrl>svn://sourcecontrol/MyProject/Trunk</trunkUrl>
      <autoGetSource>true</autoGetSource>
    </sourcecontrol>
    <labeller type="remoteProjectLabeller">
      <project>MyProject</project>
      <serverUri>tcp://edgeServer:21234/CruiseManager.rem</serverUri>
    </labeller>
    <tasks>
      <!--
              ... 
      -->
    </tasks>
    <publishers>
      <!--
              ... 
      -->
    </publishers>
  </project>
</cruisecontrol>

Details to note here are:

the labeller points to the edge server to obtain the build label; this is necessary because labels are generated during the build trigger, which on the farm server is always forced and won't include any source revision information;
the project name on the farm server matches exactly the project name on the edge server; this is a convention assumed by the plugin.

Source Code Overview

I need a FarmServer type to support the CC.NET configuration layer:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{
    [ReflectorType( "farmServer" )]
    public class FarmServer
    {
        [ReflectorProperty( "uri" )]
        public string Uri;
 
        [ReflectorProperty( "priority" )]
        public int Priority;
    }
}

No real surprises here. Each FarmServer instance holds a URI to a CC.NET farm server and it's priority in the balance algorithm.

The real meat is in the FarmPublisher class:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{
    [ReflectorType( "farmBuild" )]
    public class FarmPublisher : ITask
    {
        ICruiseManagerFactory factory;
 
        [ReflectorProperty( "Name", Required = false )]
        public string EnforcerName;
 
        [ReflectorHash( "farmServers", "uri", Required = true )]
        public Hashtable FarmServers;
 
        public FarmPublisher() : this( new RemoteCruiseManagerFactory() ) { }
 
        public FarmPublisher( ICruiseManagerFactory factory )
        {
            this.factory = factory;
            this.EnforcerName = Environment.MachineName;
        }
 
        public void Run( IIntegrationResult result )
        {
            // build a list of available farm servers
            //  based off of the plugin configuration
            Dictionary<int, ICruiseManager> servers = new Dictionary<int, ICruiseManager>();
            FindAvailableFarmServers( result, servers );
 
            if( 0 == servers.Count )
            {
                Log.Info( "No servers are available for this project at this time" );
                result.Status = IntegrationStatus.Failure;
                return;
            }
 
            // sort the available servers by priority
            List<int> keys = new List<int>( servers.Keys );
            keys.Sort();
 
            // force a build on the server with the highest 
            //  priority
            ICruiseManager availableServer = servers[ keys[ 0 ] ];
            Log.Info( "forcing build on server ..." );
            availableServer.ForceBuild( result.ProjectName, EnforcerName );
        }
        ...

FarmPublisher is configured with a list of FarmServer objects (lines 20-21). The Run method (starting on line 31) implements the simple load-balancing algorithm:

a list of farm servers which are available to build the project is constructed (lines 33-36);
if no server is available to build the project, the edge server reports a build failure (line 38-43);
the list of available farm servers is sorted by priority (lines 45-47);
the project build is started on the farm server configured with the highest priority (line 53).

Determining a list of available farm servers is pretty straightforward:

void FindAvailableFarmServers( IIntegrationResult result, IDictionary<int, ICruiseManager> servers )
{
    // predicate to locate a server that isn't actively building 
    // the current project
    Predicate<ProjectStatus> predicate = delegate( ProjectStatus prj )
    {
        return IsServerAvailableToBuildProject( result, prj );
    };
 
    // check the status of each configured farm server
    foreach( FarmServer server in FarmServers.Values )
    {
        ICruiseManager manager = null;
        try
        {
            manager = ( ICruiseManager )factory.GetCruiseManager( server.Uri );
 
            // get a local copy of server's current project status snapshot
            List<ProjectStatus> projects = new List<ProjectStatus>( manager.GetProjectStatus() );
            if( null != projects.Find( predicate ) )
            {
                // add the farm server to the list of available servers, 
                //  keyed by its configured priority
                servers[ server.Priority ] = manager;
            }
        }
        catch( Exception e )
        {
            Log.Warning( e );
        }
    }
}

Available servers are saved in the servers dictionary, keyed by their configured priority. The availability of each farm server listed in the task configuration is checked by obtaining the status of the farm server's projects, and passing them to the IsServerAvaialbleToBuildProject method:

bool IsServerAvailableToBuildProject( IIntegrationResult result, ProjectStatus prj )
{
    if( null == prj || null == result )
    {
        return false;
    }
    bool status = (          
        // project name must match
        StringComparer.InvariantCultureIgnoreCase.Equals( result.ProjectName, prj.Name ) &&
 
        // integrator status must be "running"
        prj.Status == ProjectIntegratorState.Running &&                
 
        // build activity must be "sleeping"
        prj.Activity.IsSleeping()
    );
    return status;
}

which simply returns true when:

the farm server configuration contains the project,
the project is currently running, and
the project isn't currently building.

Download

This code is basically spike-quality at this point. I fully expect to throw this away in favor of something better (or get my manager to splurge for TeamCity). There's still a lot of stuff to do. E.g., the algorithm assumes that a farm server is capable of building more than one project at a time - that is, if a farm server is busy building one project, it can still be available to build another concurrently. My assumption is that I'll manage this with the farm server priority configuration. I'd like to leverage the queuing features available in CC.NET; however, I see no way of querying the queue status of a farm server in the CC.NET API. But at least I can start disabling the post-commit hook on our repository.

The project contains the code, a test/demo, and just a few basic unit tests; I'll update the download if/when the project matures. If you use it, or have any suggestions, please let me know in the comments of this post. Enjoy!

ccnet.edgeserver.zip (607.87 kb)