TechStumbler

Loading Remote Assemblies in Powershell with .NET 4

2012-01-10T13:59:00.001-05:00

We recently updated our entire codebase to .NET 4.0 from 3.5. There are plenty of blog posts about that process in general, so I'm going to keep this one specific to our Powershell deployment scripts.

We build our code to a central server and then use Powershell to install those build on our Development server. The upgrade to .NET 4 caused 2 problems with Powershell when we called [Reflection.Assembly]::LoadFile()

First:
Powershell by default runs in .NET 2.0. When we tried to load our new 4.0 assemblies, we got this error:



Exception calling "LoadFile" with "1" argument(s): "Could not load file or assembly 'file://\\buildServer\Application\assembly.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded."

At C:\Scripts\Deployment\Deploy.ps1:XX char:XX

+ $assembly = [Reflection.Assembly]::LoadFile <<<< ($file);     + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException     + FullyQualifiedErrorId : DotNetMethodException

We need to tell Powershell to run in .NET 4.0 mode. To do that we need to create an app.config file for Powershell. In our case the file was created here: C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe.config, but YMMV based on Powershell's location. The new powershell.exe.config file looks like this:



<?xml version="1.0"?> 

<configuration> 

 <startup useLegacyV2RuntimeActivationPolicy="true"> 

  <supportedruntime version="v4.0.30319"/> 

  <supportedruntime version="v2.0.50727"/> 

 </startup>

</configuration>

Now Powershell can run with both 2.0 and 4.0 assemblies.

Second:
The security features are a bit different when running 2.0 vs 4.0. This difference caused a problem when running assemblies from our Build server on our Development server:



Exception calling "LoadFile" with "1" argument(s): "An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for

more information."

At C:\Scripts\Deployment\Deploy.ps1:XX char:XX

+ $assembly = [Reflection.Assembly]::LoadFile <<<< ($file);     + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException     + FullyQualifiedErrorId : DotNetMethodException

Now we simply add the loadFromRemoteSources switch to the powershell.exe.config file we just created so that the entire file looks like this:



<?xml version="1.0"?> 

<configuration> 

 <startup useLegacyV2RuntimeActivationPolicy="true"> 

  <supportedruntime version="v4.0.30319"/> 

  <supportedruntime version="v2.0.50727"/> 

 </startup>

<runtime>

 <loadfromremotesources enabled="true"/>

</runtime>

</configuration>

Now we are successfully loading remote .NET 4.0 assemblies in Powershell. Hope this helps.

Safari User Agent Detection in JavaScript

2011-08-26T09:58:00.003-04:00

Here's a quick snippit for detecting if a browser is Safari version 4 or 5. It works for both desktop and mobile (iPhone, iPod, iPad) versions.



function detectSafari45()

{

  var safariRegEx = /Mozilla\/5\.0 \([^\)]*\) AppleWebKit\/\d+\.\d+(\.\d+)? \(KHTML, like Gecko\) Version\/[45]\.\d+\.\d+( Mobile\/\w*)? Safari\/\d+\.\d+(\.\d+)?/i

  var match = safariRegEx.exec(navigator.userAgent);



  if (match !== null && match.length > 0)

  {

    return true;

  }

  return false;

}

Averaging Timespans in T-SQL

2011-05-10T12:27:00.004-04:00

I'm doing some work with an operations queue. All of the operations are added to a small table on our SQL Server instance. We have a start_datetime and a stop_datetime for each operation. It took me a little time but here's a select statement for profiling the wait times for operations in our queue.



select  

  CONVERT(VARCHAR(13),creation_date,120) as [hour], 

  CONVERT(VARCHAR(8), max(stop_datetime - start_datetime), 108) as MaxWaitTime,

  CONVERT(VARCHAR(8), min(stop_datetime - start_datetime), 108) as MinWaitTime,

  CONVERT(VARCHAR(8), cast(avg(cast(stop_datetime - start_datetime as float)) as datetime), 108) as AvgWaitTime

from 

  [OpQueue].[Op]

where 

  start_datetime > '2011-05-07'

group by 

  CONVERT(VARCHAR(13),start_datetime ,120)

Here are a few things to note.
This statement will spit out the min, max, and average running time for operations in the queue that start in the same hour.
The third argument for convert is pretty handy for DATETIMEs. Here is the page where it is described: http://msdn.microsoft.com/en-us/library/ms187928.aspx.
The AVG() function doesn't work for DATETIMEs, so we need to convert it to a float and then back again to get what we are looking for.

Here's the graph we ended up with.

Seems like it's time to invest in some more processing power.

Running Hudson from OS X: the .war.zip Fiasco

2011-04-22T12:10:00.000-04:00

It took me a while to figure this out and maybe it's just because I'm not a java guy, but I couldn't figure out how to start the Hudson continuous integration tool on my Mac. I downloaded the 2.0.0 version from the Hudson site (http://hudson-ci.org/). The file was hudson-2.0.0.war.zip. The instructions say to run



java -jar hudson-2.0.0.war

to start up the service. I unzipped the file and ended up with a bunch of .class files and no .war. Ah, looks like I also unzipped the .war file. So then I found this nice post (http://superuser.com/questions/159260/in-mac-os-x-how-can-i-unzip-a-zip-file-without-unzipping-its-contents) on how to unzip a file without unzipping its contents. Now I have a



hudson-2.0.0.war/

directory. I ran



$ java -jar hudson-2.0.0.war

Invalid or corrupt jarfile hudson-2.0.0.war

Hmm, not sure what to do next. Somewhere in my search I read that a .war is basically just a zip file. So I tried



$ java -jar hudson-2.0.0.war.zip

on the original file, which worked. I renamed hudson-2.0.0.war.zip



$ mv hudson-2.0.0.war.zip hudson-2.0.0.war

and I was off and running.

So I hope this helps some other java n00bs who may have run into this. The moral of the story is .war == .zip

Moving Replicated FullText Index in SQL Server

2010-12-10T10:58:00.000-05:00

We needed to move the location of our full text index (FTI) on a subscriber. I thought this would be a pain because the replication, but it turns out to be pretty straightforward. Here's the system I am working with:

SQL Server 2005
Transactional Replication with an Updateable Subscriber.

Here are the steps I took:

0. Make sure you have appropriate backups.
1. Point all of the apps to the Publisher because we will need to take the Subscriber offline.
2. On the Subscriber, open the synchronization status by right-clicking on Replication->Local Subscriptions-> ans selecting View Synchronization Status.
3. Stop the Synchronization Service. (This really just pauses updates).
4. On the Subscriber run SELECT name FROM sys.database_files WHERE type_desc = 'FULLTEXT'; to get the name of the FTI.
5. On the Subscriber run ALTER DATABASE [DB_NAME] SET OFFLINE;
6. Move the FTI where to it's new home.
7. On the subscriber run ALTER DATABASE [DB_NAME] MODIFY FILE (Name=[FTI_NAME], Filename = "'new/location/on/disk/');
8. On the Subscriber run ALTER DATABASE [DB_NAME] SET ONLINE;
9. Start the Sync Service in View Synchronization Status.

That should be it. Hope this helps.

T-SQL Query: Tables Without a Primary Keys

2010-11-24T17:29:00.003-05:00

I used this when I was setting up a Transactional Replication system on SQL Server. I needed to figure out which tables did not have primary keys assigned.



select s.name+'.'+ t.name

from

  sys.schemas s

  join sys.tables t on s.schema_id = t.schema_id

  left join sys.indexes i on i.object_id = t.object_id and

                             i.is_primary_key = 1

where i.name is null

order by (s.name+'.'+ t.name) asc;

Great Post on SQL Server Stored Procedure Variables

2010-05-27T12:15:00.000-04:00

It took me a while to find this about using variables in the TOP clause of a MSSQL stored prodedure. This post answered all my questions:

http://sqlserver2000.databases.aspfaq.com/how-do-i-use-a-variable-in-a-top-clause-in-sql-server.html

Here's the best part



CREATE PROCEDURE dbo.getFoo 

  @top INT

 AS 

  BEGIN 

    SET ROWCOUNT @top 



    SELECT foo 

      FROM blat 

      ORDER BY foo DESC 



    -- never forget to set it back to 0! 

    SET ROWCOUNT 0 

  END 

GO

Programmaticly Changing the File Attribute of an App.config File.

2010-03-25T12:57:00.001-04:00

The requirement for this assignment was to use a command line argument to change the settings of a C# console application. I didn't want to compile the settings into the app itself, but to use an app.config file instead. Here's how it works:



public static bool SetEnvironment(string env)

{

  string configFilePath = string.Empty;



  env = env.ToLowerInvariant();

  switch (env)

  {

    case "dev":

       configFilePath = @"Config\Dev.config";

       break;

    case "prod":

       configFilePath = @"Config\Prod.config";

       break;

    default:

       return false;

  }



  try

  {

    System.Configuration.Configuration config =   

       ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);



    AppSettingsSection appSetSec = config.AppSettings;

    appSetSec.File = configFilePath;

    config.Save(ConfigurationSaveMode.Modified);

    ConfigurationManager.RefreshSection("appSettings");

  }

  catch (Exception ex)

  {

    OfflineDemoTool.WriteError(ex.Message);

    return false;

  }



  return true;

}

One thing that I forgot to do was to make sure that the Config\Dev.config and Config\Prod.config files were set to "Copy if newer" in their properties list, otherwise they won't be copied to the output directory.

Most of this comes straight from the MSDN article found here: http://msdn.microsoft.com/en-us/library/system.configuration.appsettingssection.aspx

Windows Commands with Arguments in Powershell

2009-12-09T11:49:00.002-05:00

I have been trying to write a simple PowerShell script that runs (invokes?) MySqlDump.exe for two days. PowerShell syntax for running commands with arguments is not obvious and I wasn't able to find it documented anywhere. While this case is specific to mysqldump, it can be applied to any command line executable with multiple arguments. The trick is to create an array of the arguments and use that as the second "argument" for the ampersand (&) call operator. Here's what my final example looked like:



[string]$pathToExe = "C:\MySQL\MySQL Server 5.1\bin\mysqldump.exe";
[string]$user = "myUser";
[string]$password = "myPass";
[string]$dbName = "myDB";
[Array]$arguments = "-u", $user, "--password=$password", $dbName;

& $pathToExe $arguments | Out-File -FileName "out.sql";

So there you have it. That's how to run an executable with spaces and arguments from PowerShell.

While this article from PowerShell.com didn't answer my questions, I thought it was pretty useful and relevent:

http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chapter-12-command-discovery-and-scriptblocks.aspx

Changing aspnet_wp user in IIS 5.1

2009-11-10T10:59:00.003-05:00

It's possible that no one will ever have to work with IIS 5.1 in XP again, but here you go just in case:

We needed the asp.net worker process (aspnet_wp.exe) to access a share on a remote machine. By default the "Netowrk Service" runs aspnet_wp and that user doesn't have access to remote shares. The way to change this is by changing the "processModel" attribute in your machine.config file (c:\windows\microsoft.net\framework\v2*\config\machine.config).

Mine ended up looking something like this:


<system.web>
  <processmodel username="DOMAIN\username" password="password" autoconfig="true">
  </processmodel>
  ...
</system.web>

Just make sure to restart IIS for the changes to take effect.

Powershell Diff Directory

2009-05-19T12:48:00.003-04:00

I wanted to create a file diff between two different directories. Unfortunately, the Copy-Item CmdLt will not (as far as I know) create the directory structure while copying. This is a bummer. I wanted to keep all of the files in the pipe as opposed to to getting all the files first because of the number of files in the directory structure. Here's what I cam up with


$fromDir = "C:\FromDir";
$toDir = "C:\ToDir";
$diffDir = "C:\DiffDir";


Get-ChildItem -Recurse -Path $fromDir | % {
  if ((Test-Path (Join-Path -Path $toDir -ChildPath ([string]$_.FullName).Replace($fromDir, ""))) -eq $false) 
  {
    if ((Test-Path (Join-Path -Path $diffDir -ChildPath         ([string]$_.Directory).Replace($fromDir, ""))) -eq $false)
    {
      New-Item -Type "Directory" -Path (Join-Path -Path $diffDir -ChildPath ([string]$_.Directory).Replace($fromDir, ""));
    }
    Copy-Item -Recurse -Force $_.FullName -Destination (Join-Path -Path $diffDir -ChildPath ([string]$_.FullName).Replace($fromDir, ""));
  }
};

As always, if anyone know a better way to do this, let me know.

Powershell Rename-Item vs Copy-Item: Changing file names to upper case

2009-04-29T10:39:00.005-04:00

I needed recursively change all file and sub-directory names in a directory from mixed case to upper case. To me, this felt like a problem for PowerShell, but maybe I was wrong. Here was my first attempt:


Get-ChildItem -Recurse -Path "C:\media" | % {Rename-Item $_.FullName -NewName ([string]$_.Name).ToUpper()};

However, I received the following error:


Rename-Item : Source and destination path must be different.

So it looks like PowerShell refused to rename file1 to FILE1, because it thinks that they are the same file. In order to get around this I ended up copying the entire tree, and even that command isn't particularly elegant:


Get-ChildItem -Recurse -Path "C:\media" | % {Copy-Item -Force $_.FullName ([string]$_.FullName).Replace("C:\media", "C:\MEDIA2").ToUpper()};

If there is a better way to do this, I'd love to know.

XPlanner Authentication with NTLM

2008-09-18T13:50:00.003-04:00

I spent most of the day setting up XPlanner and the documentation seemed to be lacking in a few spots. I'll go over the caveats I stumbled on.

The XPlanner install was happening on the lone linux box (Fedora 8) on an NT domain. The box was almost completely empty so I had to install java first. I chose to install the jsdk 1.4.2 even though that version's is EOL'd. XPlanner hasn't been updated in a while (since 2006) and I didn't want to mess w/ a new version of the sdk.

I upacked and installed the "standalone" version of XPlanner 0.7b7. So far so good. I was immediately able to use the default username/password to login.

The hardest part was figuring out how to authenticate based on our NT credentials. Most of the configuration occurs in the xplanner-0.7b7-standalone/webapps/ROOT/WEB-INF/classes/xplanner.properties file.

Here's how I was able to get the NT authentication to work. In the xplanner.properties file, find the authentication strings:


#
# XPlanner security configuration
#
xplanner.security.login[0].module=com.technoetic.xplanner.security.module.XPlannerLoginModule
xplanner.security.login[0].name=XPlanner
xplanner.security.login[0].option.userIdCaseSensitive=true
xplanner.security.login[0].option.debug=true

#xplanner.security.login[1].module=com.technoetic.xplanner.security.module.jndi.JNDILoginModule
#xplanner.security.login[1].name=JNDI
#xplanner.security.login[1].option.userIdCaseSensitive=false
#xplanner.security.login[1].option.debug=true
#xplanner.security.login[1].option.connectionURL=
#xplanner.security.login[1].option.connectionName=cn=
#xplanner.security.login[1].option.connectionPassword=
#xplanner.security.login[1].option.digest=SHA
#xplanner.security.login[1].option.userPattern=
#xplanner.security.login[1].option.userPassword=
#xplanner.security.login[1].option.authentication=simple
#xplanner.security.login[1].option.derefAliases=never
#xplanner.security.login[1].option.userSearch=cn={0}
#xplanner.security.login[1].option.userSubtree=true
#xplanner.security.login[1].option.roleBase=
#xplanner.security.login[1].option.roleName=
#xplanner.security.login[1].option.roleSearch=(uniqueMember={0})

# NTLM login module
#xplanner.security.login[2].module=com.technoetic.xplanner.security.module.ntlm.NtlmLoginModule
#xplanner.security.login[2].name=NTLM
#xplanner.security.login[2].option.userIdCaseSensitive=false
#xplanner.security.login[2].option.domain=DOMAIN
#xplanner.security.login[2].option.controller=CONTROLLER

To use the NTLM module, comment out the first set to security strings, uncomment the NTLM strings, and change the NTLM array index to 0. Here's how my file looked after I finished.


#
# XPlanner security configuration
#
#xplanner.security.login[0].module=com.technoetic.xplanner.security.module.XPlannerLoginModule
#xplanner.security.login[0].name=XPlanner
#xplanner.security.login[0].option.userIdCaseSensitive=true
#xplanner.security.login[0].option.debug=true

#xplanner.security.login[1].module=com.technoetic.xplanner.security.module.jndi.JNDILoginModule
#xplanner.security.login[1].name=JNDI
#xplanner.security.login[1].option.userIdCaseSensitive=false
#xplanner.security.login[1].option.debug=true
#xplanner.security.login[1].option.connectionURL=
#xplanner.security.login[1].option.connectionName=cn=
#xplanner.security.login[1].option.connectionPassword=
#xplanner.security.login[1].option.digest=SHA
#xplanner.security.login[1].option.userPattern=
#xplanner.security.login[1].option.userPassword=
#xplanner.security.login[1].option.authentication=simple
#xplanner.security.login[1].option.derefAliases=never
#xplanner.security.login[1].option.userSearch=cn={0}
#xplanner.security.login[1].option.userSubtree=true
#xplanner.security.login[1].option.roleBase=
#xplanner.security.login[1].option.roleName=
#xplanner.security.login[1].option.roleSearch=(uniqueMember={0})

# NTLM login module
xplanner.security.login[0].module=com.technoetic.xplanner.security.module.ntlm.NtlmLoginModule
xplanner.security.login[0].name=NTLM
xplanner.security.login[0].option.userIdCaseSensitive=false
xplanner.security.login[0].option.domain=DOMAIN
xplanner.security.login[0].option.controller=CONTROLLER

The NTLM module uses the local DB as a fall back, so any logins that you creaed locally that aren't in the Active Directory should still work.

JPEG Marker codes

2008-09-09T16:21:00.002-04:00

I'm doing a bit of JPEG manipulation for a project and it took me longer than I would have liked to find a consise table of JPEG segment markers or frame markers or whatever you want to call them. Here's a C# enum with all the values. Hopefully that will save some typing for you.

public enum JpegMarkers

{

// Start of Frame markers, non-differential, Huffman coding

HuffBaselineDCT = 0xFFC0,

HuffExtSequentialDCT = 0xFFC1,

HuffProgressiveDCT = 0xFFC2,

HuffLosslessSeq = 0xFFC3,

// Start of Frame markers, differential, Huffman coding

HuffDiffSequentialDCT = 0xFFC5,

HuffDiffProgressiveDCT = 0xFFC6,

HuffDiffLosslessSeq = 0xFFC7,

// Start of Frame markers, non-differential, arithmetic coding

ArthBaselineDCT = 0xFFC8,

ArthExtSequentialDCT = 0xFFC9,

ArthProgressiveDCT = 0xFFCA,

ArthLosslessSeq = 0xFFCB,

// Start of Frame markers, differential, arithmetic coding

ArthDiffSequentialDCT = 0xFFCD,

ArthDiffProgressiveDCT = 0xFFCE,

ArthDiffLosslessSeq = 0xFFCF,

// Huffman table spec

HuffmanTableDef = 0xFFC4,

// Arithmetic table spec

ArithmeticTableDef = 0xFFCC,

// Restart Interval termination

RestartIntervalStart = 0xFFD0,

RestartIntervalEnd = 0xFFD7,

// Other markers

StartOfImage = 0xFFD8,

EndOfImage = 0xFFD9,

StartOfScan = 0xFFDA,

QuantTableDef = 0xFFDB,

NumberOfLinesDef = 0xFFDC,

RestartIntervalDef = 0xFFDD,

HierarchProgressionDef = 0xFFDE,

ExpandRefComponents = 0xFFDF,

// App segments

App0 = 0xFFE0,

App1 = 0xFFE1,

App2 = 0xFFE2,

App3 = 0xFFE3,

App4 = 0xFFE4,

App5 = 0xFFE5,

App6 = 0xFFE6,

App7 = 0xFFE7,

App8 = 0xFFE8,

App9 = 0xFFE9,

App10 = 0xFFEA,

App11 = 0xFFEB,

App12 = 0xFFEC,

App13 = 0xFFED,

App14 = 0xFFEE,

App15 = 0xFFEF,

// Jpeg Extensions

JpegExt0 = 0xFFF0,

JpegExt1 = 0xFFF1,

JpegExt2 = 0xFFF2,

JpegExt3 = 0xFFF3,

JpegExt4 = 0xFFF4,

JpegExt5 = 0xFFF5,

JpegExt6 = 0xFFF6,

JpegExt7 = 0xFFF7,

JpegExt8 = 0xFFF8,

JpegExt9 = 0xFFF9,

JpegExtA = 0xFFFA,

JpegExtB = 0xFFFB,

JpegExtC = 0xFFFC,

JpegExtD = 0xFFFD,

// Comments

Comment = 0xFFFE,

// Reserved

ArithTemp = 0xFF01,

ReservedStart = 0xFF02,

ReservedEnd = 0xFFBF

};

eApps and yum

2008-06-03T16:19:00.003-04:00

One of my new projects is hosted on eApps. While they've been great so far, I spent some time trying to get some real developer tools on my box using yum. The first problem was that php was failing with the following error:

PHP Warning: Module 'modulename' already loaded in Unknown on line 0

As it turns out that means that the xls package was included in the binary, and also being declared as a dynamic package. So, to fix this I commented out the following line in the /etc/php.d/xsl.ini:

;extension=xsl.so

Now on to yum itself. Doing any kind of yum update resulted in the following error:

Error: Missing Dependency: glibc-common = 2.3.4-2.36 is needed by package glibc-dummy-centos-4

After doing a bit of research, I found that the dummy-centos-4 package isn't really necessary if you're going to be upgrading your gcc libraries anyways so away it goes.

yum remove glibc-dummy-centos-4

After that I ran a full yum upgrade which worked just fine.

Stripping Accennts From Text With PHP

2008-05-02T20:43:00.002-04:00

Most of the PHP string functions don't really work with UTF-8 encoded strings (http://bugs.php.net/bug.php?id=35520). I found this post on the Symfony forums that had a solution.


$text = iconv('UTF-8', 'ASCII//TRANSLIT', $text);

Done and done.

SQL Join with count

2008-04-14T14:41:00.004-04:00

For the real estate site I've been working on, I needed to get a list of real estate offices and how many active listings they had on the MySQL DB we are using. I had a table of offices and a table of listings. I needed each office represented even if that office had no active listings. It seemed like an easy application of the left join command. Not quite as easy as I thought. Here's my first attempt:


SELECT
COUNT(listings.id), offices.id
FROM offices  LEFT  JOIN  listings ON (offices.id = listings.office_id)
WHERE
listings.status = 1

However, this only returned office/count pair where the count was greater than 0. It seemed like the WHERE condition wasn't being applied correctly, like it was being applied AFTER the tables were joined. I wanted the WHERE condition to be applied to the listings table and then have the results joined to the offices table. So, here's the solution I found: in situations like this, the WHERE clause needed to be added to the ON clause in the LEFT JOIN like this:


SELECT
COUNT(listings.id), offices.id
FROM offices  LEFT  JOIN  listings ON (offices.id = listings.office_id AND listings.status = 1)

Works like a charm.

Dynamic Credentials in Symfony

2008-04-04T11:17:00.004-04:00

For a long time now I've been using dynamic credentials in Symfony as laid out in this code snippit: http://www.symfony-project.org/snippets/snippet/18.

// to put in the actions.class.php file
function getCredential()
{
  $this->post = $this->_retrievePost(); // retrieving the object based on the request parameters
  if ($this->getUser()->isOwnerOf($this->post))
    $this->getUser()->addCredential('owner');
  else
    $this->getUser()->removeCredential('owner');

  // the hijack is over, let the normal flow continue:
  return parent::getCredential();
}

However, it seems like this isn't really the proper way to add dynamic credentials. The getCredential() method is called by the Symfony FilterChain to discover what credentials that module requires, not to add credentials to the user.

For me, the problem cam to a head when I was trying to show a link if a user had a particular credential. The problem was that the page with the link was not a secure page. Since the page was not secute (is_secure: off in the config/security.yml file) the getCretential() method was never called.

The solution is to use the preExecute() method instead of getCredential()

// to put in the actions.class.php file
function preExecute()
{
  $this->post = $this->_retrievePost(); // retrieving the object based on the request parameters
  if ($this->getUser()->isOwnerOf($this->post))
    $this->getUser()->addCredential('owner');
  else
    $this->getUser()->removeCredential('owner');

  // the hijack is over, let the normal flow continue:
  return parent::getCredential();
}

Accent mode in Emacs 22

2008-03-06T16:34:00.000-05:00

For a long time I used the emacs iso-accents-mode for my editing in Spanish. Looks like this has finally been removed from the standard emacs distros, so I had to figure out whatever they replaced it with. Here's what I (eventually) found: To get the same functionality as the old iso-accents-mode, you need to change the input mode. The official doc can be found here, but here are the basic commands.


M-x list-input-methods


C-x RETURN C-\ method RETURN

This is how to select the input method.

C-\

Toggles the input mode on/off. Pretty useful.

The mode I use is latin-1-prefix. It should be familiar to the iso-accents-mode followers.

Loading NetFlix Prize Data into MySQL

2008-03-06T09:40:00.002-05:00

I may be a year late, but I decided to enter into the NetFlix Prize contest the other day. I've already got MySQL (Lampp) running on my box (Fedora 8) so I decided to load the data into my DB to do some basic analytics. There's some good info out there, but nothing that encompassed all the steps I took, so I'm going to repeat my process here.

1. Enter the Contest and download the dataset. Nothing too complicated here, you just need a unique team name and an email address.

2. Get Ruby where it needs to be. For this I used yum to install both the ruby and ruby-devel packages.

2a. Then you need to download and install Ruby/MySQL. Here are the basic steps from http://lafcadio.rubyforge.org/manual/ch02s02.html


wget http://tmtm.org/downloads/mysql/ruby/mysql-ruby-2.7.tar.gz
tar zxvf mysql-ruby-2.7.tar.gz
cd mysql-ruby-2.7
ruby extconf.rb --with-mysql-config
make
ruby test.rb [hostname] [username] [dbpassword]
sudo make install

2b. After that is all set and you can connect to the MySQL w/ the test.rb script, download the ruby DBI from here: http://www.kitebird.com/articles/ruby-dbi.html#TOC_3


wget http://rubyforge.org/frs/download.php/12368/dbi-0.1.1.tar.gz
tar zxvf dbi-0.1.1.tar.gz
ruby setup.rb config --with=dbi,dbd_mysql
ruby setup.rb setup
sudo ruby setup.rb install

In the config command, I chose to only include the MySQL packages of the DBI. I think it's generally better to only include the packages you want as opposed to always including everything. If you don't specify which DB packages you want, the installer will assume you want everything. If you don't have all of the required development packages already on your machine (ie oracle and sybase) you'll get into errors when you run the setup command.

3. Now find a ruby script to load the data. I used this one from juretta:


#!/usr/bin/ruby
#
#  Created by Stefan Saasen on 2007-04-24.
#  Copyright (c) 2007. All rights reserved.
require 'dbi'         

# Configuration
DB_DATABASE   = 'netflix'
DB_USER       = 'user'
DB_PASSWORD   = '******'
           
# Directory with netflix dataset
NETFLIX_DOWNLOAD_DIR = File.join("/sata_ext/net_flix")

NETFLIX_MOVIE_TITLES = File.join(NETFLIX_DOWNLOAD_DIR, "movie_titles.txt")
NETFLIX_TRAINING_SET = File.join(NETFLIX_DOWNLOAD_DIR, "training_set")

# Profile...
start = Time.now

# Connect to the MySQL database
DBI.connect("DBI:Mysql:#{DB_DATABASE}", DB_USER, DB_PASSWORD) do |dbh|
# create the necessary tables
stmts = DATA.read.split(";").delete_if{|stmt| stmt.strip.empty?}
stmts.each do |stmt|
  # Execute the current SQL statement
  dbh.do(stmt)
end

# 1. Insert movie titles into movie_titles table using simple INSERT STATEMENTS
sql = "INSERT INTO movie_titles (id, year_of_release, title) VALUES(?,?,?)"
# It would be even more efficient to use INSERT statements that
# use VALUES syntax which can insert multiple rows.
# e.g. INSERT INTO movie_titles (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
dbh.prepare(sql) do |sth|
  File.open(NETFLIX_MOVIE_TITLES) do |f|
     f.each do |line|
       # Execute the prepared INSERT Statement
       id, year_of_release, title = line.split(",")
       sth.execute id.to_i, year_of_release.strip, title.strip
     end
  end
end

# 2. Insert rating data from training_set using the fast LOAD DATA INFILE syntax
# The filename contains the movie id!
sql = "LOAD DATA INFILE ? INTO TABLE ratings FIELDS TERMINATED BY ',' "+
      "IGNORE 1 LINES (customer_id, rating, date) SET movie_id = ?;"
dbh.prepare(sql) do |sth|
  Dir[NETFLIX_TRAINING_SET + "/*.txt"].each do |trs_file|
    puts "Importing #{trs_file}..."
    if trs_file =~ /([0-9]+)\.txt$/
      movie_id = $1.dup.to_i
    else
      raise "Missing movie_id (file: #{trs_file})"
    end
    # Execute the prepared statement using the .txt file and the movie_id
    sth.execute File.expand_path(trs_file), movie_id
  end
end

# Add index for certain columns
#puts "Creating index on the ratings table - may take a while"
#["CREATE INDEX mid_idx ON ratings (...)", "CREATE INDEX cid_idx ON ..."].each{|stmt| dbh.do(stmt)}           
end
puts "Import successfully finished!\n"
puts "The netflix data import took #{sprintf("%0.2f", (Time.now - start)/60.0)} minutes"

__END__
DROP DATABASE IF EXISTS netflix;
CREATE DATABASE netflix /*!40100 DEFAULT CHARACTER SET latin1 */;
USE netflix;

DROP TABLE IF EXISTS `movie_titles`;
CREATE TABLE `movie_titles` (
`id` int(11) NOT NULL,
`year_of_release` YEAR(4) default NULL,
`title` VARCHAR(250),
PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS `ratings`;
CREATE TABLE `ratings` (
`id` int(11) NOT NULL auto_increment,
`movie_id` int(11) NOT NULL,
`customer_id` int(11) NOT NULL,
`date` date default NULL,
`rating` int(1),
PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

The last part there are the SQL commands to create the tables needed for this script.

I loaded all of the data onto my machine (Dual Intel 2.13 GHz w/ 1GB RAM) in about 13 minutes.