Scott Weinstein on .Net, Linq, PowerShell, WPF, and WCF

When Add(ing)-Type, choose your method signatures wisely

Scott Weinstein — Fri, 08 May 2009 18:44:18 GMT

Powershell V2 has some great new features, in particular Add-Type and Remoting features are likely to be quite popular and work together without much issue. That said, there are edge cases which illustrate how the types returned from remoting calls. The following script demonstrates the issue

$csCode = @"
using System;
using System.Collections.Generic;
using System.Linq;
namespace Demo {
    public static class D
    {
        public static int Add(int a, int b)
        {
            return a + b;
        }
        public static int AddArray(int[] ints)
        {
            return ints.Sum();
        }
        public static int AddEnumerable(IEnumerable<object> ints)
        {
            return ints.Cast<int>().Sum();
        }
        public static int AddEnumerable(IEnumerable<int> ints)
        {
            return ints.Sum();
        }
    }
}
"@

Add-Type -TypeDefinition $csCode -Language CSharpVersion3
$oneRemote = Invoke-Command -ComputerName localhost  -ScriptBlock { return 1 }
$listRemote = Invoke-Command -ComputerName localhost  -ScriptBlock { return (1,2,3) }
$one = &{return 1}
if ($one -eq $oneRemote)
{
    Write-Host "1 == 1"
}

$v =  [Demo.D]::Add($one,$oneRemote)
Write-output "One + OneRemote = $v"  ; $v=$null
$v = [Demo.D]::AddArray(($one,$oneRemote))
Write-output "One + OneRemote via array =  $v" ; $v=$null

$v = [Demo.D]::AddArray($listRemote)
Write-output "One + OneRemote via remote array =  $v" ; $v=$null

$v = [Demo.D]::AddEnumerable((1,2,3,4))
Write-output "One + OneRemote via local IEnumerable =  $v" ; $v=$null

$v = [Demo.D]::AddEnumerable($listRemote)
Write-output "One + OneRemote via remote IEnumerable =  $v"; $v=$null



$oneRemote | Get-Member

The output from the above is

1 == 1
One + OneRemote = 2
One + OneRemote via array = 2
One + OneRemote via remote array = 6
One + OneRemote via local IEnumerable = 10 # so far as expected

Exception calling "AddEnumerable" with "1" argument(s): "Specified cast is not valid."
At typeDemo.ps1:49 char:29
+ $v = [Demo.D]::AddEnumerable <<<< ($listRemote)

   TypeName: System.Int32

Name               MemberType   Definition
----               ----------   ----------
CompareTo          Method       System.Int32 CompareTo(Object value)
Equals             Method       System.Boolean Equals(Object obj), System.Boolean Equals(Int32 obj)
GetHashCode        Method       System.Int32 GetHashCode()
GetType            Method       System.Type GetType()
GetTypeCode        Method       System.TypeCode GetTypeCode()
ToString           Method       System.String ToString()
PSComputerName     NoteProperty System.String PSComputerName=localhost
PSShowComputerName NoteProperty System.Boolean PSShowComputerName=True
RunspaceId         NoteProperty System.Guid RunspaceId=e0dc5c05-c87d-41ad-afe0-16bc1b711f08

What’s happening under the covers is that the PowerShell reporting infrastructure is returning a PSObject. By inspecting the type via Get-Member you can see that it has some extra NoteProperties. To PowerShell and .Net methods that expect integers and arrays of integers, the object looks and behaves like it should. But if your Add-Types use a more LINQ style approach, which expects an IEnumerable<T>, the PowerShell type system doesn’t properly convert the adapted type to its native underlying type and runtime exceptions are the result.

Some interesting sessions at the Microsoft Enterprise Developer and Solutions Conference

Scott Weinstein — Mon, 04 May 2009 17:13:55 GMT

There are some interesting sessions at the 2009 EntDevCon.

Three of particular interest to me are are

1. “Coral8 Engine Integrated with Microsoft Windows HPC Server 2008” at 11 on Tuesday.

I had a chance to see some of the early bits, and for anyone interesting in streaming data with high computations needs, this promises to be well worth your time. In a future post I’ll talk about how this technology can be used.

Additionally, some co-workers at Lab49 will be presenting on Wednesday:

2. Marc Jacobs and Citi on “Agile Development using Low-Fidelity Prototypes, Expression Blend and SketchFlow:A Case Study”

3. Daniel Simon on “Patterns and practices:composite development for WPF and Silverlight using Composite Application Guidance (a.k.a. Prism 2.0)”

And if I have time, I might have a few hours at the Lab49 booth, stop by to say hi.

CPU Monitoring and Alerting via Performance counters, Coral8, and PowerShell

Scott Weinstein — Wed, 22 Apr 2009 03:58:09 GMT

The PowerShell team has a short post on using V2 cmdlets to Monitor performance counters.

Building on that, and the prior work with the PoShAdapter, here’s a sample with Coral8 and PowerShell to alert you via SMS when the rolling 30 seconds average CPU hits a threshold.

create VARIABLE float CPUThreashold  = 70; 
create OUTPUT STREAM fakeout SCHEMA (x string);
CREATE SCHEMA CPUSchema (cpu FLOAT);
create INPUT STREAM inCPU SCHEMA CPUSchema;
create OUTPUT STREAM highCPU SCHEMA CPUSchema;
create local STREAM avgCPU SCHEMA CPUSchema;

ATTACH OUTPUT ADAPTER smsAdapter TYPE PoShAdapter TO STREAM highCPU
PROPERTIES
    BEGINBLOCK  = [[ 
    Add-Type -AssemblyName System.Security   
    Add-Type -Path "C:\Program Files\Coral8\Server\bin\GmailHelper.dll" 
    $pass = ".."
    ]],
PROCESSBlock     = [[
 foreach ($cpuSpike in $input)
 {
  $m = "CPU spike at {0:0.00}" -f $cpuSpike['cpu']
  [RC.Gmail.GmailMessage]::SendFromGmail("user", $pass,"6465555555@tmomail.net", 
   "Via Powershell/Coral8",$m)
 }
]];

ATTACH OUTPUT ADAPTER CPUAdapter TYPE PoShAdapter TO STREAM fakeout
PROPERTIES
    RESULTSSTREAM = "ccl://localhost:6789/Stream/Default/MonitorCPU/inCPU",
    INPUTBLOCK    = [[
Get-Counter '\Processor(*)\% Processor Time'    |
    select -expand CounterSamples               |
        ? { $_.InstanceName -eq "_total" }      | 
            % { ,,($_.CookedValue,0) } # 
]];

INSERT into avgCPU
select avg(cpu)  
from inCPU 
keep 30 SECONDS 
OUTPUT EVERY 30 SECONDS;

INSERT into highCPU 
select cpu 
from avgCPU 
WHERE cpu > CPUThreashold;

Yields:

*I lowered the threshold for testing
** Yes, I should check my voice mail

CEP, Citi, and Coral8

Scott Weinstein — Tue, 21 Apr 2009 03:36:16 GMT

The system I helped build at my previous job gets a nice write-up in Wall Street & Technology:

Citi Taps CEP for Analyzing Equity Data

I’m sure Marc, Hanno and the rest of the team are pleased.

Congrats guys!

Using Coral8 and PowerShell to receive eventing data

Scott Weinstein — Mon, 20 Apr 2009 02:54:31 GMT

I’ve updated the PowerShell Coral8 adapter so it can be used to receive input. Rather than demonstrate input from a database or an RSS stream, both of which can are supported by the native Coral8 adapters, I have a demo with WMI events. In this case allowing us to monitor process creation across a network.

Using the same general template as before we have the following two script blocks

BeginBlock

$computers = &{$args} localhost ccs01 ccs02
$jobs = $computers | % `
 {
   Register-WmiEvent -Class Win32_ProcessStartTrace -ComputerName $_ ` 
   -Action  { `
            $res = @{}
            $res['ProcessName'] = $args[1].NewEvent.ProcessName
            $res['ComputerName'] = $args[0].scope.Path.Server
            return $res
        }
    }

InputBlock

while ($true) 
{
    Start-Sleep -Milliseconds 100
    foreach ($p in (Receive-Job $jobs))
    {
        # Input adapter doesn't support hashtables yet
        ,,($p['ProcessName'],$p['ComputerName']) 
    }
}

And the payoff

Stream transforms in Coral8 via .Net

Scott Weinstein — Fri, 17 Apr 2009 03:55:00 GMT

In the first post on integrating PowerShell and Coral8 I showed how to create a message sink. In C#ish pseudocode we did the following:

OutputStream<T> => Action<T>

where the Action<T> was a PowerShell block to send a message via GMail.

Today the goal is to do a transform, something along the following

OutputStream<T> => Func<T,T2> => InputStream<T2>

In this example, T will be a pair of stock symbols, T2 will be a pair stock symbols along with the correlation of their log normal closing prices for the past 10 days.

let’s get started.

Coral8

CREATE SCHEMA           StockPairSchema( SecA STRING,SecB STRING );
CREATE SCHEMA           StockPairCorrelationSchema INHERITS from StockPairSchema (CORR  FLOAT) ;
CREATE OUTPUT   STREAM  StockPairs          SCHEMA StockPairSchema;
CREATE INPUT    STREAM  StockCorrelations   SCHEMA StockPairCorrelationSchema;
ATTACH OUTPUT ADAPTER PairstoCorrelationsFunc TYPE PoShAdapter TO STREAM StockPairs
PROPERTIES
RESULTSSTREAM = "ccl://localhost:6789/Stream/Default/TestC8/StockCorrelations",

The process block is very simple:

foreach ($t in $input)
{
    $a = Get-LogReturns $t["SecA"]
    $b = Get-LogReturns $t["SecB"]
    $c = [Demo.Stats]::Correlate($a, $b)
    ,,($t["SecA"],$t["SecB"],$c) #// double commas so the values are not flattened
}

The block path defines the Get-LogReturns and Correlate function, so naturally it’s a bit longer.
Get-LogReturns:

$wc = New-Object Net.WebClient    
function Get-LogReturns ($sec)
{
    $qry = "http://ichart.finance.yahoo.com/table.csv?s=$sec"
    $secAData = ConvertFrom-Csv $wc.DownloadString($qry) | select  -First 10 | % { $_.'Adj Close' }
    for ($i=0; $i -lt $secAData.Count-1; $i++) { [Math]::Log( $secAData[$i] / $secAData[$i+1] )  }
}

[Demo.Stats]::Correlate

$csCode = @"
using System;
using System.Collections.Generic;
using System.Linq;
namespace Demo 
{
    public static class Stats
    {
        private static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(IList<TFirst> first, 
				IList<TSecond> second, 
				Func<TFirst, TSecond, TResult> func)
        {
            for (int ii = 0; ii < Math.Min(first.Count(),second.Count()); ii++)
                yield return func(first[ii],second[ii]);
        }
        public static double Correlate(object[] s1,object[] s2)
        {
            return Correlate(s1.Cast<double>(), s2.Cast<double>());
        }
        public static double Correlate(IEnumerable<double> s1,IEnumerable<double> s2)
        {
            var sum1 = s1.Sum();
            var sum2 = s2.Sum();
            var sumSq1 = s1.Select(v=> v*v).Sum();
            var sumSq2 = s2.Select(v=> v*v).Sum();
            var pSum = Zip(s1.ToList(),s2.ToList(), ( a, b) => a*b).Sum();
            var len = s1.Count();
            var num = pSum - ((sum1 * sum2) / len);
            var denom = Math.Sqrt(((sumSq1 - (sum1 * sum1) / len) * (sumSq2 - (sum2 * sum2) / len)));
            return (denom == 0.0) ? 0 : num / denom;
        }
    }
}
"@
Add-Type -TypeDefinition $csCode -Language CSharpVersion3 -PassThru

And in case you missed it, the correlate function isn't in PowerShell at all, but rather coded up via in-line C# code, compiled at the startup of the adaptor, and running in the Coral8 server process.

Automatic properties in PowerShell… and how you can almost have them for your custom types

Scott Weinstein — Tue, 14 Apr 2009 03:37:00 GMT

For certain types, PowerShell has automatic properties, by this I mean PowerShell is able to inspect the object and then expose properties where normally you would need to use a string indexer.

Like so:

$xbooks = [xml]"<Books><Book><Title>PowerShell in Action</Title></Book></Books>" 
$xbooks.Books.Book.Title

Similar functionality is available for COM, WMI, ADO.Net, and AD types. It’s one of the core features of PowerShell, and its simplicity and elegance makes even diehard UNIX types sit up and take notice.

But if you’re writing your own cmdlet, navigation provider, or are hosting PowerShell, and you wish to provide an adapter for your own type – you will quickly find that adapters and rest of the infrastructure for automatic properties are internal classes and unavailable to you.

The best option you have is something along the following:

public PSObject CreateCustomPSObject(MyType mytype )
{
    PSObject pso = new PSObject(mytype);
    foreach (string name in MyColumnNames)     
    { 
        string getter = String.Format("$this[\"{0}\"]", name); 
        string setter = String.Format("$this[\"{0}\"] = $value", name); 
        pso.Properties.Add(new PSScriptProperty(name,                                        InvokeCommand.NewScriptBlock(getter),                                InvokeCommand.NewScriptBlock(setter))); 
    
    } 
    return pso; 
}
WriteItemObject(CreateCustomPSObject(mytype));

This will work, but the output from get-member won’t be as clean as it is for a DataRow or other adapted types. Even worse, this option isn’t available at all for hosted scenarios as there no instance of CommandInvocationIntrinsics available from a Runspace or Pipleline.

In-process PowerShell adapter for Coral8

Scott Weinstein — Sun, 12 Apr 2009 19:40:00 GMT

A few days ago a thought crept into my head; wouldn’t it be nice if I could have a Coral8 in-process adapter to run PowerShell code? If it worked it could potentially be a “universal” adapter, limiting the need for custom one-off adapters.

Coral8, out-of-the-box, supports two different types of adapters. In-process adapters which must be written in C (enough said?), and out-of-process adapters which can be written in the language of your choice, but then have startup sequence, configuration, and deployment work associated with them.

Using the PoShAdapter I’ve been developing, let’s take a look at the adapter as a message sink. In this case a “sent to Gmail” adapter.

CREATE SCHEMA TSchema ( toAddr STRING , subject string, message as string); 
CREATE OUTPUT STREAM outGmail SCHEMA TSchema; 
ATTACH OUTPUT ADAPTER PoShAdapter TYPE PoShAdapter  TO STREAM outGmail  PROPERTIES          
BEGINBLOCK  = [[   
    Add-Type -AssemblyName System.Security     
    Add-Type -Path "C:\Program Files\Coral8\Server\bin\GmailHelper.dll"   
    $encPassword = "AQQbRXQr…kg=="   
    $password =[Text.Encoding]::UTF8.GetString([Security.Cryptography.ProtectedData]::UnProtect([System.Convert]::FromBase64String($encPassword),$null,"CurrentUser"))
]],
PROCESS     = [[             
    foreach ($t in $input)             
    {                  
        [RC.Gmail.GmailMessage]::SendFromGmail("from.user", $password,
                       $t["toAddr"], $t["subject"], $t[“message”])
    }
    ]];

The posh adapter, like PowerShell scripts and functions have begin, process, and end blocks.

In the beginblock we load the assemblies for System.Security and GmailHelper, as well as decrypt the gmail password.

The process block will have as its $input set to a collection of the Coral8 tuples, so it simply loops through them sends out an email for each one. Because we the Runspace is kept for the lifetime of the adapter, context from the beginblock is available in the process block.

In later posts I’ll explore using the PoShAdapter to transform data (as in a select or map operator), and look into the possibility of using the adapter to listen for WMI or other system events.

Adding var args support to office integration functions with dynamic lambda expressions

Scott Weinstein — Wed, 11 Mar 2009 00:00:00 GMT

Anyone who’s ever done office integration has come across a method definition like so:

object Run(object Macro, object Arg1, object Arg2, …, object Arg30);

if you're working in VB it’s not such an issue, and rumor has it that it’s that C# 4 will also solve the problem of code such as:

Run(“…”, Type.Missing, Type.Missing,/*28 more times*/ …, Type.Missing);

But for everyone else, you can avoid the extra typing with an extension method and dynamic lambda expressions. For example with:

    public static class OfficeHelpers
    {
        private static MemberInfo missingMemberInfo = typeof(Type).GetMembers().Where(mi => mi.Name == "Missing").Single();
        private static MethodInfo runMethodInfo = typeof(_Application).GetMethod("Run");
        private static ParameterExpression macroNameParamExp = Expression.Parameter(typeof(string), "macroName");
        private static ParameterExpression argsParamExp = Expression.Parameter(typeof(object[]), "args");

        public static object Run(this Application App, string macroName, params object[] args)
        {
            var argsExprs = GenerateArgs(runMethodInfo, macroName, args);
            var callExpr = Expression.Call(Expression.Constant(App), runMethodInfo, argsExprs);
            var paramExprs = new ParameterExpression[] { macroNameParamExp, argsParamExp };
            var lambda = Expression.Lambda>(callExpr, paramExprs);
            return lambda.Compile().Invoke(macroName, args);
        }

        private static IEnumerable<Expression> GenerateArgs(MethodInfo runfuncMI, string macroName, object[] args)
        {
            yield return macroNameParamExp;
            for (int ii = 0; ii < args.Length; ii++)
                yield return BinaryExpression.ArrayIndex(argsParamExp, Expression.Constant(ii, typeof(int)));
            for (int ii = 0; ii < runfuncMI.GetParameters().Length - args.Length - 1; ii++)
                yield return Expression.MakeMemberAccess(null, missingMemberInfo);
        }

a call to

OfficeHelpers.Run("…", arg1,arg2,arg3);

will generate and execute an expression like so:

(macroName, args) => RunFuncTest(macroName, args[0], args[1], args[2], Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing)

Linq2Kdb+ Provider - source code update

Scott Weinstein — Wed, 04 Mar 2009 04:09:40 GMT

A few months ago, over the winter holiday, I put together a Linq provider for the Kdb+ database. The folks at KX systems have graciously provided a place to host the code, which you can find at https://code.kx.com/svn/contrib/sweinstein/linq2kdbplus. Use anonymous/anonymous for access.

As of now the list of the unimplemented items is considerably longer than implemented features.

What works

Basic projections / Selects (map)
Basic restrictions / Wheres (filter)
code generation to create strongly typed representations of a Kdb database

Still remaining to implemented

Joins
Orderbys
Grouping
Aggregates and other functions/method calls
Kdb+ columns with array types
Code generation support for Kdb+ user functions

A few notes on the code

The heavy lifting of transforming the expression trees done via IQToolkit

The unit tests uses xUnit.Net

The code generation uses T4 and requires the T4 Toolbox

run SetupCodeGen.ps1 to have the correct paths placed in the registry

Linq2Kdb+ was developed on .Net 3.5SP1 and Kdb+ 2.5 - for other versions YMMV.

Feedback welcome.

Dependency Injection for end-of-lifecycle clean up

Scott Weinstein — Tue, 03 Mar 2009 03:03:43 GMT

The benefits of using an inversion of control container for simplifying the task of initializing an object graph have been well covered. But the benefits of using an IoC container for simplifying object cleanup are not as well publicized, and is a feature that I've grown to really like taking advantage of.

The other night, at an Alt.Net Meetup on dependency injection/inversion of control I found myself being only person in the room enthusiastic about this.

The core of the problem is that objects that require cleanup, (ie implement IDisposable) can infect object hierarchies - any class with a has-a relationship to the IDisposable class really needs implement IDisposable itself. If you're not careful, you can easily end up with tens of classes that have the nearly-all-ceremony Dispose() methods that do nothing but cleanup their children.

The cleanest solution for this problem is to move the member level containment to method level and wrap the use of the disposable in a using statement. But when that isn't a viable option the next best option is to move to dependency injection - if your class didn't create a disposable object, it can't be responsible for cleaning it up.

Of course, at some point the objects do need to be disposed of, in the simplest way to do that is to use a container, such as autofac, which supports both nested containers and object cleanup.

Catalog of User Experience Patterns

Scott Weinstein — Tue, 03 Feb 2009 13:22:00 GMT

Via Brad Abrams, check out this great site: Quince: a catalog of UX design patterns.

A Linq2Kdb+ Query provider

Scott Weinstein — Sun, 04 Jan 2009 10:13:00 GMT

Over the winter break I wanted to learn more about a few technologies:

T4 - the built into Visual Studio code generation tool
The Kdb+ columnar database
Expression trees

The result ended up in a Linq to Kdb+ query provider. The following snippet translates to

using (IConnection connection = new Connection("localhost", 5001))
{
    var tkc = new TestKdbContext(connection,Console.Out);
    connection.Run(@"\l sp.q");
    IEnumerable<string> names = from row in tkc.sRecord
           where row.status == 20 && row.city == GetCity()
           select row.name;
}

which ends up being transformed to something like this Q function

{?[s;enlist (&;(=;`status;20);(=;`city;enlist x));();(enlist `name)!enlist `name]}

My thoughts -

T4, combined with the T4 toolbox proved to be a fine alternative to CodeSmith. With it I was able to generate some strongly typed classes which mapped to a Kdb tables, and also create a strongly typed KB query context.
Kdb+/Q is certainly an impressive technology. It combines a columnar database server and complete functional and query language in a 156kb executable. That said, getting proficient in reading Q code seems like it'll take five times as long as it will to learn to write it.
Expression trees are definitely fun to play with, there's tons of opportunity in compiled lambdas for custom filtering and fine-grained entitlement checks. Creating the query provider was made possible (in the time I had) with the IQToolkit and this guide.

Update: Source code is now available

Creating high performance WCF services

Scott Weinstein — Sun, 04 Jan 2009 01:14:00 GMT

I had a WCF service where I wanted to be able to support over a hundred concurrent users, and while most of the service methods had small payloads which returned quickly, the startup sequence needed to pull down 200,000 records. The out of the box WCF service had no ability to support this scenario, but with some effort I was able to squeeze orders of magnitude performance increases out of the service and hit the performance goal.

Initially performance was abysmal and there was talk of ditching WCF entirely ( and as the one pushing WCF technology on the project this didn't seem like a career enhancing change )

Here's how performance was optimized. These are listed in the order they were implemented. Some are fairly obvious, others took some time to discover. Each item represents, a significant increase in latency or scalability from the prior - and although I have internal measurement numbers, I'm not comfortable publishing them as the size of the data increased, and the testing approach changed.

Use NetTCP binding
This helps both throughput and the time it takes to open and close connections
Use DataContract Serializer instead of XMLSerializer
I started out using DataTables - POCO objects via Linq2Sql yielded a 6x increase
slow: [OperationContract] MyDataTable GetData(...);
fast: [OperationContract] IEnumerable<MyData> GetData(...);
Unthrottle your service
It's quite understanable that WCF is resistant to Denial of Service attacks out of the box, but it's too bad that it's is such a manual operation to hit the "turbo button". It would be nice if the Visual Studio tooling did this for you, or at least had some guidance (MS - hint, hint)

The items to look at here are:
1. <serviceBehaviors><serviceThrottling ...> set the max values high
2. <dataContractSerializer maxItemsInObjectGraph="2147483647" />
3. and under <netTcpBinding> setting the listenBacklog, maxConnections, and maxBuffer* value high
Cache your data
WCF, unlike ASP.Net has no built in facility to cache service responses, so you need to do it by hand. Any cache class will do.

Normalize/compress your data
this doesn't necessarily have to be done in the database, the Linq GroupBy operators make this easy to do in code. To clarify, say your data is kept in a denormalized table

string	Key1
string	Key2
string	Key3
int	val1
int	val2

the bulk of the result set ends up being duplicate data

LongKeyVal1	LongKeyVal2	LongKeyVal3	10	12
LongKeyVal1	LongKeyVal2	LongKeyVal3	11	122
LongKeyVal1	LongKeyVal2	LongKeyVal3	12	212

so normalize this into

LongKeyVal1

LongKeyVal2

LongKeyVal3

10 12

11 122

12 212

In code, given the following classes

public class MyDataDenormalized
{
    public string Key1 { get; set; }
    public string Key2 { get; set; }
    public string Key3 { get; set; }
    public int Val1 { get; set; }
    public int Val2 { get; set; }
}
public class MyDataGroup
{
    public string Key1 { get; set; }
    public string Key2 { get; set; }
    public string Key3 { get; set; }
    public MyDataItem[] Values { get; set; }
}
public class MyDataItem
{
    public int Val1 { get; set; }
    public int Val2 { get; set; }
}

you can transform an IEnumerable<MyDataDenormalized> into a IEnumerable<MyDataGroup> via the following

var keyed = from sourceItem in source
           group sourceItem by new
           {
               sourceItem.Key1,
               sourceItem.Key2,
               sourceItem.Key3,
           } into g
           select g;
var groupedList = from kItems in keyed
              let newValues = (from sourceItem in kItems select new MyDataItem() { Val1 = sourceItem.Val1, Val2= sourceItem.Val2 }).ToArray()
              select new MyDataGroup()
              {
                  Key1 = kItems.Key.Key1,
                  Key2 = kItems.Key.Key2,
                  Key3 = kItems.Key.Key3,
                  Values = newValues,
              };

Use the BinaryFormatter, and cache your serializations
If you're willing to forgo over the wire type safety, the binary formatter is the way to go for scalability. Data caching has only a limited impact if a significant amount of CPU time is spent serializing it - which is exactly what happens with the DataContract serializer.

The operation contract changes to

[OperationContract]
Byte[] GetData(...);

and the implementation to

var bf = new BinaryFormatter();
using (var ms = new MemoryStream())
{
    bf.Serialize(ms, groupeList);
    // and best to cache it too
    return ms.GetBuffer();
}

Before items 4,5, and 6 the service would max out at about 50 clients ( response time to go way up and CPU usage would hit 80% - on a 8 core box). After these changes were made, the service could handle of 100 + clients and CPU usage flattened out at 30%

Update: Shay Jacoby has reasonably suggested I show some code.

Update2: Brett asks about relative impact. Here's a summary

item	latency	scalability
2) DataContract Serializer	large	large
3) unthrottle	small	large
4) cache data	small
5) normalize data	medium
6) cache serialization	small	large

Adding users to a distribution list, in bulk

Scott Weinstein — Thu, 18 Dec 2008 23:59:00 GMT

If you've every had to add more then two users to a distribution list in outlook you know just how painful the modal dialogs can be.

Doing it in bulk is easy, if you have the right tools (and assuming you are the owner or co-owner of the group).

PowerShell
PowerShell Commands for Active Directory
a list of users you want to add

In this example, I have my list of users in a text file, but it could come from a DB, feed, or other distribution list.

The "script" is all of two lines long:

> $users = cat users.txt | Get-QADUser
> Get-QADGroup <groupName> |  Add-QADGroupMember  -member $users

How great is that?