I tried doing this declaratively:

<FieldRef ID="{fa564e0f-0c70-4ab9-b863-0177e6ddd247}" Name="Title" DisplayName="Request" Required="FALSE" />

However, I found this didn’t work in SP2010 – the field still appeared as ‘Title’. This was frustrating.

Instead, I had to set the Title’s display name in code:

foreach (SPFieldLink fldLnk in contentType.FieldLinks)
{
if (fldLnk.Id == SPBuiltInFieldId.Title)
{
fldLnk.DisplayName = "Request";
}
}
contentType.Update(false);

This was a bit strange – other fields can have a DisplayName set in CAML quite happily. This seems to just be a feature of the Title column.

EDIT: I also had to use a similar approach on setting the Title column for a content type used on a list – but using:

private void SetTitleField(SPWeb web, string listname, string contentTypeName, string newTitleName) 
{
SPList list = web.Lists[listname];
SPContentType ct = list.ContentTypes[contentTypeName];

foreach (SPFieldLink fldLnk in ct.FieldLinks)
{
//Necessary to set title field - when using content types, this seems to set it to "Title" irrespective of what is set in the Content type or list's XML.
if (fldLnk.Id == SPBuiltInFieldId.Title)
{
	fldLnk.DisplayName = newTitleName;
	ct.Update(false);
	break;
}
}
foreach (SPField fld in list.Fields)
{
if (fld.Id == SPBuiltInFieldId.Title)
{
	fld.Title = newTitleName;
	fld.Update();
	break;
}
}     
}

Well, fundamentally the issue is inefficient queries being run against SharePoint lists. An inefficient query against a list with a small number of items isn’t a big deal, but with larger lists (over 5000 items) SQL server has to escalate from a row-level lock to a table-level lock. This can hold up other users while the table is locked, and could be very slow. To overcome this, SharePoint rejects queries that breach it’s rules with a SPQueryThrottledException. Fine, but what are the factors in determining if it’s throttled? Well, that’s kind of complicated. You’ve got the following factors:

• Indexed Columns – An index makes it more likely that the query will succeed – ‘cos it doesn’t have to scan the whole List. You can still get an exception if your eventual results set is more than 5000 items. An Index on a Unique valued column will always pass, though.
• Query Throttle Limit – you can adjust this value from 5000 items – but Microsoft did chose that value for a reason. Beware!
• Who is querying? Ordinary users have one limit (normally 5000 items), Super users (users with Read or Full Control on the entire web app) a second limit, and Farm Admins are unlimited. Note that as developers typically develop using a Farm Admin account, this is risky, and means careful testing with other accounts and real volumes of data is necessary. Also not that App Pool accounts are still limited – Elevating privileges will not avoid query throttling. And Site Collection Admins are also still limited.
• How you’re querying – There are some ways around these limits by getting the results in blocks, using the ContentIterator control.
• Overrides – You can allow code to override your query throttle limits on a web app.
• “Happy Hour” – You can set a period (normally of low usage) where query throttle limits are turned off – for example, in the middle of the night, if you need to do reports or something.
• Authentication Method – New for SharePoint 2013, the Query Throttle limits are different if you use Classic or Claims authentication. Yup, the authentication mode matters. This came as something of a surprise to both Wayne and myself, and sent a shiver through me; I had visions of some of my applications not working…
• Query Throttling on External Lists – Not something I’d thought of before, but if your BCS connection limits the number of items it returns to a limit below the Query Throttle limit then even if your query on the external list should return more than that limit, you’ll only get the number specified by BCS, and the Query Throttle won’t be hit. In other words, the BCS limit is applied before the Query Throttle limit. Or so Wayne says – it’s not something I’ve tried.

So, not simple. I did ask if there was a document or a table of this stuff, but apparently there isn’t :/ . This is unfortunate; I feel this whole area is unnecessarily complex, and Wayne’s session was largely a set of questions to see what the audience knew – and it turns out that an audience of largely SharePoint devs were confused by it …

I guess, therefore, test, and test again, and remember to deal with large lists…

Naturally, I added this to my search results page, and removed the existing Search Box web part.

During test, the customer noticed that the ‘Show More’ links on the refinement panel had stopped working:

Interesting – although showing some different metadata properties, this was all pretty standard stuff. A bit of a check on Google found this interesting post by Chris Coulson. The short if it is that the Refinements panel uses JavaScript that is loaded by the Search Box web part – so you have to have a Search Box web part on the results page, even if it is hidden. Add that and the Refinement panel works again.

You can hide the web part via the web part properties > Layout > Hidden.

SharePoint 2013 introduces something called “Routing Rules”. These are rules that allow you to direct traffic to different front-end servers (or pools of servers), allowing you to isolate traffic, route to better health servers, etc.. Spence Harbar has a very good article about it.

Anyway, one some of the criteria that you can match rules on are Regular Expressions. However, SharePoint does warn you that Regex routing rules are slower – this is unsurprising. But how much slower than, say, a ‘Starts with’ or ‘Ends with’ rule? And on the Ignite training I did wonder about the efficiency of their example…

To test this I made some assumptions. First, I’m guessing that the record routing rules are written in C#, secondly, that the use the normal C# regex engine. Thirdly, I’ll assume that the ‘Ends With’ and ‘Starts with’ rules use the ordinary String class methods. If so, this would mean I could test the relative speed of the different rules, and perhaps whether the example from the training was a particularly efficient one.

I wrote a little console app, and contrived a test string of:

string target = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMicrosoft One Note 2010";

I chose this because I wanted to try the Ignite example regex, which should match faster if the target string is at the end of the input:

 Regex igniteRegex = new Regex(".*Microsoft One Note 2010*");

Regexper nicely resolves this to:

Cool. Now, the way regex engines match statements like .* is that, as they build up a stack of matching characters, this would make them capture the whole string to the end, and then they’d pop each character off the stack until it can match the next part of the regex.

My belief was that a simpler regex, such as:

 Regex igniteRegex = new Regex("Microsoft One Note 2010");

Would be faster. This wouldn’t need to ‘scan backwards’ over the string. It’s also, notably, much less obviously a Regex – there’s no character classes, or operators to match multiple characters, etc.. Effectively, we’re just looking for that text within the string.

And, finally, I’d test an ‘Ends With’ using the String.EndsWith() method.

I’d try matches with each of these on the target string 1,000,000 times, to take a decent amount of time. So, the results?

Yup, that’s right – changing the regex save about 60% of the time, and using EndsWith() was 30% faster than that, too.

I’ve no doubt that the regex used in the training was just an example, but what concerned me a little was that, as I said at the start, I’m frequently surprised by how unfamiliar some developers are with regexes, and I’m not sure it was made clear enough the impact of the regex pattern. I mean, I wouldn’t have thought such small changes would make such a large difference.

Now, obviously this test is still a bit contrived. If I was putting this into production I think I’d be tempted to try to get a sample of the URLs I’d wanted to process – both items I wanted to match and not match – and then, similarly to above, I’d write a little console application to check my regex against them. I mean, the savings we’re talking about are tiny, but are for each request, and are comparatively easy.

Conclusion – design your regexes with care if using Routing Rules in SharePoint.

Additional: Charlie (see below) asks a good question – how does “Ends With” compare with a regex that matches ‘End String’ (i.e. $ ). My gut feel was that this would still be slower than without, but barely. The reasoning behind this was that the regex would still scan forward over the whole string until it found a match. EndsWith() is probably optimised for simply checking the tail. The most likely thing to upset this logic is if the Regex engine has an optimisation to realise that ‘something$‘ is equivalent to EndsWith(“something”) and so does that instead. My results?

So you can see that the ‘End String’ character doesn’t make much difference in speed – less than 5% – which suggests that that string is still scanned forward.

Well, fortunately, Flavien Charlon has the answer. Put a shortcut to your batch file in %ProgramData%\Microsoft\Windows\Start Menu\Programs

You should then be able to add it from ‘All Apps’ on the start menu.

That’s one annoyance dealt with. Sadly, Windows 8 seems to have plenty more.

“I got 99 problems but a batch ain’t one”

This is quite cute – it gives the users a way to get the data out into something they’re familiar with, can manipulate and can print. However, I was having an issue. Using a list based on a Custom List (above), one of my columns (Enquiry) wasn’t exported:

I wondered at first if this was because of it’s column type (Hyperlink), or because I’d created the column programmatically. Then, on a hunch really, I tried changing the column order. I added another column before it in the list view. I used the ID column, but any should work. When I exported the list – all the columns I wanted came through (but the ID column didn’t).

What I think is going on here is that if you used the same export on a document library then typically – though not always – the first column is an icon for the type of file it is. My suspicion is that some muppet, when writing the export to excel, realised that they’d have these icons, and decided to avoid them by excluding the first column. This is unfortunate for two reasons:

• Not all lists have icons – witness my Custom List
• Not all document libraries have the file icon as the first column.

For now the resolution would be to have another column as the first on your list.

List:

Document Library:

Note: Allegedly, this does change if the event receiver is synchronous - the before properties are available. I’ve not checked this out yet.

Obviously, for SharePoint 2007 systems that doesn’t apply – as ‘*ed’ event receivers are always asynchronous.

I had been using it to loop over some (but not all) the items in a list. Some changes meant that I was now going to loop over all the items. I had a list with 500 items that I was testing with. My code was:

ContentIterator ci = new ContentIterator("example");
SPQuery qry = new SPQuery();
qry.Query = ContentIterator.ItemEnumerationOrderByNVPField;
ci.ProcessListItems(list,qry, ProcessItem, ProcessItemError);

This would process the first 200 items, over and over. It never reached an end. Hmm.

I was puzzled by this, so I put my code in a console app, and then tried querying the list (to check the number of items I was getting):

ContentIterator ci = new ContentIterator("Replicate");
SPQuery qry = new SPQuery();
qry.Query = ContentIterator.ItemEnumerationOrderByNVPField;
Console.WriteLine(qry.Query);
SPListItemCollection items = list.GetItems(qry);
Console.WriteLine("Item: {0}", items.Count);
ci.ProcessListItems(list,qry, ProcessItem, ProcessItemError);

Interestingly, this worked. I can’t see why though. I presume that using the SPQuery changes the query, or somehow affects the list. I’m really not sure.

Now, the observant might well note that actually, a better overload to use on the ContentIterator.ProcessListItems would be the one that doesn’t need a query:

ci.ProcessListItems(list, ProcessItem, ProcessItemError);

This made me wonder – what was different about this? Well, I used Reflector and looked within the class, and found that it was using a different OrderBy clause. When I changed my code to…

ContentIterator ci = new ContentIterator("example");
SPQuery qry = new SPQuery();
qry.Query = ContentIterator.ItemEnumerationOrderByID;
ci.ProcessListItems(list,qry, ProcessItem, ProcessItemError);

… it worked.

So, if your ContentIterator starts to run forever:

• Check you’re using the best overload for what you want. If you don’t need to use an SPQuery, don’t!
• Check your Order By clause – you may need to use a different one.

iOS 5 and OSx (Mountain Lion or later) have ‘Reminders’. I figured that it must be possible to show Tasks from Exchange – after all, I could already sync Email and Calendars. So I set this up … and no tasks came through. Puzzling.

Instead, I tried the opposite – to create an Exchange Task from iOS.

These tasks then did appear in Outlook. It was only then that I went to the Outlook ‘Tasks’ tab, and realised that only the tasks created as ‘Tasks’ were being synchronised. Check out the ‘type’ icons. ‘Followed up’ email has a mail icon – and the tasks that sync have a ‘Task’ icon:

I created a ‘Task’ item in Outlook and voila, it appeared on my phone (as you can see in the screenshot of my phone).

Further checking shows that when you flag an item for ‘Follow Up’, it creates a ‘To Do’ item. This is not the same as a Task, and isn’t synced with iOS or Mac OSx Reminders.

It’s kinda hard to say who missed the point here – Microsoft or Apple – but it is pretty annoying that an email flagged for follow up can’t be synced with Reminders. This might be worth remembering if considering Office 365 / iPad or iPhone scenarios.

(Other than that, it’s nice to be able to get email, contact and calendar information by phone)

Edit: I’ve had to break this down into parts – for some reason WordPress throws a 403 error if I don’t. I think it sees this regex as a security risk!

Before the @ sign:
^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+

The @ sign and after:
@[a-zA-Z0-9-]+(?:\.[a-zA-Z0-9-]+)*$

Reads like it should work pretty well to me. I didn’t realise before the @ was so generous. And you could shrink it by using a case insensitive operator.

If you have the Volume menu visible in the menu bar, clicking on this icon displays a simple volume slider. But if you hold the Option key when you click on the icon, you can select audio inputs and outputs directly, including nearby devices compatible with AirPlay.

SPContentType ctype = site.RootWeb.ContentTypes[SPBuiltInContentTypeId.Document];
if (ctype != null)
{
 if (ctype.EventReceivers != null)
 {
   bool bContinueDelete = true;
   while (bContinueDelete)
   {
     if (ctype.EventReceivers.Count < 1)
     {
       bContinueDelete = false;
     }
     else
     {
     bool bFoundOne = false;
     foreach (SPEventReceiverDefinition d in ctype.EventReceivers)
     {
         //Could match on the Type of the event receiver, but for this example, let's use name
         if (d.Name.Contains("Document ID"))
         {
           d.Delete();
           ctype.Update(true, false);
           bFoundOne = true;
           break;
         }
     }
     if (!bFoundOne)
     {
       bContinueDelete = false;
     }
   }
 }
}

More on why this is important in a later post.

I don’t like the iCloud UI for OSx – it’s inferior to Dropbox. As each  application has it’s own iCloud storage it’s pretty hard to store a heterogeneous set of documents relating to a project or customer. For example, a Contract document and an Expenses spreadsheet would be invisible from each other’s program – and Finder.

So, the hell with iCloud by default – let’s use the filesystem. In terminal, run:

defaults write NSGlobalDomain NSDocumentSaveNewDocumentsToCloud -bool false

I did a bit of digging and found that others have had the same problem – but no solution.

I then spoke to one of our administrators, and he suggest clearing out the ‘event cache’ table. I’d never heard of that (and I’ve no idea how he found out about it), but his advice was:

Minimise the amount of rows in the EventCache SQL table:

• set ChangeLogRetentionPeriod to 1 day (1.00:00:00) on web application
• set EventLogRetentionPeriod to 1 day (1.00:00:00) on web application
• set ‘Change Log’ timer job to run daily (default is weekly)

Okay, so I did this using STSADM:

stsadm -o setproperty -pn change-log-retention-period -pv 1
stsadm -o setproperty -pn event-log-retention-period -pv 1

…and I then changed the Change Log timer job’ schedule in central admin, and set it running right away.

The timer job took about 8 minutes to run. That was a surprisingly long time.

When I then tried creating a site, it took 7 minutes. Clearly, this is still an unwell system, but that’s a lot better than 30 minutes. I’ll update if I find more.

“Delete the database”, I thought, “It’s a database, how hard can it be to fix that?”

So I set out to fix the database without deleting it, and losing all my folders…

The database that launchpad uses is a sqlite database, so I opened Terminal, and opened the database:

andy$ sqlite3 ~/Library/Application\ Support/Dock/*.db

This command tells the terminal to open a sqlite session, connecting to any database (and there only is one) inside the Library > Application Support > Dock folder.

Next, I wanted to query the ‘apps’ table to find any called ‘iPhoto’ (I’m skipping over how I found there was an apps table – but see this blog post – and that it had a ‘title’ column)

sqlite> select * from apps where title="iPhoto";
18|iPhoto|com.apple.iPhoto||8|370119271.0|bookx
146|iPhoto|com.apple.iPhoto||8|370125850.0|book?

Hmm. Two rows returned. The | character separates the different columns, and the penultimate column – the long number – is the modified date. Therefore, I assumed that row ID 18 with the lower (earlier) modified date is actually the old icon. It was this one I wanted to delete. I ran a select to check I was only getting it, and then deleted it.

sqlite> select * from apps where title="iPhoto" and item_id=18;
18|iPhoto|com.apple.iPhoto||8|370119271.0|bookx

sqlite> delete from apps where title="iPhoto" and item_id=18;

A quick check showed one remaining iPhoto entry:

sqlite> select * from apps where title="iPhoto";
146|iPhoto|com.apple.iPhoto||8|370125850.0|book?

Finally, I ran the command

killall Dock

This is a bit terrifying – it closes the process supporting the Dock – but like Explorer in Windows, this restarts automatically, and when it does it loads the newly modified database file – without the second iPhoto icon.

Problem solved, without having to reorganise all my icons.

Encryption isn’t something that I work with every day, but as a web developer you can’t really get away from needing to secure something – and that means encryption.So, to overcome some of this it’s worth a bit of reading.

Bruce Schneier is a pretty interesting author. His books on Applied Cryptography and Practical Cryptography are excellent, and well worth a read for anyone starting to work with crypto. His blog is also an interesting discussion of security and risk in a wider context.

Troy Hunt has managed to write a number of posts that have grabbed my attention over the year or so. “Lessons in web site security anti-patterns” is just that, “A brief Sony password analysis” is fascinating, and “Our password hashing has no clothes” was eye opening. I like that his posts are strongly based on systematic analysis.

Cryptography on StackExchange can be interesting. It’s where I first heard of scrypt, which is quite interesting.

Anyway, I’ll try and update this if I find new, useful resources, or post your own favourites in the comments.

SPSite site = ...
SPFile sourceSolutionFile = ...
//Get the Solution Gallery for the SPSite
SPDocumentLibrary solutionGallery = (SPDocumentLibrary)site.GetCatalog(SPListTemplateType.SolutionCatalog);
//Add the WSP File. I've used a source that is an SPFile, but really it's a string and byte array
SPFile solutionFile = solutionGallery.RootFolder.Files.Add(sourceSolutionFile.File.Name, sourceSolutionFile.File.OpenBinary());
// Activate Solution
SPUserSolution newUserSolution = newSite.Solutions.Add(solutionFile.Item.ID);

Edit: Additional note – you can use this to copy Site Templates (as WSPs) from one Site Collection to another.

That’s fine, but here’s where things got weird.

• If you were using the form as a New form, then search worked just fine the first time you clicked it.
• If you were using the form as an Edit form, the the search button didn’t work the first time you use it – but did the second time you clicked on it.

Bizarre, and I had to fix it…

I started looking at obvious things (i.e. my code) but that looked okay. I debugged the page lifecycle and found that my code wouldn’t break – heck, it didn’t even appear to be called – during the “first click in Edit more” scenario – but it did break on the second click. That’s pretty weird – clicking the same button should run the same code each time. I tried break pointing every function in my C# page-behind, and none of those breakpoints were hit. It was like the page wasn’t running at all, and yet I could see the form.

I began to think it was something fundamentally different about the page, so I decided to compare the source of my Edit form before and after that first click.

The first thing I noticed was that the files were different sizes. This seemed surprising, but I figured that things like the ViewState could account for that. So, next, I used WinMerge to highlight differences. There were 12 differences, mostly to do with transient data, such as the viewstate, and different digests. However, when I got to the  middle of the file I found a difference in the controls being added to the page.

This is the Small Search box, which caught me off-guard. It appeared to have an extra control on the page before the first ‘search’ button click. Afterwards, it was gone.

By this point I was in a hurry, so I decided to try just removing the search box from the page. We don’t need it anyway. The search box is put in a ContentPlaceHolder control, so I decided to blank that out:

And mysteriously, my page and it’s search button started working again. I think that what’s happening is that the Search Box control loaded by the delegate is dynamically creating webcontrols. However, for some reason, on the postback it isn’t creating the same controls. This should cause an exception due to the invalid viewstate, but for some reason this wasn’t being shown. However, this exception would happen before any of my code was run – which is why none of my breakpoints would be hit on the first button press.

All in all, strange, but hopefully it might help someone else if they have the same problem.

Okay, but could I get it to link to a view of the discussion thread itself?

I tried using the LinkUrl property, but that didn’t work very well – when I tried to set it as the link in the CQWP it said that that field didn’t exist. It does though.

So I checked Google. It turns out that this has been written about before. The short of it seems to be that the XSL styles that are emitting the link on the page call an XSL template OuterTemplate.GetSafeLink :

<xsl:variable name="myLink">
<xsl:call-template name="OuterTemplate.GetSafeLink">
<xsl:with-param name="UrlColumnName" select="'LinkUrl'"/>
</xsl:call-template>
</xsl:variable>

We don’t want the safe link – it takes us to the Display Form – but rather we just want the LinkUrl value itself:

<xsl:variable name="myLink"> 
<xsl:value-of select="@LinkUrl"/> 
</xsl:variable> 

That seems to work nicely. It’s a bit of a pain that you have to modify ItemStyles.xsl – and it’s a fair while since I last had to!

It turns out that this is done via the SPWebApplication.Policies collection, which is actually pretty simple to use:

System.Uri uri = new Uri("http://sp/");
SPWebApplication webApplication = SPWebApplication.Lookup(uri)
SPPolicyCollection policyCollection = webApplication.Policies;
SPPolicyRoleCollection policyRoles = webApplication.PolicyRoles;
foreach (SPPolicy p in policyCollection)
{
    Console.WriteLine("{0} - {1}", p.DisplayName, p.UserName);                 
    foreach (SPPolicyRole pr in p.PolicyRoleBindings)                 
    {                     
        Console.WriteLine("     {0} - IsAdmin: {1}, FullControl: {2}", pr.Name, 
                pr.IsSiteAdmin, pr.GrantRightsMask == SPBasePermissions.FullMask);                 
    }             
}

Might be useful someday.